Linear Programming (continued)

Success of LP

The success of linear programming as an optimization tool can be attributed to many factors. One is that in a reasonable range of operating values, the abstractions for costs and resource usages can be taken to be linear with respect to operating variables. Even some non-linear but convex cost expressions can be written in piecewise linear form which are then amenable to linear programming formulations. A second reason is that there are now powerful general-purpose codes available to solve ‘large’ linear programming formulations in a numerically stable ways in such a way that engineers and managers can interpret the results usefully. These codes are based either on simple (extreme point) methods or on barrier (interior point) methods. For LPs with special structure (such as network flow LPs), very large problems can be set up and solved very effectively. A third reason is that the supporting theory of convex analysis and duality is so strong and elegant that a lot of additional mileage is available from a linear programming formulation and its solution. These include sensitivity information and various enhancements to computation and interpretation of solutions. Put together, these make LP among the most powerful tools of optimization theory and also in the practice of Operations Research.

As part of this course, we will see some aspects of LP that are interesting, useful and intriguing, apart from the basic material which you should read up on, in any textbook on Optimization and/or O.R. These notes are one step ahead of the current lectures, but you may please read them now.

Special structure LPs : Network flow LPs

Probably the most important sub-class of practical LPs are Network Flow LPs, which admit a surprisingly large number of specializations to interesting applications. These are defined below.

The underlying structure is that of a directed network. The specification is in terms of a set N of nodes, a set of arcs A, with each arc j having a head node i₁ and a tail node i₂. The main variables are flow variables x_j, defined on each arc j. Typically, flow variables satisfying bound constraints on arcs and satisfying a flow balance condition at nodes are to be decided so that costs associated with the flow are minimized. In addition, the following data is required to specify the Min cost flow network flow LP.

A cost c_j on each arc j, lower and upper bounds l_j and u_j on each arc j, and the flow divergence or net supply b_i at each node i.

With these, the min cost network flow or the optimal distribution problem is as follows:

Min S c_j x_j

subject to: S _{{j outgoing at i}} x_j - S _{{j incoming at j}} x_j = b_i (each i)

and l_j <= x_j <= u_j (each j)

For more details on such problems, apart from O.R. books, you can consult the book Linear Programming and Network Flows by Bazaraa, Jarvis and Sherali or one of several books on network flows and combinatorial optimization (notice that some of the problems discussed here have a combinatorial flavour, such as the shortest path problem and the assignment problem). Linear programming as a sub-problem in solving large combinatorial problems is one of the most important aspects of LP, in the last 20 years.

Exercises:

• By properly defining the various constants, show that the problem of finding the shortest path from a node s to a node t on a directed network (with distances defined on each arc) can be formulated as a min cost LP. One question which you should think about and perhaps answer later on, is how the (extreme point) solution to this LP relates to the known (discrete search) techniques for this problem. Also formulate the longest path problem on a (acyclic) network, which comes up in Project Management as the critical path, can also be formulated as an LP.

• What is the condition to be satisfied by constants bi at each node to make the network flow LP a meaningful one? Relate this condition and other correspondences to two well known problems from the O.R. literature, namely the Transportation (or Transhipment) problem and the Assignment problem. These are discussed in any standard O.R. book and also in Belegundu and Chandrupatla. Verify that these two problems are special cases of the Min cost flow LP.

• Note that for arbitrary data, the network flow LP may not have a feasible flow. For the usual shortest path problem, transportation and assignment problem verify that the formulation is feasible. What happens to the longest path formulation with positive costs with cycles in the network?

• You can see that the constraint matrix of the min cost flow LP is of a structure that admits easy computation of basic feasible solutions (starting from a feasible one). The constraint matrix is such that each m by m sub-matrix (actually any sub-matrix) that is invertible, has determinant 1 or -1 – these matrices are called totally unimodular. Verify this if you can (you can prove it by induction or other methods). What do bfs’s of this LP correspond to on the network?

• How would you model a network flow problem on an undirected network?

Linear programming and duality

Corresponding to any linear programming, there is another one in the background which is closely connected with it. What a minimization problem (with constraints) achieves as a minimum can be interpreted as the maximum of another problem. This is quite a deep principle, which holds not just for LPs but for a larger class of optimization problems. [One example of this is the following geometric problem. Given a convex set K and a point c outside it, both say in Rⁿ – although again this sort of result is valid in a more general setting – try find a point in K that is closest to c (in the Euclidean norm). This minimum norm problem turns out to be intimately related to the following maximization problem. Construct a (hyper)plane that separates c and K (the hyperplane is defined as some d^Tx and separation is achieved by saying that d^Tx < 0 for x in K and d^Tc >= 0, i.e. K and c lie on opposite sides of the hyperplane). Find the maximum distance between c and such a hyperplane. This maximization problem turns out to have the same value as the earlier minimization problem.]

Weak duality

One way to derive this is the following. Consider the LP

Min_x c^Tx subject to Ax = b, x >= 0 called the primal problem P.

Multiply constraint i by a constant wi and add the resulting set of equations. You can see that the RHS quantity S w_ib_i <= c^Tx for any feasible x, provided that for each index j, the following inequalities hold S w_i a_ij <= c_j. You can see this by direct arithmetic component-wise and even more easily by matrix vector arithmetic. Note that this bound on the objective function value is true for any x and is therefore true for the optimal solution x* to the primal problem. The quantity given by wTb is a bound on the primal objective function valid with any w satisfying the given constraints w^TA <= c and therefore {Max_w w^Tb s.t. w^TA <= c} <= c^Tx* for feasible x* (i.e. it gives a lower bound. The problem in w is also an LP (but with unconstrained variables w).

This result is called weak duality. The problem {Max_w w^Tb s.t. w^TA <= c}is called the Dual LP, and the variables w are called the dual variables. Using this itself, one can show that if the primal P is unbounded, the dual D is infeasible and vice-versa. Note that both primal and dual can be infeasible (both cannot be unbounded from the previous statement).

Strong duality

The obvious question of interest is whether there is indeed a w that achieves this bound (i.e. satisfies the constraints and has w^Tb = c^Tx*). This answer is ‘yes’ for linear programmes and some other optimization problems, but not true in general. This can be proved in many ways, and the most direct way for us is if we believe in the finite termination of the Simplex method (in the absence of degeneracy), then at optimality, the coefficients of the basic and non-basic variables satisfy the condition that (c_j – c_B^T B^-1Aj) >= 0 all j, where A_j is the j-th constraint column of A. Note that for j belonging to the basis, this is zero by definition and for the non-basic variables, this is precisely the optimality test that we saw in the Simplex method.

Now defining w^T = c_B^T B^-1 constructively gives us a feasible dual vector that satisfies w*^Tb = c^Tx* [verify this].

The main duality result is that for a linear programme that has an optimal solution, the dual LP also has an optimal solution and with the same objective function value as the primal problem.

Sometimes the dual problem is easier to solve as it may have fewer constraints or may have special structure (e.g. separability).

Dual variables and Complementary slackness

There are several interesting applications and interpretation of dual variables and the dual LP. The values of the dual variables at optimality are of particular significance as they capture the rate of change of the objective function c^Tx* with respect to small changes in the RHS variables b. This is of interest if b_i represents the availability of resource i or other interpretations of the constraint RHS. This result can be seen intuitively from the expression c^Tx* = z* = w*^Tb and so taking the partial derivative of z* w.r.t. bi gives the desired interpretation.

The other important linkage of the primal and dual problems is through complementary slackness. For the standard pair of LPs where both the primal and dual have optimal solutions (x* and w*), the complementary slackness condition is that w*^TA_i < c_i implies that x*_i = 0, i.e. the i-th constraint in the dual is satisfied with inequality only for zero variables and conversely that x*_i = 0 implies that i-th constraint in the dual is binding. Note that both these conditions are one-way conditions and not if and only if conditions.

A set of feasible vectors x and w are optimal if and only they satisfy the complementary slackness conditions.

Exercises

• You can test your understanding of complementary slackness and dual variables and some of the relevant notation and transformations by writing the dual LP to the inequality constrained LP Min_x c^Tx subject to Ax >= b, x >= 0.

• Write the dual of the transportation LP and interpret the dual variables.

• Write the dual of the shortest path LP and interpret the dual variables.

Sensitivity analysis

For the LP in standard form, Min_x c^Tx subject to Ax = b, x >= 0, the questions of interest are the following. How does the optimal solution vary with change in the data parameters A_ij, c_j and b_i (especially the latter two, because the ‘technology’ matrix A does not change as often as prices and resource availabilities and demands – in planning applications). What range of values keeps the same basis, but with different values of the variables at optimality? These are useful in design decisions and some managerial decisions, as data are often not known with certainty or are likely to change with time. These questions are well understood and answered in the case of linear programming.

Subsidiary decisions are whether an LP optimal solution remains optimal if a new constraint is added (if the original solution satisfies the new constraint, it is optimal. Why?) and how to restore optimality and feasibility if not, and whether a solution remains optimal if a new variable is added to an LP.