Convex optimization
Convex optimization is a subfield of mathematical optimization that studies the problem of minimizing convex functions over convex sets. Many classes of convex optimization problems admit polynomial-time algorithms, whereas mathematical optimization is in general NP-hard. Convex optimization has applications in a wide range of disciplines, such as automatic control systems, estimation and signal processing, communications and networks, electronic circuit design, data analysis and modeling, finance, statistics (optimal experimental design), and structural optimization, where the approximation concept has proven to be efficient. With recent advancements in computing and optimization algorithms, convex programming is nearly as straightforward as  linear programming. Definition. A convex optimization problem is an optimization problem in which the objective function is a convex function and the feasible set is a convex set. A function formula_1 mapping some subset of formula_2into formula_3 is convex if its domain is convex and for all formula_4 and all formula_5 in its domain, the following condition holds: formula_6. A set S is convex if for all members formula_7 and all formula_4, we have that formula_9. Concretely, a convex optimization problem is the problem of finding some formula_10 attaining where the objective function formula_12 is convex, as is the feasible  set formula_13. Standard form. A convex optimization problem is in "standard form" if it is written as where formula_18 is the optimization variable, the function formula_19 is convex, formula_20, formula_21, are convex, and formula_22, formula_23, are affine. This notation describes the problem of finding formula_18 that minimizes formula_25 among all formula_26 satisfying formula_27, formula_21 and formula_29, formula_23. The function formula_1 is the objective function of the problem, and the functions formula_32 and formula_33 are the constraint functions. The feasible set formula_13 of the optimization problem consists of all points formula_35 satisfying the constraints. This set is convex because formula_36 is  convex, the sublevel sets of convex functions are convex, affine sets are convex, and the intersection of convex sets is convex. A solution to a convex optimization problem is any point formula_37 attaining formula_11. In general, a convex optimization problem may have zero, one, or many solutions. Many optimization problems can be equivalently formulated in this standard form. For example, the problem of maximizing a concave function formula_1 can be re-formulated equivalently as the problem of minimizing the convex function formula_40. The problem of maximizing a concave function over a convex set is commonly called a convex optimization problem . Properties. The following are useful properties of convex optimization problems: These results are used by the theory of convex minimization along with geometric notions from functional analysis (in Hilbert spaces) such as the Hilbert projection theorem, the separating hyperplane theorem, and Farkas' lemma. Applications. Ben-Hain and Elishakoff (1990), Elishakoff et al. (1994) applied convex analysis to model uncertainty. The following problem classes are all convex optimization problems, or can be reduced to convex optimization problems via simple transformations: Convex optimization has practical applications for the following. Lagrange multipliers. Consider a convex minimization problem given in standard form by a  cost function formula_41 and inequality constraints formula_42 for formula_43. Then the domain formula_44 is: The Lagrangian function for the problem is For each point formula_47 in formula_48 that minimizes formula_1 over formula_48, there exist real numbers formula_51 called Lagrange multipliers, that satisfy these conditions simultaneously: If there exists a "strictly feasible point", that is, a point formula_58 satisfying then the statement above can be strengthened to require that formula_60. Conversely, if some formula_47 in formula_48 satisfies (1)–(3) for scalars formula_63 with formula_60 then formula_47 is certain to minimize formula_1 over formula_48. Algorithms. Unconstrained convex optimization can be easily solved with  gradient descent (a special case of steepest descent) or Newton's method, combined with line search for an appropriate step size; these can be mathematically proven to converge quickly, especially the latter method. Convex optimization with linear equality constraints can also be solved using KKT matrix techniques if the objective function is a quadratic function (which generalizes to a variation of Newton's method, which works even if the point of initialization does not satisfy the constraints), but can also generally be solved by eliminating the equality constraints with linear algebra or solving the dual problem. Finally, convex optimization with  both linear equality constraints and convex inequality constraints can be solved by applying an unconstrained convex optimization technique to the objective function plus logarithmic barrier terms. (When the starting point is not feasible - that is, satisfying the constraints - this is preceded by so-called "phase I" methods, which either find a feasible point or show that none exist. Phase I methods generally consist of reducing the search in question to yet another convex optimization problem.) Convex optimization problems can also be solved by the following contemporary methods: Subgradient methods can be implemented simply and so are widely used. Dual subgradient  methods are subgradient methods applied to a dual problem. The drift-plus-penalty method is similar to the dual subgradient method, but takes a time average of the primal variables. Implementations. Convex optimization and related algorithms have been implemented in the following software programs: Extensions. Extensions of convex optimization include the optimization of biconvex, pseudo-convex, and quasiconvex functions. Extensions of the theory of convex analysis and iterative methods for approximately solving non-convex minimization problems occur in the field of generalized convexity, also known as abstract convex analysis.