Fundamental Concepts in Optimization Theory

In general, what optimization theory handles is to minimize/maximize one or more objective functions under given constraints and requirements.

General Problem Statement

All optimization problem can be represented by the GPS form

$$\begin{array}{rllll} \min &f(\bold{x})=\dots \\ \text{where} &\bold{x}=\begin{bmatrix}x_1 \\ x_2 \\ \vdots \\ x_n\end{bmatrix} \\ \text{s.t.} &g_i(\bold{x}) \le 0, i=1,2,\dots, m \\ &h_j(\bold{x})=0, j=1,2,\dots,L \\ \end{array}$$

Of course, for any optimization problem, $\mathbf{x}$ is bounded in some region, although probably not all $\mathbf{x}$ in this region satisfies the constraint. Such region is called design space.

In addition, regions in which any $\mathbf{x}$ satisfies all constraints are called feasible region.

For a specific solution $\mathbf{x}^\ast$, and an inequality constraint $g(\mathbf{x})$, if

• $g(\mathbf{x}^\ast)\gt 0$, then the inequality is violated, and the constraint is said to be violated constraint
• $g(\mathbf{x}^\ast)=0$, then the constraint is said to be active constraint
• $g(\mathbf{x}^\ast)\lt 0$, then the constraint is said to be inactive constraint