1.4. Zero-Order Methods¶
1.4.1. Univariate Method¶
In the Univariate Method, the search direction at iteration \(k\) is defined by:
where \(\mathbf{e}_k\) is a vector with zero elements except at position \(k\), where the element is 1. This procedure is equivalent to modifying one variable at a time in the iterative process, meaning only the variable at position \(k\) of the variable vector \(\mathbf{x}\) is modified in iteration \(k\). For a problem with \(n\) variables, if the position \(\mathbf{x}\) has not converged to the solution \(\mathbf{x}^*\) after \(n\) iterations, a new cycle of iterations should start with the same directions used in the first cycle, and so on until convergence .
The implementation with optymus:
from optymus import Optimizer
# Define the objective function
f = lambda x: x[0]**2 + x[1]**2
# Define the initial point
x0 = [0.0, 2.0]
# Define the optimization problem
results = Optimizer(f_obj=f, x0=x0, method='univariate')
# Print the solution
results.report()
1.4.2. Powell’s Method¶
The Univariate Method is computationally inefficient and generally requires many iterations to reach a solution. One way to accelerate this process is to incorporate a new search direction, called the pattern movement, into the set of \(n\) search directions at the end of each iterative cycle of \(n\) iterations. During the first \(n\) cycles, a pattern direction is incorporated at the end of each cycle into the set of \(n\) search directions, replacing one of the discarded univariate directions. After \(n\) cycles, no univariate direction should remain in the set of search directions. These new search directions were proposed by Powell and are obtained according to the expression below:
where \(\mathbf{x}_n\) is the point obtained at the end of each cycle of \(n\) iterations and \(\mathbf{x}_0\) is the initial point. For each new cycle, if there is no convergence, a new pattern direction is created using the same procedure, i.e., the final point minus the initial point .
The implementation with optymus:
from optymus import Optimizer
# Define the objective function
f = lambda x: x[0]**2 + x[1]**2
# Define the initial point
x0 = [0.0, 2.0]
# Define the optimization problem
results = Optimizer(f_obj=f, x0=x0, method='powell')
# Print the solution
results.report()