Guide

This guide provide some theoretical backgound on optimization methods. With this you can understand how the optimization methods work and how to use them in the optymus module.

We focus on continuous optimization. The continuous optimization methods are used to optimize functions with continuous variables like problems with real numbers, vectors, etc.

Table of Contents

• 1. Continuous Optimization
  • 1.1. Introduction
  • 1.2. Notation and Common Definitions
    • 1.2.1. Basic Notations
      • 1.2.1.1. Vectors and Matrices
      • 1.2.1.2. Functions and Derivatives
    • 1.2.2. Sets and Indices
    • 1.2.3. Common Definitions
      • 1.2.3.1. Feasible Region
      • 1.2.3.2. Optimal Solution
      • 1.2.3.3. Basis and Basic Variables
      • 1.2.3.4. Non-basic Variables
      • 1.2.3.5. Simplex Method
      • 1.2.3.6. Lagrange Multipliers
      • 1.2.3.7. Karush-Kuhn-Tucker (KKT) Conditions
      • 1.2.3.8. Jacobian and Hessian
      • 1.2.3.9. Sensitivity Analysis
  • 1.3. Line Search Methods
    • 1.3.1. Golden Section Search
    • 1.3.2. Bisection Method
    • 1.3.3. Armijo Rule
    • 1.3.4. Wolfe Conditions
  • 1.4. Zero-Order Methods
    • 1.4.1. Univariate Method
    • 1.4.2. Powell’s Method
  • 1.5. First-Order Methods
    • 1.5.1. Steepest Descent (Gradient Descent)
    • 1.5.2. Conjugate Gradient Methods
      • 1.5.2.1. Fletcher–Reeves Method
      • 1.5.2.2. Hestenes–Stiefel Method
      • 1.5.2.3. Polak–Ribiére Method
      • 1.5.2.4. Dai-Yuan Method
    • 1.5.3. Quasi-Newton Methods
      • 1.5.3.1. Broyden-Fletcher-Goldfarb-Shanno (BFGS) Method
      • 1.5.3.2. Limited-Memory BFGS (L-BFGS) Method
  • 1.6. Second-Order Methods
    • 1.6.1. Newton-Raphson Method
  • 1.7. Applications in Engineering
    • 1.7.1. Problem Types
    • 1.7.2. Pre-built Domains
    • 1.7.3. See Also