Inequality constrained stochastic nonlinear optimization via active-set sequential quadratic programming

• Published in: Mathematical programming Vol. 202; no. 1-2; pp. 279 - 353
• Main Authors: Na, Sen, Anitescu, Mihai, Kolar, Mladen
• Format: Journal Article
• Language: English
• Published: Berlin/Heidelberg Springer Berlin Heidelberg 01-11-2023; Springer; Springer Nature B.V
• Subjects: Algorithms; Artificial neural networks; Calculus of Variations and Optimal Control; Optimization; Combinatorics; Constraints; Convergence; Full Length Paper; Mathematical and Computational Physics; Mathematical Methods in Physics; Mathematics; Mathematics and Statistics; Mathematics of Computing; Neural networks; Numerical Analysis; Optimization; Quadratic programming; Theoretical; 90C30; Sequential quadratic programming; 90C55; Exact augmented Lagrangian; 90-08; 90C90; 90C15; 90C26; Inequality constraints; Stochastic optimization
• ISSN: 0025-5610; 1436-4646
• DOI: 10.1007/s10107-023-01935-7

We study nonlinear optimization problems with a stochastic objective and deterministic equality and inequality constraints, which emerge in numerous applications including finance, manufacturing, power systems and, recently, deep neural networks. We propose an active-set stochastic sequential quadratic programming (StoSQP) algorithm that utilizes a differentiable exact augmented Lagrangian as the merit function. The algorithm adaptively selects the penalty parameters of the augmented Lagrangian, and performs a stochastic line search to decide the stepsize. The global convergence is established: for any initialization, the KKT residuals converge to zero
almost surely
. Our algorithm and analysis further develop the prior work of Na et al. (Math Program, 2022.
https://doi.org/10.1007/s10107-022-01846-z
). Specifically, we allow nonlinear inequality constraints
without
requiring the strict complementary condition; refine some of designs in Na et al. (2022) such as the feasibility error condition and the monotonically increasing sample size; strengthen the global convergence guarantee; and improve the sample complexity on the objective Hessian. We demonstrate the performance of the designed algorithm on a subset of nonlinear problems collected in CUTEst test set and on constrained logistic regression problems.