Black-Box Optimization Revisited: Improving Algorithm Selection Wizards Through Massive Benchmarking

• Published in: IEEE transactions on evolutionary computation Vol. 26; no. 3; pp. 490 - 500
• Main Authors: Meunier, Laurent, Rakotoarison, Herilalaina, Wong, Pak Kan, Roziere, Baptiste, Rapin, Jeremy, Teytaud, Olivier, Moreau, Antoine, Doerr, Carola
• Format: Journal Article
• Language: English
• Published: New York IEEE 01-06-2022; The Institute of Electrical and Electronics Engineers, Inc. (IEEE); Institute of Electrical and Electronics Engineers
• Subjects: Algorithms; Artificial Intelligence; Benchmark testing; Benchmarking; Benchmarks; black-box optimization; Computer Science; Global optimization; Heuristic algorithms; Linear programming; Machine Learning; Mixed integer; Open source software; Optimization; Parameterization; Reproducibility; Reproducibility of results; Statistics; Training
• ISSN: 1089-778X; 1941-0026
• DOI: 10.1109/TEVC.2021.3108185

Existing studies in black-box optimization suffer from low generalizability, caused by a typically selective choice of problem instances used for training and testing of different optimization algorithms. Among other issues, this practice promotes overfitting and poor-performing user guidelines. We address this shortcoming by introducing in this work a general-purpose algorithm selection wizard that was designed and tested on a previously unseen breadth of black-box optimization problems, ranging from academic benchmarks to real-world applications, from discrete over numerical to mixed-integer problems, from small to very large-scale problems, from noisy over dynamic to static problems, etc. Not only did we use the already very extensive benchmark environment available in Nevergrad, but we also extended it significantly by adding a number of additional benchmark suites, including Pyomo, Photonics, large-scale global optimization (LSGO), and MuJoCo. Our wizard achieves competitive performance on all benchmark suites. It significantly outperforms previous state-of-the-art algorithms on some of the suites, including YABBOB and LSGO. Its excellent performance is obtained without any task-specific parametrization. The algorithm selection wizard, all of its base solvers, as well as the benchmark suites are available for reproducible research in the open-source Nevergrad platform.