Open Access
June 2019 Constrained Bayesian Optimization with Noisy Experiments
Benjamin Letham, Brian Karrer, Guilherme Ottoni, Eytan Bakshy
Bayesian Anal. 14(2): 495-519 (June 2019). DOI: 10.1214/18-BA1110


Randomized experiments are the gold standard for evaluating the effects of changes to real-world systems. Data in these tests may be difficult to collect and outcomes may have high variance, resulting in potentially large measurement error. Bayesian optimization is a promising technique for efficiently optimizing multiple continuous parameters, but existing approaches degrade in performance when the noise level is high, limiting its applicability to many randomized experiments. We derive an expression for expected improvement under greedy batch optimization with noisy observations and noisy constraints, and develop a quasi-Monte Carlo approximation that allows it to be efficiently optimized. Simulations with synthetic functions show that optimization performance on noisy, constrained problems outperforms existing methods. We further demonstrate the effectiveness of the method with two real-world experiments conducted at Facebook: optimizing a ranking system, and optimizing server compiler flags.


Download Citation

Benjamin Letham. Brian Karrer. Guilherme Ottoni. Eytan Bakshy. "Constrained Bayesian Optimization with Noisy Experiments." Bayesian Anal. 14 (2) 495 - 519, June 2019.


Published: June 2019
First available in Project Euclid: 10 August 2018

zbMATH: 07045440
MathSciNet: MR3934095
Digital Object Identifier: 10.1214/18-BA1110

Keywords: Bayesian optimization , quasi-Monte Carlo methods , randomized experiments

Vol.14 • No. 2 • June 2019
Back to Top