The Annals of Statistics
- Ann. Statist.
- Volume 41, Number 3 (2013), 1516-1541.
Kullback–Leibler upper confidence bounds for optimal sequential allocation
We consider optimal sequential allocation in the context of the so-called stochastic multi-armed bandit model. We describe a generic index policy, in the sense of Gittins [J. R. Stat. Soc. Ser. B Stat. Methodol. 41 (1979) 148–177], based on upper confidence bounds of the arm payoffs computed using the Kullback–Leibler divergence. We consider two classes of distributions for which instances of this general idea are analyzed: the kl-UCB algorithm is designed for one-parameter exponential families and the empirical KL-UCB algorithm for bounded and finitely supported distributions. Our main contribution is a unified finite-time analysis of the regret of these algorithms that asymptotically matches the lower bounds of Lai and Robbins [Adv. in Appl. Math. 6 (1985) 4–22] and Burnetas and Katehakis [Adv. in Appl. Math. 17 (1996) 122–142], respectively. We also investigate the behavior of these algorithms when used with general bounded rewards, showing in particular that they provide significant improvements over the state-of-the-art.
Ann. Statist. Volume 41, Number 3 (2013), 1516-1541.
First available in Project Euclid: 1 August 2013
Permanent link to this document
Digital Object Identifier
Mathematical Reviews number (MathSciNet)
Zentralblatt MATH identifier
Cappé, Olivier; Garivier, Aurélien; Maillard, Odalric-Ambrym; Munos, Rémi; Stoltz, Gilles. Kullback–Leibler upper confidence bounds for optimal sequential allocation. Ann. Statist. 41 (2013), no. 3, 1516--1541. doi:10.1214/13-AOS1119. http://projecteuclid.org/euclid.aos/1375362558.
- Supplementary material: Technical proofs. The supplemental article contains the proofs of the results stated in the paper.