The Annals of Applied Probability

Woodroofe’s one-armed bandit problem revisited

Alexander Goldenshluger and Assaf Zeevi

Full-text: Open access

Enhanced PDF (329 KB)

Abstract

We consider the one-armed bandit problem of Woodroofe [J. Amer. Statist. Assoc. 74 (1979) 799–806], which involves sequential sampling from two populations: one whose characteristics are known, and one which depends on an unknown parameter and incorporates a covariate. The goal is to maximize cumulative expected reward. We study this problem in a minimax setting, and develop rate-optimal polices that involve suitable modifications of the myopic rule. It is shown that the regret, as well as the rate of sampling from the inferior population, can be finite or grow at various rates with the time horizon of the problem, depending on “local” properties of the covariate distribution. Proofs rely on martingale methods and information theoretic arguments.

Article information

Source
Ann. Appl. Probab., Volume 19, Number 4 (2009), 1603-1633.

Dates
First available in Project Euclid: 27 July 2009

Permanent link to this document
https://projecteuclid.org/euclid.aoap/1248700629

Digital Object Identifier
doi:10.1214/08-AAP589

Mathematical Reviews number (MathSciNet)
MR2538082

Zentralblatt MATH identifier
1168.62071

Subjects
Primary: 62L05: Sequential design
Secondary: 60G40: Stopping times; optimal stopping problems; gambling theory [See also 62L15, 91A60] 62C20: Minimax procedures

Keywords
Sequential allocation online learning estimation bandit problems regret inferior sampling rate minimax rate-optimal policy

Citation

Goldenshluger, Alexander; Zeevi, Assaf. Woodroofe’s one-armed bandit problem revisited. Ann. Appl. Probab. 19 (2009), no. 4, 1603--1633. doi:10.1214/08-AAP589. https://projecteuclid.org/euclid.aoap/1248700629


Export citation

References

  • Berry, D. A. and Fristedt, B. (1985). Bandit Problems: Sequential Allocation of Experiments. Chapman & Hall, London.
    Mathematical Reviews (MathSciNet): MR813698
  • Borovkov, A. A. and Sakhanenko, A. I. (1980). Estimates for averaged quadratic risk. Probab. Math. Statist. 1 185–195.
    Mathematical Reviews (MathSciNet): MR626310
  • Brown, L. D. and Gajek, L. (1990). Information inequalities for the Bayes risk. Ann. Statist. 18 1578–1594.
    Mathematical Reviews (MathSciNet): MR1074424
    Zentralblatt MATH: 0722.62003
    Digital Object Identifier: doi:10.1214/aos/1176347867
    Project Euclid: euclid.aos/1176347867
  • Cesa-Bianchi, N. and Lugosi, G. (2006). Prediction, Learning, and Games. Cambridge Univ. Press, Cambridge.
    Mathematical Reviews (MathSciNet): MR2409394
    Zentralblatt MATH: 1114.91001
  • Clayton, M. K. (1989). Covariate models for Bernoulli bandits. Sequential Anal. 8 405–426.
    Mathematical Reviews (MathSciNet): MR1036464
  • de la Peña, V. H., Klass, M. J. and Lai, T. L. (2004). Self-normalized processes: Exponential inequalities, moment bounds and iterated logarithm laws. Ann. Probab. 32 1902–1933.
  • de la Peña, V. H., Klass, M. J. and Lai, T. L. (2007). Pseudo-maximization and self-normalized processes. Probab. Surv. 4 172–192.
  • Devroye, L. (1987). A Course in Density Estimation. Progress in Probability and Statistics 14. Birkhäuser, Boston.
    Mathematical Reviews (MathSciNet): MR891874
    Zentralblatt MATH: 0617.62043
  • Gill, R. D. and Levit, B. Y. (1995). Applications of the Van Trees inequality: A Bayesian Cramér–Rao bound. Bernoulli 1 59–79.
    Mathematical Reviews (MathSciNet): MR1354456
    Digital Object Identifier: doi:10.2307/3318681
    Project Euclid: euclid.bj/1186078362
  • Gittins, J. C. (1989). Multi-armed Bandit Allocation Indices. Wiley, Chichester.
    Mathematical Reviews (MathSciNet): MR996417
    Zentralblatt MATH: 0699.90068
  • Goldenshluger, A. and Zeevi, A. (2008). Performance limitations in bandit problems with side observations. Unpublished manuscript.
  • Lai, T. L. (2001). Sequential analysis: Some classical problems and new challenges. Statist. Sinica 11 303–408.
    Mathematical Reviews (MathSciNet): MR1844531
    Zentralblatt MATH: 1037.62081
  • Lai, T. L. and Robbins, H. (1985). Asymptotically efficient adaptive allocation rules. Adv. in Appl. Math. 6 4–22.
    Mathematical Reviews (MathSciNet): MR776826
    Zentralblatt MATH: 0568.62074
    Digital Object Identifier: doi:10.1016/0196-8858(85)90002-8
  • Liptser, R. and Spokoiny, V. (2000). Deviation probability bound for martingales with applications to statistical estimation. Statist. Probab. Lett. 46 347–357.
    Mathematical Reviews (MathSciNet): MR1743992
  • Mammen, E. and Tsybakov, A. B. (1999). Smooth discrimination analysis. Ann. Statist. 27 1808–1829.
    Mathematical Reviews (MathSciNet): MR1765618
    Zentralblatt MATH: 0961.62058
    Digital Object Identifier: doi:10.1214/aos/1017939240
    Project Euclid: euclid.aos/1017939240
  • Robbins, H. (1952). Some aspects of the sequential design of experiments. Bull. Amer. Math. Soc. 55 527–535.
    Mathematical Reviews (MathSciNet): MR50246
    Zentralblatt MATH: 0049.37009
    Digital Object Identifier: doi:10.1090/S0002-9904-1952-09620-8
    Project Euclid: euclid.bams/1183517370
  • Sarkar, J. (1991). One-armed bandit problems with covariates. Ann. Statist. 19 1978–2002.
    Mathematical Reviews (MathSciNet): MR1135160
    Zentralblatt MATH: 0757.62038
    Digital Object Identifier: doi:10.1214/aos/1176348382
    Project Euclid: euclid.aos/1176348382
  • Tsybakov, A. B. (2004a). Optimal aggregation of classifiers in statistical learning. Ann. Statist. 32 135–166.
    Mathematical Reviews (MathSciNet): MR2051002
    Zentralblatt MATH: 1105.62353
    Digital Object Identifier: doi:10.1214/aos/1079120131
    Project Euclid: euclid.aos/1079120131
  • Tsybakov, A. (2004b). Introduction à l’estimation nonparamétrique. Springer, Berlin.
  • Wang, C.-C., Kulkarni, S. R. and Poor, H. V. (2005). Bandit problems with side observations. IEEE Trans. Automat. Control 50 799–806.
    Mathematical Reviews (MathSciNet): MR2123095
    Digital Object Identifier: doi:10.1109/TAC.2005.844079
  • Woodroofe, M. (1979). A one-armed bandit problem with a concomitant variable. J. Amer. Statist. Assoc. 74 799–806.
    Mathematical Reviews (MathSciNet): MR556471
    Zentralblatt MATH: 0442.62063
    Digital Object Identifier: doi:10.2307/2286402
  • Woodroofe, M. (1982). Sequential allocation with covariates. Sankhyā Ser. A 44 403–414.
    Mathematical Reviews (MathSciNet): MR705463
  • Yang, Y. and Zhu, D. (2002). Randomized allocation with nonparametric estimation for a multi-armed bandit problem with covariates. Ann. Statist. 30 100–121.
    Mathematical Reviews (MathSciNet): MR1892657
    Zentralblatt MATH: 1012.62088
    Digital Object Identifier: doi:10.1214/aos/1015362186
    Project Euclid: euclid.aos/1015362186

The Institute of Mathematical Statistics

The Annals of Applied Probability

New content alerts

Email RSS ToC RSS Article
  • You have access to this content.
  • You have partial access to this content.
  • You do not have access to this content.
Turn Off MathJax

What is MathJax?

More like this

+ See more