Some indexable families of restless bandit problems
K. D. Glazebrook, D. Ruiz-Hernandez, and C. Kirkbride
Source: Adv. in Appl. Probab. Volume 38, Number 3
(2006), 643-672.
Abstract
In 1988 Whittle introduced an important but intractable class of restless bandit problems which generalise the multiarmed bandit problems of Gittins by allowing state evolution for passive projects. Whittle's account deployed a Lagrangian relaxation of the optimisation problem to develop an index heuristic. Despite a developing body of evidence (both theoretical and empirical) which underscores the strong performance of Whittle's index policy, a continuing challenge to implementation is the need to establish that the competing projects all pass an indexability test. In this paper we employ Gittins' index theory to establish the indexability of (inter alia) general families of restless bandits which arise in problems of machine maintenance and stochastic scheduling problems with switching penalties. We also give formulae for the resulting Whittle indices. Numerical investigations testify to the outstandingly strong performance of the index heuristics concerned.
First Page:
Show
Hide
Keywords: Bandit problem; dynamic programming; Gittins index; machine maintenance; restless bandit; stochastic scheduling; switching cost
Full-text: Access denied (no subscription
detected)
We're sorry, but we are unable to provide
you with the full text of this article because we are not able to identify
you as a subscriber.
If you have a personal subscription to
this journal, then please login. If you are already logged in, then you
may need to update your profile to register your subscription. Read more about accessing full-text
Links and Identifiers
Permanent link to this document: http://projecteuclid.org/euclid.aap/1158684996
Digital Object Identifier: doi:10.1239/aap/1158684996
Mathematical Reviews number (MathSciNet): MR2256872
Zentralblatt MATH identifier: 1101.90079
References
Agrawal, R., Hedge, M. and Teneketzis, D. (1988). Asymptotically efficient adaptive allocation rules for the multiarmed bandit problem with switching cost. IEEE Trans. Automatic Control 33, 899--906.
Mathematical Reviews (MathSciNet): MR959012
Digital Object Identifier: doi:10.1109/9.7243
Zentralblatt MATH: 0678.93073
Ansell, P. S., Glazebrook, K. D., Niño-Mora, J. and O'Keeffe, M. (2003). Whittle's index policy for a multi-class queueing system with convex holding costs. Math. Meth. Operat. Res. 57, 21--39.
Mathematical Reviews (MathSciNet): MR1969254
Digital Object Identifier: doi:10.1007/s001860200257
Asawa, M. and Teneketzis, D. (1996). Multi-armed bandits with switching penalties. IEEE Trans. Automatic Control 41, 328--348.
Mathematical Reviews (MathSciNet): MR1382983
Digital Object Identifier: doi:10.1109/9.486316
Zentralblatt MATH: 0847.90137
Banks, J. S. and Sundaram, R. (1994). Switching costs and the Gittins index. Econometrica 62, 687--694.
Gittins, J. C. (1979). Bandit processes and dynamic allocation indices (with discussion). J. R. Statist. Soc. B 41, 148--177.
Mathematical Reviews (MathSciNet): MR547241
JSTOR: links.jstor.org
Gittins, J. C. (1989). Multi-Armed Bandit Allocation Indices. John Wiley, Chichester.
Mathematical Reviews (MathSciNet): MR996417
Zentralblatt MATH: 0699.90068
Glazebrook, K. D. (1980). On stochastic scheduling with precedence relations and switching costs. J. Appl. Prob. 17, 1016--1024.
Mathematical Reviews (MathSciNet): MR587202
Digital Object Identifier: doi:10.2307/3213211
JSTOR: links.jstor.org
Zentralblatt MATH: 0449.90050
Glazebrook, K. D., Mitchell, H. M. and Ansell, P. S. (2005). Index policies for the maintenance of a collection of machines by a set of repairmen. Europ. J. Operat. Res. 165, 267--284.
Mathematical Reviews (MathSciNet): MR2121966
Digital Object Identifier: doi:10.1016/j.ejor.2004.01.036
Zentralblatt MATH: 1112.90323
Glazebrook, K. D., Niño-Mora, J. and Ansell, P. S. (2002). Index policies for a class of discounted restless bandits. Adv. Appl. Prob. 34, 754--774.
Mathematical Reviews (MathSciNet): MR1938941
Digital Object Identifier: doi:10.1239/aap/1037990952
Project Euclid: euclid.aap/1037990952
Zentralblatt MATH: 1053.90048
Nash, P. (1979). Optimal allocation of resources between research projects. Doctoral Thesis, University of Cambridge.
Niño-Mora, J. (2001). Restless bandits, partial conservation laws and indexability. Adv. Appl. Prob. 33, 76--98.
Mathematical Reviews (MathSciNet): MR1825317
Digital Object Identifier: doi:10.1239/aap/999187898
Project Euclid: euclid.aap/999187898
Zentralblatt MATH: 1039.90019
Niño-Mora, J. (2002). Dynamic allocation indices for restless projects and queueing admission control: a polyhedral approach. Math. Program. 93, 361--413.
Mathematical Reviews (MathSciNet): MR1953254
Digital Object Identifier: doi:10.1007/s10107-002-0362-6
Zentralblatt MATH: 1023.90034
Papadimitriou, C. H. and Tsitsiklis, J. N. (1999). The complexity of optimal queuing network control. Math. Operat. Res. 24, 293--305.
Mathematical Reviews (MathSciNet): MR1853877
JSTOR: links.jstor.org
Puterman, M. L. (1994). Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley, New York.
Mathematical Reviews (MathSciNet): MR1270015
Zentralblatt MATH: 0829.90134
Reiman, M. I. and Wein, L. M. (1998). Dynamic scheduling of a two-class queue with setups. Operat. Res. 46, 532--547.
Mathematical Reviews (MathSciNet): MR1643177
JSTOR: links.jstor.org
Van Oyen, M. P. and Teneketzis, D. (1994). Optimal stochastic scheduling of forest networks with switching penalties. Adv. Appl. Prob. 26, 474--479.
Mathematical Reviews (MathSciNet): MR1272723
Digital Object Identifier: doi:10.2307/1427447
JSTOR: links.jstor.org
Zentralblatt MATH: 0804.60083
Weber, R. R. and Weiss, G. (1990). On an index policy for restless bandits. J. Appl. Prob. 27, 637--648. (Addendum: Adv. Appl. Prob. 23 (1991), 429--430.)
Mathematical Reviews (MathSciNet): MR1067028
Digital Object Identifier: doi:10.2307/3214547
JSTOR: links.jstor.org
Zentralblatt MATH: 0735.90072
Whittle, P. (1980). Multi-armed bandits and the Gittins index. J. R. Statist. Soc. B 42, 143--149.
Mathematical Reviews (MathSciNet): MR583348
JSTOR: links.jstor.org
Whittle, P. (1988). Restless bandits: activity allocation in a changing world. In A Celebration of Applied Probability (J. Appl. Prob. Spec. Vol. 25A), ed. J. Gani, Applied Probability Trust, Sheffield, pp. 287--298.
Mathematical Reviews (MathSciNet): MR974588
Digital Object Identifier: doi:10.2307/3214163
Whittle, P. (1996). Optimal Control: Basics and Beyond. John Wiley, Chichester.
Mathematical Reviews (MathSciNet): MR1416566
Zentralblatt MATH: 0880.49001
Advances in Applied Probability
