The Annals of Applied Probability

Dynamic Allocation Problems in Continuous Time

Nicole El Karoui and Ioannis Karatzas

Full-text: Open access


We present an approach to the general, non-Markovian dynamic allocation (or multiarmed bandit) problem, formulated in continuous time as a problem of stochastic control for multiparameter processes in the manner of Mandelbaum. This approach is based on a direct, martingale study of auxiliary questions in optimal stopping. Using a methodology similar to that of Whittle and relying on simple time-change arguments, we construct Gittins-index-type strategies, verify their optimality, provide explicit expressions for the values of dynamic allocation and associated optimal stopping problems, explore interesting dualities and derive various characterizations of Gittins indices. This paper extends results of our recent work on discrete-parameter dynamic allocation to the continuous time setup; it can be read independently of that work.

Article information

Ann. Appl. Probab., Volume 4, Number 2 (1994), 255-286.

First available in Project Euclid: 19 April 2007

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier


Primary: 93E20: Optimal stochastic control
Secondary: 60G60: Random fields 60G40: Stopping times; optimal stopping problems; gambling theory [See also 62L15, 91A60] 90B85: Continuous location 62L10: Sequential analysis

Multiarmed bandit problem optimal stopping stochastic control multiparameter random time-change Gittins index Brownian local time


Karoui, Nicole El; Karatzas, Ioannis. Dynamic Allocation Problems in Continuous Time. Ann. Appl. Probab. 4 (1994), no. 2, 255--286. doi:10.1214/aoap/1177005062.

Export citation