Institute of Mathematical Statistics Lecture Notes - Monograph Series

Multi-armed bandit problem with precedence relations

Hock Peng Chan, Cheng-Der Fuh, and Inchi Hu

Full-text: Open access

Abstract

Consider a multi-phase project management problem where the decision maker needs to deal with two issues: (a) how to allocate resources to projects within each phase, and (b) when to enter the next phase, so that the total expected reward is as large as possible. We formulate the problem as a multi-armed bandit problem with precedence relations. In Chan, Fuh and Hu (2005), a class of asymptotically optimal arm-pulling strategies is constructed to minimize the shortfall from perfect information payoff. Here we further explore optimality properties of the proposed strategies. First, we show that the efficiency benchmark, which is given by the regret lower bound, reduces to those in Lai and Robbins (1985), Hu and Wei (1989), and Fuh and Hu (2000). This implies that the proposed strategy is also optimal under the settings of aforementioned papers. Secondly, we establish the super-efficiency of proposed strategies when the bad set is empty. Thirdly, we show that they are still optimal with constant switching cost between arms. In addition, we prove that the Wald’s equation holds for Markov chains under Harris recurrent condition, which is an important tool in studying the efficiency of the proposed strategies.

Chapter information

Source
Hwai-Chung Ho, Ching-Kang Ing, Tze Leung Lai, eds., Time Series and Related Topics: In Memory of Ching-Zong Wei (Beachwood, Ohio, USA: Institute of Mathematical Statistics, 2006), 223-235

Dates
First available in Project Euclid: 28 November 2007

Permanent link to this document
https://projecteuclid.org/euclid.lnms/1196285977

Digital Object Identifier
doi:10.1214/074921706000001067

Mathematical Reviews number (MathSciNet)
MR2427850

Zentralblatt MATH identifier
1268.62092

Subjects
Primary: 62L05: Sequential design
Secondary: 62N99: None of the above, but in this section

Keywords
Markov chains multi-armed bandits Kullback--Leibler number likelihood ratio optimal stopping scheduling single-machine job sequencing Wald's equation

Rights
Copyright © 2006, Institute of Mathematical Statistics

Citation

Chan, Hock Peng; Fuh, Cheng-Der; Hu, Inchi. Multi-armed bandit problem with precedence relations. Time Series and Related Topics, 223--235, Institute of Mathematical Statistics, Beachwood, Ohio, USA, 2006. doi:10.1214/074921706000001067. https://projecteuclid.org/euclid.lnms/1196285977


Export citation