Open Access
Translator Disclaimer
January 2003 Continuous-time controlled Markov chains
Xianping Guo, Onésimo Hernández-Lerma
Ann. Appl. Probab. 13(1): 363-388 (January 2003). DOI: 10.1214/aoap/1042765671


This paper concerns studies on continuous-time controlled Markov chains, that is, continuous-time Markov decision processes with a denumerable state space, with respect to the discounted cost criterion. The cost and transition rates are allowed to be unbounded and the action set is a Borel space. We first study control problems in the class of deterministic stationary policies and give very weak conditions under which the existence of $\varepsilon$-optimal ($\varepsilon\geq 0)$ policies is proved using the construction of a minimum Q-process. Then we further consider control problems in the class of randomized Markov policies for (1) regular and (2) nonregular Q-processes. To study case (1), first we present a new necessary and sufficient condition for a nonhomogeneous Q-process to be regular. This regularity condition, together with the extended generatorof a nonhomogeneous Markov process, is used to prove the existence of $\varepsilon$-optimal stationary policies. Our results for case (1) are illustrated by a Schlögl model with a controlled diffusion. For case (2), we obtain a similar result using Kolmogorov's forward equation for the minimum Q-process and we also present an example in which our assumptions are satisfied, but those used in the previous literature fail to hold.


Download Citation

Xianping Guo. Onésimo Hernández-Lerma. "Continuous-time controlled Markov chains." Ann. Appl. Probab. 13 (1) 363 - 388, January 2003.


Published: January 2003
First available in Project Euclid: 16 January 2003

zbMATH: 1049.60067
MathSciNet: MR1952002
Digital Object Identifier: 10.1214/aoap/1042765671

Primary: 93E20
Secondary: 60J27 , 90C40

Keywords: controlled Q-processes , discounted criterion , Nonhomogeneous continuous-time Markov chains , optimal stationary policies , unbounded cost and transition rates

Rights: Copyright © 2003 Institute of Mathematical Statistics


Vol.13 • No. 1 • January 2003
Back to Top