The Annals of Applied Probability
- Ann. Appl. Probab.
- Volume 13, Number 1 (2003), 363-388.
Continuous-time controlled Markov chains
This paper concerns studies on continuous-time controlled Markov chains, that is, continuous-time Markov decision processes with a denumerable state space, with respect to the discounted cost criterion. The cost and transition rates are allowed to be unbounded and the action set is a Borel space. We first study control problems in the class of deterministic stationary policies and give very weak conditions under which the existence of $\varepsilon$-optimal ($\varepsilon\geq 0)$ policies is proved using the construction of a minimum Q-process. Then we further consider control problems in the class of randomized Markov policies for (1) regular and (2) nonregular Q-processes. To study case (1), first we present a new necessary and sufficient condition for a nonhomogeneous Q-process to be regular. This regularity condition, together with the extended generatorof a nonhomogeneous Markov process, is used to prove the existence of $\varepsilon$-optimal stationary policies. Our results for case (1) are illustrated by a Schlögl model with a controlled diffusion. For case (2), we obtain a similar result using Kolmogorov's forward equation for the minimum Q-process and we also present an example in which our assumptions are satisfied, but those used in the previous literature fail to hold.
Ann. Appl. Probab., Volume 13, Number 1 (2003), 363-388.
First available in Project Euclid: 16 January 2003
Permanent link to this document
Digital Object Identifier
Mathematical Reviews number (MathSciNet)
Zentralblatt MATH identifier
Guo, Xianping; Hernández-Lerma, Onésimo. Continuous-time controlled Markov chains. Ann. Appl. Probab. 13 (2003), no. 1, 363--388. doi:10.1214/aoap/1042765671. https://projecteuclid.org/euclid.aoap/1042765671