The Annals of Applied Probability

Continuous-time controlled Markov chains

Xianping Guo and Onésimo Hernández-Lerma

Full-text: Open access

PDF File (201 KB)

Abstract

This paper concerns studies on continuous-time controlled Markov chains, that is, continuous-time Markov decision processes with a denumerable state space, with respect to the discounted cost criterion. The cost and transition rates are allowed to be unbounded and the action set is a Borel space. We first study control problems in the class of deterministic stationary policies and give very weak conditions under which the existence of $\varepsilon$-optimal ($\varepsilon\geq 0)$ policies is proved using the construction of a minimum Q-process. Then we further consider control problems in the class of randomized Markov policies for (1) regular and (2) nonregular Q-processes. To study case (1), first we present a new necessary and sufficient condition for a nonhomogeneous Q-process to be regular. This regularity condition, together with the extended generatorof a nonhomogeneous Markov process, is used to prove the existence of $\varepsilon$-optimal stationary policies. Our results for case (1) are illustrated by a Schlögl model with a controlled diffusion. For case (2), we obtain a similar result using Kolmogorov's forward equation for the minimum Q-process and we also present an example in which our assumptions are satisfied, but those used in the previous literature fail to hold.

Article information

Source
Ann. Appl. Probab., Volume 13, Number 1 (2003), 363-388.

Dates
First available in Project Euclid: 16 January 2003

Permanent link to this document
https://projecteuclid.org/euclid.aoap/1042765671

Digital Object Identifier
doi:10.1214/aoap/1042765671

Mathematical Reviews number (MathSciNet)
MR1952002

Zentralblatt MATH identifier
1049.60067

Subjects
Primary: 93E20: Optimal stochastic control
Secondary: 60J27: Continuous-time Markov processes on discrete state spaces 90C40: Markov and semi-Markov decision processes

Keywords
Nonhomogeneous continuous-time Markov chains controlled Q-processes unbounded cost and transition rates discounted criterion optimal stationary policies

Citation

Guo, Xianping; Hernández-Lerma, Onésimo. Continuous-time controlled Markov chains. Ann. Appl. Probab. 13 (2003), no. 1, 363--388. doi:10.1214/aoap/1042765671. https://projecteuclid.org/euclid.aoap/1042765671


Export citation

References

  • [1] ANDERSON, W. J. (1991). Continuous Time Markov Chains. Springer, New York.
    Mathematical Reviews (MathSciNet): MR1118840
  • [2] BATHER, J. (1976). Optimal stationary policies for denumerable Markov chains in continuoustime. Adv. in Appl. Probab. 8 114-158.
    Mathematical Reviews (MathSciNet): MR52:16650
    Zentralblatt MATH: 0342.60045
    Digital Object Identifier: doi:10.2307/1426026
    JSTOR: links.jstor.org
  • [3] BERTSEKAS, D. P. (1987). Dy namic Programming: Deterministic and Stochastic Models. Prentice-Hall, Englewood Cliffs, NJ.
    Mathematical Reviews (MathSciNet): MR896902
  • [4] CAVAZOS-CADENA, R. and GAUCHERAND, E. (1996). Value iteration in a class of average controlled Markov chains with unbounded costs: Necessary and sufficient conditions for pointwise convergence. J. Appl. Probab. 33 986-1002.
    Mathematical Reviews (MathSciNet): MR97i:90106
    Zentralblatt MATH: 0869.93054
    Digital Object Identifier: doi:10.2307/3214980
    JSTOR: links.jstor.org
  • [5] CHEN, M. F. (1990). On three classical problems for Markov chains with continuous time parameters. J. Appl. Probab. 28 305-320.
    Mathematical Reviews (MathSciNet): MR1104568
    Zentralblatt MATH: 0761.60064
    Digital Object Identifier: doi:10.2307/3214868
    JSTOR: links.jstor.org
  • [6] CHUNG, K. L. (1960). Markov Chains with Stationary Transition Probabilities. Springer, Berlin.
    Mathematical Reviews (MathSciNet): MR116388
  • [7] DONG, Z. Q. (1979). Continuous time Markov decision programming with average reward criterion-countable state and action space. Sci. Sinica SP(II) 131-148.
    Mathematical Reviews (MathSciNet): MR576353
  • [8] FEINBERG, E. A. (1998). Continuous time discounted jump Markov decision processes: A discrete-event approach. Preprint.
    Mathematical Reviews (MathSciNet): MR2082616
    Zentralblatt MATH: 1082.90126
    Digital Object Identifier: doi:10.1287/moor.1040.0089
  • [9] FELLER, W. (1940). On the integro-differential equations of purely discontinuous Markoff processes. Trans. Amer. Math. Soc. 48 488-515.
    Mathematical Reviews (MathSciNet): MR2,101b
    Zentralblatt MATH: 0025.34704
    Digital Object Identifier: doi:10.2307/1990095
    JSTOR: links.jstor.org
  • [10] GUO, X. P. and LIU, K. (2001). A note on optimality conditions for continuous-time Markov decision processes with average cost criterion. IEEE Trans. Automat. Control 46 1984- 1989.
    Zentralblatt MATH: 01769508
    Mathematical Reviews (MathSciNet): MR1878226
    Digital Object Identifier: doi:10.1109/9.975505
  • [11] GUO, X. P. and ZHU, W. P. (2002). Optimality conditions for CTMDP with average cost criterion. In Markov Processes and Controlled Markov Chains (Z. T. Hou, J. A. Filar and A. Y. Chen, eds.) Chap. 10. Kluwer, Dordrecht.
    Mathematical Reviews (MathSciNet): MR2022425
    Zentralblatt MATH: 1068.90609
  • [12] GUO, X. P. and ZHU, W. P. (2002). Denumerable-state continuous-time Markov decision processes with unbounded transition and reward rates under the discounted criterion. J. Appl. Probab. 39 233-250.
    Zentralblatt MATH: 01824626
    Mathematical Reviews (MathSciNet): MR1908941
    Digital Object Identifier: doi:10.1239/jap/1025131422
    Project Euclid: euclid.jap/1025131422
  • [13] HAVIV, M. and PUTERMAN, M. L. (1998). Bias optimality in controlled queuing sy stems. J. Appl. Probab. 35 16-150.
  • [14] HERNÁNDEZ-LERMA, O. (1994). Lectures on Continuous-time Markov Control Processes. Sociedad Matemática Mexicana, México City.
  • [15] HEy MAN, D. P. and SOBEL, M. J. (1984). Stochastic Models in Operations Research 2. McGraw-Hill, New York.
  • [16] HOU, Z. T. (1994). The Q-matrix Problems on Markov Processes. Science and Technology Press of Hunan, Changsha, China (in Chinese).
  • [17] HOU, Z. T. and GUO, X. P. (1998). Markov Decision Processes. Science and Technology Press of Hunan, Changsha, China (in Chinese).
  • [18] KAKUMANU, P. (1971). Continuously discounted Markov decision model with countable state and action spaces. Ann. Math. Statist. 42 919-926.
    Mathematical Reviews (MathSciNet): MR43:8360
    Zentralblatt MATH: 0234.93027
    Digital Object Identifier: doi:10.1214/aoms/1177693321
    Project Euclid: euclid.aoms/1177693321
  • [19] LEFÈVRE, C. (1981). Optimal control of a birth and death epidemic process. Oper. Res. 29 971-982.
    Zentralblatt MATH: 0478.92014
  • [20] LEWIS, M. E. and PUTERMAN, M. (2001). A probabilistic analysis of bias optimality in unichain Markov decision processes. IEEE Trans. Automat. Control 46 96-100.
    Zentralblatt MATH: 01769418
    Mathematical Reviews (MathSciNet): MR1809468
    Digital Object Identifier: doi:10.1109/9.898698
  • [21] LEWIS, M. E. and PUTERMAN, M. (2000). A note on bias optimality in controlled queueing sy stems. J. Appl. Probab. 37 300-305.
    Mathematical Reviews (MathSciNet): MR1761681
    Zentralblatt MATH: 1018.90009
    Digital Object Identifier: doi:10.1239/jap/1014842288
    Project Euclid: euclid.jap/1014842288
  • [22] MILLER, R. L. (1968). Finite state continuous time Markov decision processes with an infinite planning horizon. J. Math. Anal. Appl. 22 552-569.
    Zentralblatt MATH: 0157.50301
    Mathematical Reviews (MathSciNet): MR228251
    Digital Object Identifier: doi:10.1016/0022-247X(68)90194-7
  • [23] PUTERMAN, M. L. (1994). Markov Decision Processes. Wiley, New York.
    Mathematical Reviews (MathSciNet): MR1270015
  • [24] SCHLÖGL, F. (1972). Chemical reaction models for phase transition. Z. Phy s. 253 147-161.
  • [25] SENNOTT, L. I. (1999). Stochastic Dy namic Programming and the Control of Queueing Sy stems. Wiley, New York.
    Mathematical Reviews (MathSciNet): MR99i:90115
    Zentralblatt MATH: 0997.93503
  • [26] SERFOZO, R. (1981). Optimal control of random walks, birth and death processes, and queues. Adv. in Appl. Probab. 13 61-83.
    Mathematical Reviews (MathSciNet): MR82d:60187
    Zentralblatt MATH: 0446.60062
    Digital Object Identifier: doi:10.2307/1426467
    JSTOR: links.jstor.org
  • [27] SONG, J. S. (1987). Continuous time Markov decision programming with nonuniformly bounded transition rates. Sci. Sinica 12 1258-1267 (in Chinese).
  • [28] TIJMS, H. C. (1994). Stochastic Models: An Algorithmic Approach. Wiley, Chichester.
    Mathematical Reviews (MathSciNet): MR96c:60111
  • [29] WALRAND, J. (1988). An Introduction to Queueing Networks. Prentice-Hall, Englewood Cliffs, NJ.
  • [30] YUSHKEVICH, A. A. and FEINBERG, E. A. (1979). On homogeneous Markov model with continuous time and finite or countable state space. Theory Probab. Appl. 24 156-161.
    Mathematical Reviews (MathSciNet): MR80i:90095

The Institute of Mathematical Statistics

The Annals of Applied Probability

New content alerts

Email RSS ToC RSS Article
  • You have access to this content.
  • You have partial access to this content.
  • You do not have access to this content.
Turn Off MathJax

What is MathJax?

More like this

+ See more