The Annals of Applied Probability

A characterization of the optimal risk-sensitive average cost in finite controlled Markov chains

Rolando Cavazos-Cadena and Daniel Hernández-Hernández

Full-text: Open access

Abstract

This work concerns controlled Markov chains with finite state and action spaces. The transition law satisfies the simultaneous Doeblin condition, and the performance of a control policy is measured by the (long-run) risk-sensitive average cost criterion associated to a positive, but otherwise arbitrary, risk sensitivity coefficient. Within this context, the optimal risk-sensitive average cost is characterized via a minimization problem in a finite-dimensional Euclidean space.

Article information

Source
Ann. Appl. Probab. Volume 15, Number 1A (2005), 175-212.

Dates
First available in Project Euclid: 28 January 2005

Permanent link to this document
http://projecteuclid.org/euclid.aoap/1106922326

Digital Object Identifier
doi:10.1214/105051604000000585

Mathematical Reviews number (MathSciNet)
MR2115041

Zentralblatt MATH identifier
02162825

Subjects
Primary: 93E20: Optimal stochastic control 60F10: Large deviations
Secondary: 93C55: Discrete-time systems

Keywords
Decreasing function along trajectories stopping time nearly optimal policies Hölder’s inequality simultaneous Doeblin condition recurrent state

Citation

Cavazos-Cadena, Rolando; Hernández-Hernández, Daniel. A characterization of the optimal risk-sensitive average cost in finite controlled Markov chains. The Annals of Applied Probability 15 (2005), no. 1A, 175--212. doi:10.1214/105051604000000585. http://projecteuclid.org/euclid.aoap/1106922326.


Export citation

References

  • Bielecki, T., Hernández-Hernández, D. and Pliska, S. (1999). Risk sensitive control of finite state Markov chains in discrete time, with applications to portfolio management. Math. Methods Oper. Res. 50 167--188.
  • Bielecki, T. and Pliska, S. (1999). Risk sensitive dynamic asset management. Appl. Math. Optim. 39 337--360.
  • Cavazos-Cadena, R. and Fernández-Gaucherand, E. (1999). Controlled Markov chains with risk-sensitive criteria: Average cost, optimality equations, and optimal solutions. Math. Methods Oper. Res. 49 299--324.
  • Cavazos-Cadena, R. and Fernández-Gaucherand, E. (2001). Risk-sensitive optimal control in communicating average Markov decision chains. In Modeling Uncertainty: An Examination of Stochastic Theory, Methods, and Applications (M. Dror, P. L'Ecuyer and F. Szydarovszky, eds.) 515--553. Kluwer Academic, Dordrecht.
  • Cavazos-Cadena, R. and Hernández-Hernández, D. (2002). Solution to the risk-sensitive average optimality equation in communicating Markov decision chains with finite state space: An alternative approach. Math. Methods Oper. Res. 56 473--479.
  • Cavazos-Cadena, R. and Hernández-Hernández, D. (2004). A characterization of exponential functionals in finite Markov chains. Math. Methods Oper. Res. To appear.
  • Dembo, A. and Zeitouni, O. (1993). Large Deviations Techniques and Applications. Jones and Bartlett, Boston.
  • Di Masi, G. B. and Stettner, L. (1999). Risk-sensitive control of discrete-time Markov processes with infinite horizon. SIAM J. Control Optim. 38 61--78.
  • Donsker, M. D. and Varadhan, S. R. (1975). Asymptotic evaluation of certain Markov process expectations for large time, I, II, III. Comm. Pure Appl. Math. 28 1--45, 279--301; 29 389--461.
  • Dynkin, E. B. and Yushkevich, A. A. (1979). Controlled Markov Processes. Springer, Berlin.
  • Fleming, W. H. and Hernández-Hernández, D. (1997). Risk sensitive control of finite state machines on an infinite horizon I. SIAM J. Control Optim. 35 1790--1810.
  • Fleming, W. H. and Sheu, S. (1999). Optimal long term growth rate of expected utility of wealth. Ann. Appl. Probab. 9 871--903.
  • Gale, D. (1968). A mathematical theory of optimal economic development. Bull. Amer. Math. Soc. 74 207--223.
  • Hernández-Hernández, D. and Marcus, S. I. (1996). Risk sensitive control of Markov processes in countable state space. Systems Control Lett. 29 147--155. [Corrigendum (1998) 34 105--106.]
  • Hernández-Lerma, O. (1988). Adaptive Markov Control Processes. Springer, New York.
  • Hordjik, A. (1974). Dynamic Programming and Potential Theory. Math. Centrum, Amsterdam.
  • Howard, A. R. and Matheson, J. E. (1972). Risk-sensitive Markov decision processes. Managment Sciences 18 356--369.
  • Puterman M. L. (1994). Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley, New York.
  • Thomas L. C. (1980). Connectedness conditions for denumerable state Markov decision processes. In Recent Advances in Markov Decision Processes (R. Hartles, L. C. Thomas and D. J. White, eds.) 181--204. Academic Press, New York.