The Annals of Applied Probability

A characterization of the optimal risk-sensitive average cost in finite controlled Markov chains

Rolando Cavazos-Cadena and Daniel Hernández-Hernández

Full-text: Open access


This work concerns controlled Markov chains with finite state and action spaces. The transition law satisfies the simultaneous Doeblin condition, and the performance of a control policy is measured by the (long-run) risk-sensitive average cost criterion associated to a positive, but otherwise arbitrary, risk sensitivity coefficient. Within this context, the optimal risk-sensitive average cost is characterized via a minimization problem in a finite-dimensional Euclidean space.

Article information

Ann. Appl. Probab., Volume 15, Number 1A (2005), 175-212.

First available in Project Euclid: 28 January 2005

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 93E20: Optimal stochastic control 60F10: Large deviations
Secondary: 93C55: Discrete-time systems

Decreasing function along trajectories stopping time nearly optimal policies Hölder’s inequality simultaneous Doeblin condition recurrent state


Cavazos-Cadena, Rolando; Hernández-Hernández, Daniel. A characterization of the optimal risk-sensitive average cost in finite controlled Markov chains. Ann. Appl. Probab. 15 (2005), no. 1A, 175--212. doi:10.1214/105051604000000585.

Export citation


  • Bielecki, T., Hernández-Hernández, D. and Pliska, S. (1999). Risk sensitive control of finite state Markov chains in discrete time, with applications to portfolio management. Math. Methods Oper. Res. 50 167–188.
  • Bielecki, T. and Pliska, S. (1999). Risk sensitive dynamic asset management. Appl. Math. Optim. 39 337–360.
  • Cavazos-Cadena, R. and Fernández-Gaucherand, E. (1999). Controlled Markov chains with risk-sensitive criteria: Average cost, optimality equations, and optimal solutions. Math. Methods Oper. Res. 49 299–324.
  • Cavazos-Cadena, R. and Fernández-Gaucherand, E. (2001). Risk-sensitive optimal control in communicating average Markov decision chains. In Modeling Uncertainty: An Examination of Stochastic Theory, Methods, and Applications (M. Dror, P. L'Ecuyer and F. Szydarovszky, eds.) 515–553. Kluwer Academic, Dordrecht.
  • Cavazos-Cadena, R. and Hernández-Hernández, D. (2002). Solution to the risk-sensitive average optimality equation in communicating Markov decision chains with finite state space: An alternative approach. Math. Methods Oper. Res. 56 473–479.
  • Cavazos-Cadena, R. and Hernández-Hernández, D. (2004). A characterization of exponential functionals in finite Markov chains. Math. Methods Oper. Res. To appear.
  • Dembo, A. and Zeitouni, O. (1993). Large Deviations Techniques and Applications. Jones and Bartlett, Boston.
  • Di Masi, G. B. and Stettner, L. (1999). Risk-sensitive control of discrete-time Markov processes with infinite horizon. SIAM J. Control Optim. 38 61–78.
  • Donsker, M. D. and Varadhan, S. R. (1975). Asymptotic evaluation of certain Markov process expectations for large time, I, II, III. Comm. Pure Appl. Math. 28 1–45, 279–301; 29 389–461.
  • Dynkin, E. B. and Yushkevich, A. A. (1979). Controlled Markov Processes. Springer, Berlin.
  • Fleming, W. H. and Hernández-Hernández, D. (1997). Risk sensitive control of finite state machines on an infinite horizon I. SIAM J. Control Optim. 35 1790–1810.
  • Fleming, W. H. and Sheu, S. (1999). Optimal long term growth rate of expected utility of wealth. Ann. Appl. Probab. 9 871–903.
  • Gale, D. (1968). A mathematical theory of optimal economic development. Bull. Amer. Math. Soc. 74 207–223.
  • Hernández-Hernández, D. and Marcus, S. I. (1996). Risk sensitive control of Markov processes in countable state space. Systems Control Lett. 29 147–155. [Corrigendum (1998) 34 105–106.]
  • Hernández-Lerma, O. (1988). Adaptive Markov Control Processes. Springer, New York.
  • Hordjik, A. (1974). Dynamic Programming and Potential Theory. Math. Centrum, Amsterdam.
  • Howard, A. R. and Matheson, J. E. (1972). Risk-sensitive Markov decision processes. Managment Sciences 18 356–369.
  • Puterman M. L. (1994). Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley, New York.
  • Thomas L. C. (1980). Connectedness conditions for denumerable state Markov decision processes. In Recent Advances in Markov Decision Processes (R. Hartles, L. C. Thomas and D. J. White, eds.) 181–204. Academic Press, New York.