Journal of Applied Probability

Sensitivity analysis in Markov decision processes with uncertain reward parameters

Chin Hon Tan and Joseph C. Hartman

Full-text: Access denied (no subscription detected)

We're sorry, but we are unable to provide you with the full text of this article because we are not able to identify you as a subscriber. If you have a personal subscription to this journal, then please login. If you are already logged in, then you may need to update your profile to register your subscription. Read more about accessing full-text


Sequential decision problems can often be modeled as Markov decision processes. Classical solution approaches assume that the parameters of the model are known. However, model parameters are usually estimated and uncertain in practice. As a result, managers are often interested in how estimation errors affect the optimal solution. In this paper we illustrate how sensitivity analysis can be performed directly for a Markov decision process with uncertain reward parameters using the Bellman equations. In particular, we consider problems involving (i) a single stationary parameter, (ii) multiple stationary parameters, and (iii) multiple nonstationary parameters. We illustrate the applicability of this work through a capacitated stochastic lot-sizing problem.

Article information

J. Appl. Probab., Volume 48, Number 4 (2011), 954-967.

First available in Project Euclid: 16 December 2011

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 90C40: Markov and semi-Markov decision processes
Secondary: 90C39: Dynamic programming [See also 49L20] 90C31: Sensitivity, stability, parametric optimization

Markov decision process dynamic programming sensitivity analysis


Tan, Chin Hon; Hartman, Joseph C. Sensitivity analysis in Markov decision processes with uncertain reward parameters. J. Appl. Probab. 48 (2011), no. 4, 954--967. doi:10.1239/jap/1324046012.

Export citation


  • Bazaraa, M. S., Jarvis, J. J. and Sherali, H. D. (2005). Linear Programming and Network Flows, 3rd edn. John Wiley, Hoboken, NJ.
  • Bellman, R. (1957). Dynamic Programming. Princeton University Press.
  • Charalambous, C. and Gittins, J. C. (2008). Optimal selection policies for a sequence of candidate drugs. Adv. Appl. Prob. 40, 359–376.
  • Erkin, Z. \et (2010). Eliciting patients' revealed preferences: an inverse Markov decision process approach. Decision Anal. 7, 358–365.
  • Gal, T. and Greenberg, H. J. (1997). Advances in Sensitivity Analysis and Parametric Programming. Kluwer, Dordrecht.
  • Glazebrook, K. D., Ansell, P. S., Dunn, R. T. and Lumley R. R. (2004). On the optimal allocation of service to impatient tasks. J. Appl. Prob. 41, 51–72.
  • Harmanec, D. (2002). Generalizing Markov decision processes to imprecise probabilities. J. Statist. Planning Infer. 105, 199–213.
  • Hopp, W. J. (1988). Sensitivity analysis in discrete dynamic programming. J. Optimization Theory Appl. 56, 257–269.
  • Iyengar, G. N. (2005). Robust dynamic programming. Math. Operat. Res. 30, 257–280.
  • Lim, C., Bearden, J. N. and Smith, J. C. (2006). Sequential search with multiattribute options. Decision Anal. 3, 3–15.
  • Manne, A. S. (1960). Linear programming and sequential decisions. Manag. Sci. 6, 259–267.
  • Mitrophanov A. Y., Lomsadze A. and Borodovsky M. (2005). Sensitivity of hidden Markov models. J. Appl. Prob. 42, 632–642.
  • Muckstadt, J. A. and Sapra A. (2010). Principles of Inventory Management. Springer, New York.
  • Nilim, A. and El Ghaoui, L. (2005). Robust control of Markov decision processes with uncertain transition matrices. Operat. Res. 53, 780–798.
  • Powell, W. B. (2007). Approximate Dynamic Programming. John Wiley, Hoboken, NJ.
  • Puterman, M. L. (1994). Markov Decision Processes. Discrete Stochastic Dynamic Programming. John Wiley, New York.
  • Sandvik, B. and Thorlund-Petersen, L. (2010). Sensitivity analysis of risk tolerance. Decision Anal. 7, 313–321.
  • Tan, C. H. and Hartman, J. C. (2010). Equipment replacement analyis with an uncertain finite horizon. IIE Trans. 42, 342–353.
  • Tan, C. H. and Hartman, J. C. (2011). Sensitivity analysis and dynamic programming. In Wiley Encyclopedia of Operations Research and Management Science, ed. J. J. Cochran, John Wiley, New York.
  • Topaloglu, H. and Powell, W. B. (2007). Sensitivity analysis of a dynamic fleet management model using approximate dynamic programming. Operat. Res. 55, 319–331.
  • Veinott, A. F., Jr. and Wagner, H. M. (1965). Computing optimal $(s, S)$ inventory policies. Manag. Sci. 11, 525–552.
  • Wallace, S. W. (2000). Decision making under uncertainty: is sensitivity analysis of any use? Operat. Res. 48, 20–25.
  • Ward, J. E. and Wendell, R. E. (1990). Approaches to sensitivity analysis in linear programming. Ann. Operat. Res. 27, 3–38.
  • Wendell, R. E. (1985). The tolerance approach to sensitivity analysis in linear programming. Manag. Sci. 31, 564–578.
  • White, C. C. and El-Deib, H. K. (1986). Parameter imprecision in finite state, finite action dynamic programs. Operat. Res. 34, 120–129.
  • White, D. J. (1993). Markov Decision Processes. John Wiley, Chichester.