Journal of Applied Probability

Sensitivity analysis in Markov decision processes with uncertain reward parameters

Chin Hon Tan and Joseph C. Hartman

Full-text: Access denied (no subscription detected)

We're sorry, but we are unable to provide you with the full text of this article because we are not able to identify you as a subscriber. If you have a personal subscription to this journal, then please login. If you are already logged in, then you may need to update your profile to register your subscription. Read more about accessing full-text

Abstract

Sequential decision problems can often be modeled as Markov decision processes. Classical solution approaches assume that the parameters of the model are known. However, model parameters are usually estimated and uncertain in practice. As a result, managers are often interested in how estimation errors affect the optimal solution. In this paper we illustrate how sensitivity analysis can be performed directly for a Markov decision process with uncertain reward parameters using the Bellman equations. In particular, we consider problems involving (i) a single stationary parameter, (ii) multiple stationary parameters, and (iii) multiple nonstationary parameters. We illustrate the applicability of this work through a capacitated stochastic lot-sizing problem.

Article information

Source
J. Appl. Probab., Volume 48, Number 4 (2011), 954-967.

Dates
First available in Project Euclid: 16 December 2011

Permanent link to this document
https://projecteuclid.org/euclid.jap/1324046012

Digital Object Identifier
doi:10.1239/jap/1324046012

Mathematical Reviews number (MathSciNet)
MR2896661

Zentralblatt MATH identifier
1231.90374

Subjects
Primary: 90C40: Markov and semi-Markov decision processes
Secondary: 90C39: Dynamic programming [See also 49L20] 90C31: Sensitivity, stability, parametric optimization

Keywords
Markov decision process dynamic programming sensitivity analysis

Citation

Tan, Chin Hon; Hartman, Joseph C. Sensitivity analysis in Markov decision processes with uncertain reward parameters. J. Appl. Probab. 48 (2011), no. 4, 954--967. doi:10.1239/jap/1324046012. https://projecteuclid.org/euclid.jap/1324046012


Export citation

References

  • Bazaraa, M. S., Jarvis, J. J. and Sherali, H. D. (2005). Linear Programming and Network Flows, 3rd edn. John Wiley, Hoboken, NJ.
  • Bellman, R. (1957). Dynamic Programming. Princeton University Press.
  • Charalambous, C. and Gittins, J. C. (2008). Optimal selection policies for a sequence of candidate drugs. Adv. Appl. Prob. 40, 359–376.
  • Erkin, Z. \et (2010). Eliciting patients' revealed preferences: an inverse Markov decision process approach. Decision Anal. 7, 358–365.
  • Gal, T. and Greenberg, H. J. (1997). Advances in Sensitivity Analysis and Parametric Programming. Kluwer, Dordrecht.
  • Glazebrook, K. D., Ansell, P. S., Dunn, R. T. and Lumley R. R. (2004). On the optimal allocation of service to impatient tasks. J. Appl. Prob. 41, 51–72.
  • Harmanec, D. (2002). Generalizing Markov decision processes to imprecise probabilities. J. Statist. Planning Infer. 105, 199–213.
  • Hopp, W. J. (1988). Sensitivity analysis in discrete dynamic programming. J. Optimization Theory Appl. 56, 257–269.
  • Iyengar, G. N. (2005). Robust dynamic programming. Math. Operat. Res. 30, 257–280.
  • Lim, C., Bearden, J. N. and Smith, J. C. (2006). Sequential search with multiattribute options. Decision Anal. 3, 3–15.
  • Manne, A. S. (1960). Linear programming and sequential decisions. Manag. Sci. 6, 259–267.
  • Mitrophanov A. Y., Lomsadze A. and Borodovsky M. (2005). Sensitivity of hidden Markov models. J. Appl. Prob. 42, 632–642.
  • Muckstadt, J. A. and Sapra A. (2010). Principles of Inventory Management. Springer, New York.
  • Nilim, A. and El Ghaoui, L. (2005). Robust control of Markov decision processes with uncertain transition matrices. Operat. Res. 53, 780–798.
  • Powell, W. B. (2007). Approximate Dynamic Programming. John Wiley, Hoboken, NJ.
  • Puterman, M. L. (1994). Markov Decision Processes. Discrete Stochastic Dynamic Programming. John Wiley, New York.
  • Sandvik, B. and Thorlund-Petersen, L. (2010). Sensitivity analysis of risk tolerance. Decision Anal. 7, 313–321.
  • Tan, C. H. and Hartman, J. C. (2010). Equipment replacement analyis with an uncertain finite horizon. IIE Trans. 42, 342–353.
  • Tan, C. H. and Hartman, J. C. (2011). Sensitivity analysis and dynamic programming. In Wiley Encyclopedia of Operations Research and Management Science, ed. J. J. Cochran, John Wiley, New York.
  • Topaloglu, H. and Powell, W. B. (2007). Sensitivity analysis of a dynamic fleet management model using approximate dynamic programming. Operat. Res. 55, 319–331.
  • Veinott, A. F., Jr. and Wagner, H. M. (1965). Computing optimal $(s, S)$ inventory policies. Manag. Sci. 11, 525–552.
  • Wallace, S. W. (2000). Decision making under uncertainty: is sensitivity analysis of any use? Operat. Res. 48, 20–25.
  • Ward, J. E. and Wendell, R. E. (1990). Approaches to sensitivity analysis in linear programming. Ann. Operat. Res. 27, 3–38.
  • Wendell, R. E. (1985). The tolerance approach to sensitivity analysis in linear programming. Manag. Sci. 31, 564–578.
  • White, C. C. and El-Deib, H. K. (1986). Parameter imprecision in finite state, finite action dynamic programs. Operat. Res. 34, 120–129.
  • White, D. J. (1993). Markov Decision Processes. John Wiley, Chichester.