## Journal of Applied Probability

### Sensitivity analysis in Markov decision processes with uncertain reward parameters

#### Abstract

Sequential decision problems can often be modeled as Markov decision processes. Classical solution approaches assume that the parameters of the model are known. However, model parameters are usually estimated and uncertain in practice. As a result, managers are often interested in how estimation errors affect the optimal solution. In this paper we illustrate how sensitivity analysis can be performed directly for a Markov decision process with uncertain reward parameters using the Bellman equations. In particular, we consider problems involving (i) a single stationary parameter, (ii) multiple stationary parameters, and (iii) multiple nonstationary parameters. We illustrate the applicability of this work through a capacitated stochastic lot-sizing problem.

#### Article information

Source
J. Appl. Probab., Volume 48, Number 4 (2011), 954-967.

Dates
First available in Project Euclid: 16 December 2011

https://projecteuclid.org/euclid.jap/1324046012

Digital Object Identifier
doi:10.1239/jap/1324046012

Mathematical Reviews number (MathSciNet)
MR2896661

Zentralblatt MATH identifier
1231.90374

#### Citation

Tan, Chin Hon; Hartman, Joseph C. Sensitivity analysis in Markov decision processes with uncertain reward parameters. J. Appl. Probab. 48 (2011), no. 4, 954--967. doi:10.1239/jap/1324046012. https://projecteuclid.org/euclid.jap/1324046012

#### References

• Bazaraa, M. S., Jarvis, J. J. and Sherali, H. D. (2005). Linear Programming and Network Flows, 3rd edn. John Wiley, Hoboken, NJ.
• Bellman, R. (1957). Dynamic Programming. Princeton University Press.
• Charalambous, C. and Gittins, J. C. (2008). Optimal selection policies for a sequence of candidate drugs. Adv. Appl. Prob. 40, 359–376.
• Erkin, Z. \et (2010). Eliciting patients' revealed preferences: an inverse Markov decision process approach. Decision Anal. 7, 358–365.
• Gal, T. and Greenberg, H. J. (1997). Advances in Sensitivity Analysis and Parametric Programming. Kluwer, Dordrecht.
• Glazebrook, K. D., Ansell, P. S., Dunn, R. T. and Lumley R. R. (2004). On the optimal allocation of service to impatient tasks. J. Appl. Prob. 41, 51–72.
• Harmanec, D. (2002). Generalizing Markov decision processes to imprecise probabilities. J. Statist. Planning Infer. 105, 199–213.
• Hopp, W. J. (1988). Sensitivity analysis in discrete dynamic programming. J. Optimization Theory Appl. 56, 257–269.
• Iyengar, G. N. (2005). Robust dynamic programming. Math. Operat. Res. 30, 257–280.
• Lim, C., Bearden, J. N. and Smith, J. C. (2006). Sequential search with multiattribute options. Decision Anal. 3, 3–15.
• Manne, A. S. (1960). Linear programming and sequential decisions. Manag. Sci. 6, 259–267.
• Mitrophanov A. Y., Lomsadze A. and Borodovsky M. (2005). Sensitivity of hidden Markov models. J. Appl. Prob. 42, 632–642.
• Muckstadt, J. A. and Sapra A. (2010). Principles of Inventory Management. Springer, New York.
• Nilim, A. and El Ghaoui, L. (2005). Robust control of Markov decision processes with uncertain transition matrices. Operat. Res. 53, 780–798.
• Powell, W. B. (2007). Approximate Dynamic Programming. John Wiley, Hoboken, NJ.
• Puterman, M. L. (1994). Markov Decision Processes. Discrete Stochastic Dynamic Programming. John Wiley, New York.
• Sandvik, B. and Thorlund-Petersen, L. (2010). Sensitivity analysis of risk tolerance. Decision Anal. 7, 313–321.
• Tan, C. H. and Hartman, J. C. (2010). Equipment replacement analyis with an uncertain finite horizon. IIE Trans. 42, 342–353.
• Tan, C. H. and Hartman, J. C. (2011). Sensitivity analysis and dynamic programming. In Wiley Encyclopedia of Operations Research and Management Science, ed. J. J. Cochran, John Wiley, New York.
• Topaloglu, H. and Powell, W. B. (2007). Sensitivity analysis of a dynamic fleet management model using approximate dynamic programming. Operat. Res. 55, 319–331.
• Veinott, A. F., Jr. and Wagner, H. M. (1965). Computing optimal $(s, S)$ inventory policies. Manag. Sci. 11, 525–552.
• Wallace, S. W. (2000). Decision making under uncertainty: is sensitivity analysis of any use? Operat. Res. 48, 20–25.
• Ward, J. E. and Wendell, R. E. (1990). Approaches to sensitivity analysis in linear programming. Ann. Operat. Res. 27, 3–38.
• Wendell, R. E. (1985). The tolerance approach to sensitivity analysis in linear programming. Manag. Sci. 31, 564–578.
• White, C. C. and El-Deib, H. K. (1986). Parameter imprecision in finite state, finite action dynamic programs. Operat. Res. 34, 120–129.
• White, D. J. (1993). Markov Decision Processes. John Wiley, Chichester.