Translator Disclaimer
December 2015 Countable state Markov decision processes with unbounded jump rates and discounted cost: optimality equation and approximations
H. Blok, F. M. Spieksma
Author Affiliations +
Adv. in Appl. Probab. 47(4): 1088-1107 (December 2015). DOI: 10.1239/aap/1449859801


This paper considers Markov decision processes (MDPs) with unbounded rates, as a function of state. We are especially interested in studying structural properties of optimal policies and the value function. A common method to derive such properties is by value iteration applied to the uniformised MDP. However, due to the unboundedness of the rates, uniformisation is not possible, and so value iteration cannot be applied in the way we need. To circumvent this, one can perturb the MDP. Then we need two results for the perturbed sequence of MDPs: 1. there exists a unique solution to the discounted cost optimality equation for each perturbation as well as for the original MDP; 2. if the perturbed sequence of MDPs converges in a suitable manner then the associated optimal policies and the value function should converge as well. We can model both the MDP and perturbed MDPs as a collection of parametrised Markov processes. Then both of the results above are essentially implied by certain continuity properties of the process as a function of the parameter. In this paper we deduce tight verifiable conditions that imply the necessary continuity properties. The most important of these conditions are drift conditions that are strongly related to nonexplosiveness.


Download Citation

H. Blok. F. M. Spieksma. "Countable state Markov decision processes with unbounded jump rates and discounted cost: optimality equation and approximations." Adv. in Appl. Probab. 47 (4) 1088 - 1107, December 2015.


Published: December 2015
First available in Project Euclid: 11 December 2015

zbMATH: 1331.90094
MathSciNet: MR3433297
Digital Object Identifier: 10.1239/aap/1449859801

Primary: 90C40
Secondary: 60J27 , 93E20

Keywords: discounted cost , drift conditions , nonexplosiveness , Parametrised Markov processes , perturbed MDPs

Rights: Copyright © 2015 Applied Probability Trust


This article is only available to subscribers.
It is not available for individual sale.

Vol.47 • No. 4 • December 2015
Back to Top