Institute of Mathematical Statistics Collections

Bounding Stationary Expectations of Markov Processes

Peter W. Glynn and Assaf Zeevi

Full-text: Open access


This paper develops a simple and systematic approach for obtaining bounds on stationary expectations of Markov processes. Given a function f which one is interested in evaluating, the main idea is to find a function g that satisfies a certain “mean drift” inequality with respect to f, which in turn leads to bounds on the stationary expectation of the latter. The approach developed in the paper is broadly applicable and can be used to bound steady-state expectations in general state space Markov chains, continuous time chains, and diffusion processes (with, or without, reflecting boundaries).

Chapter information

Stewart N. Ethier, Jin Feng and Richard H. Stockbridge, eds., Markov Processes and Related Topics: A Festschrift for Thomas G. Kurtz (Beachwood, Ohio, USA: Institute of Mathematical Statistics, 2008), 195-214

First available in Project Euclid: 28 January 2009

Permanent link to this document

Digital Object Identifier

Copyright © 2008, Institute of Mathematical Statistics


Glynn, Peter W.; Zeevi, Assaf. Bounding Stationary Expectations of Markov Processes. Markov Processes and Related Topics: A Festschrift for Thomas G. Kurtz, 195--214, Institute of Mathematical Statistics, Beachwood, Ohio, USA, 2008. doi:10.1214/074921708000000381.

Export citation


  • [1] Asmussen, S. (2003). Applied Probability and Queues. Springer, NY.
  • [2] Bertsimas, D., Gamarnik, D. and Tsitsiklis, J. (2001). Performance of multiclass Markovian queueing networks via piecewise linear Lyapunov functions. Ann. Appl. Probab. 11 1384–1428.
  • [3] Bertsimas, D., Paschalidis, I. and Tsitsiklis, J. (1994). Optimization of multiclass queueing networks: Polyhedral and nonlinear characterizations of achievable performance. Ann. Appl. Probab. 4 43–75.
  • [4] Borovkov, A. A. (2000). Ergodicity and Stability of Stochastic Processes. John Wiley & Sons, New York.
  • [5] Budhiraja, A. and Lee, C. (2007). Long time asymptotics for constrained diffusions in polyhedral domains. Stochastic Processes and Their Applications 117 1014–1036.
  • [6] Dai, J. and Harrison, J. M. (2008). Reflecting Brownian motion in the orthant: An illuminating example of instability. Math. of Oper. Res., to appear.
  • [7] Dupuis, P. and Williams, R. J. (1994). Lyapunov functions for semimartingale reflecting Brownian motions. Ann. Probab. 22 680–702.
  • [8] Fayolle, G. (1989). On random walks arising in queueing systems: Ergodicity and transience via quadratic forms as Lyapunov functions. Part I. Queueing Systems 5 167–184.
  • [9] Gamarnik, D. and Zeevi, A. (2006). Validity of heavy-traffic steady-state approximations in generalized Jackson networks. Ann. Appl. Probab. 16 56–90.
  • [10] Glynn, P. W. and Meyn, S. P. (1996). A Liapounov bound for solutions of the Poisson equation. Ann. Probab. 24 916–931.
  • [11] Hajek, B. (1982). Hitting-time and occupation-time bounds implied by drift analysis with applications. Ann. Appl. Prob. 14 502–525.
  • [12] Has’minskii, R. Z. (1980). Stochastic Stability of Differential Equations. Monographs and Textbooks on Mechanics of Solids and Fluids: Mechanics and Analysis 7. Germantown, Md.
  • [13] Karlin, S. and Taylor, H. (1981). A Second Course in Stochastic Processes. Academic Press, San Diego.
  • [14] Karatzas, I. and Shreve, S. E. (1991). Brownian Motion and Stochastic Calculus. Springer.
  • [15] Kingman, J. F. C. (1962). Some inequalities for the queue GI/G/1. Biometrica 49 315–324.
  • [16] Kumar, P. R. and Meyn, S. P. (1996). Duality and linear programs for sta- bility and performance analysis of queueing networks and scheduling policies. IEEE Trans. on Automatic Control 41 4–17.
  • [17] Kumar, S. and Kumar, P. R. (1994). Performance bounds for queueing networks and scheduling policies. IEEE Trans. on Automatic Control 39 1600–1611.
  • [18] Lasserre, J. (2002). Bounds on measures satisfying moment conditions. Ann. Appl. Probab. 12 1114–1137.
  • [19] Lions, P.-L. and Sznitman, A.-S. (1984). Stochastic differential equations with reflecting boundary conditions. Comm. Pure Appl. Math. 37 511–537.
  • [20] Meyn, S. P. and Tweedie, R. L. (1993). Markov Chains and Stochastic Stability. Communications and Control Engineering Series. Springer-Verlag, London, 1993.
  • [21] Meyn, S. P. and Tweedie, R. L. (1993). Stability of Markovian processes III: Foster–Lyapunov criteria for continuous time processes. Adv. Appl. Probab. 25 518–548.
  • [22] Sigman, K. and Yao, D. (1993). Finite moments for inventory processes. Ann. Appl. Probab. 3 765–778.
  • [23] Taylor, L. M. and Williams, R. J. (1993). Existence and uniqueness of semi-martingale reflecting Brownian motion in the orthant. Probab. Theory Related Fields 96 283–317.
  • [24] Tweedie, R. (1983). The existence of moments for stationary Markov chains. J. Appl. Probab. 20 191–196.