The Annals of Applied Probability

The ODE method for stability of skip-free Markov chains with applications to MCMC

Gersende Fort, Sean Meyn, Eric Moulines, and Pierre Priouret

Full-text: Open access

Abstract

Fluid limit techniques have become a central tool to analyze queueing networks over the last decade, with applications to performance analysis, simulation and optimization.

In this paper, some of these techniques are extended to a general class of skip-free Markov chains. As in the case of queueing models, a fluid approximation is obtained by scaling time, space and the initial condition by a large constant. The resulting fluid limit is the solution of an ordinary differential equation (ODE) in “most” of the state space. Stability and finer ergodic properties for the stochastic model then follow from stability of the set of fluid limits. Moreover, similarly to the queueing context where fluid models are routinely used to design control policies, the structure of the limiting ODE in this general setting provides an understanding of the dynamics of the Markov chain. These results are illustrated through application to Markov chain Monte Carlo methods.

Article information

Source
Ann. Appl. Probab., Volume 18, Number 2 (2008), 664-707.

Dates
First available in Project Euclid: 20 March 2008

Permanent link to this document
https://projecteuclid.org/euclid.aoap/1206018201

Digital Object Identifier
doi:10.1214/07-AAP471

Mathematical Reviews number (MathSciNet)
MR2399709

Zentralblatt MATH identifier
1148.60052

Subjects
Primary: 60J10: Markov chains (discrete-time Markov processes on discrete state spaces) 65C05: Monte Carlo methods

Keywords
Markov chain fluid limit subgeometric ergodicity state-dependent drift criteria Markov chain Monte Carlo Metropolis–Hastings algorithms

Citation

Fort, Gersende; Meyn, Sean; Moulines, Eric; Priouret, Pierre. The ODE method for stability of skip-free Markov chains with applications to MCMC. Ann. Appl. Probab. 18 (2008), no. 2, 664--707. doi:10.1214/07-AAP471. https://projecteuclid.org/euclid.aoap/1206018201


Export citation

References

  • Billingsley, P. (1999). Convergence of Probability Measures, 2nd ed. Wiley, New York.
  • Birnbaum, Z. and Marshall, A. (1961). Some multivariate Chebyshev inequalities with extensions to continuous parameter processes. Ann. Math. Statist. 32 687–703.
  • Chen, H. and Mandelbaum, A. (1991). Discrete flow networks: Bottleneck analysis and fluid approximations. Math. Oper. Res. 16 408–446.
  • Chen, H. and Yao, D. D. (2001). Fundamentals of Queueing Networks. Springer, New York.
  • Chen, R. and Meyn, S. (1999). Value iteration and optimization of multiclass queueing networks. Queueing Systems Theory Appl. 32 65–97.
  • Cruz, R. L. (1991). A calculus for network delay. II. Network analysis. IEEE Trans. Inform. Theory 37 132–141.
  • Dai, J. (1995). On positive Harris recurrence for multiclass queuing networks: A unified approach via fluid limit models. Ann. Appl. Probab. 5 49–77.
  • Dai, J. and Weiss, G. (1996). Stability and instability of fluid models for reentrant lines. Math. Oper. Res. 21 115–134.
  • Dai, J. G. (1996). A fluid limit model criterion for instability of multiclass queueing networks. Ann. Appl. Probab. 6 751–757.
  • Dai, J. G., Hasenbein, J. J. and Vande Vate, J. H. (2004). Stability and instability of a two-station queueing network. Ann. Appl. Probab. 14 326–377.
  • Dai, J. G. and Meyn, S. P. (1995). Stability and convergence of moments for multiclass queueing networks via fluid limit models. IEEE Trans. Automat. Control 40 1889–1904.
  • Douc, R., Fort, G., Moulines, E. and Soulier, P. (2004). Practical drift conditions for subgeometric rates of convergence. Ann. Appl. Probab. 14 1353–1377.
  • Filonov, Y. (1989). Criterion for ergodicity of homogeneous discrete Markov chains. Ukrainian Math. J. 41 1223–1225.
  • Fort, G., Meyn, S., Moulines, E. and Priouret, P. (2006). ODE methods for Markov chain stability with applications to MCMC. In Proc. 1st Int. Conf. on Performance Evaluation Methodology and Tools. ACM, New York.
  • Fort, G. and Moulines, E. (2003). Polynomial ergodicity of Markov transition kernels. Stochastic Process. Appl. 103 57–99.
  • Gamarnik, D. and Hasenbein, J. (2005). Instability in stochastic and fluid queueing networks. Ann. Appl. Probab. 15 1652–1690.
  • Gamarnik, D. and Meyn, S. P. (2005). On exponential ergodicity in multiclass queueing networks. In Asymptotic Analysis of Stochastic Systems, Invited Session at the INFORMS Annual Meeting.
  • Glynn, P. W. and Meyn, S. P. (1996). A Liapounov bound for solutions of the Poisson equation. Ann. Probab. 24 916–931.
  • Harrison, J. M. (2000). Brownian models of open processing networks: Canonical representation of workload. Ann. Appl. Probab. 10 75–103.
  • Henderson, S. and Meyn, S. (1997). Efficient simulation of multiclass queueing networks. In Simulation Conference Proceedings 216–223.
  • Henderson, S. G., Meyn, S. P. and Tadić, V. B. (2003). Performance evaluation and policy selection in multiclass networks. Discrete Event Dyn. Syst. 13 149–189.
  • Jarner, S. and Hansen, E. (2000). Geometric ergodicity of Metropolis algorithms. Stochastic Proc. Appl. 85 341–361.
  • Krasnosel’skij, M. and Rutitskij, Y. (1961). Convex Functions and Orlicz Spaces. P. Noordhoff Ltd., The Netherlands.
  • Kurtz, T. (1970). Solutions of ordinary differential equations as limits of pure jump Markov processes. J. Appl. Probab. 7 49–58.
  • Malyšev, V. A. and Menc’šikov, M. V. (1979). Ergodicity, continuity and analyticity of countable Markov chains. Trudy Moskov. Mat. Obshch. 39 3–48, 235.
  • Meyn, S. P. (1995). Transience of multiclass queueing networks via fluid limit models. Ann. Appl. Probab. 5 946–957.
  • Meyn, S. P. (2001). Sequencing and routing in multiclass queueing networks. I. Feedback regulation. SIAM J. Control Optim. 40 741–776 (electronic).
  • Meyn, S. P. (2005). Workload models for stochastic networks: Value functions and performance evaluation. IEEE Trans. Automat. Control 50 1106–1122.
  • Meyn, S. P. (2006). Large deviation asymptotics and control variates for simulating large functions. Ann. Appl. Probab. 16 310–339.
  • Meyn, S. P. (2007). Control Techniques for Complex Networks. Cambridge Univ. Press.
  • Meyn, S. P. and Tweedie, R. L. (1993). Markov Chains and Stochastic Stability. Springer, London.
  • Meyn, S. P. and Tweedie, R. L. (1994). State dependent criteria for convergence of Markov chains. Ann. Appl. Probab. 4 149–168.
  • Newell, G. (1982). Applications of Queueing Theory, 2nd ed. Chapman and Hall, London.
  • Nummelin, E. and Tuominen, P. (1983). The rate of convergence in Orey’s theorem for Harris recurrent Markov chains with applications to renewal theory. Stochastic Process. Appl. 15 295–311.
  • Patra, K. and Dey, D. K. (1999). A multivariate mixture of Weibull distributions in reliability modeling. Statist. Probab. Lett. 45 225–235.
  • Robert, C. P. (1998). Discretization and MCMC Convergence Assessment. Springer, New York.
  • Robert, C. P. and Casella, G. (2004). Monte Carlo Statistical Methods, 2nd ed. Springer, New York.
  • Robert, P. (2000). Réseaux et files d’attente: Méthodes probabilistes. Springer, Berlin.
  • Roberts, G. O. and Tweedie, R. L. (1996). Geometric convergence and central limit theorems for multidimensional Hastings and Metropolis algorithms. Biometrika 83 95–110.
  • Stolyar, A. L. (1995). On the stability of multiclass queueing networks: A relaxed sufficient condition via limiting fluid processes. Markov Process. Related Fields 1 491–512.
  • Tuominen, P. and Tweedie, R. (1994). Subgeometric rates of convergence of f-ergodic Markov chains. Adv. in Appl. Probab. 26 775–798.
  • Verhulst, F. (1996). Nonlinear Differential Equations and Dynamical Systems, 2nd ed. Springer, Berlin.
  • Whitt, W. (2002). Stochastic-Process Limits. An Introduction to Stochastic-Process Limits and Their Application to Queues. Springer, New York.