The Annals of Applied Probability

Ergodicity of inhomogeneous Markov chains through asymptotic pseudotrajectories

Michel Benaïm, Florian Bouguet, and Bertrand Cloez

Full-text: Access denied (no subscription detected)

We're sorry, but we are unable to provide you with the full text of this article because we are not able to identify you as a subscriber. If you have a personal subscription to this journal, then please login. If you are already logged in, then you may need to update your profile to register your subscription. Read more about accessing full-text


In this work, we consider an inhomogeneous (discrete time) Markov chain and are interested in its long time behavior. We provide sufficient conditions to ensure that some of its asymptotic properties can be related to the ones of a homogeneous (continuous time) Markov process. Renowned examples such as a bandit algorithms, weighted random walks or decreasing step Euler schemes are included in our framework. Our results are related to functional limit theorems, but the approach differs from the standard “Tightness/Identification” argument; our method is unified and based on the notion of pseudotrajectories on the space of probability measures.

Article information

Ann. Appl. Probab., Volume 27, Number 5 (2017), 3004-3049.

Received: February 2016
Revised: September 2016
First available in Project Euclid: 3 November 2017

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 60J10: Markov chains (discrete-time Markov processes on discrete state spaces)
Secondary: 60J25: Continuous-time Markov processes on general state spaces 60B10: Convergence of probability measures

Markov chain Markov process asymptotic pseudotrajectory quantitative ergodicity random walk bandit algorithm decreasing step Euler scheme


Benaïm, Michel; Bouguet, Florian; Cloez, Bertrand. Ergodicity of inhomogeneous Markov chains through asymptotic pseudotrajectories. Ann. Appl. Probab. 27 (2017), no. 5, 3004--3049. doi:10.1214/17-AAP1275.

Export citation


  • [1] Ané, C., Blachère, S., Chafaï, D., Fougères, P., Gentil, I., Malrieu, F., Roberto, C. and Scheffer, G. (2000). Sur les Inégalités de Sobolev Logarithmiques. Panoramas et Synthèses [Panoramas and Syntheses] 10. Société Mathématique de France, Paris.
  • [2] Bakry, D. (1994). L’hypercontractivité et son utilisation en théorie des semigroupes. In Lectures on Probability Theory (Saint-Flour, 1992). Lecture Notes in Math. 1581 1–114. Springer, Berlin.
  • [3] Bally, V. and Rabiet, V. (2015). Asymptotic behavior for multi-scale PDMP’s. Preprint, HAL.
  • [4] Benaï m, M. (1999). Dynamics of stochastic approximation algorithms. In Séminaire de Probabilités, XXXIII. Lecture Notes in Math. 1709 1–68. Springer, Berlin.
  • [5] Benaï m, M. and Hirsch, M. W. (1996). Asymptotic pseudotrajectories and chain recurrent flows, with applications. J. Dynam. Differential Equations 8 141–176.
  • [6] Billingsley, P. (1999). Convergence of Probability Measures, 2nd ed. Wiley, New York.
  • [7] Bouguet, F. (2015). Quantitative speeds of convergence for exposure to food contaminants. ESAIM Probab. Stat. 19 482–501.
  • [8] Bouguet, F., Malrieu, F., Panloup, F., Poquet, C. and Reygner, J. (2015). Long time behavior of Markov processes and beyond. In Modélisation Aléatoire et Statistique—Journées MAS 2014. ESAIM Proc. Surveys 51 193–211. EDP Sci., Les Ulis.
  • [9] Chafaï, D., Guédon, O., Lecué, G. and Pajor, A. (2012). Interactions Between Compressed Sensing Random Matrices and High Dimensional Geometry. Panoramas et Synthèses [Panoramas and Syntheses] 37. Société Mathématique de France, Paris.
  • [10] Chen, M.-F. (2004). From Markov Chains to Non-equilibrium Particle Systems, 2nd ed. World Scientific Co., Inc., River Edge, NJ.
  • [11] Cloez, B. (2012). Wasserstein decay of one dimensional jump-diffusions. ArXiv e-prints.
  • [12] Cloez, B. and Hairer, M. (2015). Exponential ergodicity for Markov processes with random switching. Bernoulli 21 505–536.
  • [13] Davis, M. H. A. (1993). Markov Models and Optimization. Monographs on Statistics and Applied Probability 49. Chapman & Hall, London.
  • [14] Douc, R., Fort, G. and Guillin, A. (2009). Subgeometric rates of convergence of $f$-ergodic strong Markov processes. Stochastic Process. Appl. 119 897–923.
  • [15] Duflo, M. (1996). Algorithmes Stochastiques. Mathématiques & Applications (Berlin) [Mathematics & Applications] 23. Springer, Berlin.
  • [16] Eberle, A. (2011). Reflection coupling and Wasserstein contractivity without convexity. C. R. Math. Acad. Sci. Paris 349 1101–1104.
  • [17] Ethier, S. N. and Kurtz, T. G. (1986). Markov Processes: Characterization and Convergence. Wiley, New York.
  • [18] Fort, G. (2015). Central limit theorems for stochastic approximation with controlled Markov chain dynamics. ESAIM Probab. Stat. 19 60–80.
  • [19] Gadat, S., Panloup, F. and Saadane, S. (2015). Regret bounds for Narendra–Shapiro bandit algorithms. ArXiv e-prints.
  • [20] Hairer, M. (2010). Convergence of Markov processes. Available at
  • [21] Hairer, M. (2011). On Malliavin’s proof of Hörmander’s theorem. Bull. Sci. Math. 135 650–666.
  • [22] Hairer, M., Hutzenthaler, M. and Jentzen, A. (2015). Loss of regularity for Kolmogorov equations. Ann. Probab. 43 468–527.
  • [23] Hairer, M. and Mattingly, J. C. (2011). Yet another look at Harris’ ergodic theorem for Markov chains. In Seminar on Stochastic Analysis, Random Fields and Applications VI. Progress in Probability 63 109–117. Birkhäuser/Springer Basel AG, Basel.
  • [24] Hairer, M., Mattingly, J. C. and Scheutzow, M. (2011). Asymptotic coupling and a general form of Harris’ theorem with applications to stochastic delay equations. Probab. Theory Related Fields 149 223–259.
  • [25] Jacod, J. and Shiryaev, A. N. (2003). Limit Theorems for Stochastic Processes, 2nd ed. Grundlehren der Mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences] 288. Springer, Berlin.
  • [26] Jarner, S. F. and Roberts, G. O. (2002). Polynomial convergence rates of Markov chains. Ann. Appl. Probab. 12 224–247.
  • [27] Joffe, A. and Métivier, M. (1986). Weak convergence of sequences of semimartingales with applications to multitype branching processes. Adv. in Appl. Probab. 18 20–65.
  • [28] Kallenberg, O. (2002). Foundations of Modern Probability, 2nd ed. Springer, New York.
  • [29] Kunita, H. (1984). Stochastic differential equations and stochastic flows of diffeomorphisms. In École D’été de Probabilités de Saint-Flour, XII—1982. Lecture Notes in Math. 1097 143–303. Springer, Berlin.
  • [30] Kushner, H. J. and Yin, G. G. (2003). Stochastic Approximation and Recursive Algorithms and Applications, 2nd ed. Stochastic Modelling and Applied Probability 35. Springer, New York.
  • [31] Lamberton, D. and Pagès, G. (2002). Recursive computation of the invariant distribution of a diffusion. Bernoulli 8 367–405.
  • [32] Lamberton, D. and Pagès, G. (2008). A penalized bandit algorithm. Electron. J. Probab. 13 341–373.
  • [33] Laruelle, S. and Pagès, G. (2013). Randomized urn models revisited using stochastic approximation. Ann. Appl. Probab. 23 1409–1436.
  • [34] Lemaire, V. (2005). Estimation récursive de la mesure invariante d’un processus de diffusion. Theses, Univ. de Marne la Vallée.
  • [35] Lindvall, T. (1992). Lectures on the Coupling Method. Wiley, New York.
  • [36] Meyn, S. P. and Tweedie, R. L. (1993). Stability of Markovian processes. III. Foster–Lyapunov criteria for continuous-time processes. Adv. in Appl. Probab. 25 518–548.
  • [37] Monmarché, P. (2014). Hypocoercive relaxation to equilibrium for some kinetic models. Kinet. Relat. Models 7 341–360.
  • [38] Rachev, S. T., Klebanov, L. B., Stoyanov, S. V. and Fabozzi, F. J. (2013). The Methods of Distances in the Theory of Probability and Statistics. Springer, New York.
  • [39] Talay, D. (1984). Efficient numerical schemes for the approximation of expectations of functionals of the solution of a SDE and applications. In Filtering and Control of Random Processes (Paris, 1983). Lect. Notes Control Inf. Sci. 61 294–313. Springer, Berlin.
  • [40] Talay, D. and Tubaro, L. (1990). Expansion of the global error for numerical schemes solving stochastic differential equations. Stoch. Anal. Appl. 8 483–509.