## The Annals of Applied Probability

### Ergodicity of inhomogeneous Markov chains through asymptotic pseudotrajectories

#### Abstract

In this work, we consider an inhomogeneous (discrete time) Markov chain and are interested in its long time behavior. We provide sufficient conditions to ensure that some of its asymptotic properties can be related to the ones of a homogeneous (continuous time) Markov process. Renowned examples such as a bandit algorithms, weighted random walks or decreasing step Euler schemes are included in our framework. Our results are related to functional limit theorems, but the approach differs from the standard “Tightness/Identification” argument; our method is unified and based on the notion of pseudotrajectories on the space of probability measures.

#### Article information

Source
Ann. Appl. Probab., Volume 27, Number 5 (2017), 3004-3049.

Dates
Revised: September 2016
First available in Project Euclid: 3 November 2017

Permanent link to this document
https://projecteuclid.org/euclid.aoap/1509696040

Digital Object Identifier
doi:10.1214/17-AAP1275

Mathematical Reviews number (MathSciNet)
MR3719952

Zentralblatt MATH identifier
1379.60077

#### Citation

Benaïm, Michel; Bouguet, Florian; Cloez, Bertrand. Ergodicity of inhomogeneous Markov chains through asymptotic pseudotrajectories. Ann. Appl. Probab. 27 (2017), no. 5, 3004--3049. doi:10.1214/17-AAP1275. https://projecteuclid.org/euclid.aoap/1509696040

#### References

• [1] Ané, C., Blachère, S., Chafaï, D., Fougères, P., Gentil, I., Malrieu, F., Roberto, C. and Scheffer, G. (2000). Sur les Inégalités de Sobolev Logarithmiques. Panoramas et Synthèses [Panoramas and Syntheses] 10. Société Mathématique de France, Paris.
• [2] Bakry, D. (1994). L’hypercontractivité et son utilisation en théorie des semigroupes. In Lectures on Probability Theory (Saint-Flour, 1992). Lecture Notes in Math. 1581 1–114. Springer, Berlin.
• [3] Bally, V. and Rabiet, V. (2015). Asymptotic behavior for multi-scale PDMP’s. Preprint, HAL.
• [4] Benaï m, M. (1999). Dynamics of stochastic approximation algorithms. In Séminaire de Probabilités, XXXIII. Lecture Notes in Math. 1709 1–68. Springer, Berlin.
• [5] Benaï m, M. and Hirsch, M. W. (1996). Asymptotic pseudotrajectories and chain recurrent flows, with applications. J. Dynam. Differential Equations 8 141–176.
• [6] Billingsley, P. (1999). Convergence of Probability Measures, 2nd ed. Wiley, New York.
• [7] Bouguet, F. (2015). Quantitative speeds of convergence for exposure to food contaminants. ESAIM Probab. Stat. 19 482–501.
• [8] Bouguet, F., Malrieu, F., Panloup, F., Poquet, C. and Reygner, J. (2015). Long time behavior of Markov processes and beyond. In Modélisation Aléatoire et Statistique—Journées MAS 2014. ESAIM Proc. Surveys 51 193–211. EDP Sci., Les Ulis.
• [9] Chafaï, D., Guédon, O., Lecué, G. and Pajor, A. (2012). Interactions Between Compressed Sensing Random Matrices and High Dimensional Geometry. Panoramas et Synthèses [Panoramas and Syntheses] 37. Société Mathématique de France, Paris.
• [10] Chen, M.-F. (2004). From Markov Chains to Non-equilibrium Particle Systems, 2nd ed. World Scientific Co., Inc., River Edge, NJ.
• [11] Cloez, B. (2012). Wasserstein decay of one dimensional jump-diffusions. ArXiv e-prints.
• [12] Cloez, B. and Hairer, M. (2015). Exponential ergodicity for Markov processes with random switching. Bernoulli 21 505–536.
• [13] Davis, M. H. A. (1993). Markov Models and Optimization. Monographs on Statistics and Applied Probability 49. Chapman & Hall, London.
• [14] Douc, R., Fort, G. and Guillin, A. (2009). Subgeometric rates of convergence of $f$-ergodic strong Markov processes. Stochastic Process. Appl. 119 897–923.
• [15] Duflo, M. (1996). Algorithmes Stochastiques. Mathématiques & Applications (Berlin) [Mathematics & Applications] 23. Springer, Berlin.
• [16] Eberle, A. (2011). Reflection coupling and Wasserstein contractivity without convexity. C. R. Math. Acad. Sci. Paris 349 1101–1104.
• [17] Ethier, S. N. and Kurtz, T. G. (1986). Markov Processes: Characterization and Convergence. Wiley, New York.
• [18] Fort, G. (2015). Central limit theorems for stochastic approximation with controlled Markov chain dynamics. ESAIM Probab. Stat. 19 60–80.
• [19] Gadat, S., Panloup, F. and Saadane, S. (2015). Regret bounds for Narendra–Shapiro bandit algorithms. ArXiv e-prints.
• [20] Hairer, M. (2010). Convergence of Markov processes. Available at http://www.hairer.org/notes/Convergence.pdf.
• [21] Hairer, M. (2011). On Malliavin’s proof of Hörmander’s theorem. Bull. Sci. Math. 135 650–666.
• [22] Hairer, M., Hutzenthaler, M. and Jentzen, A. (2015). Loss of regularity for Kolmogorov equations. Ann. Probab. 43 468–527.
• [23] Hairer, M. and Mattingly, J. C. (2011). Yet another look at Harris’ ergodic theorem for Markov chains. In Seminar on Stochastic Analysis, Random Fields and Applications VI. Progress in Probability 63 109–117. Birkhäuser/Springer Basel AG, Basel.
• [24] Hairer, M., Mattingly, J. C. and Scheutzow, M. (2011). Asymptotic coupling and a general form of Harris’ theorem with applications to stochastic delay equations. Probab. Theory Related Fields 149 223–259.
• [25] Jacod, J. and Shiryaev, A. N. (2003). Limit Theorems for Stochastic Processes, 2nd ed. Grundlehren der Mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences] 288. Springer, Berlin.
• [26] Jarner, S. F. and Roberts, G. O. (2002). Polynomial convergence rates of Markov chains. Ann. Appl. Probab. 12 224–247.
• [27] Joffe, A. and Métivier, M. (1986). Weak convergence of sequences of semimartingales with applications to multitype branching processes. Adv. in Appl. Probab. 18 20–65.
• [28] Kallenberg, O. (2002). Foundations of Modern Probability, 2nd ed. Springer, New York.
• [29] Kunita, H. (1984). Stochastic differential equations and stochastic flows of diffeomorphisms. In École D’été de Probabilités de Saint-Flour, XII—1982. Lecture Notes in Math. 1097 143–303. Springer, Berlin.
• [30] Kushner, H. J. and Yin, G. G. (2003). Stochastic Approximation and Recursive Algorithms and Applications, 2nd ed. Stochastic Modelling and Applied Probability 35. Springer, New York.
• [31] Lamberton, D. and Pagès, G. (2002). Recursive computation of the invariant distribution of a diffusion. Bernoulli 8 367–405.
• [32] Lamberton, D. and Pagès, G. (2008). A penalized bandit algorithm. Electron. J. Probab. 13 341–373.
• [33] Laruelle, S. and Pagès, G. (2013). Randomized urn models revisited using stochastic approximation. Ann. Appl. Probab. 23 1409–1436.
• [34] Lemaire, V. (2005). Estimation récursive de la mesure invariante d’un processus de diffusion. Theses, Univ. de Marne la Vallée.
• [35] Lindvall, T. (1992). Lectures on the Coupling Method. Wiley, New York.
• [36] Meyn, S. P. and Tweedie, R. L. (1993). Stability of Markovian processes. III. Foster–Lyapunov criteria for continuous-time processes. Adv. in Appl. Probab. 25 518–548.
• [37] Monmarché, P. (2014). Hypocoercive relaxation to equilibrium for some kinetic models. Kinet. Relat. Models 7 341–360.
• [38] Rachev, S. T., Klebanov, L. B., Stoyanov, S. V. and Fabozzi, F. J. (2013). The Methods of Distances in the Theory of Probability and Statistics. Springer, New York.
• [39] Talay, D. (1984). Efficient numerical schemes for the approximation of expectations of functionals of the solution of a SDE and applications. In Filtering and Control of Random Processes (Paris, 1983). Lect. Notes Control Inf. Sci. 61 294–313. Springer, Berlin.
• [40] Talay, D. and Tubaro, L. (1990). Expansion of the global error for numerical schemes solving stochastic differential equations. Stoch. Anal. Appl. 8 483–509.