The Annals of Applied Probability

Mixing time estimation in reversible Markov chains from a single sample path

Daniel Hsu, Aryeh Kontorovich, David A. Levin, Yuval Peres, Csaba Szepesvári, and Geoffrey Wolfer

Full-text: Access denied (no subscription detected)

We're sorry, but we are unable to provide you with the full text of this article because we are not able to identify you as a subscriber. If you have a personal subscription to this journal, then please login. If you are already logged in, then you may need to update your profile to register your subscription. Read more about accessing full-text

Abstract

The spectral gap $\gamma_{\star}$ of a finite, ergodic and reversible Markov chain is an important parameter measuring the asymptotic rate of convergence. In applications, the transition matrix $\mathbf{{P}}$ may be unknown, yet one sample of the chain up to a fixed time $n$ may be observed. We consider here the problem of estimating $\gamma_{\star}$ from this data. Let $\boldsymbol{\pi}$ be the stationary distribution of $\mathbf{{P}}$, and $\pi_{\star}=\min_{x}\pi (x)$. We show that if $n$ is at least $\frac{1}{\gamma_{\star}\pi_{\star}}$ times a logarithmic correction, then $\gamma_{\star}$ can be estimated to within a multiplicative factor with high probability. When $\pi $ is uniform on $d$ states, this nearly matches a lower bound of $\frac{d}{\gamma_{\star}}$ steps required for precise estimation of $\gamma_{\star}$. Moreover, we provide the first procedure for computing a fully data-dependent interval, from a single finite-length trajectory of the chain, that traps the mixing time $t_{\mathrm{mix}}$ of the chain at a prescribed confidence level. The interval does not require the knowledge of any parameters of the chain. This stands in contrast to previous approaches, which either only provide point estimates, or require a reset mechanism, or additional prior knowledge. The interval is constructed around the relaxation time $t_{\mathrm{relax}}=1/\gamma_{\star}$, which is strongly related to the mixing time, and the width of the interval converges to zero roughly at a $1/\sqrt{n}$ rate, where $n$ is the length of the sample path.

Article information

Source
Ann. Appl. Probab., Volume 29, Number 4 (2019), 2439-2480.

Dates
Received: November 2017
Revised: July 2018
First available in Project Euclid: 23 July 2019

Permanent link to this document
https://projecteuclid.org/euclid.aoap/1563869047

Digital Object Identifier
doi:10.1214/18-AAP1457

Mathematical Reviews number (MathSciNet)
MR3983341

Zentralblatt MATH identifier
07120713

Subjects
Primary: 60J10: Markov chains (discrete-time Markov processes on discrete state spaces) 62M05: Markov processes: estimation 62M99: None of the above, but in this section

Keywords
Markov chains mixing time spectral gap empirical confidence interval

Citation

Hsu, Daniel; Kontorovich, Aryeh; Levin, David A.; Peres, Yuval; Szepesvári, Csaba; Wolfer, Geoffrey. Mixing time estimation in reversible Markov chains from a single sample path. Ann. Appl. Probab. 29 (2019), no. 4, 2439--2480. doi:10.1214/18-AAP1457. https://projecteuclid.org/euclid.aoap/1563869047


Export citation

References

  • Atchadé, Y. F. (2016). Markov chain Monte Carlo confidence intervals. Bernoulli 22 1808–1838.
    Zentralblatt MATH: 1345.60074
    Digital Object Identifier: doi:10.3150/15-BEJ712
    Project Euclid: euclid.bj/1458133000
  • Audibert, J.-Y., Munos, R. and Szepesvári, C. (2009). Exploration-exploitation tradeoff using variance estimates in multi-armed bandits. Theoret. Comput. Sci. 410 1876–1902.
    Zentralblatt MATH: 1167.68059
    Digital Object Identifier: doi:10.1016/j.tcs.2009.01.016
  • Batu, T., Fortnow, L., Rubinfeld, R., Smith, W. D. and White, P. (2000). Testing that distributions are close. In 41st Annual Symposium on Foundations of Computer Science (Redondo Beach, CA, 2000) 259–269. IEEE Comput. Soc. Press, Los Alamitos, CA.
  • Batu, T., Fortnow, L., Rubinfeld, R., Smith, W. D. and White, P. (2013). Testing closeness of discrete distributions. J. ACM 60 Art. 4, 25.
    Zentralblatt MATH: 1281.68227
    Digital Object Identifier: doi:10.1145/2432622.2432626
  • Benítez, J. and Liu, X. (2012). On the continuity of the group inverse. Oper. Matrices 6 859–868.
  • Bernstein, S. (1927). Sur l’extension du théoréme limite du calcul des probabilités aux sommes de quantités dépendantes. Math. Ann. 97 1–59.
  • Bhatnagar, N., Bogdanov, A. and Mossel, E. (2011). The computational complexity of estimating MCMC convergence time. In Approximation, Randomization, and Combinatorial Optimization. Lecture Notes in Computer Science 6845 424–435. Springer, Heidelberg.
    Zentralblatt MATH: 1343.68289
  • Bhattacharya, B. B. and Valiant, G. (2015). Testing closeness with unequal sized samples. In Advances in Neural Information Processing Systems 28 (C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama and R. Garnett, eds.) 2611–2619. Curran Associates, Red Hook, NY.
  • Bousquet, O., Boucheron, S. and Lugosi, G. (2004). Introduction to statistical learning theory. Lecture Notes in Artificial Intelligence 3176 169–207.
    Zentralblatt MATH: 1120.68428
  • Bradley, R. C. (2005). Basic properties of strong mixing conditions. A survey and some open questions. Probab. Surv. 2 107–144. Update of, and a supplement to, the 1986 original.
    Zentralblatt MATH: 1189.60077
    Digital Object Identifier: doi:10.1214/154957805100000104
  • Cho, G. E. and Meyer, C. D. (2001). Comparison of perturbation bounds for the stationary distribution of a Markov chain. Linear Algebra Appl. 335 137–150.
    Zentralblatt MATH: 0983.60062
    Digital Object Identifier: doi:10.1016/S0024-3795(01)00320-2
  • Flegal, J. M. and Jones, G. L. (2011). Implementing MCMC: Estimating with confidence. In Handbook of Markov Chain Monte Carlo. Chapman & Hall/CRC Handb. Mod. Stat. Methods 175–197. CRC Press, Boca Raton, FL.
    Zentralblatt MATH: 1229.65012
  • Freedman, D. A. (1975). On tail probabilities for martingales. Ann. Probab. 3 100–118.
    Zentralblatt MATH: 0313.60037
    Digital Object Identifier: doi:10.1214/aop/1176996452
    Project Euclid: euclid.aop/1176996452
  • Gamarnik, D. (2003). Extension of the PAC framework to finite and countable Markov chains. IEEE Trans. Inform. Theory 49 338–345.
    Zentralblatt MATH: 1063.68060
    Digital Object Identifier: doi:10.1109/TIT.2002.806131
  • Garren, S. T. and Smith, R. L. (2000). Estimating the second largest eigenvalue of a Markov transition matrix. Bernoulli 6 215–242.
    Zentralblatt MATH: 0976.62081
    Digital Object Identifier: doi:10.2307/3318575
    Project Euclid: euclid.bj/1081788027
  • Gillman, D. (1998). A Chernoff bound for random walks on expander graphs. SIAM J. Comput. 27 1203–1220.
    Zentralblatt MATH: 0911.60016
    Digital Object Identifier: doi:10.1137/S0097539794268765
  • Gyori, B. M. and Paulin, D. (2014). Non-asymptotic confidence intervals for MCMC in practice. Available at arXiv:1212.2016.
    arXiv: 1212.2016
  • Haviv, M. and Van der Heyden, L. (1984). Perturbation bounds for the stationary probabilities of a finite Markov chain. Adv. in Appl. Probab. 16 804–818.
    Zentralblatt MATH: 0559.60055
    Digital Object Identifier: doi:10.2307/1427341
  • Hayashi, M. and Watanabe, S. (2016). Information geometry approach to parameter estimation in Markov chains. Ann. Statist. 44 1495–1535.
    Zentralblatt MATH: 1347.62182
    Digital Object Identifier: doi:10.1214/15-AOS1420
    Project Euclid: euclid.aos/1467894706
  • Hsu, D., Kontorovich, A. and Szepesvári, C. (2015). Mixing time estimation in reversible Markov chains from a single sample path. In Advances in Neural Information Processing Systems 28 (C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama and R. Garnett, eds.). Curran Associates, Red Hook, NY.
  • Jones, G. L. and Hobert, J. P. (2001). Honest exploration of intractable probability distributions via Markov chain Monte Carlo. Statist. Sci. 16 312–334.
    Zentralblatt MATH: 1127.60309
    Digital Object Identifier: doi:10.1214/ss/1015346317
    Project Euclid: euclid.ss/1015346317
  • Karandikar, R. L. and Vidyasagar, M. (2002). Rates of uniform convergence of empirical means with mixing processes. Statist. Probab. Lett. 58 297–307.
    Zentralblatt MATH: 1016.60036
    Digital Object Identifier: doi:10.1016/S0167-7152(02)00124-4
  • Kipnis, C. and Varadhan, S. R. S. (1986). Central limit theorem for additive functionals of reversible Markov processes and applications to simple exclusions. Comm. Math. Phys. 104 1–19.
    Zentralblatt MATH: 0588.60058
    Digital Object Identifier: doi:10.1007/BF01210789
    Project Euclid: euclid.cmp/1104114929
  • Kirkland, S. J., Neumann, M. and Shader, B. L. (1998). Applications of Paz’s inequality to perturbation bounds for Markov chains. Linear Algebra Appl. 268 183–196.
    Zentralblatt MATH: 0891.65147
    Digital Object Identifier: doi:10.1016/S0024-3795(97)00042-6
  • Kontorovich, A. and Weiss, R. (2014). Uniform Chernoff and Dvoretzky–Kiefer–Wolfowitz-type inequalities for Markov chains and related processes. J. Appl. Probab. 51 1100–1113.
    Zentralblatt MATH: 1320.60060
    Digital Object Identifier: doi:10.1239/jap/1421763330
    Project Euclid: euclid.jap/1421763330
  • Kontoyiannis, I., Lastras-Montaño, L. A. and Meyn, S. P. (2006). Exponential bounds and stopping rules for MCMC and general Markov chains. In VALUETOOLS 45.
  • León, C. A. and Perron, F. (2004). Optimal Hoeffding bounds for discrete reversible Markov chains. Ann. Appl. Probab. 14 958–970.
    Zentralblatt MATH: 1056.60070
    Digital Object Identifier: doi:10.1214/105051604000000170
    Project Euclid: euclid.aoap/1082737118
  • Levin, D. A. and Peres, Y. (2016). Estimating the spectral gap of a reversible Markov chain from a short trajectory. Available at arXiv:1612.05330.
    arXiv: 1612.05330
  • Levin, D. A., Peres, Y. and Wilmer, E. L. (2009). Markov Chains and Mixing Times. Amer. Math. Soc., Providence, RI.
    Zentralblatt MATH: 1160.60001
  • Li, X. and Wei, Y. (2001). An improvement on the perturbation of the group inverse and oblique projection. Linear Algebra Appl. 338 53–66.
    Zentralblatt MATH: 0991.15005
    Digital Object Identifier: doi:10.1016/S0024-3795(01)00369-X
  • Liu, J. S. (2001). Monte Carlo Strategies in Scientific Computing. Springer Series in Statistics. Springer, New York.
    Zentralblatt MATH: 0991.65001
  • McDonald, D. J., Shalizi, C. R. and Schervish, M. J. (2011). Estimating beta-mixing coefficients. In International Conference on Artificial Intelligence and Statistics 516–524.
    Zentralblatt MATH: 1330.62344
    Digital Object Identifier: doi:10.1214/15-EJS1094
  • Meyer, C. D. Jr. (1975). The role of the group generalized inverse in the theory of finite Markov chains. SIAM Rev. 17 443–464.
    Zentralblatt MATH: 0313.60044
    Digital Object Identifier: doi:10.1137/1017044
  • Meyn, S. P. and Tweedie, R. L. (1993). Markov Chains and Stochastic Stability. Communications and Control Engineering Series. Springer, London.
    Zentralblatt MATH: 0925.60001
  • Mohri, M. and Rostamizadeh, A. (2008). Stability bounds for non-iid processes. In Advances in Neural Information Processing Systems 20.
    Zentralblatt MATH: 1242.68238
  • Montenegro, R. and Tetali, P. (2006). Mathematical aspects of mixing times in Markov chains. Found. Trends Theor. Comput. Sci. 1 x+121.
  • Paulin, D. (2015). Concentration inequalities for Markov chains by Marton couplings and spectral methods. Electron. J. Probab. 20 no. 79, 32.
    Zentralblatt MATH: 1342.60121
    Digital Object Identifier: doi:10.1214/EJP.v20-4039
  • Seneta, E. (1993). Sensitivity of finite Markov chains under perturbation. Statist. Probab. Lett. 17 163–168.
    Zentralblatt MATH: 0777.60065
    Digital Object Identifier: doi:10.1016/0167-7152(93)90011-7
  • Steinwart, I. and Christmann, A. (2009). Fast learning from non-i.i.d. observations. In Advances in Neural Information Processing Systems 22.
  • Steinwart, I., Hush, D. and Scovel, C. (2009). Learning from dependent observations. J. Multivariate Anal. 100 175–194.
    Zentralblatt MATH: 1158.68040
    Digital Object Identifier: doi:10.1016/j.jmva.2008.04.001
  • Stewart, G. W. and Sun, J. G. (1990). Matrix Perturbation Theory. Computer Science and Scientific Computing. Academic Press, Boston, MA.
    Zentralblatt MATH: 0706.65013
  • Sutton, R. S. and Barto, A. G. (1998). Reinforcement Learning: An Introduction (Adaptive Computation and Machine Learning). MIT Press, Cambridge, MA.
  • Tropp, J. A. (2015). An introduction to matrix concentration inequalities. Foundations and Trends® in Machine Learning 8 1–230.
    Zentralblatt MATH: 1391.15071
    Digital Object Identifier: doi:10.1561/2200000048
  • Wolfer, G. and Kontorovich, A. (2019a). Estimating the mixing time of ergodic Markov chains. Available at arXiv:1902.01224.
    arXiv: 1902.01224
  • Wolfer, G. and Kontorovich, A. (2019b). Minimax identity testing to ergodic Markov chains. Available at arXiv:1902.00080.
    arXiv: 1902.00080
  • Yu, B. (1994). Rates of convergence for empirical processes of stationary mixing sequences. Ann. Probab. 22 94–116.
    Zentralblatt MATH: 0802.60024
    Digital Object Identifier: doi:10.1214/aop/1176988849
    Project Euclid: euclid.aop/1176988849

The Institute of Mathematical Statistics

The Annals of Applied Probability

New content alerts

Email RSS ToC RSS Article
  • You have access to this content.
  • You have partial access to this content.
  • You do not have access to this content.
Turn Off MathJax

What is MathJax?

More like this

+ See more