The Annals of Statistics

A spectral analytic comparison of trace-class data augmentation algorithms and their sandwich variants

Kshitij Khare and James P. Hobert

Full-text: Open access

Abstract

The data augmentation (DA) algorithm is a widely used Markov chain Monte Carlo algorithm that is easy to implement but often suffers from slow convergence. The sandwich algorithm is an alternative that can converge much faster while requiring roughly the same computational effort per iteration. Theoretically, the sandwich algorithm always converges at least as fast as the corresponding DA algorithm in the sense that ‖K*‖ ≤ ‖K‖, where K and K* are the Markov operators associated with the DA and sandwich algorithms, respectively, and ‖⋅‖ denotes operator norm. In this paper, a substantial refinement of this operator norm inequality is developed. In particular, under regularity conditions implying that K is a trace-class operator, it is shown that K* is also a positive, trace-class operator, and that the spectrum of K* dominates that of K in the sense that the ordered elements of the former are all less than or equal to the corresponding elements of the latter. Furthermore, if the sandwich algorithm is constructed using a group action, as described by Liu and Wu [J. Amer. Statist. Assoc. 94 (1999) 1264–1274] and Hobert and Marchev [Ann. Statist. 36 (2008) 532–554], then there is strict inequality between at least one pair of eigenvalues. These results are applied to a new DA algorithm for Bayesian quantile regression introduced by Kozumi and Kobayashi [J. Stat. Comput. Simul. 81 (2011) 1565–1578].

Article information

Source
Ann. Statist., Volume 39, Number 5 (2011), 2585-2606.

Dates
First available in Project Euclid: 22 December 2011

Permanent link to this document
https://projecteuclid.org/euclid.aos/1324563348

Digital Object Identifier
doi:10.1214/11-AOS916

Mathematical Reviews number (MathSciNet)
MR2906879

Zentralblatt MATH identifier
1259.60081

Subjects
Primary: 60J27: Continuous-time Markov processes on discrete state spaces
Secondary: 62F15: Bayesian inference

Keywords
Compact operator convergence rate eigenvalue group action Markov chain Markov operator Monte Carlo operator norm positive operator

Citation

Khare, Kshitij; Hobert, James P. A spectral analytic comparison of trace-class data augmentation algorithms and their sandwich variants. Ann. Statist. 39 (2011), no. 5, 2585--2606. doi:10.1214/11-AOS916. https://projecteuclid.org/euclid.aos/1324563348


Export citation

References

  • Buja, A. (1990). Remarks on functional canonical variates, alternating least squares methods and ACE. Ann. Statist. 18 1032–1069.
  • Conway, J. B. (1990). A Course in Functional Analysis, 2nd ed. Springer, New York.
  • Diaconis, P., Khare, K. and Saloff-Coste, L. (2008). Gibbs sampling, exponential families and orthogonal polynomials (with discussion). Statist. Sci. 23 151–200.
  • Eaton, M. L. (1989). Group Invariance Applications in Statistics. NSF-CBMS Regional Conference Series in Probability and Statistics 1. IMS, Hayward, CA.
  • Hobert, J. P. and Marchev, D. (2008). A theoretical comparison of the data augmentation, marginal augmentation and PX–DA algorithms. Ann. Statist. 36 532–554.
  • Hobert, J. P. and Román, J. C. (2011). Discussion of “To center or not to center: That is not the question—An ancillarity-sufficiency interweaving strategy (ASIS) for boosting MCMC efficiency,” by Y. Yu and X.-L. Meng. J. Comput. Graph. Statist. 20 571–580.
  • Hobert, J. P., Roy, V. and Robert, C. P. (2011). Improving the convergence properties of the data augmentation algorithm with an application to Bayesian mixture modelling. Statist. Sci. 26 332–351.
  • Kozumi, H. and Kobayashi, G. (2011). Gibbs sampling methods for Bayesian quantile regression. J. Stat. Comput. Simul. 81 1565–1578.
  • Liu, J. S. and Sabatti, C. (2000). Generalised Gibbs sampler and multigrid Monte Carlo for Bayesian computation. Biometrika 87 353–369.
  • Liu, J. S., Wong, W. H. and Kong, A. (1994). Covariance structure of the Gibbs sampler with applications to the comparisons of estimators and augmentation schemes. Biometrika 81 27–40.
  • Liu, J. S., Wong, W. H. and Kong, A. (1995). Covariance structure and convergence rate of the Gibbs sampler with various scans. J. Roy. Statist. Soc. Ser. B 57 157–169.
  • Liu, J. S. and Wu, Y. N. (1999). Parameter expansion for data augmentation. J. Amer. Statist. Assoc. 94 1264–1274.
  • Marchev, D. and Hobert, J. P. (2004). Geometric ergodicity of van Dyk and Meng’s algorithm for the multivariate Student’s t model. J. Amer. Statist. Assoc. 99 228–238.
  • Meng, X.-L. and van Dyk, D. A. (1999). Seeking efficient data augmentation schemes via conditional and marginal augmentation. Biometrika 86 301–320.
  • Meyn, S. P. and Tweedie, R. L. (1993). Markov Chains and Stochastic Stability. Springer, London.
  • Papaspiliopoulos, O. and Roberts, G. (2008). Stability of the Gibbs sampler for Bayesian hierarchical models. Ann. Statist. 36 95–117.
  • Roberts, G. O. and Rosenthal, J. S. (1997). Geometric ergodicity and hybrid Markov chains. Electron. Commun. Probab. 2 13–25 (electronic).
  • Rosenthal, J. S. (2003). Asymptotic variance and convergence rates of nearly-periodic Markov chain Monte Carlo algorithms. J. Amer. Statist. Assoc. 98 169–177.
  • Schervish, M. J. and Carlin, B. P. (1992). On the convergence of successive substitution sampling. J. Comput. Graph. Statist. 1 111–127.
  • Tan, A. (2008). Analysis of Markov chain Monte Carlo algorithms for random effects models. Ph.D. thesis, Dept. Statistics, Univ. Florida, Gainesville, FL.
  • Tanner, M. A. and Wong, W. H. (1987). The calculation of posterior distributions by data augmentation (with discussion). J. Amer. Statist. Assoc. 82 528–550.
  • van Dyk, D. A. and Meng, X.-L. (2001). The art of data augmentation (with discussions). J. Comput. Graph. Statist. 10 1–50.
  • Voss, H. (2003). Variational characterizations of eigenvalues of nonlinear eigenproblems. In Proceedings of the International Conference on Mathematical and Computer Modelling in Science and Engineering (M. Kocandrlova and V. Kelar, eds.) 379–383. Czech Technical Univ., Prague.
  • Yu, Y. and Meng, X.-L. (2011). To center or not to center: That is not the question—An ancillarity-sufficiency interweaving strategy (ASIS) for boosting MCMC efficiency (with discussion). J. Comput. Graph. Statist. 20 531–615.