Electronic Journal of Statistics

Convergence rates for MCMC algorithms for a robust Bayesian binary regression model

Vivekananda Roy

Full-text: Open access

Abstract

Most common regression models for analyzing binary random variables are logistic and probit regression models. However it is well known that the estimates of regression coefficients for these models are not robust to outliers [26]. The robit regression model [1, 16] is a robust alternative to the probit and logistic models. The robit model is obtained by replacing the normal (logistic) distribution underlying the probit (logistic) regression model with the Student’s $t-$distribution. We consider a Bayesian analysis of binary data with the robit link function. We construct a data augmentation (DA) algorithm that can be used to explore the corresponding posterior distribution. Following [10] we further improve the DA algorithm by adding a simple extra step to each iteration. Though the two algorithms are basically equivalent in terms of computational complexity, the second algorithm is theoretically more efficient than the DA algorithm. Moreover, we analyze the convergence rates of these Markov chain Monte Carlo (MCMC) algorithms. We prove that, under certain conditions, both algorithms converge at a geometric rate. The geometric convergence rate has important theoretical and practical ramifications. Indeed, the geometric ergodicity guarantees that the ergodic averages used to approximate posterior expectations satisfy central limit theorems, which in turn allows for the construction of asymptotically valid standard errors. These standard errors can be used to choose an appropriate (Markov chain) Monte Carlo sample size and allow one to use the MCMC algorithms developed in this paper with the same level of confidence that one would have using classical (iid) Monte Carlo. The results are illustrated using a simple numerical example.

Article information

Source
Electron. J. Statist., Volume 6 (2012), 2463-2485.

Dates
First available in Project Euclid: 4 January 2013

Permanent link to this document
https://projecteuclid.org/euclid.ejs/1357307946

Digital Object Identifier
doi:10.1214/12-EJS756

Mathematical Reviews number (MathSciNet)
MR3020272

Zentralblatt MATH identifier
1295.60089

Subjects
Primary: 60J27: Continuous-time Markov processes on discrete state spaces
Secondary: 62F15: Bayesian inference

Keywords
Convergence rate data augmentation algorithm geometric ergodicity Markov chain robit regression robust regression

Citation

Roy, Vivekananda. Convergence rates for MCMC algorithms for a robust Bayesian binary regression model. Electron. J. Statist. 6 (2012), 2463--2485. doi:10.1214/12-EJS756. https://projecteuclid.org/euclid.ejs/1357307946


Export citation

References

  • [1] Albert, J. H. and Chib, S. (1993). Bayesian analysis of binary and polychotomous response data., Journal of the American Statistical Association 88 669–679.
  • [2] Bednorz, W. and Latuszynski, K. (2007). A few remarks on “Fixed-width output analysis for Markov chain Monte Carlo” by Jones et al., Journal of the American Statistical Association 102 1485–1486.
  • [3] Feller, W. (1968)., An Introduction to Probability Theory and its Applications, vol. I, 3rd. ed. John Wiley & Sons, New York.
  • [4] Flegal, J. M. (2012)., mcmcse: Monte Carlo standard errors for MCMC. R package version 0.1. http://CRAN.R-project.org/package=mcmcse
  • [5] Flegal, J. M., Haran, M. and Jones, G. L. (2008). Markov Chain Monte Carlo: Can We Trust the Third Significant Figure?, Statistical Science 23 250–260.
  • [6] Flegal, J. M. and Jones, G. L. (2010). Batch means and spectral variance estimators in Markov chain Monte Carlo., The Annals of Statistics 38 1034–1070.
  • [7] Gelman, A. and Hill, J. (2007)., Data Analysis Using Regression and Multilevel/Hierarchical Models. Cambridge, Cambridge University Press.
  • [8] Hobert, J. P. (2011)., Handbook of Markov chain Monte Carlo. The data augmentation algorithm: theory and methodology, 253–293. CRC Press, Boca Raton, FL.
  • [9] Hobert, J. P., Jones, G. L., Presnell, B. and Rosenthal, J. S. (2002). On the applicability of regenerative simulation in Markov chain Monte Carlo., Biometrika 89 731–743.
  • [10] Hobert, J. P. and Marchev, D. (2008). A theoretical comparison of the data augmentation, marginal augmentation and PX-DA algorithms., The Annals of Statistics 36 532-554.
  • [11] Holmes, C. C. and Held, L. (2006). Bayesian auxiliary variable models for binary and multinomial regression., Bayesian Analysis 1 145-168.
  • [12] Jones, G. L., Haran, M., Caffo, B. S. and Neath, R. (2006). Fixed-width output analysis for Markov chain Monte Carlo., Journal of the American Statistical Association 101 1537–1547.
  • [13] Jones, G. L. and Hobert, J. P. (2001). Honest exploration of intractable probability distributions via Markov chain Monte Carlo., Statistical Science 16 312–34.
  • [14] Khare, K. and Hobert, J. P. (2011). A spectral analytic comparison of trace-class data augmentation algorithms and their sandwich variants., The Annals of Statistics 39 2585-2606.
  • [15] Kim, H. J. (2008). Moments of truncated Student-t distribution., Journal of the Korean Statistical Society 37 81-87.
  • [16] Liu, C. (2004). Robit regression: A simple robust alternative to logistic and probit regression. In, Applied Bayesian Modeling and Casual Inference from Incomplete-Data Perspectives (A. Gelman and X. L. Meng, eds.) 227-238. Wiley, London.
  • [17] Liu, J. S. and Sabatti, C. (2000). Generalised Gibbs sampler and multigrid Monte Carlo for Bayesian computation., Biometrika 87 353–369.
  • [18] Liu, J. S., Wong, W. H. and Kong, A. (1995). Covariance Structure and Convergence Rate of the Gibbs Sampler with Various Scans., Journal of the Royal Statistical Society, Series B 57 157-169.
  • [19] Liu, J. S. and Wu, Y. N. (1999). Parameter Expansion for Data Augmentation., Journal of the American Statistical Association 94 1264–1274.
  • [20] Marchev, D. (2011). Markov chain Monte Carlo algorithms for the Bayesian logistic regression model. In, Proceeding of the Annual International Conference on Operations Research and Statistics (C. B. Gupta, ed.) 154-159. Global Science and Technology Forum, Penang, Malaysia.
  • [21] Meng, X.-L. and van Dyk, D. A. (1999). Seeking Efficient Data Augmentation Schemes via Conditional and Marginal Augmentation., Biometrika 86 301–320.
  • [22] Meyn, S. P. and Tweedie, R. L. (1993)., Markov Chains and Stochastic Stability. Springer Verlag, London.
  • [23] Mira, A. and Geyer, C. J. (1999). Ordering Monte Carlo Markov chains. Technical Report No. 632, School of Statistics, University of, Minnesota.
  • [24] Mudholkar, G. S. and George, E. O. (1978). A remark on the shape of the logistic distribution., Biometrika 65 667-668.
  • [25] Pinkham, R. S. and Wilk, M. B. (1963). Tail areas of the t-distribution from a Mills’-ratio-like expansion., Annals of Mathematical Statistics 34 335-337.
  • [26] Pregibon, D. (1982). Resistant fits for some commonly used logistic models with medical applications., Biometrics 38 485-498.
  • [27] R Development Core Team (2011)., R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria ISBN 3-900051-07-0. http://www.R-project.org
  • [28] Rao, C. R. (1973)., Linear statistical inference and its applications, 2nd. ed. John Wiley & Sons, New York.
  • [29] Roberts, G. O. and Rosenthal, J. S. (1997). Geometric ergodicity and hybrid Markov chains., Electronic Communications in Probability 2 13-25.
  • [30] Roberts, G. O. and Tweedie, R. L. (2001). Geometric $L2$ and $L1$ convergence are equivalent for reversible Markov chains., Journal of Applied Probability 38A 37–41.
  • [31] Roy, V. (2012). Spectral analytic comparisons for data augmentation., Stat. and Prob. Letters 82 103-108.
  • [32] Roy, V. and Hobert, J. P. (2007). Convergence rates and asymptotic standard errors for MCMC algorithms for Bayesian probit regression., Journal of the Royal Statistical Society, Series B 69 607-623.
  • [33] Roy, V. and Hobert, J. P. (2010). On Monte Carlo methods for Bayesian regression models with heavy-tailed errors., Journal of Multivariate Analysis 101 1190-1202.
  • [34] Tan, A. and Hobert, J. P. (2009). Block Gibbs sampling for Bayesian random effects models with improper priors: convergence and regeneration., Journal of Computational and Graphical Statistics 18 861-878.
  • [35] Tanner, M. A. and Wong, W. H. (1987). The calculation of posterior distributions by data augmentation(with discussion)., Journal of the American Statistical Association 82 528–550.
  • [36] van Dyk, D. A. and Meng, X.-L. (2001). The Art of Data Augmentation (with discussion)., Journal of Computational and Graphical Statistics 10 1–50.
  • [37] Yu, Y. and Meng, X. L. (2011). To center or not to center: that is not the question - An ancillarity-sufficiency interweaving strategy (ASIS) for boosting MCMC efficiency., Journal of Computational and Graphical Statistics 20 531-570.
  • [38] Zellner, A. (1983). Applications of Bayesian analysis in econometrics., The Statistician 32 23-34.