The Annals of Statistics

Bayesian analysis of mixture models with an unknown number of components—an alternative to reversible jump methods

Matthew Stephens

Full-text: Open access

Abstract

Richardson and Green present a method of performing a Bayesian analysis of data from a finite mixture distribution with an unknown number of components. Their method is a Markov Chain Monte Carlo (MCMC) approach, which makes use of the “reversible jump” methodology described by Green. We describe an alternative MCMC method which views the parameters of the model as a (marked) point process, extending methods suggested by Ripley to create a Markov birth-death process with an appropriate stationary distribution. Our method is easy to implement, even in the case of data in more than one dimension, and we illustrate it on both univariate and bivariate data. There appears to be considerable potential for applying these ideas to other contexts, as an alternative to more general reversible jump methods, and we conclude with a brief discussion of how this might be achieved.

Article information

Source
Ann. Statist., Volume 28, Number 1 (2000), 40-74.

Dates
First available in Project Euclid: 14 March 2002

Permanent link to this document
https://projecteuclid.org/euclid.aos/1016120364

Digital Object Identifier
doi:10.1214/aos/1016120364

Mathematical Reviews number (MathSciNet)
MR1762903

Zentralblatt MATH identifier
1106.62316

Subjects
Primary: 62F15: Bayesian inference

Keywords
Bayesian analysis birth-death process Markov process MCMC mixture model model choice reversible jump spatial point process

Citation

Stephens, Matthew. Bayesian analysis of mixture models with an unknown number of components—an alternative to reversible jump methods. Ann. Statist. 28 (2000), no. 1, 40--74. doi:10.1214/aos/1016120364. https://projecteuclid.org/euclid.aos/1016120364


Export citation

References

  • Anderson, E. (1935). The irises of the Gaspe Peninsula. Bulletin of the American Iris Society 59 2-5.
  • Baddeley, A. J. and van Lieshout, M. N. M. (1993). Stochastic geometry models in high-level vision. In Advances in Applied Statistics (K. V. Mardia and G. K. Kanji, eds.) 1 231-256. Carfax, Abingdon, UK.
  • Brooks, S. P. and Roberts, G. O. (1998). Convergence assessment techniques for Markov chain Monte Carlo. Statist. Comput. 8 319-335.
  • Carlin, B. P. and Chib, S. (1995). Bayesian model choice via Markov Chain Monte Carlo methods. J. Roy. Statist. Soc. Ser. B 57 473-484.
  • Chib, S. (1995). Marginal likelihood from the Gibbs output. J. Amer. Statist. Assoc. 90 1313-1321.
  • Cowles, M. K. and Carlin, B. P. (1996). Markov Chain Monte Carlo convergence diagnostics: a comparative review. J. Amer. Statist. Assoc. 91 883-904.
  • Crawford, S. L. (1994). An application of the Laplace method to finite mixture distributions. J. Amer. Statist. Assoc. 89 259-267.
  • Dawid, A. P. (1997). Contribution to the discussion of paper by Richardson and Green (1997). J. Roy. Statist. Soc. Ser. B 59 772-773.
  • DiCiccio, T., Kass, R., Raftery, A. and Wasserman, L. (1997). Computing Bayes factors by posterior simulation and asymptotic approximations. J. Amer. Statist. Assoc. 92 903- 915.
  • Diebolt, J. and Robert, C. P. (1994). Estimation of finite mixture distributions through Bayesian sampling. J. Roy. Statist. Soc. Ser. B 56 363-375.
  • Escobar, M. D. and West, M. (1995). Bayesian density estimation and inference using mixtures. J. Amer. Statist. Assoc. 90 577-588.
  • Gelman, A. and Rubin, D. B. (1992). Inference from iterative simulation using multiple sequences (with discussion). Statist. Sci. 7 457-511.
  • Gelman, A. G., Carlin, J. B., Stern, H. S. and Rubin, D. B. (1995). Bayesian Data Analysis. Chapman & Hall, London.
  • George, E. I. and McCulloch, R. E. (1996). Stochastic search variable selection. In Markov Chain Monte Carlo in Practice (W. R. Gilks, S. Richardson and D. J. Spiegelhalter, eds.) Chapman & Hall, London.
  • Geyer, C. J. and Møller, J. (1994). Simulation procedures and likelihood inference for spatial point processes. Scand. J. Statist. 21 359-373.
  • Gilks, W. R., Richardson, S. and Spiegelhalter, D. J., eds. (1996). Markov Chain Monte Carlo in Practice. Chapman & Hall, London.
  • Gl ¨otzl, E. (1981). Time-reversible and Gibbsian point processes I. Markovian spatial birth and death process on a general phase space. Math. Nach. 102 217-222.
  • Green, P. J. (1995). Reversible jump Markov chain Monte Carlo computation and Bayesian model determination. Biometrika 82 711-732.
  • H¨ardle, W. (1991). Smoothing techniques with implementation in S. Springer, New York.
  • Jennison, C. (1997). Comment on "On Bayesian analysis of mixtures with an unknown number of components," by S. Richardson and P. J. Green. J. Roy. Statist. Soc. Ser. B 59 778-779. Kelly, F. P. and Ripley, B. D. (1976) A note on Strauss's model for clustering. Biometrika 63 357-360.
  • Lawson, A. B. (1996). Markov chain Monte Carlo methods for spatial cluster processes. In Computer Science and Statistics: Proceedings of the 27th Symposium on the Interface 314-319.
  • Mathieson, M. J. (1997). Ordinal models and predictive methods in pattern recognition. Ph.D. thesis, Univ. Oxford.
  • McLachlan, G. J. (1992). Discriminant Analysis and Statistical Pattern Recognition. Wiley, New York.
  • McLachlan, G. J. and Basford, K. E. (1988). Mixture Models: Inference and Applications to Clustering. Dekker, New York.
  • Phillips, D. B. and Smith, A. F. M. (1996). Bayesian model comparison via jump diffusions. In
  • Postman, M., Huchra, J. P. and Geller, M. J. (1986). Probes of large-scale structure in the Corona Borealis region. The Astronomical Journal 92 1238-1247.
  • Preston, C. J. (1976). Spatial birth-and-death processes. Bull. Inst. Internat. Statist. 46 371-391.
  • Priebe, C. E. (1994). Adaptive mixtures. J. Amer. Statist. Assoc. 89 796-806.
  • Richardson, S. and Green, P. J. (1997). On Bayesian analysis of mixtures with an unknown number of components (with discussion). J. Roy. Statist. Soc. Ser. B 59 731-792.
  • Ripley, B. D. (1977). Modelling spatial patterns (with discussion). J. Roy. Statist. Soc. Ser. B 39 172-212.
  • Ripley, B. D. (1987). Stochastic Simulation. Wiley, New York.
  • Robert, C. P. (1994). The Bayesian Choice: a Decision-Theoretic Motivation. Springer, New York.
  • Robert, C. P. (1996). Mixtures of distributions: Inference and estimation. In Markov Chain Monte Carlo in Practice (W. R. Gilks, S. Richardson and D. J. Spiegelhalter, eds.) Chapman & Hall, London.
  • Roeder, K. (1990). Density estimation with confidence sets exemplified by superclusters and voids in the galaxies. J. Amer. Statist. Assoc. 85 617-624.
  • Sheather, S. J. and Jones, M. C. (1991). A reliable data-based bandwidth selection method for kernel density estimation. J. Roy. Statist. Soc. Ser. B 53 683-690.
  • Stephens, D. A. and Fisch, R. D. (1998). Bayesian analysis of quantitative trait locus data using reversible jump Markov chain Monte Carlo. Biometrics 54 1334-1367.
  • Stephens, M. (1997). Bayesian Methods for Mixtures of Normal Distributions. Ph.D. thesis, Univ. Oxford. Available from www.stats.ox.ac.uk/ stephens.
  • Stoyan, D., Kendall, W. S. and Mecke, J. (1987). Stochastic Geometry and Its Applications, 1st ed. Wiley, New York.
  • Tierney, L. (1996). Introduction to general state-space Markov chain theory. In Markov Chain Monte Carlo in Practice (W. R. Gilks, S. Richardson and D. J. Spiegelhalter, eds.) 59-74. Chapman & Hall, London.
  • Titterington, D. M., Smith, A. F. M. and Makov, U. E. (1985). Statistical Analysis of Finite Mixture Distributions. Wiley, New York.
  • Venables, W. N. and Ripley, B. D. (1994). Modern Applied Statistics with S-Plus. Springer, New York.
  • Venables, W. N. and Ripley, B. D. (1997). Modern Applied Statistics with S-Plus, 2nd ed. Springer, New York.
  • West, M. (1993). Approximating posterior distributions by mixtures. J. Roy. Statist. Soc. Ser. B 55 409-422.
  • Wilson, S. R. (1982). Sound and exploratory data analysis. In COMPSTAT 1982. Proceedings in Computational Statistics (H. Caussinus, P. Ettinger and R. Tamassone, eds.) 447-450. Physica, Vienna.