Bayesian Analysis

Bayesian Nonparametric Inference – Why and How

Peter Müller and Riten Mitra

Full-text: Open access

Abstract

We review inference under models with nonparametric Bayesian (BNP) priors. The discussion follows a set of examples for some common inference problems. The examples are chosen to highlight problems that are challenging for standard parametric inference. We discuss inference for density estimation, clustering, regression and for mixed effects models with random effects distributions. While we focus on arguing for the need for the flexibility of BNP models, we also review some of the more commonly used BNP models, thus hopefully answering a bit of both questions, why and how to use BNP.

This review was sponsored by the Bayesian Nonparametrics Section of ISBA (ISBA/BNP). The authors thank the section officers for the support and encouragement.

Article information

Source
Bayesian Anal. Volume 8, Number 2 (2013), 269-302.

Dates
First available in Project Euclid: 24 May 2013

Permanent link to this document
https://projecteuclid.org/euclid.ba/1369407550

Digital Object Identifier
doi:10.1214/13-BA811

Mathematical Reviews number (MathSciNet)
MR3066939

Zentralblatt MATH identifier
1329.62171

Keywords
Nonparametric models Dirichlet process Polya tree dependent Dirichlet process

Citation

Müller, Peter; Mitra, Riten. Bayesian Nonparametric Inference – Why and How. Bayesian Anal. 8 (2013), no. 2, 269--302. doi:10.1214/13-BA811. https://projecteuclid.org/euclid.ba/1369407550


Export citation

References

  • Baladandayuthapani, V., Mallick, B. K., and Carroll, R. J. (2005). “Spatially Adaptive Bayesian Penalized Regression Splines (P-splines).” Journal of Computational and Graphical Statistics, 14(2):378–394.
  • Barnes, T. G., Jefferys, W. H., Berger, J. O., Müller, P., Orr, K., and Rodríguez, R. (2003). “A Bayesian Analysis of the Cepheid Distance Scale.” The Astrophysical Journal, 592(1):539.
  • Barrios, E. J., Lijoi, A., Nieto-Barajas, L. E., and Prünster, I. (2013). “Modeling with normalized random measure mixture models.” Statistical Science, to appear.
  • Box, G. E. P. (1979). “Some Problems of Statistics and Everyday Life.” Journal of the American Statistical Association, 74:1–4.
  • Branscum, A. J., Johnson, W. O., Hanson, T. E., and Gardner, I. A. (2008). “Bayesian semiparametric ROC curve estimation and disease diagnosis.” Statistics in Medicine, 27(13):2474–2496. URL http://dx.doi.org/10.1002/sim.3250
  • Broderick, T., Pitman, J., and Jordan, M. I. (2013). “Feature allocations, probability functions, and paintboxes.” arXiv: 1301.6647.
  • Bush, C. A. and MacEachern, S. N. (1996). “A semiparametric Bayesian model for randomised block designs.” Biometrika, 83:275–285.
  • Chipman, H., Kolaczyk, E., and McCulloch, R. (1997). “Adaptive Bayesian Wavelet Shrinkage.” Journal of the American Statistical Association, 92:440.
  • Clyde, M., Parmigiani, G., and Vidakovic, B. (1998). “Multiple Shrinkage and Subset Selection in Wavelets.” Biometrika, 85:391–402.
  • Dahl, D. B. (2006). “Model-Based Clustering for Expression Data via a Dirichlet Process Mixture Model.” In Vannucci, M., Do, K.-A., and Müller, P. (eds.), Bayesian Inference for Gene Expression and Proteomics. Cambridge University Press.
  • De Iorio, M., Johnson, W. O., Müller, P., and Rosner, G. L. (2009). “Bayesian Nonparametric Nonproportional Hazards Survival Modeling.” Biometrics, 65(3):762–771.
  • De la Cruz, R., Quintana, F. A., and Müller, P. (2007). “Semiparametric Bayesian classification with longitudinal markers.” Applied Statistics, 56(2):119–137.
  • Dunson, D. B. and Park, J.-H. (2007). “Kernel stick-breaking processes.” Biometrika, 95:307–323.
  • Dunson, D. B., Pillai, N., and Park, J.-H. (2007). “Bayesian Density Regression.” Journal of the Royal Statistical Society, Series B: Statistical Methodology, 69(2):163–183.
  • Dunson, D. B., Xue, Y., and Carin, L. (2008). “The Matrix Stick-Breaking Process: Flexible Bayes Meta-Analysis.” Journal of the American Statistical Association, 103(481):317–327.
  • Favaro, S. and Teh, Y. W. (2013). “MCMC for Normalized Random Measure Mixture Models.” Statistical Science, to appear.
  • Ferguson, T. S. (1973). “A Bayesian analysis of some nonparametric problems.” The Annals of Statistics, 1:209–230.
  • Freedman, D. (1963). “On the asymptotic behavior of Bayes’ estimates in the discrete case.” The Annals of Mathematical Statistics, 34(4):1386–1403.
  • Gelfand, A. E., Kottas, A., and MacEachern, S. N. (2005). “Bayesian Nonparametric Spatial Modeling With Dirichlet Process Mixing.” Journal of the American Statistical Association, 100:1021–1035.
  • Ghahramani, Z., Griffiths, T., and Sollich, P. (2007). “Bayesian nonparametric latent feature models.” In Bernardo, J. M., Bayarri, M., Berger, J. O., Dawid, A. P., Heckerman, D., and Smith, A. F. M. (eds.), Bayesian Statistics 8, 201–226. Oxford University Press.
  • Ghosal, S. (2010). “The Dirichlet process, related priors and posterior asymptotics.” In Hjort et al. (2010), 22–34.
  • Ghosal, S., Ghosh, J., and Ramamoorthi, R. (1999). “Posterior consistency of Dirichlet mixtures in density estimation.” Annals of Statistics, 27(1):143–158.
  • Ghosal, S. and Roy, A. (2006). “Posterior consistency of Gaussian process prior for nonparametric binary regression.” The Annals of Statistics, 34(5):2413–2429.
  • Ghosal, S. and van der Vaart, A. (2001). “Entropies and rates of convergence for maximum likelihood and Bayes estimation for mixtures of normal densities.” The Annals of Statistics, 29(5):1233–1263.
  • — (2007). “Posterior convergence rates of Dirichlet mixtures at smooth densities.” The Annals of Statistics, 35(2):697–723.
  • Griffiths, T. and Ghahramani, Z. (2006). “Infinite latent feature models and the Indian buffet process.” In Weiss, Y., Schölkopf, B., and Platt, J. (eds.), Advances in Neural Information Processing Systems 18, 475–482. MIT Press.
  • Guindani, M., Sepúlveda, N., Paulino, C. D., and Müller, P. (2012). “A Bayesian Semi-parametric Approach for the Differential Analysis of Sequence Counts Data.” Technical report, M.D. Anderson Cancer Center.
  • Hanson, T. and Jara, A. (2013). “Surviving fully Bayesian nonparametric regression.” In Damien, P., Dellaportas, P., Polson, N. G., and Stephens, D. A. (eds.), Bayesian Theory and Applications, 593–618. Oxford University Press.
  • Hanson, T. and Johnson, W. (2002). “Modeling Regression Error with a Mixture of Polya Trees.” Journal of the American Statistical Association, 97:1020–1033.
  • Hanson, T. E. (2006). “Inference for Mixtures of Finite Polya Tree Models.” Journal of the American Statistical Association, 101(476):1548–1565.
  • Hartigan, J. A. (1990). “Partition Models.” Communications in Statistics: Theory and Methods, 19:2745–2756.
  • Hjort, N. L. (2003). “Topics in nonparametric Bayesian statistics.” In Green, P., Hjort, N., and Richardson, S. (eds.), Highly Structured Stochastic Systems, 455–487. Oxford University Press.
  • Hjort, N. L., Holmes, C., Müller, P., and Walker, S. G. (2010). Bayesian Nonparametrics. Cambridge University Press.
  • Ishwaran, H. and James, L. F. (2001). “Gibbs Sampling Methods for Stick-Breaking Priors.” Journal of the American Statistical Association, 96:161–173.
  • James, L. F., Lijoi, A., and Prünster, I. (2009). “Posterior Analysis for Normalized Random Measures with Independent Increments.” Scandinavian Journal of Statistics, 36(1):76–97.
  • Jang, G. H., Lee, J., and Lee, S. (2010). “Posterior consistency of species sampling priors.” Statistica Sinica, 20(2):581.
  • Jara, A., Hanson, T., Quintana, F., Müller, P., and Rosner, G. (2011). “DPpackage: Bayesian Semi- and Nonparametric Modeling in R.” Journal of Statistical Software, 40(5):1–30.
  • Jara, A. and Hanson, T. E. (2011). “A class of mixtures of dependent tail-free processes.” Biometrika, 98(3):553–566. URL http://biomet.oxfordjournals.org/content/98/3/553.abstr act
  • Kennedy, M. C. and O’Hagan, A. (2001). “Bayesian Calibration of Computer Models.” Journal of the Royal Statistical Society. Series B (Statistical Methodology), 63(3):pp. 425–464. URL http://www.jstor.org/stable/2680584
  • Kingman, J. F. C. (1993). Poisson Processes. Oxford University Press.
  • Kleinman, K. and Ibrahim, J. (1998). “A Semi-parametric Bayesian Approach to the Random Effects Model.” Biometrics, 54:921–938.
  • Lavine, M. (1992). “Some aspects of Polya tree distributions for statistical modelling.” The Annals of Statistics, 20:1222–1235.
  • — (1994). “More aspects of Polya tree distributions for statistical modelling.” The Annals of Statistics, 22:1161–1176.
  • Lee, J., Quintana, F., Mueller, P., and Trippa, L. (2013). “Defining Predictive Probability Functions for Species Sampling Models.” Statistical Science, to appear.
  • Leon-Novelo, L., Bekele, B., Müller, P., Quintana, F., and Wathen, K. (2012). “Borrowing Strength with Non-Exchangeable Priors over Subpopulations.” Biometrics, 68:550–558.
  • Leon-Novelo, L. G., Müller, P., Arap, W., Kolonin, M., Sun, J., Pasqualini, R., and Do, K.-A. (2013). “Semiparametric Bayesian Inference for Phage Display Data.” Biometrics, to appear. URL http://dx.doi.org/10.1111/j.1541-0420.2012.01817.x
  • Li, Y., Müller, P., and Lin, X. (2011). “Center-Adjusted Inference for a Nonparametric Bayesian Random Effect Distribution.” Statistica Sinica, 21(3):1201–23.
  • Lijoi, A., Mena, R. H., and Prünster, I. (2005a). “Hierarchical Mixture Modeling with Normalized Inverse-Gaussian Priors.” Journal of the American Statistical Association, 100(472):pp. 1278–1291.
  • — (2007). “Controlling the reinforcement in Bayesian non-parametric mixture models.” Journal of the Royal Statistical Society: Series B (Statistical Methodology), 69(4):715–740.
  • Lijoi, A. and Prünster, I. (2010). “Models Beyond the Dirichlet process.” In Hjort et al. (2010), 80–136.
  • Lijoi, A., Prünster, I., and Walker, S. G. (2005b). “On consistency of nonparametric normal mixtures for Bayesian density estimation.” Journal of the American Statistical Association, 100(472):1292–1296.
  • MacEachern, S. (1999). “Dependent nonparametric processes.” In ASA Proceedings of the Section on Bayesian Statistical Science. Alexandria, VA: American Statistical Association.
  • Mauldin, R. D., Sudderth, W. D., and Williams, S. C. (1992). “Polya Trees and Random Distributions.” The Annals of Statistics, 20:1203–1221.
  • Morris, J., Baggerly, K., and Coombes, K. (2003). “Bayesian Shrinkage Estimators of the Relative Abundance of mRNA Transcripts using SAGE.” Biometrics, 59:476–486.
  • Morris, J. S. and Carroll, R. J. (2006). “Wavelet-based Functional Mixed Models.” Journal of the Royal Statistical Society, Series B: Statistical Methodology, 68(2):179–199.
  • Mukhopadhyay, S. and Gelfand, A. (1997). “Dirichlet Process Mixed Generalized Linear Models.” Journal of the American Statistical Association, 92:633–639.
  • Müller, P., Quintana, F., and Rosner, G. (2011). “A product partition model with regression on covariates.” Journal of Computational and Graphical Statistics, 20:260–278.
  • Müller, P. and Quintana, F. A. (2004). “Nonparametric Bayesian Data Analysis.” Statistical Science, 19:95–110.
  • Müller, P. and Rodríguez, A. (2013). Nonparametric Bayesian Inference. IMS-CBMS Lecture Notes. IMS.
  • Müller, P. and Rosner, G. (1997). “A Bayesian population model with hierarchical mixture priors applied to blood count data.” Journal of the American Statistical Association, 92:1279–1292.
  • O’Hagan, T. (1978). “Curve Fitting and Optimal Design for Prediction.” Journal of the Royal Statistical Society. Series B (Methodological), 40(1):pp. 1–42. URL http://www.jstor.org/stable/2984861
  • Paddock, S. M., Ruggeri, F., Lavine, M., and West, M. (2003). “Randomized Polya Tree Models for Nonparametric Bayesian Inference.” Statistica Sinica, 13(2):443–460.
  • Pepe, M. S., Etzioni, R., Feng, Z., Potter, J. D., Thompson, M. L., Thornquist, M., Winget, M., and Yasui, Y. (2001). “Phases of Biomarker Development for Early Detection of Cancer.” Journal of the National Cancer Institute, 93(14):1054–1061. URL http://jnci.oxfordjournals.org/content/93/14/1054.short
  • Pitman, J. (1996). “Some Developments of the Blackwell-MacQueen Urn Scheme.” In Ferguson, T. S., Shapeley, L. S., and MacQueen, J. B. (eds.), Statistics, Probability and Game Theory. Papers in Honor of David Blackwell, 245–268. Hayward, California: IMS Lecture Notes - Monograph Series.
  • Pitman, J. and Yor, M. (1997). “The Two-Parameter Poisson-Dirichlet Distribution Derived from a Stable Subordinator.” The Annals of Probability, 25:855–900.
  • Quintana, F. A. (2006). “A predictive view of Bayesian clustering.” Journal of Statistical Planning and Inference, 136:2407–2429.
  • Quintana, F. A. and Iglesias, P. L. (2003). “Bayesian clustering and product partition models.” Journal of the Royal Statistical Society: Series B (Statistical Methodology), 65(2):557–574. URL http://dx.doi.org/10.1111/1467-9868.00402
  • Regazzini, E., Lijoi, A., and Prünster, I. (2003). “Distributional Results for Means of Normalized Random Measures with Independent Increments.” The Annals of Statistics, 31(2):560–585.
  • Rodríguez, A., Dunson, D. B., and Gelfand, A. E. (2008). “The nested Dirichlet process, with Discussion.” Journal of the American Statistical Association, 103:1131–1144.
  • Rosner, G. L. (2005). “Bayesian monitoring of clinical trials with failure-time endpoints.” Biometrics, 61:239–245.
  • Schwartz, L. (1965). “On Bayes procedures.” Probability Theory and Related Fields, 4(1):10–26.
  • Sethurman, J. (1994). “A constructive definition of Dirichlet priors.” Statistica Sinica, 4:639–650.
  • Teh, Y. W., Jordan, M. I., Beal, M. J., and Blei, D. M. (2006). “Sharing Clusters among Related Groups: Hierarchical Dirichlet Processes.” Journal of the American Statistical Association, 101:1566–1581.
  • Tokdar, S. T. (2006). “Posterior consistency of Dirichlet location-scale mixture of normals in density estimation and regression.” Sankhyā: The Indian Journal of Statistics, 68:90–110.
  • Trippa, L., Müller, P., and Johnson, W. (2011). “The Multivariate Beta Process and an Extension of the Polya Tree Model.” Biometrika, 98(1):17–34.
  • van der Vaart, A. and van Zanten, J. (2008). “Rates of contraction of posterior distributions based on Gaussian process priors.” The Annals of Statistics, 36(3):1435–1463.
  • Vidakovic, B. (1998). “Nonlinear wavelet shrinkage with Bayes rules and Bayes Factors.” Journal of the American Statistical Association, 93:173–179.
  • Wade, S., Mongelluzzo, S., and Petrone, S. (2011). “An enriched conjugate prior for Bayesian nonparametric inference.” Bayesian Analysis, 6(3):359–385.
  • Walker, S. (2003). “On sufficient conditions for Bayesian consistency.” Biometrika, 90(2):482–488.
  • — (2004). “New approaches to Bayesian consistency.” The Annals of Statistics, 32(5):2028–2043.
  • — (2013). “Bayeisan nonparametrics.” In Damien, P., Dellaportas, P., Polson, N. G., and Stephens, D. A. (eds.), Bayesian Theory and Applications, 249–270. Oxford University Press.
  • Walker, S., Damien, P., Laud, P., and Smith, A. (1999). “Bayesian nonparametric inference for distributions and related functions (with discussion).” Journal of the Royal Statistical Society, Series B, 61:485–527.
  • Walker, S. and Hjort, N. (2002). “On Bayesian consistency.” Journal of the Royal Statistical Society: Series B (Statistical Methodology), 63(4):811–821.
  • Walker, S., Lijoi, A., and Prünster, I. (2007). “On rates of convergence for posterior distributions in infinite-dimensional models.” The Annals of Statistics, 35(2):738–746.
  • Wang, Y. and Taylor, J. M. (2001). “Jointly modeling longitudinal and event time data with application to acquired immunodeficiency syndrome.” Journal of the American Statistical Association, 96:895–903.
  • Zhang, S., Müller, P., and Do, K.-A. (2010). “A Bayesian Semiparametric Survival Model with Longitudinal Markers.” Biometrics, 66(2):435–443.

See also

  • Related item: Bradley P. Carlin, Thomas A. Murray. Comment on Article by Müller and Mitra. Bayesian Anal., Vol. 8, Iss. 2 (2013) 303–310.
  • Related item: Peter D. Hoff. Comment on Article by Müller and Mitra. Bayesian Anal., Vol. 8, Iss. 2 (2013) 311–318.
  • Related item: Anthony O’Hagan. Comment on Article by Müller and Mitra. Bayesian Anal., Vol. 8, Iss. 2 (2013) 319–322.
  • Related item: Contributed Discussion on Article by Müller and Mitra. Bayesian Anal., Vol. 8, Iss. 2 (2013) 323–356.
  • Related item: Peter Müller, Riten Mitra. Rejoinder. Bayesian Anal., Vol. 8, Iss. 2 (2013) 357–360.