The Annals of Statistics

Nonparametric estimation of convex models via mixtures

Peter D. Hoff

Full-text: Open access


We present a general approach to estimating probability measures constrained to lie in a convex set. We represent constrained measures as mixtures of simple, known extreme measures, and so the problem of estimating a constrained measure becomes one of estimating an unconstrained mixing measure. Convex constraints arise in many modeling situations, such as estimation of the mean and estimation under stochastic ordering constraints. We describe mixture representation techniques for these and other situations, and discuss applications to maximum likelihood and Bayesian estimation.

Article information

Ann. Statist., Volume 31, Number 1 (2003), 174-200.

First available in Project Euclid: 26 February 2003

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62G07: Density estimation
Secondary: 62C10: Bayesian problems; characterization of Bayes procedures

Choquet's theorem convex constraints stochastic ordering nonparametric estimation Bayesian inference


Hoff, Peter D. Nonparametric estimation of convex models via mixtures. Ann. Statist. 31 (2003), no. 1, 174--200. doi:10.1214/aos/1046294461.

Export citation


  • ANTONIAK, C. E. (1974). Mixtures of Dirichlet processes with applications to Bayesian nonparametric problems. Ann. Statist. 2 1152-1174.
  • ARJAS, E. and GASBARRA, D. (1996). Bayesian inference of survival probabilities, under stochastic ordering constraints. J. Amer. Statist. Assoc. 91 1101-1109.
  • ASH, R. B. (1972). Real Analy sis and Probability. Academic Press, New York.
  • BERTIN, E. M. J., CUCULESCU, I. and THEODORESCU, R. (1997). Unimodality of Probability Measures. Kluwer, Dordrecht.
  • BLACKWELL, D. and MACQUEEN, J. B. (1973). Ferguson distributions via Póly a urn schemes. Ann. Statist. 1 353-355.
  • BÖHNING, D. (1995). A review of reliable maximum likelihood algorithms for semiparametric mixture models. J. Statist. Plann. Inference 47 5-28.
  • BRUNK, H. D., FRANCK, W. E., HANSON, D. L. and HOGG, R. V. (1966). Maximum likelihood estimation of the distributions of two stochastically ordered random variables. J. Amer. Statist. Assoc. 61 1067-1080.
  • BRUNNER, L. J. and LO, A. Y. (1989). Bay es methods for a sy mmetric unimodal density and its mode. Ann. Statist. 17 1550-1566.
  • DARDANONI, V. and FORCINA, A. (1998). A unified approach to likelihood inference on stochastic orderings in a nonparametric context. J. Amer. Statist. Assoc. 93 1112-1123.
  • DHARMADHIKARI, S. and JOAG-DEV, K. (1988). Unimodality, Convexity, and Applications. Academic Press, Boston.
  • DIACONIS, P. and FREEDMAN, D. (1980). Finite exchangeable sequences. Ann. Probab. 8 745-764.
  • DIACONIS, P. and FREEDMAN, D. (1986). On the consistency of Bay es estimates. Ann. Statist. 14 1-26.
  • DOSS, H. (1985). Bayesian nonparametric estimation of the median. I. Computation of the estimates. Ann. Statist. 13 1432-1444.
  • Dy KSTRA, R. L. and FELTZ, C. J. (1989). Nonparametric maximum likelihood estimation of survival functions with a general stochastic ordering and its dual. Biometrika 76 331- 341.
  • Dy NKIN, E. B. (1978). Sufficient statistics and extreme points. Ann. Probab. 6 705-730.
  • EL BARMI, H. and Dy KSTRA, R. L. (1994). Restricted multinomial maximum likelihood estimation based upon Fenchel duality. Statist. Probab. Lett. 21 121-130.
  • EL BARMI, H. and Dy KSTRA, R. (1998). Maximum likelihood estimates via duality for log-convex models when cell probabilities are subject to convex constraints. Ann. Statist. 26 1878- 1893.
  • FERGUSON, T. S. (1973). A Bayesian analysis of some nonparametric problems. Ann. Statist. 1 209-230.
  • FERGUSON, T. S. (1974). Prior distributions on spaces of probability measures. Ann. Statist. 2 615-629.
  • HOFF, P. D. (2000). Constrained nonparametric maximum likelihood via mixtures. J. Comput. Graph. Statist. 9 633-641.
  • HOFF, P. D., HALBERG, R. B., SHEDLOVSKY, A., DOVE, W. F. and NEWTON, M. A. (2001). Identifying carriers of a genetic modifier using nonparametric Bay es methods. Case Studies in Bayesian Statistics 5. Lecture Notes on Statist. 162 327-342.
  • KAMAE, T., KRENGEL, U. and O'BRIEN, G. L. (1977). Stochastic inequalities on partially ordered spaces. Ann. Probab. 5 899-912.
  • KARR, A. F. (1991). Point Processes and Their Statistical Inference, 2nd ed. Dekker, New York.
  • KORWAR, R. M. and HOLLANDER, M. (1973). Contributions to the theory of Dirichlet processes. Ann. Probab. 1 705-711.
  • KULLBACK, S. (1968). Probability densities with given marginals. Ann. Math. Statist. 39 1236- 1243.
  • LEHMANN, E. L. (1997). Testing Statistical Hy potheses, 2nd ed. Springer, New York.
  • LEROUX, B. G. (1992). Consistent estimation of a mixing distribution. Ann. Statist. 20 1350-1360.
  • LESPERANCE, M. L. and KALBFLEISCH, J. D. (1992). An algorithm for computing the nonparametric MLE of a mixing distribution. J. Amer. Statist. Assoc. 87 120-126.
  • LINDSAY, B. G. (1983). The geometry of mixture likelihoods: A general theory. Ann. Statist. 11 86-94.
  • LINDSAY, B. G. (1995). Mixture Models: Theory, Geometry and Applications. IMS, Hay ward, CA.
  • LINDSAY, B. G. and ROEDER, K. (1993). Uniqueness of estimation and identifiability in mixture models. Canad. J. Statist. 21 139-147.
  • LO, A. Y. (1984). On a class of Bayesian nonparametric estimates. I. Density estimates. Ann. Statist. 12 351-357.
  • MACEACHERN, S. N. and MÜLLER, P. (1998). Estimating mixture of Dirichlet process models. J. Comput. Graph. Statist. 7 223-238.
  • NEAL, R. M. (2000). Markov chain sampling methods for Dirichlet process mixture models. J. Comput. Graph. Statist. 9 249-265.
  • NEWTON, M. A., CZADO, C. and CHAPPELL, R. (1996). Bayesian inference for semiparametric binary regression. J. Amer. Statist. Assoc. 91 142-153.
  • OWEN, A. B. (1988). Empirical likelihood ratio confidence intervals for a single functional. Biometrika 75 237-249.
  • OWEN, A. B. (1990). Empirical likelihood ratio confidence regions. Ann. Statist. 18 90-120.
  • OWEN, A. B. (2001). Empirical Likelihood. Chapman and Hall, London.
  • PARTHASARATHY, K. R. (1967). Probability Measures on Metric Spaces. Academic Press, New York.
  • PETERS, C. and COBERLY, W. A. (1976). The numerical evaluation of the maximum-likelihood estimate of mixture proportions. Comm. Statist. Theory Methods 5 1127-1135.
  • PETRONE, S. and RAFTERY, A. E. (1997). A note on the Dirichlet process prior in Bayesian nonparametric inference with partial exchangeability. Statist. Probab. Lett. 36 69-83.
  • PFANZAGL, J. (1979). Conditional distributions as derivatives. Ann. Probab. 7 1046-1050.
  • PFANZAGL, J. (1988). Consistency of maximum likelihood estimators for certain nonparametric families, in particular: Mixtures. J. Statist. Plann. Inference 19 137-158.
  • VAN DE GEER, S. (1995). Asy mptotic normality in mixture models. ESAIM Probab. Statist. 1 17-33.
  • VON WEIZSÄCKER, H. and WINKLER, G. (1979). Integral representation in the set of solutions of a generalized moment problem. Math. Ann. 246 23-32.
  • VON WEIZSÄCKER, H. and WINKLER, G. (1980). Noncompact extremal integral representations: Some probabilistic aspects. In Functional Analy sis: Survey s and Recent Results II (K.-D. Bierstedt and B. Fuchssteiner, eds.) 115-148. North-Holland, Amsterdam.
  • WEST, M., MÜLLER, P. and ESCOBAR, M. D. (1994). Hierarchical priors and mixture models, with application in regression and density estimation. In Aspects of Uncertainty (P. R. Freeman and A. F. M. Smith, eds.) 363-386. Wiley, Chichester.
  • WU, C. F. (1978). Some iterative procedures for generating nonsingular optimal designs. Comm. Statist. Theory Methods 7 1399-1412.
  • YERSHOV, M. P. (1973). Extensions of measures. Stochastic equations. Proc. Second Japan-USSR Sy mposium on Probability Theory. Lecture Notes in Math. 330 516-526. Springer, Berlin.