The Annals of Statistics

Distribution theory for hierarchical processes

Federico Camerlenghi, Antonio Lijoi, Peter Orbanz, and Igor Prünster

Full-text: Access denied (no subscription detected)

We're sorry, but we are unable to provide you with the full text of this article because we are not able to identify you as a subscriber. If you have a personal subscription to this journal, then please login. If you are already logged in, then you may need to update your profile to register your subscription. Read more about accessing full-text

Abstract

Hierarchies of discrete probability measures are remarkably popular as nonparametric priors in applications, arguably due to two key properties: (i) they naturally represent multiple heterogeneous populations; (ii) they produce ties across populations, resulting in a shrinkage property often described as “sharing of information.” In this paper, we establish a distribution theory for hierarchical random measures that are generated via normalization, thus encompassing both the hierarchical Dirichlet and hierarchical Pitman–Yor processes. These results provide a probabilistic characterization of the induced (partially exchangeable) partition structure, including the distribution and the asymptotics of the number of partition sets, and a complete posterior characterization. They are obtained by representing hierarchical processes in terms of completely random measures, and by applying a novel technique for deriving the associated distributions. Moreover, they also serve as building blocks for new simulation algorithms, and we derive marginal and conditional algorithms for Bayesian inference.

Article information

Source
Ann. Statist., Volume 47, Number 1 (2019), 67-92.

Dates
Received: July 2016
Revised: December 2017
First available in Project Euclid: 30 November 2018

Permanent link to this document
https://projecteuclid.org/euclid.aos/1543568582

Digital Object Identifier
doi:10.1214/17-AOS1678

Mathematical Reviews number (MathSciNet)
MR3909927

Zentralblatt MATH identifier
07036195

Subjects
Primary: 60G57: Random measures 62G05: Estimation 62F15: Bayesian inference

Keywords
Bayesian nonparametrics distribution theory hierarchical processes partition structure posterior distribution prediction random measures species sampling models

Citation

Camerlenghi, Federico; Lijoi, Antonio; Orbanz, Peter; Prünster, Igor. Distribution theory for hierarchical processes. Ann. Statist. 47 (2019), no. 1, 67--92. doi:10.1214/17-AOS1678. https://projecteuclid.org/euclid.aos/1543568582


Export citation

References

  • [1] Blei, D. M., Ng, A. Y. and Jordan, M. I. (2003). Latent Dirichlet allocation. J. Mach. Learn. Res. 3 993–1022.
  • [2] Camerlenghi, F., Lijoi, A., Orbanz, P. and Prünster, I. (2019). Distribution theory for hierarchical processes: Supplementary material. DOI:10.1214/17-AOS1678SUPP.
  • [3] Camerlenghi, F., Lijoi, A. and Prünster, I. (2017). Bayesian prediction with multiple-samples information. J. Multivariate Anal. 156 18–28.
  • [4] Caron, F., Teh, Y. W. and Murphy, T. B. (2014). Bayesian nonparametric Plackett–Luce models for the analysis of preferences for college degree programmes. Ann. Appl. Stat. 8 1145–1181.
  • [5] Daley, D. J. and Vere-Jones, D. (2008). An Introduction to the Theory of Point Processes. Vol. II, 2nd ed. Springer, New York.
  • [6] De Blasi, P., Favaro, S., Lijoi, A., Mena, R. H., Ruggiero, M. and Prünster, I. (2015). Are Gibbs-type priors the most natural generalization of the Dirichlet process? IEEE Trans. Pattern Anal. Mach. Intell. 37 212–229.
  • [7] Dunson, D. B. (2010). Nonparametric Bayes applications to biostatistics. In Bayesian Nonparametrics 223–273. Cambridge Univ. Press, Cambridge.
  • [8] Favaro, S., Lijoi, A., Mena, R. H. and Prünster, I. (2009). Bayesian nonparametric inference for species variety with a two parameter Poisson–Dirichlet process prior. J. Roy. Statist. Soc. Ser. B 71 993–1008.
  • [9] Ferguson, T. S. and Klass, M. J. (1972). A representation of independent increment processes without Gaussian components. Ann. Math. Stat. 43 1634–1643.
  • [10] Foti, N. J. and Williamson, S. A. (2015). A survey of non-exchangeable priors for Bayesian nonparametric models. IEEE Trans. Pattern Anal. Mach. Intell. 37 359–371.
  • [11] Gasthaus, J. and Teh, Y. W. (2010). Improvements to the sequence memoizer. Adv. Neuronal Inf. Process. Syst. 23.
  • [12] Gnedin, A. V. and Pitman, J. (2005). Exchangeable Gibbs partitions and Stirling triangles. Zap. Nauchn. Sem. S.-Peterburg. Otdel. Mat. Inst. Steklov. (POMI) 325 83–102.
  • [13] Griffin, J. E., Kolossiatis, M. and Steel, M. F. J. (2013). Comparing distributions by using dependent normalized random-measure mixtures. J. Roy. Statist. Soc. Ser. B 75 499–529.
  • [14] James, L. F. (2005). Bayesian Poisson process partition calculus with an application to Bayesian Lévy moving averages. Ann. Statist. 33 1771–1799.
  • [15] James, L. F. (2006). Poisson calculus for spatial neutral to the right processes. Ann. Statist. 34 416–440.
  • [16] James, L. F., Lijoi, A. and Prünster, I. (2009). Posterior analysis for normalized random measures with independent increments. Scand. J. Stat. 36 76–97.
  • [17] Kallenberg, O. (2005). Probabilistic Symmetries and Invariance Principles. Springer, New York.
  • [18] Kingman, J. F. C. (1967). Completely random measures. Pacific J. Math. 21 59–78.
  • [19] Kingman, J. F. C. (1982). The coalescent. Stochastic Process. Appl. 13 235–248.
  • [20] Kingman, J. F. C. (1993). Poisson Processes. Oxford Univ. Press, Oxford.
  • [21] Lijoi, A., Mena, R. H. and Prünster, I. (2007). Bayesian nonparametric estimation of the probability of discovering new species. Biometrika 94 769–786.
  • [22] Lijoi, A., Mena, R. H. and Prünster, I. (2007). Controlling the reinforcement in Bayesian nonparametric mixture models. J. Roy. Statist. Soc. Ser. B 69 715–740.
  • [23] Lijoi, A., Nipoti, B. and Prünster, I. (2014). Bayesian inference with dependent normalized completely random measures. Bernoulli 20 1260–1291.
  • [24] Lijoi, A., Nipoti, B. and Prünster, I. (2014). Dependent mixture models: Clustering and borrowing information. Comput. Statist. Data Anal. 71 417–433.
  • [25] Lijoi, A., Prünster, I. and Walker, S. G. (2008). Bayesian nonparametric estimators derived from conditional Gibbs structures. Ann. Appl. Probab. 18 1519–1547.
  • [26] MacEachern, S. N. (1999). Dependent nonparametric processes. In ASA Proceedings of the SBSS 50–55. Amer. Statist. Assoc., Alexandria, VA.
  • [27] MacEachern, S. N. (2000). Dependent Dirichlet processes. Technical Report, Dept. Statistics, Ohio State Univ., Columbus, OH.
  • [28] Nguyen, X. (2010). Inference of global clusters from locally distributed data. Bayesian Anal. 5 817–846.
  • [29] Nguyen, X. (2016). Borrowing strength in hierarchical Bayes: Convergence of the Dirichlet base measure. Bernoulli 22 1535–1571.
  • [30] Pitman, J. (1996). Some developments of the Blackwell–MacQueen urn scheme. In Statistics, Probability and Game Theory (T. S. Ferguson, L. S. Shapley and J. B. MacQueen, eds.). Institute of Mathematical Statistics Lecture Notes—Monograph Series 30 245–267. IMS, Hayward, CA.
  • [31] Pitman, J. (2006). Combinatorial Stochastic Processes. Ecole d’Eté de Probabilités de Saint-Flour XXXII. Lecture Notes in Mathematics N. 1875. Springer, Berlin.
  • [32] Pitman, J. and Yor, M. (1997). The two-parameter Poisson–Dirichlet distribution derived from a stable subordinator. Ann. Probab. 25 855–900.
  • [33] Regazzini, E., Lijoi, A. and Prünster, I. (2003). Distributional results for means of normalized random measures with independent increments. Ann. Statist. 31 560–585.
  • [34] Teh, Y. W. and Jordan, M. I. (2010). Hierarchical Bayesian nonparametric models with applications. In Bayesian Nonparametrics 158–207. Cambridge Univ. Press, Cambridge.
  • [35] Teh, Y. W., Jordan, M. I., Beal, M. J. and Blei, D. M. (2006). Hierarchical Dirichlet processes. J. Amer. Statist. Assoc. 101 1566–1581.
  • [36] Walker, S. and Damien, P. (2000). Representations of Lévy processes without Gaussian components. Biometrika 87 477–483.
  • [37] Wood, F., Gasthaus, J., Archambeau, C., James, L. F. and Teh, Y. W. (2011). The sequence memoizer. Commun. ACM 54 91–98.
  • [38] Zhou, M. and Carin, L. (2015). Negative binomial process count and mixture modeling. IEEE Trans. Pattern Anal. Mach. Intell. 37 307–320.

Supplemental materials

  • Distribution theory for hierarchical processes: Supplementary material. We provide the proofs of the theoretical results and specialize the Blackwell–MacQueen urn scheme of Section 6.1 to the case of hierarchies of Pitman–Yor processes.