The Annals of Statistics

Identifiability of parameters in latent structure models with many observed variables

Elizabeth S. Allman, Catherine Matias, and John A. Rhodes

Full-text: Open access

Enhanced PDF (290 KB)

Abstract

While hidden class models of various types arise in many statistical applications, it is often difficult to establish the identifiability of their parameters. Focusing on models in which there is some structure of independence of some of the observed variables conditioned on hidden ones, we demonstrate a general approach for establishing identifiability utilizing algebraic arguments. A theorem of J. Kruskal for a simple latent-class model with finite state space lies at the core of our results, though we apply it to a diverse set of models. These include mixtures of both finite and nonparametric product distributions, hidden Markov models and random graph mixture models, and lead to a number of new results and improvements to old ones.

In the parametric setting, this approach indicates that for such models, the classical definition of identifiability is typically too strong. Instead generic identifiability holds, which implies that the set of nonidentifiable parameters has measure zero, so that parameter inference is still meaningful. In particular, this sheds light on the properties of finite mixtures of Bernoulli products, which have been used for decades despite being known to have nonidentifiable parameters. In the nonparametric setting, we again obtain identifiability only when certain restrictions are placed on the distributions that are mixed, but we explicitly describe the conditions.

Article information

Source
Ann. Statist. Volume 37, Number 6A (2009), 3099-3132.

Dates
First available in Project Euclid: 17 August 2009

Permanent link to this document
https://projecteuclid.org/euclid.aos/1250515381

Digital Object Identifier
doi:10.1214/09-AOS689

Mathematical Reviews number (MathSciNet)
MR2549554

Zentralblatt MATH identifier
1191.62003

Subjects
Primary: 62E10: Characterization and structure theory
Secondary: 62F99: None of the above, but in this section 62G99: None of the above, but in this section

Keywords
Identifiability finite mixture latent structure conditional independence multivariate Bernoulli mixture nonparametric mixture contingency table algebraic statistics

Citation

Allman, Elizabeth S.; Matias, Catherine; Rhodes, John A. Identifiability of parameters in latent structure models with many observed variables. Ann. Statist. 37 (2009), no. 6A, 3099--3132. doi:10.1214/09-AOS689. https://projecteuclid.org/euclid.aos/1250515381


Export citation

References

  • [1] Abo, H., Ottaviani, G. and Peterson, C. (2009). Induction for secant varieties of Segre varieties. Trans. Amer. Math. Soc. 361 767–792. Available at arXiv:math.AG/0607191.
    Mathematical Reviews (MathSciNet): MR2452824
    Zentralblatt MATH: 1170.14036
    Digital Object Identifier: doi:10.1090/S0002-9947-08-04725-9
  • [2] Allman, E. S. and Rhodes, J. A. (2009). The identifiability of tree topology for phylogenetic models, including covarion and mixture models. J. Comput. Biol. 13 1101–1113.
    Mathematical Reviews (MathSciNet): MR2255411
    Digital Object Identifier: doi:10.1089/cmb.2006.13.1101
  • [3] Allman, E. S. and Rhodes, J. A. (2009). The identifiability of covarion models in phylogenetics. IEEE/ACM Trans. Comput. Biol. Bioinformatics 6 76–88.
  • [4] Benaglia, T., Chauveau, D. and Hunter, D. R. (2009). An EM-like algorithm for semi and nonparametric estimation in multivariate mixtures. J. Comput. Graph. Statist. To appear.
  • [5] Cappé, O., Moulines, E. and Rydén, T. (2005). Inference in Hidden Markov Models. Springer, New York.
    Mathematical Reviews (MathSciNet): MR2159833
  • [6] Carreira-Perpiñán, M. and Renals, S. (2000). Practical identifiability of finite mixtures of multivariate Bernoulli distributions. Neural Comput. 12 141–152.
  • [7] Catalisano, M. V., Geramita, A. V. and Gimigliano, A. (2002). Ranks of tensors, secant varieties of Segre varieties and fat points. Linear Algebra Appl. 355 263–285.
    Mathematical Reviews (MathSciNet): MR1930149
    Digital Object Identifier: doi:10.1016/S0024-3795(02)00352-X
  • [8] Catalisano, M. V., Geramita, A. V. and Gimigliano, A. (2005). Higher secant varieties of the Segre varieties ℙ1×⋯×ℙ1. J. Pure Appl. Algebra 201 367–380.
  • [9] Chambaz, A. and Matias, C. (2009). Number of hidden states and memory: A joint order estimation problem for Markov chains with Markov regime. ESAIM Probab. Stat. 13 38–50.
    Mathematical Reviews (MathSciNet): MR2493854
    Zentralblatt MATH: 1180.62117
    Digital Object Identifier: doi:10.1051/ps:2007048
  • [10] Cox, D., Little, J. and O’Shea, D. (1997). Ideals, Varieties, and Algorithms, 2nd ed. Springer, New York.
    Mathematical Reviews (MathSciNet): MR1417938
  • [11] Cruz-Medina, I. R., Hettmansperger, T. P. and Thomas, H. (2004). Semiparametric mixture models and repeated measures: The multinomial cut point model. J. Roy. Statist. Soc. Ser. C 53 463–474.
    Mathematical Reviews (MathSciNet): MR2075502
    Zentralblatt MATH: 1111.62302
    Digital Object Identifier: doi:10.1111/j.1467-9876.2004.05203.x
    JSTOR: links.jstor.org
  • [12] Daudin, J.-J., Picard, F. and Robin, S. (2008). A mixture model for random graphs. Statist. Comput. 18 173–183.
    Mathematical Reviews (MathSciNet): MR2390817
    Digital Object Identifier: doi:10.1007/s11222-007-9046-7
  • [13] Drton, M. (2006). Algebraic techniques for Gaussian models. In Prague Stochastics (M. Huskova and M. Janzura, eds.) 81–90. Matfyz Press, Prague.
  • [14] Elmore, R., Hall, P. and Neeman, A. (2005). An application of classical invariant theory to identifiability in nonparametric mixtures. Ann. Inst. Fourier (Grenoble) 55 1–28.
    Mathematical Reviews (MathSciNet): MR2141286
  • [15] Elmore, R. T., Hettmansperger, T. P. and Thomas, H. (2004). Estimating component cumulative distribution functions in finite mixture models. Comm. Statist. Theory Methods 33 2075–2086.
    Mathematical Reviews (MathSciNet): MR2103063
    Zentralblatt MATH: 02128328
    Digital Object Identifier: doi:10.1081/STA-200026574
  • [16] Ephraim, Y. and Merhav, N. (2002). Hidden Markov processes. IEEE Trans. Inform. Theory 48 1518–1569.
    Mathematical Reviews (MathSciNet): MR1909472
    Digital Object Identifier: doi:10.1109/TIT.2002.1003838
  • [17] Finesso, L. (1991). Consistent estimation of the order for Markov and hidden Markov chains. Ph.D. thesis, Univ. Maryland.
  • [18] Frank, O. and Harary, F. (1982). Cluster inference by using transitivity indices in empirical graphs. J. Amer. Statist. Assoc. 77 835–840.
    Mathematical Reviews (MathSciNet): MR686407
    Zentralblatt MATH: 0505.62043
    Digital Object Identifier: doi:10.2307/2287315
    JSTOR: links.jstor.org
  • [19] Garcia, L. D., Stillman, M. and Sturmfels, B. (2005). Algebraic geometry of Bayesian networks. J. Symbolic Comput. 39 331–355.
    Mathematical Reviews (MathSciNet): MR2168286
    Zentralblatt MATH: 1126.68102
    Digital Object Identifier: doi:10.1016/j.jsc.2004.11.007
  • [20] Gilbert, E. J. (1959). On the identifiability problem for functions of finite Markov chains. Ann. Math. Statist. 30 688–697.
    Mathematical Reviews (MathSciNet): MR107304
    Zentralblatt MATH: 0089.34503
    Digital Object Identifier: doi:10.1214/aoms/1177706199
    Project Euclid: euclid.aoms/1177706199
  • [21] Glick, N. (1973). Sample-based multinomial classification. Biometrics 29 241–256.
    Mathematical Reviews (MathSciNet): MR347009
    Digital Object Identifier: doi:10.2307/2529389
    JSTOR: links.jstor.org
  • [22] Goodman, L. A. (1974). Exploratory latent structure analysis using both identifiable and unidentifiable models. Biometrika 61 215–231.
    Mathematical Reviews (MathSciNet): MR370936
    Zentralblatt MATH: 0281.62057
    Digital Object Identifier: doi:10.1093/biomet/61.2.215
    JSTOR: links.jstor.org
  • [23] Gyllenberg, M., Koski, T., Reilink, E. and Verlaan, M. (1994). Nonuniqueness in probabilistic numerical identification of bacteria. J. Appl. Probab. 31 542–548.
    Mathematical Reviews (MathSciNet): MR1274807
    Digital Object Identifier: doi:10.2307/3215044
    JSTOR: links.jstor.org
  • [24] Hall, P., Neeman, A., Pakyari, R. and Elmore, R. (2005). Nonparametric inference in multivariate mixtures. Biometrika 92 667–678.
    Mathematical Reviews (MathSciNet): MR2202653
    Zentralblatt MATH: 1152.62327
    Digital Object Identifier: doi:10.1093/biomet/92.3.667
  • [25] Hall, P. and Zhou, X.-H. (2003). Nonparametric estimation of component distributions in a multivariate mixture. Ann. Statist. 31 201–224.
    Mathematical Reviews (MathSciNet): MR1962504
    Zentralblatt MATH: 1018.62021
    Digital Object Identifier: doi:10.1214/aos/1046294462
    Project Euclid: euclid.aos/1046294462
  • [26] Hettmansperger, T. P. and Thomas, H. (2000). Almost nonparametric inference for repeated measures in mixture models. J. R. Stat. Soc. Ser. B Stat. Methodol. 62 811–825.
    Mathematical Reviews (MathSciNet): MR1796294
    Zentralblatt MATH: 0957.62026
    Digital Object Identifier: doi:10.1111/1467-9868.00266
    JSTOR: links.jstor.org
  • [27] Koopmans, T. C. and Reiersøl, O. (1950). The identification of structural characteristics. Ann. Math. Statist. 21 165–181.
  • [28] Koopmans, T. C., ed. (1950). Statistical Inference in Dynamic Economic Models. Wiley, New York.
    Mathematical Reviews (MathSciNet): MR38640
  • [29] Kruskal, J. B. (1976). More factors than subjects, tests and treatments: An indeterminacy theorem for canonical decomposition and individual differences scaling. Psychometrika 41 281–293.
    Mathematical Reviews (MathSciNet): MR488592
    Digital Object Identifier: doi:10.1007/BF02293554
  • [30] Kruskal, J. B. (1977). Three-way arrays: Rank and uniqueness of trilinear decompositions, with application to arithmetic complexity and statistics. Linear Algebra Appl. 18 95–138.
    Mathematical Reviews (MathSciNet): MR444690
    Zentralblatt MATH: 0364.15021
    Digital Object Identifier: doi:10.1016/0024-3795(77)90069-6
  • [31] Lauritzen, S. L. (1996). Graphical Models Oxford Statistical Science Series 17. Clarendon Press, New York.
    Mathematical Reviews (MathSciNet): MR1419991
  • [32] Leroux, B. G. (1992). Maximum-likelihood estimation for hidden Markov models. Stochastic Process. Appl. 40 127–143.
    Mathematical Reviews (MathSciNet): MR1145463
    Zentralblatt MATH: 0738.62081
    Digital Object Identifier: doi:10.1016/0304-4149(92)90141-C
  • [33] Lindsay, B. G. (1995). Mixture Models: Theory, Geometry and Applications. NSF-CBMS Regional Conference Series in Probability and Statistics 5. IMS, Hayward, CA.
  • [34] McLachlan, G. and Peel, D. (2000). Finite Mixture Models. Wiley, New York.
    Mathematical Reviews (MathSciNet): MR1789474
  • [35] Nigam, K., McCallum, A. K., Thrun, S. and Mitchell, T. M. (2000). Text classification from labeled and unlabeled documents using EM. Machine Learning 39 103–134.
  • [36] Nowicki, K. and Snijders, T. A. B. (2001). Estimation and prediction for stochastic blockstructures. J. Amer. Statist. Assoc. 96 1077–1087.
    Mathematical Reviews (MathSciNet): MR1947255
    Zentralblatt MATH: 1072.62542
    Digital Object Identifier: doi:10.1198/016214501753208735
    JSTOR: links.jstor.org
  • [37] Pachter, L. and Sturmfels, B. (2005). Algebraic Statistics for Computational Biology. Cambridge Univ. Press, Cambridge.
    Mathematical Reviews (MathSciNet): MR2205865
  • [38] Paz, A. (1971). Introduction to Probabilistic Automata. Academic Press, New York.
    Mathematical Reviews (MathSciNet): MR289222
    Zentralblatt MATH: 0234.94055
  • [39] Petrie, T. (1969). Probabilistic functions of finite state Markov chains. Ann. Math. Statist. 40 97–115.
    Mathematical Reviews (MathSciNet): MR239662
    Digital Object Identifier: doi:10.1214/aoms/1177697807
    Project Euclid: euclid.aoms/1177697807
  • [40] Rothenberg, T. J. (1971). Identification in parametric models. Econometrica 39 577–591.
    Mathematical Reviews (MathSciNet): MR436944
    Digital Object Identifier: doi:10.2307/1913267
    JSTOR: links.jstor.org
  • [41] Tallberg, C. (2005). A Bayesian approach to modeling stochastic blockstructures with covariates. J. Math. Soc. 29 1–23.
  • [42] Teicher, H. (1967). Identifiability of mixtures of product measures. Ann. Math. Statist. 38 1300–1302.
    Mathematical Reviews (MathSciNet): MR216635
    Zentralblatt MATH: 0153.47904
    Digital Object Identifier: doi:10.1214/aoms/1177698805
    Project Euclid: euclid.aoms/1177698805
  • [43] Yakowitz, S. J. and Spragins, J. D. (1968). On the identifiability of finite mixtures. Ann. Math. Statist. 39 209–214.
    Mathematical Reviews (MathSciNet): MR224204
    Zentralblatt MATH: 0155.25703
    Digital Object Identifier: doi:10.1214/aoms/1177698520
    Project Euclid: euclid.aoms/1177698520
  • [44] Zanghi, H., Ambroise, C. and Miele, V. (2008). Fast online graph clustering via Erdös–Rényi mixture. Pattern Recognition 41 3592–3599.

The Institute of Mathematical Statistics

The Annals of Statistics

New content alerts

Email RSS ToC RSS Article
  • You have access to this content.
  • You have partial access to this content.
  • You do not have access to this content.
Turn Off MathJax

What is MathJax?

More like this

+ See more