The Annals of Statistics

Estimating multivariate latent-structure models

Stéphane Bonhomme, Koen Jochmans, and Jean-Marc Robin

Full-text: Open access


A constructive proof of identification of multilinear decompositions of multiway arrays is presented. It can be applied to show identification in a variety of multivariate latent structures. Examples are finite-mixture models and hidden Markov models. The key step to show identification is the joint diagonalization of a set of matrices in the same nonorthogonal basis. An estimator of the latent-structure model may then be based on a sample version of this joint-diagonalization problem. Algorithms are available for computation and we derive distribution theory. We further develop asymptotic theory for orthogonal-series estimators of component densities in mixture models and emission densities in hidden Markov models.

Article information

Ann. Statist., Volume 44, Number 2 (2016), 540-563.

Received: May 2015
Revised: August 2015
First available in Project Euclid: 17 March 2016

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 15A69: Multilinear algebra, tensor products 62G05: Estimation
Secondary: 15A18: Eigenvalues, singular values, and eigenvectors 15A23: Factorization of matrices 62G20: Asymptotic properties 62H17: Contingency tables 62H30: Classification and discrimination; cluster analysis [See also 68T10, 91C20]

Finite mixture model hidden Markov model latent structure multilinear restrictions multivariate data nonparametric estimation simultaneous matrix diagonalization


Bonhomme, Stéphane; Jochmans, Koen; Robin, Jean-Marc. Estimating multivariate latent-structure models. Ann. Statist. 44 (2016), no. 2, 540--563. doi:10.1214/15-AOS1376.

Export citation


  • Allman, E. S., Matias, C. and Rhodes, J. A. (2009). Identifiability of parameters in latent structure models with many observed variables. Ann. Statist. 37 3099–3132.
  • Allman, E. S., Matias, C. and Rhodes, J. A. (2011). Parameter identifiability in a class of random graph mixture models. J. Statist. Plann. Inference 141 1719–1736.
  • Anandkumar, A., Ge, R., Hsu, D., Kakade, S. M. and Telgarsky, M. (2014). Tensor decompositions for learning latent variable models. J. Mach. Learn. Res. 15 2773–2832.
  • Anderson, T. W. (1954). On estimation of parameters in latent structure analysis. Psychometrika 19 1–10.
  • Azzalini, A. (1985). A class of distributions which includes the normal ones. Scand. J. Stat. 12 171–178.
  • Benaglia, T., Chauveau, D. and Hunter, D. R. (2009). An EM-like algorithm for semi- and nonparametric estimation in multivariate mixtures. J. Comput. Graph. Statist. 18 505–526.
  • Bonhomme, S., Jochmans, K. and Robin, J. M. (2014). Nonparametric estimation of finite mixtures from repeated measurements. J. R. Stat. Soc. Ser. B 78 211–229.
  • Bonhomme, S., Jochmans, K. and Robin, J. M. (2015). Supplement to “Estimating multivariate latent-structure models.” DOI:10.1214/15-AOS1376SUPP.
  • Bordes, L., Mottelet, S. and Vandekerkhove, P. (2006). Semiparametric estimation of a two-component mixture model. Ann. Statist. 34 1204–1232.
  • Cappé, O., Moulines, E. and Rydén, T. (2005). Inference in Hidden Markov Models. Springer, New York.
  • Chiantini, L., Ottaviani, G. and Vannieuwenhoven, N. (2014). An algorithm for generic and low-rank specific identifiability of complex tensors. SIAM J. Matrix Anal. Appl. 35 1265–1287.
  • Chiantini, L., Ottaviani, G. and Vannieuwenhoven, N. (2015). On generic identifiability of symmetric tensors of subgeneric rank. Mimeo.
  • Comon, P. and Jutten, C. (2010). Handbook of Blind Source Separation: Independent Component Analysis and Applications. Academic Press, San Diego, CA.
  • De Lathauwer, L. (2006). A link between the canonical decomposition in multilinear algebra and simultaneous matrix diagonalization. SIAM J. Matrix Anal. Appl. 28 642–666.
  • De Lathauwer, L., De Moor, B. and Vandewalle, J. (2004). Computation of the canonical decomposition by means of a simultaneous generalized Schur decomposition. SIAM J. Matrix Anal. Appl. 26 295–327.
  • Domanov, I. and De Lathauwer, L. (2013a). On the uniqueness of the canonical polyadic decomposition of third-order tensors—Part I: Basic results and uniqueness of one factor matrix. SIAM J. Matrix Anal. Appl. 34 855–875.
  • Domanov, I. and De Lathauwer, L. (2013b). On the uniqueness of the canonical polyadic decomposition of third-order tensors—Part II: Uniqueness of the overall decomposition. SIAM J. Matrix Anal. Appl. 34 876–903.
  • Domanov, I. and De Lathauwer, L. (2014a). Canonical polyadic decomposition of third-order tensors: Reduction to generalized eigenvalue decomposition. SIAM J. Matrix Anal. Appl. 35 636–660.
  • Domanov, I. and De Lathauwer, L. (2014b). Generic uniqueness conditions for the canonical polyadic decomposition and INDSCAL. Mimeo.
  • Efromovich, S. (1999). Nonparametric Curve Estimation: Methods, Theory, and Applications. Springer, New York.
  • Fu, T. and Gao, X. Q. (2006). Simultaneous diagonalization with similarity transformation for non-defective matrices. Proceedings of the IEEE ICA SSP 2006 4 1137–1140.
  • Gassiat, E., Cleynen, A. and Robin, S. (2016). Inference in finite state space non parametric Hidden Markov Models and applications. Stat. Comput. 26 61–71.
  • Gassiat, E. and Rousseau, J. (2014). Non parametric finite translation mixtures and extensions. Bernoulli 22 193–212.
  • Green, B. (1951). A general solution for the latent class model of latent structure analysis. Psychometrika 16 151–166.
  • Hall, P. (1987). Cross-validation and the smoothing of orthogonal series density estimators. J. Multivariate Anal. 21 189–206.
  • Hall, P. and Zhou, X.-H. (2003). Nonparametric estimation of component distributions in a multivariate mixture. Ann. Statist. 31 201–224.
  • Hall, P., Neeman, A., Pakyari, R. and Elmore, R. (2005). Nonparametric inference in multivariate mixtures. Biometrika 92 667–678.
  • Hettmansperger, T. P. and Thomas, H. (2000). Almost nonparametric inference for repeated measures in mixture models. J. R. Stat. Soc. Ser. B 62 811–825.
  • Hunter, D. R., Wang, S. and Hettmansperger, T. P. (2007). Inference for mixtures of symmetric distributions. Ann. Statist. 35 224–251.
  • Iferroudjene, R., Abed Meraim, K. and Belouchrani, A. (2010). Joint diagonalization of non defective matrices using generalized Jacobi rotations. In 10th International Conference on Information Sciences Signal Processing and Their Applications (ISSPA), 2010 345–348.
  • Iferroudjene, R., Abed Meraim, K. and Belouchrani, A. (2009). A new Jacobi-like method for joint diagonalization of arbitrary non-defective matrices. Appl. Math. Comput. 211 363–373.
  • Jiang, T. and Sidiropoulos, N. D. (2004). Kruskal’s permutation lemma and the identification of CANDECOMP/PARAFAC and bilinear models with constant modulus constraints. IEEE Trans. Signal Process. 52 2625–2636.
  • Jochmans, K., Henry, M. and Salanié, B. (2014). Inference on mixtures under tail restrictions. Discussion Paper No. 2014-01, Dept. Economics, Sciences Po, Paris.
  • Kasahara, H. and Shimotsu, K. (2009). Nonparametric identification of finite mixture models of dynamic discrete choices. Econometrica 77 135–175.
  • Kasahara, H. and Shimotsu, K. (2014). Non-parametric identification and estimation of the number of components in multivariate mixtures. J. R. Stat. Soc. Ser. B 76 97–111.
  • Kruskal, J. B. (1976). More factors than subjects, tests and treatments: An indeterminacy theorem for canonical decomposition and individual differences scaling. Psychometrika 41 281–293.
  • Kruskal, J. B. (1977). Three-way arrays: Rank and uniqueness of trilinear decompositions, with application to arithmetic complexity and statistics. Linear Algebra Appl. 18 95–138.
  • Levine, M., Hunter, D. R. and Chauveau, D. (2011). Maximum smoothed likelihood for multivariate mixtures. Biometrika 98 403–416.
  • Liebscher, E. (1990). Hermite series estimators for probability densities. Metrika 37 321–343.
  • Luciani, X. and Albera, L. (2010). Joint eigenvalue decomposition using polar matrix factorization. In Latent Variable Analysis and Signal Separation. Lecture Notes in Computer Sciences 6365 555–562. Springer, Berlin.
  • Luciani, X. and Albera, L. (2014). Canonical polyadic decomposition based on joint eigenvalue decomposition. Chemom. Intell. Lab. Syst. 132 152–167.
  • McLachlan, G. and Peel, D. (2000). Finite Mixture Models. Wiley, New York.
  • Newey, W. K. (1997). Convergence rates and asymptotic normality for series estimators. J. Econometrics 79 147–168.
  • Petrie, T. (1969). Probabilistic functions of finite state Markov chains. Ann. Math. Statist 40 97–115.
  • Rohe, K., Chatterjee, S. and Yu, B. (2011). Spectral clustering and the high-dimensional stochastic blockmodel. Ann. Statist. 39 1878–1915.
  • Sidiropoulos, N. D. and Bro, R. (2000). On the uniqueness of multilinear decomposition of $N$-way arrays. J. Chemom. 14 229–239.
  • Snijders, T. A. B. and Nowicki, K. (1997). Estimation and prediction for stochastic blockmodels for graphs with latent block structure. J. Classification 14 75–100.

Supplemental materials

  • Supplement to “Estimating multivariate latent-structure models”. The supplement to this paper [Bonhomme, Jochmans and Robin (2015)] contains additional details and discussion, omitted proofs and additional simulation results.