Translator Disclaimer
December 2020 Singularity, misspecification and the convergence rate of EM
Raaz Dwivedi, Nhat Ho, Koulik Khamaru, Martin J. Wainwright, Michael I. Jordan, Bin Yu
Ann. Statist. 48(6): 3161-3182 (December 2020). DOI: 10.1214/19-AOS1924


A line of recent work has analyzed the behavior of the Expectation-Maximization (EM) algorithm in the well-specified setting, in which the population likelihood is locally strongly concave around its maximizing argument. Examples include suitably separated Gaussian mixture models and mixtures of linear regressions. We consider over-specified settings in which the number of fitted components is larger than the number of components in the true distribution. Such mis-specified settings can lead to singularity in the Fisher information matrix, and moreover, the maximum likelihood estimator based on $n$ i.i.d. samples in $d$ dimensions can have a nonstandard $\mathcal{O}((d/n)^{\frac{1}{4}})$ rate of convergence. Focusing on the simple setting of two-component mixtures fit to a $d$-dimensional Gaussian distribution, we study the behavior of the EM algorithm both when the mixture weights are different (unbalanced case), and are equal (balanced case). Our analysis reveals a sharp distinction between these two cases: in the former, the EM algorithm converges geometrically to a point at Euclidean distance of $\mathcal{O}((d/n)^{\frac{1}{2}})$ from the true parameter, whereas in the latter case, the convergence rate is exponentially slower, and the fixed point has a much lower $\mathcal{O}((d/n)^{\frac{1}{4}})$ accuracy. Analysis of this singular case requires the introduction of some novel techniques: in particular, we make use of a careful form of localization in the associated empirical process, and develop a recursive argument to progressively sharpen the statistical rate.


Download Citation

Raaz Dwivedi. Nhat Ho. Koulik Khamaru. Martin J. Wainwright. Michael I. Jordan. Bin Yu. "Singularity, misspecification and the convergence rate of EM." Ann. Statist. 48 (6) 3161 - 3182, December 2020.


Received: 1 September 2019; Published: December 2020
First available in Project Euclid: 11 December 2020

MathSciNet: MR4185804
Digital Object Identifier: 10.1214/19-AOS1924

Primary: 62F15 , 62G05
Secondary: 62G20

Keywords: empirical process , expectation-maximization , Fisher information matrix , localization argument , Mixture models , nonasymptotic convergence guarantees

Rights: Copyright © 2020 Institute of Mathematical Statistics


This article is only available to subscribers.
It is not available for individual sale.

Vol.48 • No. 6 • December 2020
Back to Top