Translator Disclaimer
December 2020 Singularity, misspecification and the convergence rate of EM
Raaz Dwivedi, Nhat Ho, Koulik Khamaru, Martin J. Wainwright, Michael I. Jordan, Bin Yu
Ann. Statist. 48(6): 3161-3182 (December 2020). DOI: 10.1214/19-AOS1924

Abstract

A line of recent work has analyzed the behavior of the Expectation-Maximization (EM) algorithm in the well-specified setting, in which the population likelihood is locally strongly concave around its maximizing argument. Examples include suitably separated Gaussian mixture models and mixtures of linear regressions. We consider over-specified settings in which the number of fitted components is larger than the number of components in the true distribution. Such mis-specified settings can lead to singularity in the Fisher information matrix, and moreover, the maximum likelihood estimator based on $n$ i.i.d. samples in $d$ dimensions can have a nonstandard $\mathcal{O}((d/n)^{\frac{1}{4}})$ rate of convergence. Focusing on the simple setting of two-component mixtures fit to a $d$-dimensional Gaussian distribution, we study the behavior of the EM algorithm both when the mixture weights are different (unbalanced case), and are equal (balanced case). Our analysis reveals a sharp distinction between these two cases: in the former, the EM algorithm converges geometrically to a point at Euclidean distance of $\mathcal{O}((d/n)^{\frac{1}{2}})$ from the true parameter, whereas in the latter case, the convergence rate is exponentially slower, and the fixed point has a much lower $\mathcal{O}((d/n)^{\frac{1}{4}})$ accuracy. Analysis of this singular case requires the introduction of some novel techniques: in particular, we make use of a careful form of localization in the associated empirical process, and develop a recursive argument to progressively sharpen the statistical rate.

Citation

Download Citation

Raaz Dwivedi. Nhat Ho. Koulik Khamaru. Martin J. Wainwright. Michael I. Jordan. Bin Yu. "Singularity, misspecification and the convergence rate of EM." Ann. Statist. 48 (6) 3161 - 3182, December 2020. https://doi.org/10.1214/19-AOS1924

Information

Received: 1 September 2019; Published: December 2020
First available in Project Euclid: 11 December 2020

MathSciNet: MR4185804
Digital Object Identifier: 10.1214/19-AOS1924

Subjects:
Primary: 62F15, 62G05
Secondary: 62G20

Rights: Copyright © 2020 Institute of Mathematical Statistics

JOURNAL ARTICLE
22 PAGES

This article is only available to subscribers.
It is not available for individual sale.
+ SAVE TO MY LIBRARY

SHARE
Vol.48 • No. 6 • December 2020
Back to Top