Electronic Journal of Statistics
- Electron. J. Statist.
- Volume 12, Number 2 (2018), 3473-3518.
Noise contrastive estimation: Asymptotic properties, formal comparison with MC-MLE
A statistical model is said to be un-normalised when its likelihood function involves an intractable normalising constant. Two popular methods for parameter inference for these models are MC-MLE (Monte Carlo maximum likelihood estimation), and NCE (noise contrastive estimation); both methods rely on simulating artificial data-points to approximate the normalising constant. While the asymptotics of MC-MLE have been established under general hypotheses (Geyer, 1994), this is not so for NCE. We establish consistency and asymptotic normality of NCE estimators under mild assumptions. We compare NCE and MC-MLE under several asymptotic regimes. In particular, we show that, when $m\rightarrow \infty $ while $n$ is fixed ($m$ and $n$ being respectively the number of artificial data-points, and actual data-points), the two estimators are asymptotically equivalent. Conversely, we prove that, when the artificial data-points are IID, and when $n\rightarrow \infty $ while $m/n$ converges to a positive constant, the asymptotic variance of a NCE estimator is always smaller than the asymptotic variance of the corresponding MC-MLE estimator. We illustrate the variance reduction brought by NCE through a numerical study.
Electron. J. Statist., Volume 12, Number 2 (2018), 3473-3518.
Received: February 2018
First available in Project Euclid: 17 October 2018
Permanent link to this document
Digital Object Identifier
Riou-Durand, Lionel; Chopin, Nicolas. Noise contrastive estimation: Asymptotic properties, formal comparison with MC-MLE. Electron. J. Statist. 12 (2018), no. 2, 3473--3518. doi:10.1214/18-EJS1485. https://projecteuclid.org/euclid.ejs/1539741651