Translator Disclaimer
February 2020 Statistical inference for model parameters in stochastic gradient descent
Xi Chen, Jason D. Lee, Xin T. Tong, Yichen Zhang
Ann. Statist. 48(1): 251-273 (February 2020). DOI: 10.1214/18-AOS1801

Abstract

The stochastic gradient descent (SGD) algorithm has been widely used in statistical estimation for large-scale data due to its computational and memory efficiency. While most existing works focus on the convergence of the objective function or the error of the obtained solution, we investigate the problem of statistical inference of true model parameters based on SGD when the population loss function is strongly convex and satisfies certain smoothness conditions.

Our main contributions are twofold. First, in the fixed dimension setup, we propose two consistent estimators of the asymptotic covariance of the average iterate from SGD: (1) a plug-in estimator, and (2) a batch-means estimator, which is computationally more efficient and only uses the iterates from SGD. Both proposed estimators allow us to construct asymptotically exact confidence intervals and hypothesis tests.

Second, for high-dimensional linear regression, using a variant of the SGD algorithm, we construct a debiased estimator of each regression coefficient that is asymptotically normal. This gives a one-pass algorithm for computing both the sparse regression coefficients and confidence intervals, which is computationally attractive and applicable to online data.

Citation

Download Citation

Xi Chen. Jason D. Lee. Xin T. Tong. Yichen Zhang. "Statistical inference for model parameters in stochastic gradient descent." Ann. Statist. 48 (1) 251 - 273, February 2020. https://doi.org/10.1214/18-AOS1801

Information

Received: 1 October 2017; Revised: 1 July 2018; Published: February 2020
First available in Project Euclid: 17 February 2020

zbMATH: 07196538
MathSciNet: MR4065161
Digital Object Identifier: 10.1214/18-AOS1801

Subjects:
Primary: 62J10, 62M02
Secondary: 60K35

Rights: Copyright © 2020 Institute of Mathematical Statistics

JOURNAL ARTICLE
23 PAGES

This article is only available to subscribers.
It is not available for individual sale.
+ SAVE TO MY LIBRARY

SHARE
Vol.48 • No. 1 • February 2020
Back to Top