Translator Disclaimer
October 2021 Set structured global empirical risk minimizers are rate optimal in general dimensions
Qiyang Han
Author Affiliations +
Ann. Statist. 49(5): 2642-2671 (October 2021). DOI: 10.1214/21-AOS2049

Abstract

Entropy integrals are widely used as a powerful empirical process tool to obtain upper bounds for the rates of convergence of global empirical risk minimizers (ERMs), in standard settings such as density estimation and regression. The upper bound for the convergence rates thus obtained typically matches the minimax lower bound when the entropy integral converges, but admits a strict gap compared to the lower bound when it diverges. Birgé and Massart (Probab. Theory Related Fields 97 (1993) 113–150) provided a striking example showing that such a gap is real with the entropy structure alone: for a variant of the natural Hölder class with low regularity, the global ERM actually converges at the rate predicted by the entropy integral that substantially deviates from the lower bound. The counter-example has spawned a long-standing negative position on the use of global ERMs in the regime where the entropy integral diverges, as they are heuristically believed to converge at a suboptimal rate in a variety of models.

The present paper demonstrates that this gap can be closed if the models admit certain degree of “set structures” in addition to the entropy structure. In other words, the global ERMs in such set structured models will indeed be rate-optimal, matching the lower bound even when the entropy integral diverges. The models with set structures we investigate include (i) image and edge estimation, (ii) binary classification, (iii) multiple isotonic regression, (iv) s-concave density estimation, all in general dimensions when the entropy integral diverges. Here, set structures are interpreted broadly in the sense that the complexity of the underlying models can be essentially captured by the size of the empirical process over certain class of measurable sets, for which matching upper and lower bounds are obtained to facilitate the derivation of sharp convergence rates for the associated global ERMs.

Funding Statement

The author’s research is supported in part by NSF Grant DMS-1916221.

Acknowledgments

The major part of this work (materials presented before Section 3.3 and their proofs) is based on Chapter 4 of the author’s University of Washington Ph.D. thesis in 2018. The author would like to thank Jon Wellner and Cun-Hui Zhang for helpful discussion and encouragements. He would also like to thank four referees, an Associate Editor and the Editor for their very helpful comments and suggestions that significantly improved the quality of the paper.

Citation

Download Citation

Qiyang Han. "Set structured global empirical risk minimizers are rate optimal in general dimensions." Ann. Statist. 49 (5) 2642 - 2671, October 2021. https://doi.org/10.1214/21-AOS2049

Information

Received: 1 May 2019; Revised: 1 September 2020; Published: October 2021
First available in Project Euclid: 12 November 2021

Digital Object Identifier: 10.1214/21-AOS2049

Subjects:
Primary: 60E15
Secondary: 62G05

Keywords: ‎classification‎ , Density estimation , empirical process , empirical risk minimization , non-Donsker , Nonparametric regression

Rights: Copyright © 2021 Institute of Mathematical Statistics

JOURNAL ARTICLE
30 PAGES

This article is only available to subscribers.
It is not available for individual sale.
+ SAVE TO MY LIBRARY

SHARE
Vol.49 • No. 5 • October 2021
Back to Top