Open Access
Translator Disclaimer
2021 Spectrally-truncated kernel ridge regression and its free lunch
Arash A. Amini
Author Affiliations +
Electron. J. Statist. 15(2): 3743-3761 (2021). DOI: 10.1214/21-EJS1873

Abstract

Kernel ridge regression (KRR) is a well-known and popular nonparametric regression approach with many desirable properties, including minimax rate-optimality in estimating functions that belong to common reproducing kernel Hilbert spaces (RKHS). The approach, however, is computationally intensive for large data sets, due to the need to operate on a dense n×n kernel matrix, where n is the sample size. Recently, various approximation schemes for solving KRR have been considered, and some analyzed. Some approaches such as Nyström approximation and sketching have been shown to preserve the rate optimality of KRR. In this paper, we consider the simplest approximation, namely, spectrally truncating the kernel matrix to its largest r<n eigenvalues. We derive an exact expression for the maximum risk of this truncated KRR, over the unit ball of the RKHS. This result can be used to study the exact trade-off between the level of spectral truncation and the regularization parameter. We show that, as long as the RKHS is infinite-dimensional, there is a threshold on r, above which, the spectrally-truncated KRR surprisingly outperforms the full KRR in terms of the minimax risk, where the minimum is taken over the regularization parameter. This strengthens the existing results on approximation schemes, by showing that not only one does not lose in terms of the rates, truncation can in fact improve the performance, for all finite samples (above the threshold). Moreover, we show that the implicit regularization achieved by spectral truncation is not a substitute for Hilbert norm regularization. Both are needed to achieve the best performance.

Acknowledgments

We thank Chad Hazlett and Linfan Zhang for helpful discussions and Zahra Razaee for comments on the manuscript.

Citation

Download Citation

Arash A. Amini. "Spectrally-truncated kernel ridge regression and its free lunch." Electron. J. Statist. 15 (2) 3743 - 3761, 2021. https://doi.org/10.1214/21-EJS1873

Information

Received: 1 July 2020; Published: 2021
First available in Project Euclid: 15 July 2021

Digital Object Identifier: 10.1214/21-EJS1873

Subjects:
Primary: 62G08 , 62G20
Secondary: 62C20

Keywords: kernel methods , minimax estimation , Nonparametric regression , Ridge regression , spectral truncation

JOURNAL ARTICLE
19 PAGES


SHARE
Vol.15 • No. 2 • 2021
Back to Top