Open Access
Translator Disclaimer
June 2017 Randomized sketches for kernels: Fast and optimal nonparametric regression
Yun Yang, Mert Pilanci, Martin J. Wainwright
Ann. Statist. 45(3): 991-1023 (June 2017). DOI: 10.1214/16-AOS1472

Abstract

Kernel ridge regression (KRR) is a standard method for performing nonparametric regression over reproducing kernel Hilbert spaces. Given $n$ samples, the time and space complexity of computing the KRR estimate scale as $\mathcal{O}(n^{3})$ and $\mathcal{O}(n^{2})$, respectively, and so is prohibitive in many cases. We propose approximations of KRR based on $m$-dimensional randomized sketches of the kernel matrix, and study how small the projection dimension $m$ can be chosen while still preserving minimax optimality of the approximate KRR estimate. For various classes of randomized sketches, including those based on Gaussian and randomized Hadamard matrices, we prove that it suffices to choose the sketch dimension $m$ proportional to the statistical dimension (modulo logarithmic factors). Thus, we obtain fast and minimax optimal approximations to the KRR estimate for nonparametric regression. In doing so, we prove a novel lower bound on the minimax risk of kernel regression in terms of the localized Rademacher complexity.

Citation

Download Citation

Yun Yang. Mert Pilanci. Martin J. Wainwright. "Randomized sketches for kernels: Fast and optimal nonparametric regression." Ann. Statist. 45 (3) 991 - 1023, June 2017. https://doi.org/10.1214/16-AOS1472

Information

Received: 1 September 2015; Revised: 1 April 2016; Published: June 2017
First available in Project Euclid: 13 June 2017

zbMATH: 1371.62039
MathSciNet: MR3662446
Digital Object Identifier: 10.1214/16-AOS1472

Subjects:
Primary: 62G08
Secondary: 68W20

Keywords: Convex optimization , dimensionality reduction , kernel method , Nonparametric regression , random projection

Rights: Copyright © 2017 Institute of Mathematical Statistics

JOURNAL ARTICLE
33 PAGES


SHARE
Vol.45 • No. 3 • June 2017
Back to Top