Electronic Journal of Statistics

Kernel ridge vs. principal component regression: Minimax bounds and the qualification of regularization operators

Lee H. Dicker, Dean P. Foster, and Daniel Hsu

Full-text: Open access

Enhanced PDF (426 KB)

Abstract

Regularization is an essential element of virtually all kernel methods for nonparametric regression problems. A critical factor in the effectiveness of a given kernel method is the type of regularization that is employed. This article compares and contrasts members from a general class of regularization techniques, which notably includes ridge regression and principal component regression. We derive an explicit finite-sample risk bound for regularization-based estimators that simultaneously accounts for (i) the structure of the ambient function space, (ii) the regularity of the true regression function, and (iii) the adaptability (or qualification) of the regularization. A simple consequence of this upper bound is that the risk of the regularization-based estimators matches the minimax rate in a variety of settings. The general bound also illustrates how some regularization techniques are more adaptable than others to favorable regularity properties that the true regression function may possess. This, in particular, demonstrates a striking difference between kernel ridge regression and kernel principal component regression. Our theoretical results are supported by numerical experiments.

Article information

Source
Electron. J. Statist., Volume 11, Number 1 (2017), 1022-1047.

Dates
Received: August 2016
First available in Project Euclid: 30 March 2017

Permanent link to this document
https://projecteuclid.org/euclid.ejs/1490860815

Digital Object Identifier
doi:10.1214/17-EJS1258

Mathematical Reviews number (MathSciNet)
MR3629418

Zentralblatt MATH identifier
1362.62087

Subjects
Primary: 62G08: Nonparametric regression

Keywords
Nonparametric regression reproducing kernel Hilbert space

Rights
Creative Commons Attribution 4.0 International License.

Citation

Dicker, Lee H.; Foster, Dean P.; Hsu, Daniel. Kernel ridge vs. principal component regression: Minimax bounds and the qualification of regularization operators. Electron. J. Statist. 11 (2017), no. 1, 1022--1047. doi:10.1214/17-EJS1258. https://projecteuclid.org/euclid.ejs/1490860815


Export citation

References

  • Aronszajn, N. (1950). Theory of reproducing kernels., T. A. Math. Soc. 68 337–404.
    Zentralblatt MATH: 0037.20701
    Digital Object Identifier: doi:10.1090/S0002-9947-1950-0051437-7
  • Bauer, F., Pereverzev, S. and Rosasco, L. (2007). On regularization algorithms in learning theory., J. Complexity 23 52–72.
    Zentralblatt MATH: 1109.68088
    Digital Object Identifier: doi:10.1016/j.jco.2006.07.001
  • Blanchard, G. and Mücke, N. (2016). Optimal rates for regularization of statistical inverse learning problems., arXiv preprint arXiv:1604.04054.
  • Caponnetto, A. and De Vito, E. (2007). Optimal rates for the regularized least-squares algorithm., Found. Comput. Math. 7 331–368.
    Zentralblatt MATH: 1129.68058
    Digital Object Identifier: doi:10.1007/s10208-006-0196-8
  • Caponnetto, A. and Yao, Y. (2010). Cross-validation based adaptation for regularization operators in learning theory., Anal. Appl. 8 161–183.
    Mathematical Reviews (MathSciNet): MR2646909
    Zentralblatt MATH: 1209.68405
    Digital Object Identifier: doi:10.1142/S0219530510001564
  • Carmeli, C., De Vito, E. and Toigo, A. (2006). Vector valued reproducing kernel Hilbert spaces of integrable functions and Mercer theorem., Anal. Appl. 4 377–408.
    Zentralblatt MATH: 1116.46019
    Digital Object Identifier: doi:10.1142/S0219530506000838
  • Davis, C. and Kahan, W. M. (1970). The rotation of eigenvectors by a perturbation. III., SIAM J. Numer. Anal. 7 1–46.
    Zentralblatt MATH: 0198.47201
    Digital Object Identifier: doi:10.1137/0707001
  • Dhillon, P. S., Foster, D. P., Kakade, S. M. and Ungar, L. H. (2013). A risk comparison of ordinary least squares vs ridge regression., J. Mach. Learn. Res. 14 1505–1511.
    Zentralblatt MATH: 1317.68152
  • Dicker, L. H. (2016). Ridge regression and asymptotic minimax estimation over spheres of growing dimension., Bernoulli 22 1–37.
    Zentralblatt MATH: 1388.62205
    Digital Object Identifier: doi:10.3150/14-BEJ609
    Project Euclid: euclid.bj/1443620842
  • Engl, H. W., Hanke, M. and Neubauer, A. (1996)., Regularization of Inverse Problems. Mathematics and Its Applications, Vol. 375, Springer.
    Zentralblatt MATH: 0859.65054
  • Guo, Z. C., Lin, S. B. and Zhou, D. X. (2016). Learning theory of distributed spectral algorithms., Preprint.
  • Györfi, L., Kohler, M., Krzyzak, A. and Walk, H. (2002)., A Distribution Free Theory of Nonparametric Regression. Springer.
  • Hsu, D., Kakade, S. M. and Zhang, T. (2014). Random design analysis of ridge regression., Found. Comput. Math. 14 569–600.
    Zentralblatt MATH: 1298.62120
    Digital Object Identifier: doi:10.1007/s10208-014-9192-1
  • Koltchinskii, V. (2006). Local Rademacher complexities and oracle inequalities in risk minimization., Ann. Stat. 34 2593–2656.
    Zentralblatt MATH: 1118.62065
    Digital Object Identifier: doi:10.1214/009053606000001019
    Project Euclid: euclid.aos/1179935055
  • Lin, S. B., Guo, X. and Zhou, D. X. (2016). Distributed learning with regularized least squares., arXiv preprint arXiv:1608.03339.
    Zentralblatt MATH: 06860797
  • Lo Gerfo, L., Rosasco, L., Odone, F., De Vito, E. and Verri, A. (2008). Spectral algorithms for supervised learning., Neural Comput. 7 1873–1897.
    Zentralblatt MATH: 1147.68643
    Digital Object Identifier: doi:10.1162/neco.2008.05-07-517
  • Mathé, P. (2005). Saturation of regularization methods for linear ill-posed problems in Hilbert spaces., SIAM J. Numer. Anal. 42 968–973.
  • Minsker, S. (2011). On some extensions of Bernstein’s inequality for self-adjoint operators., arXiv preprint arXiv:1112.5448.
  • Neubauer, A. (1997). On converse and saturation results for Tikhonov regularization of linear ill-posed problems., SIAM J. Numer. Anal. 34 517–527.
    Zentralblatt MATH: 0878.65038
    Digital Object Identifier: doi:10.1137/S0036142993253928
  • Pinsker, M. S. (1980). Optimal filtration of square-integrable signals in Gaussian noise., Probl. Inf. Transm. 16 52–68.
  • Reed, M. and Simon, B. (1980)., Methods of Modern Mathematical Physics; Volume 1, Functional Analysis. Academic Press.
    Zentralblatt MATH: 0459.46001
  • Rosasco, L., De Vito, E. and Verri, A. (2005). Spectral methods for regularization in learning theory. Tech. Rep. DISI-TR-05-18, Universitá degli Studi di Genova, Italy.
  • Sonnenburg, S. (2008)., Machine Learning for Genomic Sequence Analysis. Ph.D. thesis, Fraunhofer Institute FIRST. Supervised by K.-R. Müller and G. Rätsch.
  • Steinwart, I., Hush, D. and Scovel, C. (2009). Optimal rates for regularized least squares regression. In, Conference on Learning Theory.
    Zentralblatt MATH: 1127.68090
  • Tropp, J. A. (2015). An introduction to matrix concentration inequalities., arXiv preprint arXiv:1501.01571.
    Zentralblatt MATH: 1391.15071
    Digital Object Identifier: doi:10.1561/2200000048
  • Wasserman, L. (2006)., All of Nonparametric Statistics. Springer.
    Zentralblatt MATH: 1099.62029
  • Zhang, T. (2005). Learning bounds for kernel regression using effective data dimensionality., Neural Comput. 17 2077–2098.
    Zentralblatt MATH: 1080.68044
    Digital Object Identifier: doi:10.1162/0899766054323008
  • Zhang, Y., Duchi, J. C. and Wainwright, M. J. (2013). Divide and conquer kernel ridge regression. In, Conference on Learning Theory.
    Zentralblatt MATH: 1351.62142

The Institute of Mathematical Statistics and the Bernoulli Society

Electronic Journal of Statistics

New content alerts

Email RSS ToC RSS Article
  • You have access to this content.
  • You have partial access to this content.
  • You do not have access to this content.
Turn Off MathJax

What is MathJax?

More like this

+ See more