Institute of Mathematical Statistics Collections

Local polynomial regression and variable selection

Hugh Miller and Peter Hall

Full-text: Open access

Abstract

We propose a method for incorporating variable selection into local polynomial regression. This can improve the accuracy of the regression by extending the bandwidth in directions corresponding to those variables judged to be unimportant. It also increases our understanding of the dataset by highlighting areas where these variables are redundant. The approach has the potential to effect complete variable removal as well as perform partial removal when a variable redundancy applies only to particular regions of the data. We define a nonparametric oracle property and show that this is more than satisfied by our approach under asymptotic analysis. The usefulness of the method is demonstrated through simulated and real data numerical examples.

Chapter information

Source
James O. Berger, T. Tony Cai and Iain M. Johnstone, eds., Borrowing Strength: Theory Powering Applications – A Festschrift for Lawrence D. Brown (Beachwood, Ohio, USA: Institute of Mathematical Statistics, 2010), 216-233

Dates
First available in Project Euclid: 26 October 2010

Permanent link to this document
https://projecteuclid.org/euclid.imsc/1288099022

Digital Object Identifier
doi:10.1214/10-IMSCOLL615

Mathematical Reviews number (MathSciNet)
MR2798521

Subjects
Primary: 62G08: Nonparametric regression
Secondary: 62G20: Asymptotic properties

Keywords
variable selection local regression adaptive bandwidth local variable significance

Rights
Copyright © 2010, Institute of Mathematical Statistics

Citation

Miller, Hugh; Hall, Peter. Local polynomial regression and variable selection. Borrowing Strength: Theory Powering Applications – A Festschrift for Lawrence D. Brown, 216--233, Institute of Mathematical Statistics, Beachwood, Ohio, USA, 2010. doi:10.1214/10-IMSCOLL615. https://projecteuclid.org/euclid.imsc/1288099022


Export citation

References

  • [1] Bertin, K. and Lecué, G. (2008). Selection of variables and dimension reduction in high-dimensional non-parametric regression. Electron. J. Statist. 2 1224–1241.
  • [2] Bickel, P. J., Ritov, Y. and Tsybakov, A. B. (2009). Simultaneous analysis of lasso and Dantzig selector. Ann. Statist. 37 1705–1732.
  • [3] Candes, E. and Tao, T. (2007). The Dantzig selector: Statistical estimation when p is much larger than n. Ann. Statist. 35 2313–2351.
  • [4] Fan, J. and Gijbels, I. (1995). Data-driven bandwidth selection in local polynomial fitting: Variable bandwidth and spatial adaptation. J. Roy. Statist. Soc. Ser. B 57 371–394.
  • [5] Hall, P., Racine, J. and Li, Qi (2004). Cross-validation and the estimation of conditional probability densities. J. Amer. Statist. Assoc. 99 1015–1026.
  • [6] Hastie, T., Tibshirani, R. and Friedman, J. (2001). The Elements of Statistical Learning. Springer, New York.
  • [7] Lafferty, J. and Wasserman, L. (2008). Rodeo: sparse, greedy nonparametric regression. J. Amer. Statist. Assoc. 36 28–63.
  • [8] Lin, Y. and Zhang, H. H. (2006). Component selection and smoothing in smoothing spline analysis of variance models. Ann. Statist. 34 2272–2297.
  • [9] Loader, C. (1999). Local Regression and Likelihood. Springer, New York.
  • [10] Masry, E. (1996). Multivariate local polynomial regression for time series: Uniform strong consistency and rates. J. Time Series Anal. 17 571–600.
  • [11] Meinshausen, N. and Yu, B. (2009). Lasso-type recovery of sparse representations for high-dimensional data. Ann. Statist. 37 246–270.
  • [12] Meinshausen, N., Rocha, G. and Yu, B. (2007). A tale of three cousins: Lasso, L2 Boosting and Dantzig. Discussion of Candes and Tao (2007). Ann. Statist. 35 2373–2384.
  • [13] Miller, H. and Hall, P. (2010). Local polynomial regression and variable selection (long version). Manuscript.
  • [14] Nadaraya, E. A. (1964). On estimating regression. Theory Probab. Appl. 9 141–142.
  • [15] Ruppert, D. and Wand, M. P. (1994). Multivariate locally weighted least squares regression. Ann. Statist. 22 1346–1370.
  • [16] Simonoff, J. S. (1996). Smoothing Methods in Statistics. Springer, New York.
  • [17] Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. J. Roy. Statist. Soc. Ser. B 58 267–288.
  • [18] Tropp, J. A. (2004). Greed is good: Algorithmic results for sparse approximation. IEEE Trans. Inform. Theory 50 2231–2242.
  • [19] Wand, M. P. and Jones, M. C. (1995). Kernel Smoothing. Chapman and Hall, London.
  • [20] Wasserman, L. and Roeder, K. (2009). High-dimensional variable selection. Ann. Statist. 37 2178–2201.
  • [21] Watson, G. S. (1964). Smooth regression analysis. Sankhyā Ser. A 26 359–372.
  • [22] Yuan, M. and Lin, Yi (2006). Model selection and estimation in regression with grouped variables. J. R. Stat. Soc. Ser. B Stat. Methodol. 68 49–67.
  • [23] Zhao, P. and Yu, B. (2007). Stagewise lasso. J. Mach. Learn. Res. 8 2701–2726.
  • [24] Zou, H. (2006). The adaptive lasso and its oracle properties. J. Amer. Statist. Assoc. 101 1418–1429.