Institute of Mathematical Statistics Collections

Local polynomial regression and variable selection

Hugh Miller and Peter Hall

Full-text: Open access


We propose a method for incorporating variable selection into local polynomial regression. This can improve the accuracy of the regression by extending the bandwidth in directions corresponding to those variables judged to be unimportant. It also increases our understanding of the dataset by highlighting areas where these variables are redundant. The approach has the potential to effect complete variable removal as well as perform partial removal when a variable redundancy applies only to particular regions of the data. We define a nonparametric oracle property and show that this is more than satisfied by our approach under asymptotic analysis. The usefulness of the method is demonstrated through simulated and real data numerical examples.

Chapter information

James O. Berger, T. Tony Cai and Iain M. Johnstone, eds., Borrowing Strength: Theory Powering Applications – A Festschrift for Lawrence D. Brown (Beachwood, Ohio, USA: Institute of Mathematical Statistics, 2010), 216-233

First available in Project Euclid: 26 October 2010

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Primary: 62G08: Nonparametric regression
Secondary: 62G20: Asymptotic properties

variable selection local regression adaptive bandwidth local variable significance

Copyright © 2010, Institute of Mathematical Statistics


Miller, Hugh; Hall, Peter. Local polynomial regression and variable selection. Borrowing Strength: Theory Powering Applications – A Festschrift for Lawrence D. Brown, 216--233, Institute of Mathematical Statistics, Beachwood, Ohio, USA, 2010. doi:10.1214/10-IMSCOLL615.

Export citation


  • [1] Bertin, K. and Lecué, G. (2008). Selection of variables and dimension reduction in high-dimensional non-parametric regression. Electron. J. Statist. 2 1224–1241.
  • [2] Bickel, P. J., Ritov, Y. and Tsybakov, A. B. (2009). Simultaneous analysis of lasso and Dantzig selector. Ann. Statist. 37 1705–1732.
  • [3] Candes, E. and Tao, T. (2007). The Dantzig selector: Statistical estimation when p is much larger than n. Ann. Statist. 35 2313–2351.
  • [4] Fan, J. and Gijbels, I. (1995). Data-driven bandwidth selection in local polynomial fitting: Variable bandwidth and spatial adaptation. J. Roy. Statist. Soc. Ser. B 57 371–394.
  • [5] Hall, P., Racine, J. and Li, Qi (2004). Cross-validation and the estimation of conditional probability densities. J. Amer. Statist. Assoc. 99 1015–1026.
  • [6] Hastie, T., Tibshirani, R. and Friedman, J. (2001). The Elements of Statistical Learning. Springer, New York.
  • [7] Lafferty, J. and Wasserman, L. (2008). Rodeo: sparse, greedy nonparametric regression. J. Amer. Statist. Assoc. 36 28–63.
  • [8] Lin, Y. and Zhang, H. H. (2006). Component selection and smoothing in smoothing spline analysis of variance models. Ann. Statist. 34 2272–2297.
  • [9] Loader, C. (1999). Local Regression and Likelihood. Springer, New York.
  • [10] Masry, E. (1996). Multivariate local polynomial regression for time series: Uniform strong consistency and rates. J. Time Series Anal. 17 571–600.
  • [11] Meinshausen, N. and Yu, B. (2009). Lasso-type recovery of sparse representations for high-dimensional data. Ann. Statist. 37 246–270.
  • [12] Meinshausen, N., Rocha, G. and Yu, B. (2007). A tale of three cousins: Lasso, L2 Boosting and Dantzig. Discussion of Candes and Tao (2007). Ann. Statist. 35 2373–2384.
  • [13] Miller, H. and Hall, P. (2010). Local polynomial regression and variable selection (long version). Manuscript.
  • [14] Nadaraya, E. A. (1964). On estimating regression. Theory Probab. Appl. 9 141–142.
  • [15] Ruppert, D. and Wand, M. P. (1994). Multivariate locally weighted least squares regression. Ann. Statist. 22 1346–1370.
  • [16] Simonoff, J. S. (1996). Smoothing Methods in Statistics. Springer, New York.
  • [17] Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. J. Roy. Statist. Soc. Ser. B 58 267–288.
  • [18] Tropp, J. A. (2004). Greed is good: Algorithmic results for sparse approximation. IEEE Trans. Inform. Theory 50 2231–2242.
  • [19] Wand, M. P. and Jones, M. C. (1995). Kernel Smoothing. Chapman and Hall, London.
  • [20] Wasserman, L. and Roeder, K. (2009). High-dimensional variable selection. Ann. Statist. 37 2178–2201.
  • [21] Watson, G. S. (1964). Smooth regression analysis. Sankhyā Ser. A 26 359–372.
  • [22] Yuan, M. and Lin, Yi (2006). Model selection and estimation in regression with grouped variables. J. R. Stat. Soc. Ser. B Stat. Methodol. 68 49–67.
  • [23] Zhao, P. and Yu, B. (2007). Stagewise lasso. J. Mach. Learn. Res. 8 2701–2726.
  • [24] Zou, H. (2006). The adaptive lasso and its oracle properties. J. Amer. Statist. Assoc. 101 1418–1429.