We propose a thresholding least-squares method of inference for high-dimensional regression models when the number of parameters, $p$, tends to infinity with the sample size, $n$. Extending the asymptotic behavior of the F-test in high dimensions, we establish the oracle property of the thresholding least-squares estimator when $p=o(n)$. We propose two automatic selection procedures for the thresholding parameter using Scheffé and Bonferroni methods. We show that, under additional regularity conditions, the results continue to hold even if $p=\exp(o(n))$. Lastly, we show that, if properly centered, the residual-bootstrap estimator of the distribution of thresholding least-squares estimator is consistent, while a naive bootstrap estimator is inconsistent. In an intensive simulation study, we assess the finite sample properties of the proposed methods for various sample sizes and model parameters. The analysis of a real world data set illustrates an application of the methods in practice.
Electron. J. Statist.
10(2):
2124-2156
(2016).
DOI: 10.1214/16-EJS1160
Bickel, P. J. and Freedman, D. A. (1983). Bootstrapping regression models with many parameters. In, A Festschrift for Erich L. Lehmann in Honour of his Sixty-fifth Birtday (P. J. Bickel, K. A. Doksum and J. L. Hodges, eds.) 28–48. Wadsworth, Belmont, CA.Bickel, P. J. and Freedman, D. A. (1983). Bootstrapping regression models with many parameters. In, A Festschrift for Erich L. Lehmann in Honour of his Sixty-fifth Birtday (P. J. Bickel, K. A. Doksum and J. L. Hodges, eds.) 28–48. Wadsworth, Belmont, CA.
Chatterjee, A., Gupta, S. and Lahiri, S. N. (2015). On the residual empirical process based on the ALASSO in high dimensions and its functional oracle property., Journal of Econometrics 186 317–324.Chatterjee, A., Gupta, S. and Lahiri, S. N. (2015). On the residual empirical process based on the ALASSO in high dimensions and its functional oracle property., Journal of Econometrics 186 317–324.
Chatterjee, A. and Lahiri, S. N. (2013). Rates of convergence of the adaptive Lasso estimators to the oracle distribution and higher order refinements by the bootstrap., The Annals of Statistics 41 1232-1259.Chatterjee, A. and Lahiri, S. N. (2013). Rates of convergence of the adaptive Lasso estimators to the oracle distribution and higher order refinements by the bootstrap., The Annals of Statistics 41 1232-1259.
Donoho, D. L., Johnstone, I. M., Kerkyacharian, G. and Picard, D. (1996). Density estimation by wavelet thresholding., The Annals of Statistics 24 508-539.Donoho, D. L., Johnstone, I. M., Kerkyacharian, G. and Picard, D. (1996). Density estimation by wavelet thresholding., The Annals of Statistics 24 508-539.
El Ghaoui, L., Viallon, V. and Rabbani, T. (2012). Safe feature elimination for the lasso and sparse supervised learning problems., Pacific Journal of Optimization 8(4) 667–698.El Ghaoui, L., Viallon, V. and Rabbani, T. (2012). Safe feature elimination for the lasso and sparse supervised learning problems., Pacific Journal of Optimization 8(4) 667–698.
Fan, J. and Li, R. (2001). Variable selection via nonconcave penalized likelihood and its oracle properties., Journal of the American Statistical Association 96 1348–1360.Fan, J. and Li, R. (2001). Variable selection via nonconcave penalized likelihood and its oracle properties., Journal of the American Statistical Association 96 1348–1360.
Fan, J. and Lv, J. (2008). Sure independence screening for ultrahigh dimensional feature space., Journal of the Royal Statistical Society. Series B 96 1348–1360.Fan, J. and Lv, J. (2008). Sure independence screening for ultrahigh dimensional feature space., Journal of the Royal Statistical Society. Series B 96 1348–1360.
Friedman, J., Hastie, T. and Tibshirani, R. (2010). Regularization paths for generalized linear models via coordinate descent., Journal of Statistical Software 33 1-22.Friedman, J., Hastie, T. and Tibshirani, R. (2010). Regularization paths for generalized linear models via coordinate descent., Journal of Statistical Software 33 1-22.
Huang, J., Horowitz, J. L. and Ma, S. (2008). Asymptotic properties of Bridge estimators in sparse high-dimensional regression models., Annals of Statistics 36 587–613.Huang, J., Horowitz, J. L. and Ma, S. (2008). Asymptotic properties of Bridge estimators in sparse high-dimensional regression models., Annals of Statistics 36 587–613.
Knight, K. and Fu, W. (2000). Asymptotics for LASSO-type estimators., The Annals of Statistics 5 1356–1378. MR1805787 10.1214/aos/1015957397 euclid.aos/1015957397
Knight, K. and Fu, W. (2000). Asymptotics for LASSO-type estimators., The Annals of Statistics 5 1356–1378. MR1805787 10.1214/aos/1015957397 euclid.aos/1015957397
Mammen, E. (1989). Asymptotics with increasing dimension for robust regression with applications to the bootstrap., The Annals of Statistics 17 382–400.Mammen, E. (1989). Asymptotics with increasing dimension for robust regression with applications to the bootstrap., The Annals of Statistics 17 382–400.
Portnoy, S. (1984). Asymptotic behavior of M-estimators of $p$ regression parameters when $p^2/n$ is large; I. Consistency., The Annals of Statistics 12 1298-1309.Portnoy, S. (1984). Asymptotic behavior of M-estimators of $p$ regression parameters when $p^2/n$ is large; I. Consistency., The Annals of Statistics 12 1298-1309.
Portnoy, S. (1985). Asymptotic behavior of M-estimators of $p$ regression parameters when $p^2/n$ is large; II. Normal approximation., The Annals of Statistics 13 1403-1417.Portnoy, S. (1985). Asymptotic behavior of M-estimators of $p$ regression parameters when $p^2/n$ is large; II. Normal approximation., The Annals of Statistics 13 1403-1417.
van de Geer, S., Bühlmann, P. and Zhou, S. (2011). The adaptive and the thresholded Lasso for potentially misspecified models (and a lower bound for the Lasso)., Electronic Journal of Statistics 5 688–749.van de Geer, S., Bühlmann, P. and Zhou, S. (2011). The adaptive and the thresholded Lasso for potentially misspecified models (and a lower bound for the Lasso)., Electronic Journal of Statistics 5 688–749.
Wang, X. and Leng, C. (2016). High dimensional ordinary least squares projection for screening variables., Journal of the Royal Statistical Society. Series B. 78 589–611.Wang, X. and Leng, C. (2016). High dimensional ordinary least squares projection for screening variables., Journal of the Royal Statistical Society. Series B. 78 589–611.
Zelen, M. and Severo, N. (1972). Probability Functions. In, Handbook of Mathematical Functions with Formula, Graphs, and Mathematical Tables, (M. Abramowitz and I. Stegun, eds.). Applied Mathematics Series 55 925–997. National Bureau of Standards, Washington.Zelen, M. and Severo, N. (1972). Probability Functions. In, Handbook of Mathematical Functions with Formula, Graphs, and Mathematical Tables, (M. Abramowitz and I. Stegun, eds.). Applied Mathematics Series 55 925–997. National Bureau of Standards, Washington.