Thresholding least-squares inference in high-dimensional regression models

Mihai Giurcanu

doi:10.1214/16-EJS1160

2016 Thresholding least-squares inference in high-dimensional regression models

Mihai Giurcanu

Electron. J. Statist. 10(2): 2124-2156 (2016). DOI: 10.1214/16-EJS1160

Abstract

We propose a thresholding least-squares method of inference for high-dimensional regression models when the number of parameters, $p$, tends to infinity with the sample size, $n$. Extending the asymptotic behavior of the F-test in high dimensions, we establish the oracle property of the thresholding least-squares estimator when $p=o(n)$. We propose two automatic selection procedures for the thresholding parameter using Scheffé and Bonferroni methods. We show that, under additional regularity conditions, the results continue to hold even if $p=\exp(o(n))$. Lastly, we show that, if properly centered, the residual-bootstrap estimator of the distribution of thresholding least-squares estimator is consistent, while a naive bootstrap estimator is inconsistent. In an intensive simulation study, we assess the finite sample properties of the proposed methods for various sample sizes and model parameters. The analysis of a real world data set illustrates an application of the methods in practice.

References

1.

Athreya, K. B. and Lahiri, S. N. (2006)., Measure Theory and Probability Theory. Springer, New York.Athreya, K. B. and Lahiri, S. N. (2006)., Measure Theory and Probability Theory. Springer, New York.

2.

Bhatia, R. (1997)., Matrix Analysis. Springer, New York.Bhatia, R. (1997)., Matrix Analysis. Springer, New York.

3.

Bickel, P. J. and Freedman, D. A. (1983). Bootstrapping regression models with many parameters. In, A Festschrift for Erich L. Lehmann in Honour of his Sixty-fifth Birtday (P. J. Bickel, K. A. Doksum and J. L. Hodges, eds.) 28–48. Wadsworth, Belmont, CA.Bickel, P. J. and Freedman, D. A. (1983). Bootstrapping regression models with many parameters. In, A Festschrift for Erich L. Lehmann in Honour of his Sixty-fifth Birtday (P. J. Bickel, K. A. Doksum and J. L. Hodges, eds.) 28–48. Wadsworth, Belmont, CA.

4.

Bickel, P. J. and Levina, E. (2008). Covariance Regularization by thresholding., The Annals of Statistics 36 2577–2604.Bickel, P. J. and Levina, E. (2008). Covariance Regularization by thresholding., The Annals of Statistics 36 2577–2604.

5.

Bühlmann, P. (2006). Boosting for high-dimensional linear models., The Annals of Statistics 34 559–583.Bühlmann, P. (2006). Boosting for high-dimensional linear models., The Annals of Statistics 34 559–583.

6.

Chatterjee, A., Gupta, S. and Lahiri, S. N. (2015). On the residual empirical process based on the ALASSO in high dimensions and its functional oracle property., Journal of Econometrics 186 317–324.Chatterjee, A., Gupta, S. and Lahiri, S. N. (2015). On the residual empirical process based on the ALASSO in high dimensions and its functional oracle property., Journal of Econometrics 186 317–324.

7.

Chatterjee, A. and Lahiri, S. N. (2011). Bootstrapping Lasso estimators., Journal of the American Statistical Association 106 608-625.Chatterjee, A. and Lahiri, S. N. (2011). Bootstrapping Lasso estimators., Journal of the American Statistical Association 106 608-625.

8.

Chatterjee, A. and Lahiri, S. N. (2013). Rates of convergence of the adaptive Lasso estimators to the oracle distribution and higher order refinements by the bootstrap., The Annals of Statistics 41 1232-1259.Chatterjee, A. and Lahiri, S. N. (2013). Rates of convergence of the adaptive Lasso estimators to the oracle distribution and higher order refinements by the bootstrap., The Annals of Statistics 41 1232-1259.

9.

Chow, Y. S. and Teicher, H. (1997)., Probability Theory: independence, interchangeability, martingales. Springer, New York.Chow, Y. S. and Teicher, H. (1997)., Probability Theory: independence, interchangeability, martingales. Springer, New York.

10.

Davison, A. C. and Hinkley, D. V. (1997)., Bootstrap Methods and their Application. Cambridge University Press, Cambridge.Davison, A. C. and Hinkley, D. V. (1997)., Bootstrap Methods and their Application. Cambridge University Press, Cambridge.

11.

Donoho, D. L. and Johnstone, I. M. (1994). Ideal spatial adaptation by wavelet shrinkage., Biometrika 81 425-455.Donoho, D. L. and Johnstone, I. M. (1994). Ideal spatial adaptation by wavelet shrinkage., Biometrika 81 425-455.

12.

Donoho, D. L., Johnstone, I. M., Kerkyacharian, G. and Picard, D. (1996). Density estimation by wavelet thresholding., The Annals of Statistics 24 508-539.Donoho, D. L., Johnstone, I. M., Kerkyacharian, G. and Picard, D. (1996). Density estimation by wavelet thresholding., The Annals of Statistics 24 508-539.

13.

Dudley, R. M. (2002)., Real Analysis and Probability. Cambridge University Press, Cambridge.Dudley, R. M. (2002)., Real Analysis and Probability. Cambridge University Press, Cambridge.

14.

Efron, B. (1979). Bootstrap methods: another look at Jackknife., The Annals of Statistics 7 1–26.Efron, B. (1979). Bootstrap methods: another look at Jackknife., The Annals of Statistics 7 1–26.

15.

El Ghaoui, L., Viallon, V. and Rabbani, T. (2012). Safe feature elimination for the lasso and sparse supervised learning problems., Pacific Journal of Optimization 8(4) 667–698.El Ghaoui, L., Viallon, V. and Rabbani, T. (2012). Safe feature elimination for the lasso and sparse supervised learning problems., Pacific Journal of Optimization 8(4) 667–698.

16.

El Karoui, N. (2008). Operator norm consistent estimation of large-dimensional sparse covariance matrices., The Annals of Statistics 36 2717–2756.El Karoui, N. (2008). Operator norm consistent estimation of large-dimensional sparse covariance matrices., The Annals of Statistics 36 2717–2756.

17.

Fan, J. and Li, R. (2001). Variable selection via nonconcave penalized likelihood and its oracle properties., Journal of the American Statistical Association 96 1348–1360.Fan, J. and Li, R. (2001). Variable selection via nonconcave penalized likelihood and its oracle properties., Journal of the American Statistical Association 96 1348–1360.

18.

Fan, J. and Lv, J. (2008). Sure independence screening for ultrahigh dimensional feature space., Journal of the Royal Statistical Society. Series B 96 1348–1360.Fan, J. and Lv, J. (2008). Sure independence screening for ultrahigh dimensional feature space., Journal of the Royal Statistical Society. Series B 96 1348–1360.

19.

Frank, I. E. and Friedman, J. H. (1993). A statistical view of some chemometrics regression tools (with discussion)., Technometrics 35 109–148.Frank, I. E. and Friedman, J. H. (1993). A statistical view of some chemometrics regression tools (with discussion)., Technometrics 35 109–148.

20.

Freedman, D. A. (1981). Bootstrapping regression models., The Annals of Statistics 9 1218–1228.Freedman, D. A. (1981). Bootstrapping regression models., The Annals of Statistics 9 1218–1228.

21.

Friedman, J., Hastie, T. and Tibshirani, R. (2010). Regularization paths for generalized linear models via coordinate descent., Journal of Statistical Software 33 1-22.Friedman, J., Hastie, T. and Tibshirani, R. (2010). Regularization paths for generalized linear models via coordinate descent., Journal of Statistical Software 33 1-22.

22.

Hall, P. (1992)., The Bootstrap and Edgeworth Expansion. Springer-Verlag, New York.Hall, P. (1992)., The Bootstrap and Edgeworth Expansion. Springer-Verlag, New York.

23.

Hastie, T., Tibshirani, R. and Friedman, J. (2008)., The Elements of Statistical Learning. Springer, New York.Hastie, T., Tibshirani, R. and Friedman, J. (2008)., The Elements of Statistical Learning. Springer, New York.

24.

Huang, J., Horowitz, J. L. and Ma, S. (2008). Asymptotic properties of Bridge estimators in sparse high-dimensional regression models., Annals of Statistics 36 587–613.Huang, J., Horowitz, J. L. and Ma, S. (2008). Asymptotic properties of Bridge estimators in sparse high-dimensional regression models., Annals of Statistics 36 587–613.

25.

Huang, J., Ma, S. and Zhang, C.-H. (2008). Adaptive LASSO for sparse high-dimensional regression models., Statistica Sinica 18 1603–1618.Huang, J., Ma, S. and Zhang, C.-H. (2008). Adaptive LASSO for sparse high-dimensional regression models., Statistica Sinica 18 1603–1618.

26.

Huber, P. (1973). Robust regression: asymptotics, conjectures, and Monte Carlo., The Annals of Statistics 1 799–821.Huber, P. (1973). Robust regression: asymptotics, conjectures, and Monte Carlo., The Annals of Statistics 1 799–821.

27.

Khuri, A. I. (2010)., Linear Model Methodology. Chapman & Hall/CRC, Boca Raton, Florida.Khuri, A. I. (2010)., Linear Model Methodology. Chapman & Hall/CRC, Boca Raton, Florida.

28.

Knight, K. and Fu, W. (2000). Asymptotics for LASSO-type estimators., The Annals of Statistics 5 1356–1378. MR1805787 10.1214/aos/1015957397 euclid.aos/1015957397 Knight, K. and Fu, W. (2000). Asymptotics for LASSO-type estimators., The Annals of Statistics 5 1356–1378. MR1805787 10.1214/aos/1015957397 euclid.aos/1015957397

29.

Mammen, E. (1989). Asymptotics with increasing dimension for robust regression with applications to the bootstrap., The Annals of Statistics 17 382–400.Mammen, E. (1989). Asymptotics with increasing dimension for robust regression with applications to the bootstrap., The Annals of Statistics 17 382–400.

30.

Mammen, E. (1993). Bootstrap and wild bootstrap for high dimensional linear models., The Annals of Statistics 21 255–285.Mammen, E. (1993). Bootstrap and wild bootstrap for high dimensional linear models., The Annals of Statistics 21 255–285.

31.

Portnoy, S. (1984). Asymptotic behavior of M-estimators of $p$ regression parameters when $p^2/n$ is large; I. Consistency., The Annals of Statistics 12 1298-1309.Portnoy, S. (1984). Asymptotic behavior of M-estimators of $p$ regression parameters when $p^2/n$ is large; I. Consistency., The Annals of Statistics 12 1298-1309.

32.

Portnoy, S. (1985). Asymptotic behavior of M-estimators of $p$ regression parameters when $p^2/n$ is large; II. Normal approximation., The Annals of Statistics 13 1403-1417.Portnoy, S. (1985). Asymptotic behavior of M-estimators of $p$ regression parameters when $p^2/n$ is large; II. Normal approximation., The Annals of Statistics 13 1403-1417.

33.

Tibshirani, R. (1996). Regression shrinkage and selection estimator via the Lasso., Journal of the Royal Statistical Society: Series B 58 267–288.Tibshirani, R. (1996). Regression shrinkage and selection estimator via the Lasso., Journal of the Royal Statistical Society: Series B 58 267–288.

34.

van de Geer, S., Bühlmann, P. and Zhou, S. (2011). The adaptive and the thresholded Lasso for potentially misspecified models (and a lower bound for the Lasso)., Electronic Journal of Statistics 5 688–749.van de Geer, S., Bühlmann, P. and Zhou, S. (2011). The adaptive and the thresholded Lasso for potentially misspecified models (and a lower bound for the Lasso)., Electronic Journal of Statistics 5 688–749.

35.

van der Laan, M. and Bryan, J. (2001). Gene expression analysis with the parametric bootstrap., Biostatistics 2 445–461.van der Laan, M. and Bryan, J. (2001). Gene expression analysis with the parametric bootstrap., Biostatistics 2 445–461.

36.

van der Vaart, A. W. (1998)., Asymptotic Statistics. Springer-Verlag, New York.van der Vaart, A. W. (1998)., Asymptotic Statistics. Springer-Verlag, New York.

37.

Wang, X. and Leng, C. (2016). High dimensional ordinary least squares projection for screening variables., Journal of the Royal Statistical Society. Series B. 78 589–611.Wang, X. and Leng, C. (2016). High dimensional ordinary least squares projection for screening variables., Journal of the Royal Statistical Society. Series B. 78 589–611.

38.

Wasserman, L. and Roeder, K. (2009). High-dimensional variable selection., The Annals of Statistics 37 2178–2201.Wasserman, L. and Roeder, K. (2009). High-dimensional variable selection., The Annals of Statistics 37 2178–2201.

39.

Zelen, M. and Severo, N. (1972). Probability Functions. In, Handbook of Mathematical Functions with Formula, Graphs, and Mathematical Tables, (M. Abramowitz and I. Stegun, eds.). Applied Mathematics Series 55 925–997. National Bureau of Standards, Washington.Zelen, M. and Severo, N. (1972). Probability Functions. In, Handbook of Mathematical Functions with Formula, Graphs, and Mathematical Tables, (M. Abramowitz and I. Stegun, eds.). Applied Mathematics Series 55 925–997. National Bureau of Standards, Washington.

40.

Zou, H. (2006). The adaptive lasso and its oracle properties., Journal of the American Statistical Association 101 1418–1429.Zou, H. (2006). The adaptive lasso and its oracle properties., Journal of the American Statistical Association 101 1418–1429.

Citation Download Citation

Mihai Giurcanu "Thresholding least-squares inference in high-dimensional regression models," Electronic Journal of Statistics 10(2), 2124-2156, (2016). https://doi.org/10.1214/16-EJS1160

Received: 1 January 2016; Published: 2016

Access the abstract

JOURNAL ARTICLE
33 PAGES

DOWNLOAD PDF + SAVE TO MY LIBRARY