Open Access
June 2019 Bootstrap tuning in Gaussian ordered model selection
Vladimir Spokoiny, Niklas Willrich
Ann. Statist. 47(3): 1351-1380 (June 2019). DOI: 10.1214/18-AOS1717
Abstract

The paper focuses on the problem of model selection in linear Gaussian regression with unknown possibly inhomogeneous noise. For a given family of linear estimators $\{\widetilde{\boldsymbol{{\theta}}}_{m},m\in\mathscr{M}\}$, ordered by their variance, we offer a new “smallest accepted” approach motivated by Lepski’s device and the multiple testing idea. The procedure selects the smallest model which satisfies the acceptance rule based on comparison with all larger models. The method is completely data-driven and does not use any prior information about the variance structure of the noise: its parameters are adjusted to the underlying possibly heterogeneous noise by the so-called “propagation condition” using a wild bootstrap method. The validity of the bootstrap calibration is proved for finite samples with an explicit error bound. We provide a comprehensive theoretical study of the method, describe in details the set of possible values of the selected model $\widehat{m}\in\mathscr{M}$ and establish some oracle error bounds for the corresponding estimator $\widehat{\boldsymbol{{\theta}}}=\widetilde{\boldsymbol{{\theta}}}_{\widehat{m}}$.

References

1.

Arlot, S. (2009). Model selection by resampling penalization. Electron. J. Stat. 3 557–624. 1326.62097 10.1214/08-EJS196Arlot, S. (2009). Model selection by resampling penalization. Electron. J. Stat. 3 557–624. 1326.62097 10.1214/08-EJS196

2.

Baraud, Y., Huet, S. and Laurent, B. (2003). Adaptive tests of linear hypotheses by model selection. Ann. Statist. 31 225–251. 1018.62037 10.1214/aos/1046294463 euclid.aos/1046294463Baraud, Y., Huet, S. and Laurent, B. (2003). Adaptive tests of linear hypotheses by model selection. Ann. Statist. 31 225–251. 1018.62037 10.1214/aos/1046294463 euclid.aos/1046294463

3.

Barron, A., Birgé, L. and Massart, P. (1999). Risk bounds for model selection via penalization. Probab. Theory Related Fields 113 301–413. 0946.62036 10.1007/s004400050210Barron, A., Birgé, L. and Massart, P. (1999). Risk bounds for model selection via penalization. Probab. Theory Related Fields 113 301–413. 0946.62036 10.1007/s004400050210

4.

Beran, R. (1986). Discussion: Jackknife, bootstrap and other resampling methods in regression analysis. Ann. Statist. 14 1295–1298.Beran, R. (1986). Discussion: Jackknife, bootstrap and other resampling methods in regression analysis. Ann. Statist. 14 1295–1298.

5.

Birgé, L. (2001). An alternative point of view on Lepski’s method. In State of the Art in Probability and Statistics (Leiden, 1999). Institute of Mathematical Statistics Lecture Notes—Monograph Series 36 113–133. IMS, Beachwood, OH. MR1836557Birgé, L. (2001). An alternative point of view on Lepski’s method. In State of the Art in Probability and Statistics (Leiden, 1999). Institute of Mathematical Statistics Lecture Notes—Monograph Series 36 113–133. IMS, Beachwood, OH. MR1836557

6.

Birgé, L. and Massart, P. (2007). Minimal penalties for Gaussian model selection. Probab. Theory Related Fields 138 33–73. 1112.62082 10.1007/s00440-006-0011-8Birgé, L. and Massart, P. (2007). Minimal penalties for Gaussian model selection. Probab. Theory Related Fields 138 33–73. 1112.62082 10.1007/s00440-006-0011-8

7.

Cavalier, L. and Golubev, Y. (2006). Risk hull method and regularization by projections of ill-posed inverse problems. Ann. Statist. 34 1653–1677. 1246.62082 10.1214/009053606000000542 euclid.aos/1162567628Cavalier, L. and Golubev, Y. (2006). Risk hull method and regularization by projections of ill-posed inverse problems. Ann. Statist. 34 1653–1677. 1246.62082 10.1214/009053606000000542 euclid.aos/1162567628

8.

Chernozhukov, V., Chetverikov, D. and Kato, K. (2014). Anti-concentration and honest, adaptive confidence bands. Ann. Statist. 42 1787–1818. MR3262468 1305.62161 10.1214/14-AOS1235 euclid.aos/1410440625Chernozhukov, V., Chetverikov, D. and Kato, K. (2014). Anti-concentration and honest, adaptive confidence bands. Ann. Statist. 42 1787–1818. MR3262468 1305.62161 10.1214/14-AOS1235 euclid.aos/1410440625

9.

Dalalyan, A. S. and Salmon, J. (2012). Sharp oracle inequalities for aggregation of affine estimators. Ann. Statist. 40 2327–2355. MR3059085 1257.62038 10.1214/12-AOS1038 euclid.aos/1358951384Dalalyan, A. S. and Salmon, J. (2012). Sharp oracle inequalities for aggregation of affine estimators. Ann. Statist. 40 2327–2355. MR3059085 1257.62038 10.1214/12-AOS1038 euclid.aos/1358951384

10.

Gach, F., Nickl, R. and Spokoiny, V. (2013). Spatially adaptive density estimation by localised Haar projections. Ann. Inst. Henri Poincaré Probab. Stat. 49 900–914. 1355.62009 10.1214/12-AIHP485 euclid.aihp/1372772649Gach, F., Nickl, R. and Spokoiny, V. (2013). Spatially adaptive density estimation by localised Haar projections. Ann. Inst. Henri Poincaré Probab. Stat. 49 900–914. 1355.62009 10.1214/12-AIHP485 euclid.aihp/1372772649

11.

Giné, E. and Nickl, R. (2010). Confidence bands in density estimation. Ann. Statist. 38 1122–1170. 1183.62062 10.1214/09-AOS738 euclid.aos/1266586625Giné, E. and Nickl, R. (2010). Confidence bands in density estimation. Ann. Statist. 38 1122–1170. 1183.62062 10.1214/09-AOS738 euclid.aos/1266586625

12.

Goeman, J. J. and Solari, A. (2010). The sequential rejection principle of familywise error control. Ann. Statist. 38 3782–3810. 1204.62140 10.1214/10-AOS829 euclid.aos/1291126973Goeman, J. J. and Solari, A. (2010). The sequential rejection principle of familywise error control. Ann. Statist. 38 3782–3810. 1204.62140 10.1214/10-AOS829 euclid.aos/1291126973

13.

Goldenshluger, A. (2009). A universal procedure for aggregating estimators. Ann. Statist. 37 542–568. 1155.62018 10.1214/00-AOS576 euclid.aos/1232115945Goldenshluger, A. (2009). A universal procedure for aggregating estimators. Ann. Statist. 37 542–568. 1155.62018 10.1214/00-AOS576 euclid.aos/1232115945

14.

Härdle, W. and Mammen, E. (1993). Comparing nonparametric versus parametric regression fits. Ann. Statist. 21 1926–1947. MR1245774 0795.62036 10.1214/aos/1176349403 euclid.aos/1176349403Härdle, W. and Mammen, E. (1993). Comparing nonparametric versus parametric regression fits. Ann. Statist. 21 1926–1947. MR1245774 0795.62036 10.1214/aos/1176349403 euclid.aos/1176349403

15.

Ibragimov, I. A. and Has’minskiĭ, R. Z. (1981). Statistical Estimation: Asymptotic Theory. Applications of Mathematics 16. Springer, New York.Ibragimov, I. A. and Has’minskiĭ, R. Z. (1981). Statistical Estimation: Asymptotic Theory. Applications of Mathematics 16. Springer, New York.

16.

Kneip, A. (1994). Ordered linear smoothers. Ann. Statist. 22 835–866. 0815.62022 10.1214/aos/1176325498 euclid.aos/1176325498Kneip, A. (1994). Ordered linear smoothers. Ann. Statist. 22 835–866. 0815.62022 10.1214/aos/1176325498 euclid.aos/1176325498

17.

Lepski, O. V., Mammen, E. and Spokoiny, V. G. (1997). Optimal spatial adaptation to inhomogeneous smoothness: An approach based on kernel estimates with variable bandwidth selectors. Ann. Statist. 25 929–947. 0885.62044 10.1214/aos/1069362731 euclid.aos/1069362731Lepski, O. V., Mammen, E. and Spokoiny, V. G. (1997). Optimal spatial adaptation to inhomogeneous smoothness: An approach based on kernel estimates with variable bandwidth selectors. Ann. Statist. 25 929–947. 0885.62044 10.1214/aos/1069362731 euclid.aos/1069362731

18.

Lepski, O. V. and Spokoiny, V. G. (1997). Optimal pointwise adaptive methods in nonparametric estimation. Ann. Statist. 25 2512–2546. 0894.62041 10.1214/aos/1030741083 euclid.aos/1030741083Lepski, O. V. and Spokoiny, V. G. (1997). Optimal pointwise adaptive methods in nonparametric estimation. Ann. Statist. 25 2512–2546. 0894.62041 10.1214/aos/1030741083 euclid.aos/1030741083

19.

Lepskiĭ, O. V. (1990). A problem of adaptive estimation in Gaussian white noise. Teor. Veroyatn. Primen. 35 459–470.Lepskiĭ, O. V. (1990). A problem of adaptive estimation in Gaussian white noise. Teor. Veroyatn. Primen. 35 459–470.

20.

Lepskiĭ, O. V. (1991). Asymptotically minimax adaptive estimation. I. Upper bounds. Optimally adaptive estimates. Teor. Veroyatn. Primen. 36 645–659. 0738.62045Lepskiĭ, O. V. (1991). Asymptotically minimax adaptive estimation. I. Upper bounds. Optimally adaptive estimates. Teor. Veroyatn. Primen. 36 645–659. 0738.62045

21.

Lepskiĭ, O. V. (1992). Asymptotically minimax adaptive estimation. II. Schemes without optimal adaptation. Adaptive estimates. Teor. Veroyatn. Primen. 37 468–481.Lepskiĭ, O. V. (1992). Asymptotically minimax adaptive estimation. II. Schemes without optimal adaptation. Adaptive estimates. Teor. Veroyatn. Primen. 37 468–481.

22.

Mammen, E. (1993). Bootstrap and wild bootstrap for high-dimensional linear models. Ann. Statist. 21 255–285. 0771.62032 10.1214/aos/1176349025 euclid.aos/1176349025Mammen, E. (1993). Bootstrap and wild bootstrap for high-dimensional linear models. Ann. Statist. 21 255–285. 0771.62032 10.1214/aos/1176349025 euclid.aos/1176349025

23.

Marcus, R., Peritz, E. and Gabriel, K. R. (1976). On closed testing procedures with special reference to ordered analysis of variance. Biometrika 63 655–660. 0353.62037 10.1093/biomet/63.3.655Marcus, R., Peritz, E. and Gabriel, K. R. (1976). On closed testing procedures with special reference to ordered analysis of variance. Biometrika 63 655–660. 0353.62037 10.1093/biomet/63.3.655

24.

Massart, P. (2007). Concentration Inequalities and Model Selection. Lecture Notes in Math. 1896. Springer, Berlin. 1170.60006Massart, P. (2007). Concentration Inequalities and Model Selection. Lecture Notes in Math. 1896. Springer, Berlin. 1170.60006

25.

Pinsker, M. S. (1980). Optimal filtration of square-integrable signals in Gaussian noise. Probl. Inf. Transm. 16 52–68.Pinsker, M. S. (1980). Optimal filtration of square-integrable signals in Gaussian noise. Probl. Inf. Transm. 16 52–68.

26.

Romano, J. P. and Wolf, M. (2005). Stepwise multiple testing as formalized data snooping. Econometrica 73 1237–1282. 1153.62310 10.1111/j.1468-0262.2005.00615.xRomano, J. P. and Wolf, M. (2005). Stepwise multiple testing as formalized data snooping. Econometrica 73 1237–1282. 1153.62310 10.1111/j.1468-0262.2005.00615.x

27.

Spokoiny, V. G. (1996). Adaptive hypothesis testing using wavelets. Ann. Statist. 24 2477–2498. MR1425962 0898.62056 10.1214/aos/1032181163 euclid.aos/1032181163Spokoiny, V. G. (1996). Adaptive hypothesis testing using wavelets. Ann. Statist. 24 2477–2498. MR1425962 0898.62056 10.1214/aos/1032181163 euclid.aos/1032181163

28.

Spokoiny, V. (2012). Parametric estimation. Finite sample theory. Ann. Statist. 40 2877–2909. 1296.62051 10.1214/12-AOS1054 euclid.aos/1360332187Spokoiny, V. (2012). Parametric estimation. Finite sample theory. Ann. Statist. 40 2877–2909. 1296.62051 10.1214/12-AOS1054 euclid.aos/1360332187

29.

Spokoiny, V. and Vial, C. (2009). Parameter tuning in pointwise adaptation using a propagation approach. Ann. Statist. 37 2783–2807. 1173.62028 10.1214/08-AOS607 euclid.aos/1247836669Spokoiny, V. and Vial, C. (2009). Parameter tuning in pointwise adaptation using a propagation approach. Ann. Statist. 37 2783–2807. 1173.62028 10.1214/08-AOS607 euclid.aos/1247836669

30.

Spokoiny, V., Wang, W. and Härdle, W. K. (2013). Local quantile regression. J. Statist. Plann. Inference 143 1109–1129. 1279.62098 10.1016/j.jspi.2013.03.008Spokoiny, V., Wang, W. and Härdle, W. K. (2013). Local quantile regression. J. Statist. Plann. Inference 143 1109–1129. 1279.62098 10.1016/j.jspi.2013.03.008

31.

Spokoiny, V. and Willrich, N. (2019). Supplement to “Bootstrap tuning in Gaussian ordered model selection.”  DOI:10.1214/18-AOS1717SUPP.Spokoiny, V. and Willrich, N. (2019). Supplement to “Bootstrap tuning in Gaussian ordered model selection.”  DOI:10.1214/18-AOS1717SUPP.

32.

Spokoiny, V. and Zhilova, M. (2015). Bootstrap confidence sets under model misspecification. Ann. Statist. 43 2653–2675. 1327.62179 10.1214/15-AOS1355 euclid.aos/1444222088Spokoiny, V. and Zhilova, M. (2015). Bootstrap confidence sets under model misspecification. Ann. Statist. 43 2653–2675. 1327.62179 10.1214/15-AOS1355 euclid.aos/1444222088

33.

Wu, C.-F. J. (1986). Jackknife, bootstrap and other resampling methods in regression analysis. Ann. Statist. 14 1261–1350. 0618.62072 10.1214/aos/1176350142 euclid.aos/1176350142Wu, C.-F. J. (1986). Jackknife, bootstrap and other resampling methods in regression analysis. Ann. Statist. 14 1261–1350. 0618.62072 10.1214/aos/1176350142 euclid.aos/1176350142
Copyright © 2019 Institute of Mathematical Statistics
Vladimir Spokoiny and Niklas Willrich "Bootstrap tuning in Gaussian ordered model selection," The Annals of Statistics 47(3), 1351-1380, (June 2019). https://doi.org/10.1214/18-AOS1717
Received: 1 July 2015; Published: June 2019
Vol.47 • No. 3 • June 2019
Back to Top