The paper focuses on the problem of model selection in linear Gaussian regression with unknown possibly inhomogeneous noise. For a given family of linear estimators $\{\widetilde{\boldsymbol{{\theta}}}_{m},m\in\mathscr{M}\}$, ordered by their variance, we offer a new “smallest accepted” approach motivated by Lepski’s device and the multiple testing idea. The procedure selects the smallest model which satisfies the acceptance rule based on comparison with all larger models. The method is completely data-driven and does not use any prior information about the variance structure of the noise: its parameters are adjusted to the underlying possibly heterogeneous noise by the so-called “propagation condition” using a wild bootstrap method. The validity of the bootstrap calibration is proved for finite samples with an explicit error bound. We provide a comprehensive theoretical study of the method, describe in details the set of possible values of the selected model $\widehat{m}\in\mathscr{M}$ and establish some oracle error bounds for the corresponding estimator $\widehat{\boldsymbol{{\theta}}}=\widetilde{\boldsymbol{{\theta}}}_{\widehat{m}}$.
Ann. Statist.
47(3):
1351-1380
(June 2019).
DOI: 10.1214/18-AOS1717
Arlot, S. (2009). Model selection by resampling penalization. Electron. J. Stat. 3 557–624. 1326.62097 10.1214/08-EJS196Arlot, S. (2009). Model selection by resampling penalization. Electron. J. Stat. 3 557–624. 1326.62097 10.1214/08-EJS196
Baraud, Y., Huet, S. and Laurent, B. (2003). Adaptive tests of linear hypotheses by model selection. Ann. Statist. 31 225–251. 1018.62037 10.1214/aos/1046294463 euclid.aos/1046294463Baraud, Y., Huet, S. and Laurent, B. (2003). Adaptive tests of linear hypotheses by model selection. Ann. Statist. 31 225–251. 1018.62037 10.1214/aos/1046294463 euclid.aos/1046294463
Barron, A., Birgé, L. and Massart, P. (1999). Risk bounds for model selection via penalization. Probab. Theory Related Fields 113 301–413. 0946.62036 10.1007/s004400050210Barron, A., Birgé, L. and Massart, P. (1999). Risk bounds for model selection via penalization. Probab. Theory Related Fields 113 301–413. 0946.62036 10.1007/s004400050210
Birgé, L. (2001). An alternative point of view on Lepski’s method. In State of the Art in Probability and Statistics (Leiden, 1999). Institute of Mathematical Statistics Lecture Notes—Monograph Series 36 113–133. IMS, Beachwood, OH. MR1836557Birgé, L. (2001). An alternative point of view on Lepski’s method. In State of the Art in Probability and Statistics (Leiden, 1999). Institute of Mathematical Statistics Lecture Notes—Monograph Series 36 113–133. IMS, Beachwood, OH. MR1836557
Birgé, L. and Massart, P. (2007). Minimal penalties for Gaussian model selection. Probab. Theory Related Fields 138 33–73. 1112.62082 10.1007/s00440-006-0011-8Birgé, L. and Massart, P. (2007). Minimal penalties for Gaussian model selection. Probab. Theory Related Fields 138 33–73. 1112.62082 10.1007/s00440-006-0011-8
Cavalier, L. and Golubev, Y. (2006). Risk hull method and regularization by projections of ill-posed inverse problems. Ann. Statist. 34 1653–1677. 1246.62082 10.1214/009053606000000542 euclid.aos/1162567628Cavalier, L. and Golubev, Y. (2006). Risk hull method and regularization by projections of ill-posed inverse problems. Ann. Statist. 34 1653–1677. 1246.62082 10.1214/009053606000000542 euclid.aos/1162567628
Chernozhukov, V., Chetverikov, D. and Kato, K. (2014). Anti-concentration and honest, adaptive confidence bands. Ann. Statist. 42 1787–1818. MR3262468 1305.62161 10.1214/14-AOS1235 euclid.aos/1410440625Chernozhukov, V., Chetverikov, D. and Kato, K. (2014). Anti-concentration and honest, adaptive confidence bands. Ann. Statist. 42 1787–1818. MR3262468 1305.62161 10.1214/14-AOS1235 euclid.aos/1410440625
Dalalyan, A. S. and Salmon, J. (2012). Sharp oracle inequalities for aggregation of affine estimators. Ann. Statist. 40 2327–2355. MR3059085 1257.62038 10.1214/12-AOS1038 euclid.aos/1358951384Dalalyan, A. S. and Salmon, J. (2012). Sharp oracle inequalities for aggregation of affine estimators. Ann. Statist. 40 2327–2355. MR3059085 1257.62038 10.1214/12-AOS1038 euclid.aos/1358951384
Gach, F., Nickl, R. and Spokoiny, V. (2013). Spatially adaptive density estimation by localised Haar projections. Ann. Inst. Henri Poincaré Probab. Stat. 49 900–914. 1355.62009 10.1214/12-AIHP485 euclid.aihp/1372772649Gach, F., Nickl, R. and Spokoiny, V. (2013). Spatially adaptive density estimation by localised Haar projections. Ann. Inst. Henri Poincaré Probab. Stat. 49 900–914. 1355.62009 10.1214/12-AIHP485 euclid.aihp/1372772649
Giné, E. and Nickl, R. (2010). Confidence bands in density estimation. Ann. Statist. 38 1122–1170. 1183.62062 10.1214/09-AOS738 euclid.aos/1266586625Giné, E. and Nickl, R. (2010). Confidence bands in density estimation. Ann. Statist. 38 1122–1170. 1183.62062 10.1214/09-AOS738 euclid.aos/1266586625
Goeman, J. J. and Solari, A. (2010). The sequential rejection principle of familywise error control. Ann. Statist. 38 3782–3810. 1204.62140 10.1214/10-AOS829 euclid.aos/1291126973Goeman, J. J. and Solari, A. (2010). The sequential rejection principle of familywise error control. Ann. Statist. 38 3782–3810. 1204.62140 10.1214/10-AOS829 euclid.aos/1291126973
Goldenshluger, A. (2009). A universal procedure for aggregating estimators. Ann. Statist. 37 542–568. 1155.62018 10.1214/00-AOS576 euclid.aos/1232115945Goldenshluger, A. (2009). A universal procedure for aggregating estimators. Ann. Statist. 37 542–568. 1155.62018 10.1214/00-AOS576 euclid.aos/1232115945
Härdle, W. and Mammen, E. (1993). Comparing nonparametric versus parametric regression fits. Ann. Statist. 21 1926–1947. MR1245774 0795.62036 10.1214/aos/1176349403 euclid.aos/1176349403Härdle, W. and Mammen, E. (1993). Comparing nonparametric versus parametric regression fits. Ann. Statist. 21 1926–1947. MR1245774 0795.62036 10.1214/aos/1176349403 euclid.aos/1176349403
Kneip, A. (1994). Ordered linear smoothers. Ann. Statist. 22 835–866. 0815.62022 10.1214/aos/1176325498 euclid.aos/1176325498Kneip, A. (1994). Ordered linear smoothers. Ann. Statist. 22 835–866. 0815.62022 10.1214/aos/1176325498 euclid.aos/1176325498
Lepski, O. V., Mammen, E. and Spokoiny, V. G. (1997). Optimal spatial adaptation to inhomogeneous smoothness: An approach based on kernel estimates with variable bandwidth selectors. Ann. Statist. 25 929–947. 0885.62044 10.1214/aos/1069362731 euclid.aos/1069362731Lepski, O. V., Mammen, E. and Spokoiny, V. G. (1997). Optimal spatial adaptation to inhomogeneous smoothness: An approach based on kernel estimates with variable bandwidth selectors. Ann. Statist. 25 929–947. 0885.62044 10.1214/aos/1069362731 euclid.aos/1069362731
Lepski, O. V. and Spokoiny, V. G. (1997). Optimal pointwise adaptive methods in nonparametric estimation. Ann. Statist. 25 2512–2546. 0894.62041 10.1214/aos/1030741083 euclid.aos/1030741083Lepski, O. V. and Spokoiny, V. G. (1997). Optimal pointwise adaptive methods in nonparametric estimation. Ann. Statist. 25 2512–2546. 0894.62041 10.1214/aos/1030741083 euclid.aos/1030741083
Lepskiĭ, O. V. (1991). Asymptotically minimax adaptive estimation. I. Upper bounds. Optimally adaptive estimates. Teor. Veroyatn. Primen. 36 645–659. 0738.62045Lepskiĭ, O. V. (1991). Asymptotically minimax adaptive estimation. I. Upper bounds. Optimally adaptive estimates. Teor. Veroyatn. Primen. 36 645–659. 0738.62045
Lepskiĭ, O. V. (1992). Asymptotically minimax adaptive estimation. II. Schemes without optimal adaptation. Adaptive estimates. Teor. Veroyatn. Primen. 37 468–481.Lepskiĭ, O. V. (1992). Asymptotically minimax adaptive estimation. II. Schemes without optimal adaptation. Adaptive estimates. Teor. Veroyatn. Primen. 37 468–481.
Mammen, E. (1993). Bootstrap and wild bootstrap for high-dimensional linear models. Ann. Statist. 21 255–285. 0771.62032 10.1214/aos/1176349025 euclid.aos/1176349025Mammen, E. (1993). Bootstrap and wild bootstrap for high-dimensional linear models. Ann. Statist. 21 255–285. 0771.62032 10.1214/aos/1176349025 euclid.aos/1176349025
Marcus, R., Peritz, E. and Gabriel, K. R. (1976). On closed testing procedures with special reference to ordered analysis of variance. Biometrika 63 655–660. 0353.62037 10.1093/biomet/63.3.655Marcus, R., Peritz, E. and Gabriel, K. R. (1976). On closed testing procedures with special reference to ordered analysis of variance. Biometrika 63 655–660. 0353.62037 10.1093/biomet/63.3.655
Massart, P. (2007). Concentration Inequalities and Model Selection. Lecture Notes in Math. 1896. Springer, Berlin. 1170.60006Massart, P. (2007). Concentration Inequalities and Model Selection. Lecture Notes in Math. 1896. Springer, Berlin. 1170.60006
Romano, J. P. and Wolf, M. (2005). Stepwise multiple testing as formalized data snooping. Econometrica 73 1237–1282. 1153.62310 10.1111/j.1468-0262.2005.00615.xRomano, J. P. and Wolf, M. (2005). Stepwise multiple testing as formalized data snooping. Econometrica 73 1237–1282. 1153.62310 10.1111/j.1468-0262.2005.00615.x
Spokoiny, V. G. (1996). Adaptive hypothesis testing using wavelets. Ann. Statist. 24 2477–2498. MR1425962 0898.62056 10.1214/aos/1032181163 euclid.aos/1032181163Spokoiny, V. G. (1996). Adaptive hypothesis testing using wavelets. Ann. Statist. 24 2477–2498. MR1425962 0898.62056 10.1214/aos/1032181163 euclid.aos/1032181163
Spokoiny, V. (2012). Parametric estimation. Finite sample theory. Ann. Statist. 40 2877–2909. 1296.62051 10.1214/12-AOS1054 euclid.aos/1360332187Spokoiny, V. (2012). Parametric estimation. Finite sample theory. Ann. Statist. 40 2877–2909. 1296.62051 10.1214/12-AOS1054 euclid.aos/1360332187
Spokoiny, V. and Vial, C. (2009). Parameter tuning in pointwise adaptation using a propagation approach. Ann. Statist. 37 2783–2807. 1173.62028 10.1214/08-AOS607 euclid.aos/1247836669Spokoiny, V. and Vial, C. (2009). Parameter tuning in pointwise adaptation using a propagation approach. Ann. Statist. 37 2783–2807. 1173.62028 10.1214/08-AOS607 euclid.aos/1247836669
Spokoiny, V., Wang, W. and Härdle, W. K. (2013). Local quantile regression. J. Statist. Plann. Inference 143 1109–1129. 1279.62098 10.1016/j.jspi.2013.03.008Spokoiny, V., Wang, W. and Härdle, W. K. (2013). Local quantile regression. J. Statist. Plann. Inference 143 1109–1129. 1279.62098 10.1016/j.jspi.2013.03.008
Spokoiny, V. and Willrich, N. (2019). Supplement to “Bootstrap tuning in Gaussian ordered model selection.” DOI:10.1214/18-AOS1717SUPP.Spokoiny, V. and Willrich, N. (2019). Supplement to “Bootstrap tuning in Gaussian ordered model selection.” DOI:10.1214/18-AOS1717SUPP.
Spokoiny, V. and Zhilova, M. (2015). Bootstrap confidence sets under model misspecification. Ann. Statist. 43 2653–2675. 1327.62179 10.1214/15-AOS1355 euclid.aos/1444222088Spokoiny, V. and Zhilova, M. (2015). Bootstrap confidence sets under model misspecification. Ann. Statist. 43 2653–2675. 1327.62179 10.1214/15-AOS1355 euclid.aos/1444222088
Wu, C.-F. J. (1986). Jackknife, bootstrap and other resampling methods in regression analysis. Ann. Statist. 14 1261–1350. 0618.62072 10.1214/aos/1176350142 euclid.aos/1176350142Wu, C.-F. J. (1986). Jackknife, bootstrap and other resampling methods in regression analysis. Ann. Statist. 14 1261–1350. 0618.62072 10.1214/aos/1176350142 euclid.aos/1176350142