The Annals of Statistics

A simple bootstrap method for constructing nonparametric confidence bands for functions

Peter Hall and Joel Horowitz

Full-text: Open access


Standard approaches to constructing nonparametric confidence bands for functions are frustrated by the impact of bias, which generally is not estimated consistently when using the bootstrap and conventionally smoothed function estimators. To overcome this problem it is common practice to either undersmooth, so as to reduce the impact of bias, or oversmooth, and thereby introduce an explicit or implicit bias estimator. However, these approaches, and others based on nonstandard smoothing methods, complicate the process of inference, for example, by requiring the choice of new, unconventional smoothing parameters and, in the case of undersmoothing, producing relatively wide bands. In this paper we suggest a new approach, which exploits to our advantage one of the difficulties that, in the past, has prevented an attractive solution to the problem—the fact that the standard bootstrap bias estimator suffers from relatively high-frequency stochastic error. The high frequency, together with a technique based on quantiles, can be exploited to dampen down the stochastic error term, leading to relatively narrow, simple-to-construct confidence bands.

Article information

Ann. Statist. Volume 41, Number 4 (2013), 1892-1921.

First available in Project Euclid: 5 September 2013

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62G07: Density estimation 62G08: Nonparametric regression
Secondary: 62G09: Resampling methods

Bandwidth bias bootstrap confidence interval conservative coverage coverage error kernel methods statistical smoothing


Hall, Peter; Horowitz, Joel. A simple bootstrap method for constructing nonparametric confidence bands for functions. Ann. Statist. 41 (2013), no. 4, 1892--1921. doi:10.1214/13-AOS1137.

Export citation


  • Beran, R. (1987). Prepivoting to reduce level error of confidence sets. Biometrika 74 457–468.
  • Berry, S. M., Carroll, R. J. and Ruppert, D. (2002). Bayesian smoothing and regression splines for measurement error problems. J. Amer. Statist. Assoc. 97 160–169.
  • Bjerve, S., Doksum, K. A. and Yandell, B. S. (1985). Uniform confidence bounds for regression based on a simple moving average. Scand. J. Stat. 12 159–169.
  • Brown, L. D. and Levine, M. (2007). Variance estimation in nonparametric regression via the difference sequence method. Ann. Statist. 35 2219–2232.
  • Buckley, M. J., Eagleson, G. K. and Silverman, B. W. (1988). The estimation of residual variance in nonparametric regression. Biometrika 75 189–199.
  • Cai, T. T., Levine, M. and Wang, L. (2009). Variance function estimation in multivariate nonparametric regression with fixed design. J. Multivariate Anal. 100 126–136.
  • Cai, T. T. and Low, M. G. (2006). Adaptive confidence balls. Ann. Statist. 34 202–228.
  • Chen, S. X. (1996). Empirical likelihood confidence intervals for nonparametric density estimation. Biometrika 83 329–341.
  • Chen, S. X., Härdle, W. and Li, M. (2003). An empirical likelihood goodness-of-fit test for time series. J. R. Stat. Soc. Ser. B Stat. Methodol. 65 663–678.
  • Claeskens, G. and Van Keilegom, I. (2003). Bootstrap confidence bands for regression curves and their derivatives. Ann. Statist. 31 1852–1884.
  • Dette, H., Munk, A. and Wagner, T. (1998). Estimating the variance in nonparametric regression—what is a reasonable choice? J. R. Stat. Soc. Ser. B Stat. Methodol. 60 751–764.
  • Efron, B. and Tibshirani, R. J. (1993). An Introduction to the Bootstrap. Monographs on Statistics and Applied Probability 57. Chapman & Hall, New York.
  • Eubank, R. L. and Speckman, P. L. (1993). Confidence bands in nonparametric regression. J. Amer. Statist. Assoc. 88 1287–1301.
  • Eubank, R. L. and Wang, S. (1994). Confidence regions in non-parametric regression. Scand. J. Stat. 21 147–158.
  • Fan, J. and Yao, Q. (1998). Efficient estimation of conditional variance functions in stochastic regression. Biometrika 85 645–660.
  • Gasser, T., Sroka, L. and Jennen-Steinmetz, C. (1986). Residual variance and residual pattern in nonlinear regression. Biometrika 73 625–633.
  • Genovese, C. R. and Wasserman, L. (2005). Confidence sets for nonparametric wavelet regression. Ann. Statist. 33 698–729.
  • Genovese, C. and Wasserman, L. (2008). Adaptive confidence bands. Ann. Statist. 36 875–905.
  • Giné, E. and Nickl, R. (2010). Confidence bands in density estimation. Ann. Statist. 38 1122–1170.
  • Hall, P. (1986). On the bootstrap and confidence intervals. Ann. Statist. 14 1431–1452.
  • Hall, P. (1992a). Effect of bias estimation on coverage accuracy of bootstrap confidence intervals for a probability density. Ann. Statist. 20 675–694.
  • Hall, P. (1992b). On bootstrap confidence intervals in nonparametric regression. Ann. Statist. 20 695–711.
  • Hall, P. and Horowitz, J. (2013). Supplement to “A simple bootstrap method for constructing nonparametric confidence bands for functions.” DOI:10.1214/13-AOS1137SUPP.
  • Hall, P., Kay, J. W. and Titterington, D. M. (1990). Asymptotically optimal difference-based estimation of variance in nonparametric regression. Biometrika 77 521–528.
  • Hall, P. and Marron, J. S. (1990). On variance estimation in nonparametric regression. Biometrika 77 415–419.
  • Hall, P. and Owen, A. B. (1993). Empirical likelihood confidence bands in density estimation. J. Comput. Graph. Statist. 2 273–289.
  • Hall, P. and Titterington, D. M. (1988). On confidence bands in nonparametric density estimation and regression. J. Multivariate Anal. 27 228–254.
  • Härdle, W. and Bowman, A. W. (1988). Bootstrapping in nonparametric regression: Local adaptive smoothing and confidence bands. J. Amer. Statist. Assoc. 83 102–110.
  • Härdle, W., Huet, S. and Jolivet, E. (1995). Better bootstrap confidence intervals for regression curve estimation. Statistics 26 287–306.
  • Härdle, W. and Marron, J. S. (1991). Bootstrap simultaneous error bars for nonparametric regression. Ann. Statist. 19 778–796.
  • Härdle, W., Huet, S., Mammen, E. and Sperlich, S. (2004). Bootstrap inference in semiparametric generalized additive models. Econometric Theory 20 265–300.
  • Hoffmann, M. and Nickl, R. (2011). On adaptive inference and confidence bands. Ann. Statist. 39 2383–2409.
  • Horowitz, J. L. and Spokoiny, V. G. (2001). An adaptive, rate-optimal test of a parametric mean-regression model against a nonparametric alternative. Econometrica 69 599–631.
  • Komlós, J., Major, P. and Tusnády, G. (1976). An approximation of partial sums of independent RV’s, and the sample DF. II. Z. Wahrsch. Verw. Gebiete 34 33–58.
  • Li, K.-C. (1989). Honest confidence regions for nonparametric regression. Ann. Statist. 17 1001–1008.
  • Loh, W.-Y. (1987). Calibrating confidence coefficients. J. Amer. Statist. Assoc. 82 155–162.
  • Low, M. G. (1997). On nonparametric confidence intervals. Ann. Statist. 25 2547–2554.
  • Massart, P. (1989). Strong approximation for multivariate empirical and related processes, via KMT constructions. Ann. Probab. 17 266–291.
  • McMurry, T. L. and Politis, D. N. (2008). Bootstrap confidence intervals in nonparametric regression with built-in bias correction. Statist. Probab. Lett. 78 2463–2469.
  • Mendez, G. and Lohr, S. (2011). Estimating residual variance in random forest regression. Comput. Statist. Data Anal. 55 2937–2950.
  • Müller, U. U., Schick, A. and Wefelmeyer, W. (2003). Estimating the error variance in nonparametric regression by a covariate-matched $U$-statistic. Statistics 37 179–188.
  • Müller, H.-G. and Stadtmüller, U. (1987). Estimation of heteroscedasticity in regression analysis. Ann. Statist. 15 610–625.
  • Müller, H.-G. and Stadtmüller, U. (1993). On variance function estimation with quadratic forms. J. Statist. Plann. Inference 35 213–231.
  • Müller, H.-G. and Zhao, P. L. (1995). On a semiparametric variance function model and a test for heteroscedasticity. Ann. Statist. 23 946–967.
  • Munk, A., Bissantz, N., Wagner, T. and Freitag, G. (2005). On difference-based variance estimation in nonparametric regression when the covariate is high dimensional. J. R. Stat. Soc. Ser. B Stat. Methodol. 67 19–41.
  • Neumann, M. H. (1994). Fully data-driven nonparametric variance estimators. Statistics 25 189–212.
  • Neumann, M. H. (1995). Automatic bandwidth choice and confidence intervals in nonparametric regression. Ann. Statist. 23 1937–1959.
  • Neumann, M. H. and Polzehl, J. (1998). Simultaneous bootstrap confidence bands in nonparametric regression. J. Nonparametr. Stat. 9 307–333.
  • Picard, D. and Tribouley, K. (2000). Adaptive confidence interval for pointwise curve estimation. Ann. Statist. 28 298–335.
  • Rice, J. (1984). Bandwidth choice for nonparametric regression. Ann. Statist. 12 1215–1230.
  • Ruppert, D., Sheather, S. J. and Wand, M. P. (1995). An effective bandwidth selector for local least squares regression. J. Amer. Statist. Assoc. 90 1257–1270.
  • Ruppert, D. and Wand, M. P. (1994). Multivariate locally weighted least squares regression. Ann. Statist. 22 1346–1370.
  • Ruppert, D., Wand, M. P. and Carroll, R. J. (2003). Semiparametric Regression. Cambridge Series in Statistical and Probabilistic Mathematics 12. Cambridge Univ. Press, Cambridge.
  • Schucany, W. R. and Sommers, J. P. (1977). Improvement of kernel type density estimators. J. Amer. Statist. Assoc. 72 420–423.
  • Seifert, B., Gasser, T. and Wolf, A. (1993). Nonparametric estimation of residual variance revisited. Biometrika 80 373–383.
  • Sun, J. and Loader, C. R. (1994). Simultaneous confidence bands for linear regression and smoothing. Ann. Statist. 22 1328–1345.
  • Tong, T. and Wang, Y. (2005). Estimating residual variance in nonparametric regression using least squares. Biometrika 92 821–830.
  • Tusnády, G. (1977). A remark on the approximation of the sample $DF$ in the multidimensional case. Period. Math. Hungar. 8 53–55.
  • Wang, Y. D. and Wahba, G. (1995). Bootstrap confidence-intervals and for smoothing splines and their comparison to Bayesian confidence-intervals. Comm. Statist. Simulation Comput. 51 263–279.
  • Xia, Y. (1998). Bias-corrected confidence bands in nonparametric regression. J. R. Stat. Soc. Ser. B Stat. Methodol. 60 797–811.

Supplemental materials

  • Supplementary material: Appendix B. The supplementary material in Appendix B.1 outlines theoretical properties underpinning our methodology, while Appendix B.2 contains a proof of Theorem 4.1.