Statistical Science

Models as Approximations I: Consequences Illustrated with Linear Regression

Andreas Buja, Lawrence Brown, Richard Berk, Edward George, Emil Pitkin, Mikhail Traskin, Kai Zhang, and Linda Zhao

Full-text: Access denied (no subscription detected)

We're sorry, but we are unable to provide you with the full text of this article because we are not able to identify you as a subscriber. If you have a personal subscription to this journal, then please login. If you are already logged in, then you may need to update your profile to register your subscription. Read more about accessing full-text


In the early 1980s, Halbert White inaugurated a “model-robust” form of statistical inference based on the “sandwich estimator” of standard error. This estimator is known to be “heteroskedasticity-consistent,” but it is less well known to be “nonlinearity-consistent” as well. Nonlinearity, however, raises fundamental issues because in its presence regressors are not ancillary, hence cannot be treated as fixed. The consequences are deep: (1) population slopes need to be reinterpreted as statistical functionals obtained from OLS fits to largely arbitrary joint ${x\textrm{-}y}$ distributions; (2) the meaning of slope parameters needs to be rethought; (3) the regressor distribution affects the slope parameters; (4) randomness of the regressors becomes a source of sampling variability in slope estimates of order $1/\sqrt{N}$; (5) inference needs to be based on model-robust standard errors, including sandwich estimators or the ${x\textrm{-}y}$ bootstrap. In theory, model-robust and model-trusting standard errors can deviate by arbitrary magnitudes either way. In practice, significant deviations between them can be detected with a diagnostic test.

Article information

Statist. Sci., Volume 34, Number 4 (2019), 523-544.

First available in Project Euclid: 8 January 2020

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Ancillarity of regressors misspecification econometrics sandwich estimator bootstrap


Buja, Andreas; Brown, Lawrence; Berk, Richard; George, Edward; Pitkin, Emil; Traskin, Mikhail; Zhang, Kai; Zhao, Linda. Models as Approximations I: Consequences Illustrated with Linear Regression. Statist. Sci. 34 (2019), no. 4, 523--544. doi:10.1214/18-STS693.

Export citation


  • Berk, R. H. (1966). Limiting behavior of posterior distributions when the model is incorrect. Ann. Math. Stat. 37 51–58.
  • Berk, R. H. (1970). Consistency a posteriori. Ann. Math. Stat. 41 894–906.
  • Berk, R., Brown, L., Buja, A., Zhang, K. and Zhao, L. (2013). Valid post-selection inference. Ann. Statist. 41 802–837.
  • Berk, R., Kriegler, B. and Ylvisaker, D. (2008). Counting the homeless in Los Angeles county. In Probability and Statistics: Essays in Honor of David A. Freedman. Inst. Math. Stat. (IMS) Collect. 2 127–141. IMS, Beachwood, OH.
  • Berman, M. (1988). A theorem of Jacobi and its generalization. Biometrika 75 779–783.
  • Bickel, P. J., Götze, F. and van Zwet, W. R. (1997). Resampling fewer than $n$ observations: Gains, losses, and remedies for losses. Statist. Sinica 7 1–31.
  • Box, G. E. P. (1979). Robustness in the strategy of scientific model building. In Robustness in Statistics: Proceedings of a Workshop (R. L. Launer and G. N. Wilkinson, eds.) Academic Press (Elsevier), Amsterdam.
  • Bühlmann, P. and van de Geer, S. (2015). High-dimensional inference in misspecified linear models. Electron. J. Stat. 9 1449–1473.
  • Buja, A., Brown, L., Berk, R., George, E., Pitkin, E., Traskin, M., Zhang, K., and Zhao, L. (2019). Supplement to “Models as Approximations I: Consequences Illustrated with Linear Regression.” 10.1214/18-STS693SUPP.
  • Cox, D. R. (1962). Further results on tests of separate families of hypotheses (1962). J. Roy. Statist. Soc. Ser. B 24 406–424.
  • Cox, D. R. (1995). Discussion of Chatfield (1995). J. Roy. Statist. Soc. Ser. A 158 455–456.
  • Cox, D. R. and Hinkley, D. V. (1974). Theoretical Statistics. CRC Press, London.
  • Davies, L. (2014). Data Analysis and Approximate Models. Monographs on Statistics and Applied Probability 133. CRC Press, Boca Raton, FL.
  • De Blasi, P. (2013). Discussion on article “Bayesian inference with misspecified models” by Stephen G. Walker. J. Statist. Plann. Inference 143 1634–1637.
  • Diggle, P. J., Heagerty, P. J., Liang, K.-Y. and Zeger, S. L. (2002). Analysis of Longitudinal Data, 2nd ed. Oxford Statistical Science Series 25. Oxford Univ. Press, Oxford.
  • Donoho, D. and Montanari, A. (2014). Variance Breakdown of Huber (M)-estimators: $n/p\rightarrow m\in(1,\infty)$. Available at arXiv:1503.02106.
  • Efron, B. (1982). The Jackknife, the Bootstrap and Other Resampling Plans. CBMS-NSF Regional Conference Series in Applied Mathematics 38. SIAM, Philadelphia, PA.
  • Efron, B. and Tibshirani, R. J. (1993). An Introduction to the Bootstrap. Monographs on Statistics and Applied Probability 57. CRC Press, New York.
  • Eicker, F. (1963). Asymptotic normality and consistency of the least squares estimators for families of linear regressions. Ann. Math. Stat. 34 447–456.
  • El Karoui, N., Bean, D., Bickel, P. and Yu, B. (2013). Optimal M-estimation in high-dimensional regression. Proc. Natl. Acad. Sci. 110 14563–14568.
  • Freedman, D. A. (1981). Bootstrapping regression models. Ann. Statist. 9 1218–1228.
  • Freedman, D. A. (2006). On the so-called “Huber sandwich estimator” and “robust standard errors”. Amer. Statist. 60 299–302.
  • Gelman, A. and Park, D. K. (2009). Splitting a predictor at the upper quarter or third and the lower quarter or third. Amer. Statist. 63 1–8.
  • Hall, P. (1992). The Bootstrap and Edgeworth Expansion. Springer Series in Statistics. Springer, New York.
  • Hall, A. R. (2005). Generalized Method of Moments. Advanced Texts in Econometrics. Oxford Univ. Press, Oxford.
  • Hampel, F. R., Ronchetti, E. M., Rousseeuw, P. J. and Stahel, W. A. (1986). Robust Statistics: The Approach Based on Influence Functions. Wiley Series in Probability and Mathematical Statistics: Probability and Mathematical Statistics. Wiley, New York.
  • Hansen, L. P. (1982). Large sample properties of generalized method of moments estimators. Econometrica 50 1029–1054.
  • Hartigan, J. A. (1983). Asymptotic normality of posterior distributions. In Bayes Theory. Springer Series in Statistics 107–118. Springer, New York, NY.
  • Hausman, J. A. (1978). Specification tests in econometrics. Econometrica 46 1251–1271.
  • Hinkley, D. V. (1977). Jacknifing in unbalanced situations. Technometrics 19 285–292.
  • Hoff, P. and Wakefield, J. (2013). Bayesian sandwich posteriors for pseudo-true parameters. A discussion of “Bayesian inference with misspecified models” by Stephen Walker. J. Statist. Plann. Inference 143 1638–1642.
  • Huber, P. J. (1967). The behavior of maximum likelihood estimates under nonstandard conditions. In Proc. Fifth Berkeley Sympos. Math. Statist. and Probability (Berkeley, Calif., 1965/66), Vol. I: Statistics 221–233. Univ. California Press, Berkeley, CA.
  • Huber, P. J. and Ronchetti, E. M. (2009). Robust Statistics, 2nd ed. Wiley Series in Probability and Statistics. Wiley, Hoboken, NJ.
  • Kauermann, G. and Carroll, R. J. (2001). A note on the efficiency of sandwich covariance matrix estimation. J. Amer. Statist. Assoc. 96 1387–1396.
  • Krasker, W. S. and Welsch, R. E. (1982). Efficient bounded-influence regression estimation. J. Amer. Statist. Assoc. 77 595–604.
  • Lee, J. D., Sun, D. L., Sun, Y. and Taylor, J. E. (2016). Exact post-selection inference, with application to the lasso. Ann. Statist. 44 907–927.
  • Lehmann, E. L. and Romano, J. P. (2008). Testing Statistical Hypotheses, 3rd ed. Springer Texts in Statistics. Springer, New York.
  • Liang, K. Y. and Zeger, S. L. (1986). Longitudinal data analysis using generalized linear models. Biometrika 73 13–22.
  • Loh, P.-L. (2017). Statistical consistency and asymptotic normality for high-dimensional robust $M$-estimators. Ann. Statist. 45 866–896.
  • Long, J. S. and Ervin, L. H. (2000). Using heteroscedasticity consistent standard errors in the linear model. Amer. Statist. 54 217–224.
  • MacKinnon, J. and White, H. (1985). Some heteroskedasticity-consistent covariance matrix estimators with improved finite sample properties. J. Econometrics 29 305–325.
  • Mammen, E. (1993). Bootstrap and wild bootstrap for high-dimensional linear models. Ann. Statist. 21 255–285.
  • Mammen, E. (1996). Empirical process of residuals for high-dimensional linear models. Ann. Statist. 24 307–335.
  • McCarthy, D., Zhang, K., Brown, L. D., Berk, R., Buja, A., George, E. I. and Zhao, L. (2018). Calibrated percentile double bootstrap for robust linear regression inference. Statist. Sinica 28 2565–2589.
  • Newey, W. K. and West, K. D. (1987). A simple, positive semidefinite, heteroskedasticity and autocorrelation consistent covariance matrix. Econometrica 55 703–708.
  • O’Hagan, A. (2013). Bayesian inference with misspecified models: Inference about what?. J. Statist. Plann. Inference 143 1643–1648.
  • Owen, A. B. (2001). Empirical Likelihood. Chapman & Hall/CRC, Boca Raton, FL.
  • Politis, D. N. and Romano, J. P. (1994). Large sample confidence regions based on subsamples under minimal assumptions. Ann. Statist. 22 2031–2050.
  • R Development Core Team (2008). R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. Available at
  • Stigler, S. M. (2001). Ancillary history. In State of the Art in Probability and Statistics: Festschrift for W. R. van Zwet (M. DeGunst, C. Klaassen and A. van der Vaart, eds.) 555–567.
  • Szpiro, A. A., Rice, K. M. and Lumley, T. (2010). Model-robust regression and a Bayesian “sandwich” estimator. Ann. Appl. Stat. 4 2099–2113.
  • Walker, S. G. (2013). Bayesian inference with misspecified models. J. Statist. Plann. Inference 143 1621–1633.
  • Wasserman, L. (2011). Low assumptions, high dimensions. Ration. Mark. Moral. 2 201–209.
  • Weber, N. C. (1986). The jackknife and heteroskedasticity: Consistent variance estimation for regression models. Econom. Lett. 20 161–163.
  • White, H. (1980a). Using least squares to approximate unknown regression functions. Internat. Econom. Rev. 21 149–170.
  • White, H. (1980b). A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity. Econometrica 48 817–838.
  • White, H. (1981). Consequences and detection of misspecified nonlinear regression models. J. Amer. Statist. Assoc. 76 419–433.
  • White, H. (1982). Maximum likelihood estimation of misspecified models. Econometrica 50 1–25.
  • White, H. (1994). Estimation, Inference and Specification Analysis. Econometric Society Monographs 22. Cambridge Univ. Press, Cambridge.
  • Wu, C.-F. J. (1986). Jackknife, bootstrap and other resampling methods in regression analysis. Ann. Statist. 14 1261–1350.

See also

Supplemental materials

  • Supplement to “Models as Approximations I: Consequences Illustrated with Linear Regression”. This supplement contains Appendices A-F.