Statistical Science

Workshop on statistical approaches for the evaluation of complex computer models

Richard A. Berk, Peter Bickel, Katherine Campbell, Robert Fovell, Sallie Keller-McNulty, Elizabeth Kelly, Rodman Linn, Byungkyu Park, Alan Perelson, Nagui Rouphail, Jerome Sacks, and Frederic Schoenberg

Full-text: Open access

Abstract

As decision- and policy-makers come to rely increasingly on estimates and simulations produced by computerized models of the world, in areas as diverse as climate prediction, transportation planning, economic policy and civil engineering, the need for objective evaluation of the accuracy and utility of such models likewise becomes more urgent. This article summarizes a two-day workshop that took place in Santa Fe, New Mexico in December 1999, whose focus was the evaluation of complex computer models. Approximately half of the workshop was taken up with formal presentation of four computer models by their creators, each paired with an initial assessment by a statistician. These prepared papers are presented, in shortened form, in Section 3 of this paper. The remainder of the workshop was devoted to introductory and summary comments, short contributed descriptions of related models and a great deal of floor discussion, which was recorded by assigned rapporteurs. These are presented in Sections 2 and 4 in the paper. In the introductory and concluding sections we attempt to summarize the progress made by the workshop and suggest next steps.

Article information

Source
Statist. Sci., Volume 17, Issue 2 (2002), 173-192.

Dates
First available in Project Euclid: 28 August 2002

Permanent link to this document
https://projecteuclid.org/euclid.ss/1030550860

Digital Object Identifier
doi:10.1214/ss/1030550860

Mathematical Reviews number (MathSciNet)
MR1939339

Zentralblatt MATH identifier
1032.62102

Keywords
Model accuracy model evaluation model validation uncertainty analysis computer experiments

Citation

Berk, Richard A.; Bickel, Peter; Campbell, Katherine; Fovell, Robert; Keller-McNulty, Sallie; Kelly, Elizabeth; Linn, Rodman; Park, Byungkyu; Perelson, Alan; Rouphail, Nagui; Sacks, Jerome; Schoenberg, Frederic. Workshop on statistical approaches for the evaluation of complex computer models. Statist. Sci. 17 (2002), no. 2, 173--192. doi:10.1214/ss/1030550860. https://projecteuclid.org/euclid.ss/1030550860


Export citation

References

  • BAy ARRI, M. J. and BERGER, J. O. (1999). Quantifying surprise in the data and model verification. In Bayesian Statistics VI (J. M. Bernardo, J. O. Berger, A. P. Dawid and A. F. M. Smith, eds.) 53-82. Oxford Univ. Press.
  • BEy ER, W. E. P., DEBRUIJN, I. A., PALACHE, A. M., WESTEN
  • DORP, R. G. J. and OSTERHAUS, A. D. M. E. (1998). The plea against annual influenza vaccination? `The Hoskins' Paradox' revisited. Vaccine 16 1929-1932.
  • BOSSERT, J., REISNER, J. M., LINN R. R., WINTERKAMP,
  • J. L., SCHAUB, R. and RIGGAN, P. J. (1998). Validation of coupled atmosphere-fire behavior models. Preprints of the 14th Conference on Fire and Forest Meteorology, November 16-20, 1998, Luso-Coimbra, Portugal.
  • BOX, G. E. P. and DRAPER, N. (1987). Empirical Model Building and Response Surfaces. Wiley, New York.
  • BRIEMAN, L. (2001). Statistical modeling: The two cultures. Statist. Sci. 16 199-231.
  • CLARK, T. L., RADKE, L., COEN, J. and MIDDLETON, D. (1999). Analy sis of small-scale convective dy namics in a crown fire using infrared video camera imagery. J. Appl. Meteor. 38 1401-1420.
  • CLICK, S. and ROUPHAIL, N. (1999). Lane group level field evaluation of computer-based signal timing models. Paper presented at the 78th Annual Meeting of the Transportation Research Board, Washington, DC, January 1999.
  • CORSIM USER'S MANUAL. (1997). FHWA, U.S. Department of Transportation, Office of Safety and Traffic Operation R&D, Intelligent Sy stems and Technology Division, McLean, VA.
  • EFRON, B. and TIBSHIRANI, R. (1993). An Introduction to the Bootstrap. Chapman and Hall, London.
  • GOLDBERG, D. E. (1989). Genetic Algorithms in Search, Optimization, and Machine Learning. Addison-Wesley, Reading, MA.
  • HOSKINS, T. W., DAVIS, J. R., SMITH, A. J., MILLER, C. L. and
  • ALLCHIN, A. (1979). Assessment of inactivated influenza-A vaccine after three outbreaks of influenza-A at Christ's Hospital. The Lancet 1 (8106) 33-35.
  • KEITEL, W. A., CATE, T. R., COUCH, R. B., HUGGINS, L.
  • L. and HESS, K. R. (1997). Continued efficacy of annual immunization with inactivated influenza virus vaccine over a five year period: A placebo-controlled trial in healthy adults. Vaccine 15 1114-1122.
  • KENNEDY, M. and O'HAGAN, A. (2000). Bayesian calibration of computer models. J. Roy. Statist. Soc. Ser. B 63 425-464.
  • KIEFER, J. and WOLFOWITZ, J. (1952). Stochastic estimation of the maximum in a regression. Ann. Math. Statist. 23 462-466.
  • KUSHNER, H. and YIN, G. (1997). Stochastic Approximation Algorithms and Applications. Springer, New York.
  • LINN, R. R. (1997). Transport model for prediction of wildfire behavior. Los Alamos National Laboratory Scientific Report LA13334-T.
  • MILLER, R. G. (1966). Simultaneous Statistical Inference. McGraw Hill, New York.
  • PERELSON, A. S. and OSTER, G. F. (1979). Theoretical studies of clonal selection: minimal antibody repertoire size and reliability of self-nonself discrimination. J. Theoret. Biol. 81 645-670.
  • SCHOENBERG, F., BERK, R., FOVELL, R., LI, C., LU, R. and
  • WEISS, R. (2001). Approximation and inversion of a complex meteorological sy stem via local linear filters. J. Appl. Meteor. 40 446-458.
  • SMITH, D. J., FORREST, S., ACKLEY, D. H. and PERELSON, A. S.
  • (1999). Variable efficacy of repeated annual influenza vaccination. Proc. Natl. Acad. Sci. USA 96 14001-14006.
  • VON STORCH, H. and NAVARRA, A., eds. (1999). Analy sis of Climate Variability: Applications of Statistical Techniques. Springer, New York.
  • ZEIGLER, B. (1976). Theory of Modelling and Simulation. Wiley, New York.