The Annals of Statistics

Computer model validation with functional output

M. J. Bayarri, J. O. Berger, J. Cafeo, G. Garcia-Donato, F. Liu, J. Palomo, R. J. Parthasarathy, R. Paulo, J. Sacks, and D. Walsh

Full-text: Open access

Abstract

A key question in evaluation of computer models is Does the computer model adequately represent reality? A six-step process for computer model validation is set out in Bayarri et al. [Technometrics 49 (2007) 138–154] (and briefly summarized below), based on comparison of computer model runs with field data of the process being modeled. The methodology is particularly suited to treating the major issues associated with the validation process: quantifying multiple sources of error and uncertainty in computer models; combining multiple sources of information; and being able to adapt to different, but related scenarios.

Two complications that frequently arise in practice are the need to deal with highly irregular functional data and the need to acknowledge and incorporate uncertainty in the inputs. We develop methodology to deal with both complications. A key part of the approach utilizes a wavelet representation of the functional data, applies a hierarchical version of the scalar validation methodology to the wavelet coefficients, and transforms back, to ultimately compare computer model output with field output. The generality of the methodology is only limited by the capability of a combination of computational tools and the appropriateness of decompositions of the sort (wavelets) employed here.

The methods and analyses we present are illustrated with a test bed dynamic stress analysis for a particular engineering system.

Article information

Source
Ann. Statist. Volume 35, Number 5 (2007), 1874-1906.

Dates
First available in Project Euclid: 7 November 2007

Permanent link to this document
http://projecteuclid.org/euclid.aos/1194461715

Digital Object Identifier
doi:10.1214/009053607000000163

Mathematical Reviews number (MathSciNet)
MR2363956

Zentralblatt MATH identifier
1144.62368

Subjects
Primary: 62Q05: Statistical tables

Keywords
Computer models validation functional data bias Bayesian analysis uncertain inputs

Citation

Bayarri, M. J.; Berger, J. O.; Cafeo, J.; Garcia-Donato, G.; Liu, F.; Palomo, J.; Parthasarathy, R. J.; Paulo, R.; Sacks, J.; Walsh, D. Computer model validation with functional output. Ann. Statist. 35 (2007), no. 5, 1874--1906. doi:10.1214/009053607000000163. http://projecteuclid.org/euclid.aos/1194461715.


Export citation

References

  • Aslett, R., Buck, R. J., Duvall, S. G., Sacks, J. and Welch, W. J. (1998). Circuit optimization via sequential computer experiments: Design of an output buffer. Appl. Statist. 47 31--48.
  • Bayarri, M., Berger, J., Cafeo, J., Garcia-Donato, G., Liu, F., Palomo, J., Parthasarathy, R., Paulo, R., Sacks, J. and Walsh, D. (2006). Computer model validation with functional outputs. Technical Report 165, National Institute of Statistical Sciences.
  • Bayarri, M. J., Berger, J. O., Kennedy, M., Kottas, A., Paulo, R., Sacks, J., Cafeo, J. A., Lin, C. H. and Tu, J. (2005). Bayesian validation of a computer model for vehicle crashworthiness. Technical Report 163, National Institute of Statistical Sciences.
  • Bayarri, M. J., Berger, J. O., Paulo, R., Sacks, J., Cafeo, J. A., Cavendish, J., Lin, C.-H. and Tu, J. (2007). A framework for validation of computer models. Technometrics 49 138--154.
  • Cafeo, J. and Cavendish, J. (2001). A framework for verification and validation of computer models and simulations. Internal General Motors document, GM Research and Development Center.
  • Craig, P. S., Goldstein, M., Rougier, J. C. and Seheult, A. H. (2001). Bayesian forecasting for complex systems using computer simulators. J. Amer. Statist. Assoc. 96 717--729.
  • Craig, P. S., Goldstein, M., Seheult, A. H. and Smith, J. A. (1997). Pressure matching for hydrocarbon reservoirs: A case study in the use of Bayes linear strategies for large computer experiments (with discussion). In Case Studies in Bayesian Statistics III (C. Gatsonis, J. S. Hodges, R. E. Kass, R. McCulloch, P. Rossi and N. D. Singpurwalla, eds.) 36--93. Springer, New York.
  • Currin, C., Mitchell, T., Morris, M. and Ylvisaker, D. (1991). Bayesian prediction of deterministic functions, with applications to the design and analysis of computer experiments. J. Amer. Statist. Assoc. 86 953--963.
  • Easterling, R. G. (2001). Measuring the predictive capability of computational models: Principles and methods, issues and illustrations. Technical Report SAND2001-0243, Sandia National Laboratories.
  • Fuentes, M., Guttorp, P. and Challenor, P. (2003). Statistical assessment of numerical models. Internat. Statist. Rev. 71 201--221.
  • Goldstein, M. and Rougier, J. C. (2003). Calibrated Bayesian forecasting using large computer simulators. Technical report, Statistics and Probability Group, Univ. Durham. Available at www.maths.dur.ac.uk/stats/physpred/papers/ CalibratedBayesian.ps.
  • Goldstein, M. and Rougier, J. C. (2004). Probabilistic formulations for transferring inferences from mathematical models to physical systems. SIAM J. Sci. Comput. 26 467--487.
  • Higdon, D., Gattiker, J., Williams, B. and Rightley, M. (2007). Computer model validation using high-dimensional outputs. In Bayesian Statistics 8 (J. Bernardo, M. J. Bayarri, J. O. Berger, A. P. Dawid, D. Heckerman, A. F. M. Smith and M. West, eds.). Oxford Univ. Press.
  • Kennedy, M. C. and O'Hagan, A. (2001). Bayesian calibration of computer models (with discussion) J. R. Stat. Soc. Ser. B Stat. Methodol. 63 425--464.
  • Kennedy, M. C., O'Hagan, A. and Higgins, N. (2002). Bayesian analysis of computer code outputs. In Quantitative Methods for Current Environmental Issues (C. W. Anderson, V. Barnett, P. C. Chatwin and A. H. El-Shaarawi, eds.) 227--243. Springer, London.
  • Morris, M. D., Mitchell, T. J. and Ylvisaker, D. (1993). Bayesian design and analysis of computer experiments: Use of derivatives in surface prediction. Technometrics 35 243--255.
  • Morris, J., Vannucci, M., Brown, P. and Carroll, R. (2003). Wavelet-based nonparametric modeling of hierarchical functions in colon carcinogenesis (with discussion). J. Amer. Statist. Assoc. 98 573--597.
  • Müller, P. and Vidakovic, B., eds. (1999). Bayesian Inference in Wavelet-Based Models. Lecture Notes in Statist. 141. Springer, New York.
  • Nagy, B., Loeppky, J. and Welch, W. J. (2007). Fast Bayesian inference for Gaussian process models. Technical Report 230, Dept. Statistics, Univ. British Columbia.
  • Oberkampf, W. and Trucano, T. (2000). Validation methodology in computational fluid dynamics. Technical Report 2000-2549, American Institute of Aeronautics and Astronautics.
  • Pilch, M., Trucano, T., Moya, J. L., Froehlich, G., Hodges, A. and Peercy, D. (2001). Guidelines for Sandia ASCI verification and validation plans---content and format: Version 2.0. Technical Report SAND 2001-3101, Sandia National Laboratories.
  • Roache, P. (1998). Verification and Validation in Computational Science and Engineering. Hermosa, Albuquerque, NM.
  • Sacks, J., Welch, W. J., Mitchell, T. J. and Wynn, H. P. (1989). Design and analysis of computer experiments (with discussion). Statist. Sci. 4 409--435.
  • Santner, T., Williams, B. and Notz, W. (2003). The Design and Analysis of Computer Experiments. Springer, New York.
  • Schonlau, M. and Welch, W. J. (2005). Screening the input variables to a computer model via analysis of variance and visualization. In Screening Methods for Experimentation in Industry, Drug Discovery and Genetics (A. Dean and S. Lewis, eds.) 308--327. Springer, New York.
  • Trucano, T., Pilch, M. and Oberkampf, W. O. (2002). General concepts for experimental validation of ASCII code applications. Technical Report SAND 2002-0341, Sandia National Laboratories.
  • Vidakovic, B. (1999). Statistical Modeling by Wavelets. Wiley, New York.
  • Welch, W. J., Buck, R. J., Sacks, J., Wynn, H. P., Mitchell, T. J. and Morris, M. D. (1992). Screening, predicting and computer experiments. Technometrics 34 15--25.