The Annals of Applied Statistics

Open statistical issues in Particle Physics

Louis Lyons

Full-text: Open access


Many statistical issues arise in the analysis of Particle Physics experiments. We give a brief introduction to Particle Physics, before describing the techniques used by Particle Physicists for dealing with statistical problems, and also some of the open statistical questions.

Article information

Ann. Appl. Stat. Volume 2, Number 3 (2008), 887-915.

First available in Project Euclid: 13 October 2008

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Particle Physics parameter determination goodness of fit p-values hypothesis testing nuisance parameters upper limits blind analysis signal-background separation combining results


Lyons, Louis. Open statistical issues in Particle Physics. Ann. Appl. Stat. 2 (2008), no. 3, 887--915. doi:10.1214/08-AOAS163.

Export citation


  • Aslan, B. and Zech, G. (2004). A multivariate two-sample test based on the concept of minimum energy. J. Stat. Comp. Simul. 75 109.
  • Aslan, B. and Zech, G. (2005). Statistical energy as a tool for binning-free multivariate goodness of fit tests, two-sample comparison and unfolding. Nuclear Instruments and Methods A537 626.
  • Belikov, J. (2007). ALICE statistical wish-list. Available at
  • CDF Statistics Committee (2007). Frequently asked questions. Available at
  • Cheung, H. and Lyons, L. (2000). FNAL confidence limits workshop. Available at
  • Cousins, R. (1998). Improved central confidence intervals for the ratio of Poisson means. Nuclear Instruments and Methods A417 391–399.
  • Cousins, R. (2007). Annotated bibliography on some papers on combining significances or p-values. Available at arXiv:0705.2209.
  • Cousins, R. D. and Highland, V. L. (1992). Incorporating systematic uncertainties into an upper limit. Nuclear Instruments and Methods A320 331.
  • Cox, D. R. (1958). Some problems connected with statistical inference. Ann. Math. Statist. 29 357–372.
  • Cranmer, K. (2007). Progress, challenges and future of statistics at the LHC. Available at
  • Cuadras, C. M., Fortiana, J. and Oliva, F. (1997). The proximity of an individual to a population with applications to discriminant analysis. J. Classification 14 117–136.
  • Cuadras, C. M. and Fortiana, J. (2003). Distance-based multivariate two sample tests. Available at
  • Demortier, L. (2005). Bayesian reference analysis. Available at
  • Demortier, L. (2006). Setting the scene for p-values. Available at 2006/06w5054/report06w5054.pdf.
  • Demortier, L. (2007). P-values and nuisance parameters. Available at
  • Drton, M. (2007). Likelihood ratio tests and singularities. Available at
  • Feldman, G. J. and Cousins, R. D. (1998). Unified approach to the classical statistical analysis of small signals. Phys. Rev. D 57 3873–3889.
  • Friedman, J. H. (2003). Recent advances in predictive (machine) learning. Available at
  • Friedman, J. H. (2005). Separating signal from background using ensembles of rules. Available at
  • Gross, E. (2007). ATLAS and CMS statistical wish-list. Available at
  • Heinrich, J. (2003a). Coverage of error bars for Poisson data. Available at
  • Heinrich, J. (2003b). Pitfalls of goodness-of-fit from likelihood. Available at
  • Heinrich, J. (2005). The Bayesian approach to setting limits: What to avoid. Available at
  • Heinrich, J. (2007). Review of Banff challenge on upper limits. Available at
  • Heinrich, J. and Lyons, L. (2007). Systematic errors. Annual Reviews of Nuclear and Particle Science 57 145–169.
  • Heinrich, J. et al. (2004). Interval estimation in the presence of nuisance parameters. 1. Bayesian approach. CDF Note 7117. Available at
  • Höcker, A. et al. (2007). TMVA—Toolkit for multivariate data analysis. Available at
  • James, F., Lyons, L. and Perrin, Y. (2000). Workshop on confidence limits. CERN Yellow Report 2000-05.
  • Klein, J. R. and Roodman, A. (2005). Blind analysis in nuclear and particle physics. Annual Review of Nuclear and Particle Physics 55 141.
  • Linnemann, J. (2007). A pitfall in evaluating systematic errors. Available at
  • Lyons, L. (1999). Comparing two hypotheses. Available at
  • Lyons, L. (2008). Supplement to “Open statistical issues in particle physics.” DOI: 10.1214/08-AOAS163SUPP.
  • Lyons, L., Martin, A. and Saxon, D. (1990). On the determination of the B lifetime by combining the results of different experiments. Phy. Rev. D 41 982.
  • Lyons, L., Mount, R. and Reitmeyer, R., eds. (2003). PHYSTAT 20003. eConf C030908, SLAC-R-703. Available at
  • Lyons, L. and Ünel, M. K. (2005). Statistical problems in particle physics, astrophysics and cosmology. Imperial College Press, London. Available at
  • Narsky, I. (2000). Comparison of upper limits. Available at
  • Narsky, I. (2006). StatPatternRecognition: A C++ package for multi-variate classification. Available at
  • Neal, R. (2007). Computing likelihood functions when distributions are defined by simulators with nuisance parameters. Available at
  • Nicolo, D. and Signorelli, G. (2002). Application of strong confidence to the CHOOZ experiment with frequentist inclusion of nuisance parameters. Available at
  • Particle Data Group (2006). J. Phys. G: Nucl. Part. Phys. 33 1 (see page 14).
  • Prosper, H. B. (2002). Multivariate analysis: A unified perspective. Available at
  • Prosper, H. B., Lyons, L. and De Roeck, A. (2007). PHYSTAT-LHC Workshop at CERN on Statistical Issues for LHC Physics. Available at
  • Protassov, R. et al. (2002). Statistics: Handle with care. Detecting multiple model components with the likelihood ratio test. Astrophysics J. 571 545–559.
  • Punzi, G. (2003). Sensitivity of searches for new signals and its optimization. Available at
  • Punzi, G. (2005). Ordering algorithms and confidence intervals in the presence of nuisance parameters. Available at
  • Read, A. L. (2000). Modified frequentist analysis of search results (the CLs method). Available at
  • Read, A. L. (2004). Presentation of search results—the CLs method. J. Phys. G: Nucl. Part. Phys. 28 2693–2704.
  • Reid, N. (2007). Some aspects of design of experiments. Available at
  • Reid, N., Linnemann, J. and Lyons, L. (2006). Workshop on statistical inference problems in high energy physics and astronomy. Available at
  • Roe, B. P. (2007). Statistical errors in Monte Carlo estimates of systematic errors. Nuclear Instruments and Methods A570 159–164.
  • Rolke, W. A., Lopez, A. M. and Conrad, J. (2005). Limits and confidence intervals in the presence of nuisance parameters. Nuclear Instruments and Methods A551 493–503.
  • Self, S. G. and Liang, K. Y. (1987). Asymptotic properties of maximum likelihood estimators and likelihood ratio test under non-standard conditions. J. Amer. Statist. Assoc. 82 605–610.
  • Sen, B., Walker, M. and Woodroofe, M. (2008). On the unified method with nuisance parameters. Statist. Sinica. To appear.
  • Trotta, R. (2008). Bayes in the sky: Bayesian inference and model selection in cosmology. Contemporary Physics 49 71–104.
  • Whalley, M. and Lyons, L., eds. (2002). Advanced statistical techniques in particle physics, Durham IPPP/02/39. Available at
  • Wilks, S. S. (1938). The large-sample distribution of the likelihood ratio for testing composite hypotheses. Ann. Math. Statist. 9 60–62.
  • Xie, Y. (2007). LHCb statistical wishlist. Available at

Supplemental materials