Electronic Journal of Statistics

Some non-asymptotic properties of parametric bootstrap P-values in discrete models

Chris J. Lloyd

Full-text: Open access


For discrete data especially, standard P-values can misreport the true significance, even for moderately large sample sizes. The bootstrap P-value is the exact tail probability of an appropriate test statistic, calculated assuming the nuisance parameter equals the null maximum likelihood (ML) estimate. For basic discrete models and standard test statistics, bootstrap P-values have been found to be extremely close to uniformly distributed under the null ([1]). Detailed numerical results reported there suggest that this phenomenon is not explained by asymptotics. In this paper, we identify several desirable non-asymptotic properties of bootstrap P-values and provide arguments for why bootstrap P-values are so close to exact. The most important of these is that bootstrap will correct ‘incorrect’ ordering of the sample space and that this leads to a more pivotal distribution. Most of these arguments only hold for discrete models and when the null ML estimate is used.

Article information

Electron. J. Statist., Volume 6 (2012), 2449-2462.

First available in Project Euclid: 4 January 2013

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62F03: Hypothesis testing
Secondary: 62F05: Asymptotic properties of tests

Bootstrap exact test exact P-value nuisance parameters size accuracy


Lloyd, Chris J. Some non-asymptotic properties of parametric bootstrap P-values in discrete models. Electron. J. Statist. 6 (2012), 2449--2462. doi:10.1214/12-EJS751. https://projecteuclid.org/euclid.ejs/1357307945

Export citation


  • [1] Lloyd, C.J. (2013). A numerical investigation of the accuracy of parametric bootstrap for discrete data., Comp. Statist. Data Anal. 57. To appear.
  • [2] Cressie, N. and Read, T.R.C. (1984). Multinomial goodness-of-fit tests., J. Roy. Statist. Soc. B 46 157–214.
  • [3] Perkins, W., Tygert, M. and Ward, R. (2011). Computing the confidence levels for a root-mean-square test of goodness-of-fit, Appl. Math. Comput. 217 9072–9084.
  • [4] Beran, R. (1988). Prepivoting test statistics: a bootstrap view of asymptotic refinements., J. Amer. Statist. Assoc. 83 687–697.
  • [5] Rohmel, J. and Mansmann, U. (1999). Unconditional non-asymptotic one-sided tests for independent binomial proportions when the interest lies in showing non-inferiority or superiority., Biometrical Journal 41 149–170.
  • [6] DiCiccio, T.J., Martin, M.A. and Stern, S.E. (1984). Simple and accurate one-sided inference from signed roots of likelihood ratios., Canad. J. Statist. 29 167–76.
  • [7] DiCiccio, T.J. and Stern, S.E. (1994). Constructing approximate standard normal pivots from signed roots of adjusted likelihood ratio statistics., Scand. J. Statist. 21 447–460.
  • [8] Fraser, D.A.S and Rousseau, J. (2008.) Studentization and deriving accurate P-values., Biometrika 95 1–16.
  • [9] Freeman, G.H. and Halton, J.H. (1951). Note on exact treatment of contingency, goodness of fit and other problems of significance., Biometrika 38 141–149.
  • [10] Lee, S.M.S. and Young, G.A. (2005). Parametric bootstrapping with nuisance parameters., Stat. Prob. Letters 71 143–153.
  • [11] Lloyd, C.J. (2008). Exact P-values for discrete models obtained by estimation and maximisation., Austral. and New Zealand J. Statist. 50 329–346.
  • [12] Lloyd, C.J. (2012). Computing Highly Accurate or Exact P-values using Importance Sampling., Comp. Statist. Data Anal. 56 1784–1794.
  • [13] Berger, R.L. and Boos, D.D. (1994). P values maximised over a confidence set for the nuisance parameter., J. Amer. Statist. Assoc. 89 1012–1016.