The Annals of Statistics

How do bootstrap and permutation tests work?

Arnold Janssen and Thorsten Pauls

Full-text: Open access


Resampling methods are frequently used in practice to adjust critical values of nonparametric tests. In the present paper a comprehensive and unified approach for the conditional and unconditional analysis of linear resampling statistics is presented. Under fairly mild assumptions we prove tightness and an asymptotic series representation for their weak accumulation points. From this series it becomes clear which part of the resampling statistic is responsible for asymptotic normality. The results leads to a discussion of the asymptotic correctness of resampling methods as well as their applications in testing hypotheses. They are conditionally correct iff a central limit theorem holds for the original test statistic. We prove unconditional correctness iff the central limit theorem holds or when symmetric random variables are resampled by a scheme of asymptotically random signs. Special cases are the m (n) out of k (n) bootstrap, the weighted bootstrap, the wild bootstrap and all kinds of permutation statistics. The program is carried out for convergent partial sums of rowwise independent infinitesimal triangular arrays in detail. These results are used to compare power functions of conditional resampling tests and their unconditional counterparts. The proof uses the method of random scores for permutation type statistics.

Article information

Ann. Statist., Volume 31, Number 3 (2003), 768-806.

First available in Project Euclid: 25 June 2003

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62G09: Resampling methods 62G10: Hypothesis testing

Sample mean bootstrap weighted bootstrap wild bootstrap permutation statistics resampling tests conditional tests random scores rank statistics infinitely divisible laws two-sample test exchangeable variables


Janssen, Arnold; Pauls, Thorsten. How do bootstrap and permutation tests work?. Ann. Statist. 31 (2003), no. 3, 768--806. doi:10.1214/aos/1056562462.

Export citation


  • ARAUJO, A. and GINÉ, E. (1980). The Central Limit Theorem for Real and Banach Valued Random Variables. Wiley, New York.
  • ARCONES, M. and GINÉ, E. (1989). The bootstrap of the mean with arbitrary bootstrap sample size. Ann. Inst. H. Poincaré Probab. Statist. 25 457-481.
  • ARCONES, M. and GINÉ, E. (1991). Additions and corrections to "The bootstrap of the mean with arbitrary bootstrap sample size." Ann. Inst. H. Poincaré Probab. Statist. 27 583-595.
  • ATHREy A, K. B. (1987). Bootstrap of the mean in the infinite variance case. Ann. Statist. 15 724-731.
  • BARBE, P. and BERTAIL, P. (1995). The Weighted Bootstrap. Lecture Notes in Statist. 98. Springer, New York.
  • BERAN, R. (1988). Prepivoting test statistics: A bootstrap view of asy mptotic refinements. J. Amer. Statist. Assoc. 83 687-697.
  • BICKEL, P. J., GÖTZE, F. and VAN ZWET, W. R. (1997). Resampling fewer than n observations: Gains, losses, and remedies for losses. Statist. Sinica 7 1-31.
  • BILLINGSLEY, P. (1971). Weak Convergence of Measures: Applications in Probability. SIAM, Philadelphia.
  • CHERNOFF, H. and TEICHER, H. (1958). A central limit theorem for sums of interchangeable random variables. Ann. Math. Statist. 29 118-130.
  • CSÖRG O, M., CSÖRG O, S., HORVÁTH, L. and MASON, D. M. (1986). Normal and stable convergence of integral functions of the emirical distribution function. Ann. Probab. 14 86-118.
  • CSÖRG O, S., HÄUSLER, E. and MASON, D. M. (1988). A probabilistic approach to the asy mptotic distribution of sums of independent, identically distributed random variables. Adv. in Appl. Math. 9 259-333.
  • CSÖRG O, S. and MASON, D. M. (1989). Bootstrapping empirical functions. Ann. Statist. 17 1447- 1471.
  • CUESTA-ALBERTOS, J. A. and MATRÁN, C. (1998). The asy mptotic distribution of the bootstrap sample mean of an infinitesimal array. Ann. Inst. H. Poincaré Probab. Statist. 34 23-48.
  • DEHEUVELS, P., MASON, D. M. and SHORACK, G. R. (1993). Some results on the influence of extremes on the bootstrap. Ann. Inst. H. Poincaré Probab. Statist. 29 83-103.
  • DEL BARRIO, E. and MATRÁN, C. (2000). The weighted bootstrap mean for heavy-tailed distributions. J. Theoret. Probab. 13 547-569.
  • DEL BARRIO, E., MATRÁN, C. and CUESTA-ALBERTOS, J. A. (1999). Neccessary conditions for the bootstrap of the mean of a triangular array. Ann. Inst. H. Poincaré Probab. Statist. 35 371-386.
  • DUDLEY, R. M. (1989). Real Analy sis and Probability. Wadsworth, Pacific Grove, CA.
  • EINMAHL, U. and MASON, D. M. (1992). Approximations to permutation and exchangeable processes. J. Theoret. Probab. 5 101-126.
  • GINÉ, E. and ZINN, J. (1989). Necessary conditions for the bootstrap of the mean. Ann. Statist. 17 684-691.
  • GNEDENKO, B. V. and KOLMOGOROV, A. N. (1954). Limit Distributions for Sums of Independent Random Variables. Addison-Wesley, Cambridge, MA.
  • HÁJEK, J. (1961). Some extensions of the Wald-Wolfowitz-Noether theorem. Ann. Math. Statist. 32 506-523.
  • HÁJEK, J. (1968). Asy mptotic normality of simple linear rank statistics under alternatives. Ann. Math. Statist. 39 325-346.
  • HÁJEK, J., SIDÁK, Z. and SEN, P. K. (1999). Theory of Rank Tests, 2nd ed. Academic Press, Orlando, FL.
  • HALL, P. (1990). Asy mptotic properties of the bootstrap for heavy-tailed distributions. Ann. Probab. 18 1342-1360.
  • HALL, P. (1992). The Bootstrap and Edgeworth Expansion. Springer, New York.
  • HARTMAN, P. and WINTNER, A. (1942). On the infinitesimal generators of integral convolutions. Amer. J. Math. 64 273-298.
  • JANSSEN, A. (1989). Local asy mptotic normality for randomly censored models with applications to rank tests. Statist. Neerlandica 43 109-125.
  • JANSSEN, A. (1991). Conditional rank tests for randomly censored data. Ann. Statist. 19 1434-1456.
  • JANSSEN, A. (1994). Sums of independent triangular array s and extreme order statistics. Ann. Probab. 22 1766-1793.
  • JANSSEN, A. (1997). Studentized permutation tests for non-i.i.d. hy potheses and the generalized Behrens-Fisher problem. Statist. Probab. Lett. 36 9-21.
  • JANSSEN, A. (1999). Nonparametric sy mmetry tests for statistical functionals. Math. Methods Statist. 8 320-343.
  • JANSSEN, A. (2000). Invariance principles for sums of extreme sequential order statistics attracted to Lévy processes. Stochastic Process. Appl. 85 255-277.
  • JANSSEN, A. and MASON, D. M. (1990). Non-standard Rank Tests. Lecture Notes in Statist. 65. Springer, New York.
  • KINATEDER, J. (1992). An invariance principle applicable to the bootstrap. In Exploring the Limits of Bootstrap (R. LePage and L. Billard, eds.) 157-181. Wiley, New York.
  • KNIGHT, K. (1989). On the bootstrap of the sample mean in the infinite variance case. Ann. Statist. 17 1168-1175.
  • LEPAGE, R., WOODROOFE, M. and ZINN, J. (1981). Convergence to a stable distribution via order statistics. Ann. Probab. 9 624-632.
  • MAMMEN, E. (1992a). When Does Bootstrap Work? Asy mptotic Results and Simulations. Lecture Notes in Statist. 77. Springer, New York.
  • MAMMEN, E. (1992b). Bootstrap, wild bootstrap, and asy mptotic normality. Probab. Theory Related Fields 93 439-455.
  • MASON, D. M. and NEWTON, M. A. (1992). A rank statistics approach to the consistency of a general bootstrap. Ann. Statist. 20 1611-1624.
  • MASON, D. M. and SHAO, Q.-M. (2001). Bootstrapping the Student t-statistic. Ann. Probab. 29 1435-1450.
  • NEUHAUS, G. (1988). Asy mptotically optimal rank tests for the two-sample problem with randomly censored data. Comm. Statist. Theory Methods 17 2037-2058.
  • NEUHAUS, G. (1993). Conditional rank tests for the two-sample problem under random censorship. Ann. Statist. 21 1760-1779.
  • PETROV, V. V. (1995). Limit Theorems of Probability Theory. Clarendon Press, Oxford.
  • RAIKOV, D. A. (1938). On a connection between the central limit law of the theory of probability and the law of large numbers. Bull. Acad. Sci. URSS Ser. Math. 2 323-338.
  • ROMANO, J. P. (1989). Bootstrap and randomization tests of some nonparametric hy potheses. Ann. Statist. 17 141-159.
  • SHORACK, G. R. (1996). Linear rank statistics, finite sampling, permutation tests and winsorizing. Ann. Statist. 24 1371-1385.
  • SHORACK, G. R. (2000). Probability for Statisticians. Springer, New York.
  • STRASSER, H. and WEBER, C. (1999). The asy mptotic theory of permutation statistics. Math. Methods Statist. 8 220-250.
  • TUCKER, H. G. (1975). The supports of infinitely divisible distribution functions. Proc. Amer. Math. Soc. 49 436-440.
  • VAN DER VAART, A. W. and WELLNER, J. A. (1996). Weak Convergence and Empirical Processes. With Applications to Statistics. Springer, New York.
  • WELLNER, J. (2001). Some converse limit theorems for exchangeable bootstraps. In State of the Art in Probability and Statistics. Festschrift for Willem R. van Zwet (M. de Gunst et al., eds.) 593-606. IMS, Beachwood, OA.
  • WITTING, H. and NÖLLE, G. (1970). Angewandte Mathematische Statistik. Optimale finite und asy mptotische Verfahren. Teubner, Stuttgart.
  • ZOLOTAREV, V. M. and KRUGLOV, V. M. (1975). The structure of infinitely divisible distributions on a bicompact Abelian group. Theory Probab. Appl. 20 698-709.