The Annals of Statistics

Exact and asymptotically robust permutation tests

EunYi Chung and Joseph P. Romano

Full-text: Open access


Given independent samples from $P$ and $Q$, two-sample permutation tests allow one to construct exact level tests when the null hypothesis is $P=Q$. On the other hand, when comparing or testing particular parameters $\theta$ of $P$ and $Q$, such as their means or medians, permutation tests need not be level $\alpha$, or even approximately level $\alpha$ in large samples. Under very weak assumptions for comparing estimators, we provide a general test procedure whereby the asymptotic validity of the permutation test holds while retaining the exact rejection probability $\alpha$ in finite samples when the underlying distributions are identical. The ideas are broadly applicable and special attention is given to the $k$-sample problem of comparing general parameters, whereby a permutation test is constructed which is exact level $\alpha$ under the hypothesis of identical distributions, but has asymptotic rejection probability $\alpha$ under the more general null hypothesis of equality of parameters. A Monte Carlo simulation study is performed as well. A quite general theory is possible based on a coupling construction, as well as a key contiguity argument for the multinomial and multivariate hypergeometric distributions.

Article information

Ann. Statist., Volume 41, Number 2 (2013), 484-507.

First available in Project Euclid: 16 April 2013

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62E20: Asymptotic distribution theory
Secondary: 62G10: Hypothesis testing

Behrens–Fisher problem coupling permutation test


Chung, EunYi; Romano, Joseph P. Exact and asymptotically robust permutation tests. Ann. Statist. 41 (2013), no. 2, 484--507. doi:10.1214/13-AOS1090.

Export citation


  • [1] Chung, E. and Romano, P. J. (2011). Asymptotically valid and exact permutation tests based on two-sample $U$-statistics. Technical Report 2011-09, Dept. Statistics, Stanford Univ.
  • [2] Chung, E. and Romano, P. J. (2013). Supplement to “Exact and asymptotically robust permutation tests.” DOI:10.1214/13-AOS1090SUPP.
  • [3] Delaigle, A., Hall, P. and Jin, J. (2011). Robustness and accuracy of methods for high dimensional data analysis based on Student’s $t$-statistic. J. R. Stat. Soc. Ser. B Stat. Methodol. 73 283–301.
  • [4] Devroye, L. P. and Wagner, T. J. (1980). The strong uniform consistency of kernel density estimates. In Multivariate Analysis, V (Proc. Fifth Internat. Sympos., Univ. Pittsburgh, Pittsburgh, Pa., 1978) 59–77. North-Holland, Amsterdam.
  • [5] Efron, B. (1979). Bootstrap methods: Another look at the jackknife. Ann. Statist. 7 1–26.
  • [6] Hall, P., DiCiccio, T. J. and Romano, J. P. (1989). On smoothing and the bootstrap. Ann. Statist. 17 692–704.
  • [7] Hall, P. and Wilson, S. R. (1991). Two guidelines for bootstrap hypothesis testing. Biometrics 47 757–762.
  • [8] Hoeffding, W. (1952). The large-sample power of tests based on permutations of observations. Ann. Math. Statist. 23 169–192.
  • [9] Janssen, A. (1997). Studentized permutation tests for non-i.i.d. hypotheses and the generalized Behrens–Fisher problem. Statist. Probab. Lett. 36 9–21.
  • [10] Janssen, A. (1999). Testing nonparametric statistical functionals with applications to rank tests. J. Statist. Plann. Inference 81 71–93.
  • [11] Janssen, A. (2005). Resampling Student’s $t$-type statistics. Ann. Inst. Statist. Math. 57 507–529.
  • [12] Janssen, A. and Pauls, T. (2003). How do bootstrap and permutation tests work? Ann. Statist. 31 768–806.
  • [13] Janssen, A. and Pauls, T. (2005). A Monte Carlo comparison of studentized bootstrap and permutation tests for heteroscedastic two-sample problems. Comput. Statist. 20 369–383.
  • [14] Janssen, A. and Pauly, M. (2009). Asymptotics and effectiveness of conditional tests with applications to randomization tests. Technical report, Univ. Duesseldorf.
  • [15] Krishnamoorthy, K., Lu, F. and Mathew, T. (2007). A parametric bootstrap approach for ANOVA with unequal variances: Fixed and random models. Comput. Statist. Data Anal. 51 5731–5742.
  • [16] Lehmann, E. L. and Romano, J. P. (2005). Testing Statistical Hypotheses, 3rd ed. Springer, New York.
  • [17] Neubert, K. and Brunner, E. (2007). A studentized permutation test for the non-parametric Behrens–Fisher problem. Comput. Statist. Data Anal. 51 5192–5204.
  • [18] Neuhaus, G. (1993). Conditional rank tests for the two-sample problem under random censorship. Ann. Statist. 21 1760–1779.
  • [19] Omelka, M. and Pauly, M. (2012). Testing equality of correlation coefficients in a potentially unbalanced two-sample problem via permutation methods. J. Statist. Plann. Inference 142 1396–1406.
  • [20] Pauly, M. (2011). Discussion about the quality of F-ratio resampling tests for comparing variances. TEST 20 163–179.
  • [21] Rice, W. R. and Gaines, S. D. (1989). One-way analysis of variance with unequal variances. Proc. Natl. Acad. Sci. USA 86 8183–8184.
  • [22] Romano, J. P. (1989). Bootstrap and randomization tests of some nonparametric hypotheses. Ann. Statist. 17 141–159.
  • [23] Romano, J. P. (1990). On the behavior of randomization tests without a group invariance assumption. J. Amer. Statist. Assoc. 85 686–692.
  • [24] Serfling, R. J. (1980). Approximation Theorems of Mathematical Statistics. Wiley, New York.

Supplemental materials