The Annals of Statistics

Sequential multiple testing with generalized error control: An asymptotic optimality theory

Yanglei Song and Georgios Fellouris

Full-text: Access denied (no subscription detected)

We're sorry, but we are unable to provide you with the full text of this article because we are not able to identify you as a subscriber. If you have a personal subscription to this journal, then please login. If you are already logged in, then you may need to update your profile to register your subscription. Read more about accessing full-text


The sequential multiple testing problem is considered under two generalized error metrics. Under the first one, the probability of at least $k$ mistakes, of any kind, is controlled. Under the second, the probabilities of at least $k_{1}$ false positives and at least $k_{2}$ false negatives are simultaneously controlled. For each formulation, the optimal expected sample size is characterized, to a first-order asymptotic approximation as the error probabilities go to 0, and a novel multiple testing procedure is proposed and shown to be asymptotically efficient under every signal configuration. These results are established when the data streams for the various hypotheses are independent and each local log-likelihood ratio statistic satisfies a certain strong law of large numbers. In the special case of i.i.d. observations in each stream, the gains of the proposed sequential procedures over fixed-sample size schemes are quantified.

Article information

Ann. Statist., Volume 47, Number 3 (2019), 1776-1803.

Received: August 2016
Revised: April 2018
First available in Project Euclid: 13 February 2019

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62L10: Sequential analysis

Multiple testing sequential analysis asymptotic optimality generalized familywise error rates misclassification rate


Song, Yanglei; Fellouris, Georgios. Sequential multiple testing with generalized error control: An asymptotic optimality theory. Ann. Statist. 47 (2019), no. 3, 1776--1803. doi:10.1214/18-AOS1737.

Export citation


  • [1] Bartroff, J. (2018). Multiple hypothesis tests controlling generalized error rates for sequential data. Statist. Sinica 28 363–398.
  • [2] Bartroff, J. and Lai, T. L. (2010). Multistage tests of multiple hypotheses. Comm. Statist. Theory Methods 39 1597–1607.
  • [3] Bartroff, J. and Song, J. (2014). Sequential tests of multiple hypotheses controlling type I and II familywise error rates. J. Statist. Plann. Inference 153 100–114.
  • [4] Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. J. Roy. Statist. Soc. Ser. B 57 289–300.
  • [5] Benjamini, Y. and Yekutieli, D. (2001). The control of the false discovery rate in multiple testing under dependency. Ann. Statist. 29 1165–1188.
  • [6] Bogdan, M., Chakrabarti, A., Frommlet, F. and Ghosh, J. K. (2011). Asymptotic Bayes-optimality under sparsity of some multiple testing procedures. Ann. Statist. 39 1551–1579.
  • [7] De, S. K. and Baron, M. (2012). Sequential Bonferroni methods for multiple hypothesis testing with strong control of family-wise error rates I and II. Sequential Anal. 31 238–262.
  • [8] De, S. K. and Baron, M. (2012). Step-up and step-down methods for testing multiple hypotheses in sequential experiments. J. Statist. Plann. Inference 142 2059–2070.
  • [9] De, S. K. and Baron, M. (2015). Sequential tests controlling generalized familywise error rates. Stat. Methodol. 23 88–102.
  • [10] Dembo, A. and Zeitouni, O. (1998). Large Deviations Techniques and Applications, 2nd ed. Applications of Mathematics (New York) 38. Springer, New York.
  • [11] Foresti, G. L., Regazzoni, C. S. and Varshney, P. K. (2003). Multisensor Surveillance Systems: The Fusion Perspective. Springer, New York.
  • [12] Guo, W., He, L. and Sarkar, S. K. (2014). Further results on controlling the false discovery proportion. Ann. Statist. 42 1070–1101.
  • [13] Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scand. J. Stat. 6 65–70.
  • [14] Hommel, G. (1988). A stagewise rejective multiple test procedure based on a modified Bonferroni test. Biometrika 75 383–386.
  • [15] Hommel, G. and Hoffmann, T. (1988). Controlled uncertainty. In Multiple Hypothesenprüfung/Multiple Hypotheses Testing 154–161. Springer, Berlin.
  • [16] Hsu, P. L. and Robbins, H. (1947). Complete convergence and the law of large numbers. Proc. Natl. Acad. Sci. USA 33 25–31.
  • [17] Kittur, A., Chi, E. H. and Suh, B. (2008). Crowdsourcing user studies with Mechanical Turk. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems 453–456. ACM, New York.
  • [18] Lehmann, E. L. and Romano, J. P. (2005). Generalizations of the familywise error rate. Ann. Statist. 33 1138–1154.
  • [19] Lehmann, E. L., Romano, J. P. and Shaffer, J. P. (2005). On optimality of stepdown and stepup multiple test procedures. Ann. Statist. 33 1084–1108.
  • [20] Li, Y., Nitinawarat, S. and Veeravalli, V. V. (2014). Universal outlier hypothesis testing. IEEE Trans. Inform. Theory 60 4066–4082.
  • [21] Malloy, M. L. and Nowak, R. D. (2014). Sequential testing for sparse recovery. IEEE Trans. Inform. Theory 60 7862–7873.
  • [22] Marcus, R., Peritz, E. and Gabriel, K. R. (1976). On closed testing procedures with special reference to ordered analysis of variance. Biometrika 63 655–660.
  • [23] Peña, E. A., Habiger, J. D. and Wu, W. (2011). Power-enhanced multiple decision functions controlling family-wise error and false discovery rates. Ann. Statist. 39 556–583.
  • [24] Rappaport, T. S. (1996). Wireless Communications: Principles and Practice, 2nd ed. Prentice Hall, Upper Saddle River, NJ.
  • [25] Romano, J. P. and Shaikh, A. M. (2006). Stepup procedures for control of generalizations of the familywise error rate. Ann. Statist. 34 1850–1873.
  • [26] Romano, J. P. and Wolf, M. (2007). Control of generalized error rates in multiple testing. Ann. Statist. 35 1378–1408.
  • [27] Song, Y. and Fellouris, G. (2016). Logarithmically efficient simulation for misclassification probabilities in sequential multiple testing. In 2016 Winter Simulation Conference (WSC) 314–325.
  • [28] Song, Y. and Fellouris, G. (2017). Asymptotically optimal, sequential, multiple testing procedures with prior information on the number of signals. Electron. J. Stat. 11 338–363.
  • [29] Song, Y. and Fellouris, G. (2019). Supplement to “Sequential multiple testing with generalized error control: An asymptotic optimality theory.” DOI:10.1214/18-AOS1737SUPP.
  • [30] Storey, J. D. (2007). The optimal discovery procedure: A new approach to simultaneous significance testing. J. R. Stat. Soc. Ser. B. Stat. Methodol. 69 347–368.
  • [31] Sun, W. and Cai, T. T. (2009). Large-scale multiple testing under dependence. J. R. Stat. Soc. Ser. B. Stat. Methodol. 71 393–424.
  • [32] Tartakovsky, A., Nikiforov, I. and Basseville, M. (2015). Sequential Analysis: Hypothesis Testing and Changepoint Detection. Monographs on Statistics and Applied Probability 136. CRC Press, Boca Raton, FL.
  • [33] Tartakovsky, A. G. (1998). Asymptotic optimality of certain multihypothesis sequential tests: Non-i.i.d. case. Stat. Inference Stoch. Process. 1 265–295.
  • [34] Tartakovsky, A. G., Li, X. R. and Yaralov, G. (2003). Sequential detection of targets in multichannel systems. IEEE Trans. Inform. Theory 49 425–445.
  • [35] Wald, A. (1945). Sequential tests of statistical hypotheses. Ann. Math. Stat. 16 117–186.

Supplemental materials

  • Supplement to “Sequential multiple testing with generalized error control: An asymptotic optimality theory”. In the supplementary file, we present (i) more simulation studies, (ii) proofs of all results in this paper and (iii) additional technical lemmas.