The Annals of Statistics

Asymptotic properties of the sequential empirical ROC, PPV and NPV curves under case-control sampling

Joseph S. Koopmeiners and Ziding Feng

Full-text: Open access


The receiver operating characteristic (ROC) curve, the positive predictive value (PPV) curve and the negative predictive value (NPV) curve are three measures of performance for a continuous diagnostic biomarker. The ROC, PPV and NPV curves are often estimated empirically to avoid assumptions about the distributional form of the biomarkers. Recently, there has been a push to incorporate group sequential methods into the design of diagnostic biomarker studies. A thorough understanding of the asymptotic properties of the sequential empirical ROC, PPV and NPV curves will provide more flexibility when designing group sequential diagnostic biomarker studies. In this paper, we derive asymptotic theory for the sequential empirical ROC, PPV and NPV curves under case-control sampling using sequential empirical process theory. We show that the sequential empirical ROC, PPV and NPV curves converge to the sum of independent Kiefer processes and show how these results can be used to derive asymptotic results for summaries of the sequential empirical ROC, PPV and NPV curves.

Article information

Ann. Statist., Volume 39, Number 6 (2011), 3234-3261.

First available in Project Euclid: 5 March 2012

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62L12: Sequential estimation
Secondary: 62G05: Estimation

Group sequential methods empirical process theory diagnostic testing


Koopmeiners, Joseph S.; Feng, Ziding. Asymptotic properties of the sequential empirical ROC, PPV and NPV curves under case-control sampling. Ann. Statist. 39 (2011), no. 6, 3234--3261. doi:10.1214/11-AOS937.

Export citation


  • Csörgő, M. and Szyszkowicz, B. (1998). Sequential quantile and Bahadur–Kiefer processes. In Order Statistics: Theory and Methods. Handbook of Statistics 16 631–688. North-Holland, Amsterdam.
  • Dorfman, D. and Alf, E. (1960). Maximum likelihood estimation of parameters of signal detection theory and determination of confidence intervals-rating method data. J. Math. Psych. 6 487–496.
  • Hsieh, F. and Turnbull, B. W. (1996). Nonparametric and semiparametric estimation of the receiver operating characteristic curve. Ann. Statist. 24 25–40.
  • Hwang, I. K., Shih, W. J. and De Cani, J. S. (1990). Group sequential designs using a family of type I error probability spending functions. Stat. Med. 9 1439–1445.
  • Jennison, C. and Turnbull, B. W. (2000). Group Sequential Methods With Applications to Clinical Trials. Chapman & Hall/CRC, Boca Raton, FL.
  • Koopmeiners, J. S. and Feng, Z. (2010). Asymptotic properties of the sequential empirical ROC and PPV curves. Working Paper 345, Univ. Washington Biostatistics Working Paper Series.
  • Liu, A. and Hall, W. J. (2001). Unbiased estimation of secondary parameters following a sequential test. Biometrika 88 895–900.
  • Marrero, J. A., Feng, Z., Wang, Y., Nguyen, M. H., Befeler, A. S., Roberts, L. R., Reddy, K. R., Harnois, D., Llovet, J. M., Normolle, D., Dalhgren, J., Chia, D., Lok, A. S., Wagner, P. D., Srivastava, S. and Schwartz, M. (2009). [alpha]-Fetoprotein, des-[gamma] carboxyprothrombin, and lectin-bound [alpha]-fetoprotein in early hepatocellular carcinoma. Gastroenterology 137 110–118.
  • Moskowitz, C. S. and Pepe, M. S. (2004). Quantifying and comparing the predictive accuracy of continuous prognostic factors for binary outcomes. Biostatistics 5 113–127.
  • Pepe, M. S. (2003). The Statistical Evaluation of Medical Tests for Classification and Prediction. Oxford Statistical Science Series 28. Oxford Univ. Press, Oxford.
  • Pepe, M. S., Feng, Z., Longton, G. and Koopmeiners, J. (2009). Conditional estimation of sensitivity and specificity from a phase 2 biomarker study allowing early termination for futility. Stat. Med. 28 762–779.
  • Pyke, R. and Shorack, G. R. (1968). Weak convergence of a two-sample empirical process and a new approach to Chernoff–Savage theorems. Ann. Math. Statist. 39 755–771.
  • Tang, L., Emerson, S. S. and Zhou, X.-H. (2008). Nonparametric and semiparametric group sequential methods for comparing accuracy of diagnostic tests. Biometrics 64 1137–1145.
  • Tang, L. L. and Liu, A. (2010). Sample size recalculation in sequential diagnostic trials. Biostatistics 11 151–163.
  • Zheng, Y., Cai, T., Pepe, M. S. and Levy, W. C. (2008). Time-dependent predictive values of prognostic biomarkers with failure time outcome. J. Amer. Statist. Assoc. 103 362–368.
  • Zhu, H. and Hu, F. (2010). Sequential monitoring of response-adaptive randomized clinical trials. Ann. Statist. 38 2218–2241.