The Annals of Statistics

Rank verification for exponential families

Kenneth Hung and William Fithian

Full-text: Access denied (no subscription detected)

We're sorry, but we are unable to provide you with the full text of this article because we are not able to identify you as a subscriber. If you have a personal subscription to this journal, then please login. If you are already logged in, then you may need to update your profile to register your subscription. Read more about accessing full-text


Many statistical experiments involve comparing multiple population groups. For example, a public opinion poll may ask which of several political candidates commands the most support; a social scientific survey may report the most common of several responses to a question; or, a clinical trial may compare binary patient outcomes under several treatment conditions to determine the most effective treatment. Having observed the “winner” (largest observed response) in a noisy experiment, it is natural to ask whether that candidate, survey response or treatment is actually the “best” (stochastically largest response). This article concerns the problem of rank verification—post hoc significance tests of whether the orderings discovered in the data reflect the population ranks. For exponential family models, we show under mild conditions that an unadjusted two-tailed pairwise test comparing the first two-order statistics (i.e., comparing the “winner” to the “runner-up”) is a valid test of whether the winner is truly the best. We extend our analysis to provide equally simple procedures to obtain lower confidence bounds on the gap between the winning population and the others, and to verify ranks beyond the first.

Article information

Ann. Statist., Volume 47, Number 2 (2019), 758-782.

Received: February 2017
Revised: June 2017
First available in Project Euclid: 11 January 2019

Permanent link to this document

Digital Object Identifier

Primary: 62F07: Ranking and selection
Secondary: 62F03: Hypothesis testing

Ranking selective inference exponential family multiple comparison sample best


Hung, Kenneth; Fithian, William. Rank verification for exponential families. Ann. Statist. 47 (2019), no. 2, 758--782. doi:10.1214/17-AOS1634.

Export citation


  • Alikhani, L. (2011). Study: Tween TV today is all about fame. Available at
  • Berger, R. L. (1980). Minimax subset selection for the multinomial distribution. J. Statist. Plann. Inference 4 391–402.
  • Berger, R. L. (1982). Multiparameter hypothesis testing and acceptance sampling. Technometrics 24 295–300.
  • Besag, J. and Clifford, P. (1989). Generalized Monte Carlo significance tests. Biometrika 76 633–642.
  • Bofinger, E. (1991). Selecting “demonstrably best” or “demonstrably worst” exponential populations. Aust. N. Z. J. Stat. 33 183–190.
  • Edwards, D. G. and Hsu, J. C. (1983). Multiple comparisons with the best treatment. J. Amer. Statist. Assoc. 78 965–971.
  • Finner, H. and Strassburger, K. (2002). The partitioning principle: A powerful tool in multiple decision theory. Ann. Statist. 30 1194–1213.
  • Fithian, W., Sun, D. L. and Taylor, J. E. (2014). Optimal inference after model selection. Preprint. Available at arXiv:1410.2597.
  • Fithian, W., Taylor, J. E. and Tibshirani, R. J. (2015). Selective sequential model selection. Preprint. Available at arXiv:1512.02565.
  • Gupta, S. S., Huang, D.-Y. and Panchapakesan, S. (1984). On some inequalities and monotonicity results in selection and ranking theory. In Inequalities in Statistics and Probability (Lincoln, Neb., 1982) 211–227. IMS, Hayward, CA.
  • Gupta, S. S. and Liang, T. (1989). Selecting the best binomial population: Parametric empirical Bayes approach. J. Statist. Plann. Inference 23 21–31.
  • Gupta, S. S. and Nagel, K. (1967). On selection and ranking procedures and order statistics from the multinomial distribution. Sankhyā 29 1–34.
  • Gupta, S. S. and Panchapakesan, S. (1971). On multiple decision (subset selection) procedures. Technical report, Purdue Univ., West Lafayette, IN.
  • Gupta, S. S. and Panchapakesan, S. (1985). Subset selection procedures: Review and assessment. Amer. J. Math. Management Sci. 5 235–311.
  • Gupta, S. S. and Wong, W.-Y. (1976). On subset selection procedures for Poisson processes and some applications to the binomial and multinomial problems. Technical report.
  • Gutmann, S. and Maymin, Z. (1987). Is the selected population the best? Ann. Statist. 15 456–461.
  • Hsu, J. C. (1984). Constrained simultaneous confidence intervals for multiple comparisons with the best. Ann. Statist. 12 1136–1144.
  • Hsu, J. (1996). Multiple Comparisons: Theory and Methods. CRC Press, Boca Raton, FL.
  • Kannan, N. and Panchapakesan, S. (2009). Does the selected normal population have the smallest variance? Amer. J. Math. Management Sci. 29 109–123.
  • Marshall, A. W., Olkin, I. and Arnold, B. C. (2011). Inequalities: Theory of Majorization and Its Applications, 2nd ed. Springer, New York.
  • Maymin, Z. and Gutmann, S. (1992). Testing retrospective hypotheses. Canad. J. Statist. 20 335–345.
  • Nettleton, D. (2009). Testing for the supremacy of a multinomial cell probability. J. Amer. Statist. Assoc. 104 1052–1059.
  • Ng, H. K. T. and Panchapakesan, S. (2007). Is the selected multinomial cell the best?. Sequential Anal. 26 415–423.
  • Quinnipiac University Poll Institute (2016). First-timers put Trump ahead in Iowa GOP caucus, Quinnipiac University poll finds; Sanders needs first-timers to tie Clinton in Dem caucus. Available at
  • Stefansson, G., Kim, W.-C. and Hsu, J. C. (1988). On confidence sets in multiple comparisons. In Statistical Decision Theory and Related Topics IV 2 89–104. Springer, New York.
  • Uhls, Y. T. and Greenfield, P. M. (2012). The value of fame: Preadolescent perceptions of popular media and their relationship to future aspirations. Dev. Psychol. 48 315–326.