The Annals of Applied Statistics

A simple forward selection procedure based on false discovery rate control

Yoav Benjamini and Yulia Gavrilov

Full-text: Open access


We propose the use of a new false discovery rate (FDR) controlling procedure as a model selection penalized method, and compare its performance to that of other penalized methods over a wide range of realistic settings: nonorthogonal design matrices, moderate and large pool of explanatory variables, and both sparse and nonsparse models, in the sense that they may include a small and large fraction of the potential variables (and even all). The comparison is done by a comprehensive simulation study, using a quantitative framework for performance comparisons in the form of empirical minimaxity relative to a “random oracle”: the oracle model selection performance on data dependent forward selected family of potential models. We show that FDR based procedures have good performance, and in particular the newly proposed method, emerges as having empirical minimax performance. Interestingly, using FDR level of 0.05 is a global best.

Article information

Ann. Appl. Stat. Volume 3, Number 1 (2009), 179-198.

First available in Project Euclid: 16 April 2009

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Linear regression multiple testing random oracle


Benjamini, Yoav; Gavrilov, Yulia. A simple forward selection procedure based on false discovery rate control. Ann. Appl. Stat. 3 (2009), no. 1, 179--198. doi:10.1214/08-AOAS194.

Export citation


  • Abramovich, F. and Benjamini, Y. (1995). Thresholding of wavelet coefficients as multiple hypotheses testing procedure. In Wavelets and Statistics. Lecture Notes in Statistics 103 5–14. Springer, Berlin.
  • Abramovich, F., Benjamini, Y., Donoho, D. and Johnstone, I. (2006). Adapting to unknown sparsity by controlling the false discovery rate. Ann. Statist. 34 584–653.
  • Akaike, H. (1973). Information theory and an extension of the maximum likelihood principle. In Second International Symposium on Information Theory (B. N. Petrov and F. Czaki, eds.) 267–281. Akademia Kiadó, Budapest.
  • Andrews, D. F., Bickel, P. J., Hampel, F. R., Huber, P. J., Rogers, W. H. and Tukey, J. W. (1972). Robust Estimates of Location: Survey and Advances. Princeton Univ. Press, Princeton, NJ.
  • Benjamini, Y. and Gavrilov, Y. (2009). Supplement to “A simple forward selection procedure based on false discovery rate control.” DOI: 10.1214/08-AOAS194SUPP.
  • Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. J. Roy. Statist. Soc. Ser. B 57 289–300.
  • Benjamini, Y. and Hochberg, Y. (2000). On the adaptive control of the false discovery fate in multiple testing with independent statistics. J. Educ. Behav. Stat. 25 60–83.
  • Benjamini, Y., Krieger, A. and Yekutieli, D. (2006). Adaptive linear step-up procedures that control the false discovery rate. Biometrica 93 491–507.
  • Bickel, P., Ritov, Y. and Tsybakov, A. (2008a). Simultaneous analysis of Lasso and Dantzig selector. Ann. Statist. To appear. Available at
  • Bickel, P., Ritov, Y. and Tsybakov, A. (2008b). Hierarchical selection of variables in sparse high-dimensional regression. Working paper. Available at
  • Birgé, L. and Massart, P. (2001). A generalized Cp criterion for Gaussian model. Technical report, Lab. De Probabilitiés, Univ. Paris VI. Available at
  • Candes, E. and Tao, T. (2007). The Dantzig selector: Statistical estimation when p is much larger than n. Ann. Statist. 35 2313–2351.
  • Cochran, W. G. (1977). Sampling Techniques, 3rd ed. Wiley, New York.
  • Donoho, D. L. and Johnstone, I. M. (1994). Ideal spatial adaptation by wavelet shrinkage. Biometrika 81 425–455.
  • Draper, N. R. and Smith, H. (1998). Applied Regression Analyses, 3rd ed. Wiley, New York.
  • Efron, B., Hastie, T., Johnstone, I. and Tibshirani, R. (2004). Least angle regression. Ann. Statist. 32 407–499.
  • Finner, H., Dickhaus, T. and Roters, M. (2009). On the false discovery rate and asymptotically optimal rejection curve. Ann. Statist. To appear.
  • Foster, D. and Stine, R. (2004). Variable selection in data mining: Building a predictive model for bankruptcy. J. Amer. Statist. Assoc. 99 303–313.
  • Gavrilov, Y., Benjamini, Y. and Sarkar, S. (2009). An adaptive step-down procedure with proven FDR control under independence. Ann. Statist. 37 619–629.
  • Genovese, C. and Wasserman, L. (2002). Operating characteristics and extensions of the false discovery rate procedure. J. Roy. Statist. Soc. Ser. B 64 499–517.
  • George, E. I. and Foster, D. P. (2000). Calibration and empirical Bayes variable selection. Biometrika 87 731–747.
  • Johnstone, I. and Silverman, B. W. (2005). Empirical Bayes selection of wavelet thresholds. Ann. Statist. 33 1700–1752.
  • Mallows, C. L. (1973). Some comments on Cp. Technometrics 12 661–675.
  • Sarkar, S. K. (2002). Some results on false discovery rate in stepwise multiple testing procedures. Ann. Statist. 30 239–257.
  • Shen, X. and Ye, J. (2002). Adaptive model selection. J. Amer. Statist. Assoc. 97 210–221.
  • Storey, J. D. (2002). A direct approach to false discovery rates. J. Roy. Statist. Soc. Ser. B 64 479–98.
  • Storey, J. D., Taylor, J. E. and Siegmund, D. (2004). Strong control, conservative point estimation and simultaneous conservative consistency of false discovery rates: A unified approach. J. Roy. Statist. Soc. Ser. B 66 187–205.
  • Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. J. Roy. Statist. Soc. Ser. B 58 267–288.
  • Tibshirani, R. and Knight, K. (1999). The covariance inflation criterion for adaptive model selection. J. Roy. Statist. Soc. Ser. B 61 529–546.
  • Wu, Y., Boos, D. D. and Stefanski, L. A. (2007). Controlling variable selection by the addition of pseudovariables. J. Amer. Statist. Assoc. 102 235–243.
  • Yuan, M., Ekici, A., Lu, Z. and Monteiro, R. (2007). Dimension reduction and coefficient estimation in multivariate linear regression. J. Roy. Statist. Soc. Ser. B 69 329–346.

Supplemental materials