The Annals of Statistics

Asymptotic properties of bridge estimators in sparse high-dimensional regression models

Jian Huang, Joel L. Horowitz, and Shuangge Ma

Full-text: Open access

Abstract

We study the asymptotic properties of bridge estimators in sparse, high-dimensional, linear regression models when the number of covariates may increase to infinity with the sample size. We are particularly interested in the use of bridge estimators to distinguish between covariates whose coefficients are zero and covariates whose coefficients are nonzero. We show that under appropriate conditions, bridge estimators correctly select covariates with nonzero coefficients with probability converging to one and that the estimators of nonzero coefficients have the same asymptotic distribution that they would have if the zero coefficients were known in advance. Thus, bridge estimators have an oracle property in the sense of Fan and Li [J. Amer. Statist. Assoc. 96 (2001) 1348–1360] and Fan and Peng [Ann. Statist. 32 (2004) 928–961]. In general, the oracle property holds only if the number of covariates is smaller than the sample size. However, under a partial orthogonality condition in which the covariates of the zero coefficients are uncorrelated or weakly correlated with the covariates of nonzero coefficients, we show that marginal bridge estimators can correctly distinguish between covariates with nonzero and zero coefficients with probability converging to one even when the number of covariates is greater than the sample size.

Article information

Source
Ann. Statist. Volume 36, Number 2 (2008), 587-613.

Dates
First available in Project Euclid: 13 March 2008

Permanent link to this document
https://projecteuclid.org/euclid.aos/1205420512

Digital Object Identifier
doi:10.1214/009053607000000875

Mathematical Reviews number (MathSciNet)
MR2396808

Zentralblatt MATH identifier
1133.62048

Subjects
Primary: 62J05: Linear regression 62J07: Ridge regression; shrinkage estimators
Secondary: 62E20: Asymptotic distribution theory 60F05: Central limit and other weak theorems

Keywords
Penalized regression high-dimensional data variable selection asymptotic normality oracle property

Citation

Huang, Jian; Horowitz, Joel L.; Ma, Shuangge. Asymptotic properties of bridge estimators in sparse high-dimensional regression models. Ann. Statist. 36 (2008), no. 2, 587--613. doi:10.1214/009053607000000875. https://projecteuclid.org/euclid.aos/1205420512


Export citation

References

  • Bair, E., Hastie, T., Paul, D. and Tibshirani, R. (2006). Prediction by supervised principal components. J. Amer. Statist. Assoc. 101 119–137.
  • Bülhman, P. (2006). Boosting for high-dimensional linear models. Ann. Statist. 34 559–583.
  • Fan, J. (1997). Comment on “Wavelets in statistics: A review,” by A. Antoniadis. J. Italian Statist. Soc. 6 131–138.
  • Fan, J. and Li, R. (2001). Variable selection via nonconcave penalized likelihood and its oracle properties. J. Amer. Statist. Assoc. 96 1348–1360.
  • Fan, J. and Li, R. (2006). Statistical challenges with high dimensionality: Feature extraction in knowledge discovery. In International Congress of Mathematicians III 595–622. Eur. Math. Soc., Zürich.
  • Fan, J. and Lv, J. (2006). Sure independence screening for ultra-high dimensional feature space. Preprint, Dept. Operational Research and Financial Engineering, Princeton Univ.
  • Fan, J. and Peng, H. (2004). Nonconcave penalized likelihood with a diverging number of parameters. Ann. Statist. 32 928–961.
  • Fan, J., Peng, H. and Huang, T. (2005). Semilinear high-dimensional model for normalization of microarray data: A theoretical analysis and partial consistency (with discussion). J. Amer. Statist. Assoc. 100 781–813.
  • Frank, I. E. and Friedman, J. H. (1993). A statistical view of some chemometrics regression tools (with discussion). Technometrics 35 109–148.
  • Fu, W. J. (1998). Penalized regressions: The bridge versus the Lasso. J. Comput. Graph. Statist. 7 397–416.
  • Hoerl, A. E. and Kennard, R. W. (1970). Ridge regression: Biased estimation for nonorthogonal problems. Technometrics 12 55–67.
  • Huang, J., Wang, D. L. and Zhang, C.-H. (2005). A two-way semilinear model for normalization and analysis of cDNA microarray data. J. Amer. Statist. Assoc. 100 814–829.
  • Huang, J. and Zhang, C.-H. (2005). Asymptotic analysis of a two-way semiparametric regression model for microarray data. Statist. Sinica 15 597–618.
  • Huber, P. J. (1981). Robust Statistics. Wiley, New York.
  • Hunter, D. R. and Li, R. (2005). Variable selection using MM algorithms. Ann. Statist. 33 1617–1642.
  • Knight, K. and Fu, W. J. (2000). Asymptotics for lasso-type estimators. Ann. Statist. 28 1356–1378.
  • Kosorok, M. R. and Ma, S. (2007). Marginal asymptotics for the “large p, small n” paradigm: With applications to microarray data. Ann. Statist. 35 1456–1486.
  • Portnoy, S. (1984). Asymptotic behavior of M estimators of p regression parameters when p2/n is large. I. Consistency. Ann. Statist. 12 1298–1309.
  • Portnoy, S. (1985). Asymptotic behavior of M estimators of p regression parameters when p2/n is large. II. Normal approximation. Ann. Statist. 13 1403–1417.
  • Tibshirani, R. (1996). Regression shrinkage and selection via the Lasso. J. Roy. Statist. Soc. Ser. B 58 267–288.
  • van der Laan, M. J. and Bryan, J. (2001). Gene expression analysis with the parametric bootstrap. Biostatistics 2 445–461.
  • Van der Vaart, A. W. and Wellner, J. A. (1996). Weak Convergence and Empirical Processes: With Applications to Statistics. Springer, New York.
  • Zhao, P. and YU, B. (2006). On model selection consistency of Lasso. J. Mach. Learn. Res. 7 2541–2563.
  • Zou, H. and Hastie, T. (2005). Regularization and variable selection via the elastic net. J. Roy. Statist. Soc. Ser. B 67 301–320.