The Annals of Applied Statistics

Sparse estimation of large covariance matrices via a nested Lasso penalty

Elizaveta Levina, Adam Rothman, and Ji Zhu

Full-text: Access has been disabled (more information)

Abstract

The paper proposes a new covariance estimator for large covariance matrices when the variables have a natural ordering. Using the Cholesky decomposition of the inverse, we impose a banded structure on the Cholesky factor, and select the bandwidth adaptively for each row of the Cholesky factor, using a novel penalty we call nested Lasso. This structure has more flexibility than regular banding, but, unlike regular Lasso applied to the entries of the Cholesky factor, results in a sparse estimator for the inverse of the covariance matrix. An iterative algorithm for solving the optimization problem is developed. The estimator is compared to a number of other covariance estimators and is shown to do best, both in simulations and on a real data example. Simulations show that the margin by which the estimator outperforms its competitors tends to increase with dimension.

Article information

Source
Ann. Appl. Stat. Volume 2, Number 1 (2008), 245-263.

Dates
First available in Project Euclid: 24 March 2008

Permanent link to this document
http://projecteuclid.org/euclid.aoas/1206367820

Digital Object Identifier
doi:10.1214/07-AOAS139

Mathematical Reviews number (MathSciNet)
MR2415602

Zentralblatt MATH identifier
1137.62338

Keywords
Covariance matrix high dimension low sample size large p small n Lasso sparsity Cholesky decomposition

Citation

Levina, Elizaveta; Rothman, Adam; Zhu, Ji. Sparse estimation of large covariance matrices via a nested Lasso penalty. Ann. Appl. Stat. 2 (2008), no. 1, 245--263. doi:10.1214/07-AOAS139. http://projecteuclid.org/euclid.aoas/1206367820.


Export citation

References

  • Adam, B., Qu, Y., Davis, J., Ward, M., Clements, M., Cazares, L., Semmes, O., Schellhammer, P., Yasui, Y., Feng, Z. and Wright, G. (2002). Serum protein fingerprinting coupled with a pattern-matching algorithm distinguishes prostate cancer from benign prostate hyperplasia and healthy men., Cancer Research 62 3609–3614.
  • Anderson, T. W. (1958)., An Introduction to Multivariate Statistical Analysis. Wiley, New York.
    Mathematical Reviews (MathSciNet): MR91588
  • Banerjee, O., d’Aspremont, A. and El Ghaoui, L. (2006). Sparse covariance selection via robust maximum likelihood estimation. In, Proceedings of ICML.
  • Bickel, P. J. and Levina, E. (2004). Some theory for Fisher’s linear discriminant function, “naive Bayes,” and some alternatives when there are many more variables than observations., Bernoulli 10 989–1010.
    Mathematical Reviews (MathSciNet): MR2108040
    Digital Object Identifier: doi:10.3150/bj/1106314847
    Project Euclid: euclid.bj/1106314847
  • Bickel, P. J. and Levina, E. (2007). Regularized estimation of large covariance matrices., Ann. Statist. To appear.
    Mathematical Reviews (MathSciNet): MR2387969
    Zentralblatt MATH: 1132.62040
    Digital Object Identifier: doi:10.1214/009053607000000758
    Project Euclid: euclid.aos/1201877299
  • Dempster, A. (1972). Covariance selection., Biometrics 28 157–175.
  • Dey, D. K. and Srinivasan, C. (1985). Estimation of a covariance matrix under Stein’s loss., Ann. Statist. 13 1581–1591.
    Mathematical Reviews (MathSciNet): MR811511
    Zentralblatt MATH: 0582.62042
    Digital Object Identifier: doi:10.1214/aos/1176349756
    Project Euclid: euclid.aos/1176349756
  • Diggle, P. and Verbyla, A. (1998). Nonparametric estimation of covariance structure in longitudinal data., Biometrics 54 401–415.
  • Djavan, B., Zlotta, A., Kratzik, C., Remzi, M., Seitz, C., Schulman, C. and Marberger, M. (1999). Psa, psa density, psa density of transition zone, free/total psa ratio, and psa velocity for early detection of prostate cancer in men with serum psa 2.5 to 4.0 ng/ml., Urology 54 517–522.
  • Fan, J., Fan, Y. and Lv, J. (2008). High dimensional covariance matrix estimation using a factor model., J. Econometrics. To appear.
    Mathematical Reviews (MathSciNet): MR2472991
    Digital Object Identifier: doi:10.1016/j.jeconom.2008.09.017
  • Fan, J. and Li, R. (2001). Variable selection via nonconcave penalized likelihood and its oracle properties., J. Amer. Statist. Assoc. 96 1348–1360.
    Mathematical Reviews (MathSciNet): MR1946581
    Zentralblatt MATH: 1073.62547
    Digital Object Identifier: doi:10.1198/016214501753382273
    JSTOR: links.jstor.org
  • Friedman, J. (1989). Regularized discriminant analysis., J. Amer. Statist. Assoc. 84 165–175.
    Mathematical Reviews (MathSciNet): MR999675
    Digital Object Identifier: doi:10.2307/2289860
    JSTOR: links.jstor.org
  • Friedman, J., Hastie, T., Höfling, H. G. and Tibshirani, R. (2007). Pathwise coordinate optimization., Ann. Appl. Statist. 1 302–332.
    Mathematical Reviews (MathSciNet): MR2415737
    Zentralblatt MATH: 05226935
    Digital Object Identifier: doi:10.1214/07-AOAS131
    Project Euclid: euclid.aoas/1196438020
  • Fu, W. (1998). Penalized regressions: The bridge versus the lasso., J. Comput. Graph. Statist. 7 397–416.
    Mathematical Reviews (MathSciNet): MR1646710
    Digital Object Identifier: doi:10.2307/1390712
    JSTOR: links.jstor.org
  • Furrer, R. and Bengtsson, T. (2007). Estimation of high-dimensional prior and posterior covariance matrices in Kalman filter variants., J. Multivariate Anal. 98 227–255.
    Mathematical Reviews (MathSciNet): MR2301751
    Digital Object Identifier: doi:10.1016/j.jmva.2006.08.003
  • Haff, L. R. (1980). Empirical Bayes estimation of the multivariate normal covariance matrix., Ann. Statist. 8 586–597.
    Mathematical Reviews (MathSciNet): MR568722
    Zentralblatt MATH: 0441.62045
    Digital Object Identifier: doi:10.1214/aos/1176345010
    Project Euclid: euclid.aos/1176345010
  • Hastie, T., Tibshirani, R. and Friedman, J. (2001)., The Elements of Statistical Learning. Springer, Berlin.
    Mathematical Reviews (MathSciNet): MR1851606
  • Hoerl, A. E. and Kennard, R. W. (1970). Ridge regression: Biased estimation for nonorthogonal problems., Technometrics 12 55–67.
  • Huang, J., Liu, N., Pourahmadi, M. and Liu, L. (2006). Covariance matrix selection and estimation via penalised normal likelihood., Biometrika 93 85–98.
    Mathematical Reviews (MathSciNet): MR2277742
    Zentralblatt MATH: 1152.62346
    Digital Object Identifier: doi:10.1093/biomet/93.1.85
  • Johnstone, I. M. (2001). On the distribution of the largest eigenvalue in principal components analysis., Ann. Statist. 29 295–327.
    Mathematical Reviews (MathSciNet): MR1863961
    Zentralblatt MATH: 1016.62078
    Digital Object Identifier: doi:10.1214/aos/1009210544
    Project Euclid: euclid.aos/1009210544
  • Johnstone, I. M. and Lu, A. Y. (2007). Sparse principal components analysis., J. Amer. Statist. Assoc. To appear.
    Mathematical Reviews (MathSciNet): MR2751448
    Digital Object Identifier: doi:10.1198/jasa.2009.0121
  • Ledoit, O. and Wolf, M. (2003). A well-conditioned estimator for large-dimensional covariance matrices., J. Multivariate Anal. 88 365–411.
    Mathematical Reviews (MathSciNet): MR2026339
    Zentralblatt MATH: 1032.62050
    Digital Object Identifier: doi:10.1016/S0047-259X(03)00096-4
  • Mardia, K. V., Kent, J. T. and Bibby, J. M. (1979)., Multivariate Analysis. Academic Press, New York.
    Mathematical Reviews (MathSciNet): MR560319
  • Pannek, J. and Partin, A. (1998). The role of psa and percent free psa for staging and prognosis prediction in clinically localized prostate cancer., Semin. Urol. Oncol. 16 100–105.
  • Pourahmadi, M. (1999). Joint mean-covariance models with applications to longitudinal data: Unconstrained parameterisation., Biometrika 86 677–690.
    Mathematical Reviews (MathSciNet): MR1723786
    Zentralblatt MATH: 0949.62066
    Digital Object Identifier: doi:10.1093/biomet/86.3.677
    JSTOR: links.jstor.org
  • Smith, M. and Kohn, R. (2002). Parsimonious covariance matrix estimation for longitudinal data., J. Amer. Statist. Assoc. 97 1141–1153.
    Mathematical Reviews (MathSciNet): MR1951266
    Zentralblatt MATH: 1041.62044
    Digital Object Identifier: doi:10.1198/016214502388618942
    JSTOR: links.jstor.org
  • Stamey, T., Johnstone, I., McNeal, J., Lu, A. and Yemoto, C. (2002). Preoperative serum prostate specific antigen levels between 2 and 22 ng/ml correlate poorly with post-radical prostatectomy cancer morphology: Prostate specific antigen cure rates appear constant between 2 and 9 ng/ml., J. Urol. 167 103–111.
  • Tibshirani, R. (1996). Regression shrinkage and selection via the lasso., J. Roy. Statist. Soc. Ser. B 58 267–288.
    Mathematical Reviews (MathSciNet): MR1379242
    JSTOR: links.jstor.org
  • Tibshirani, R., Hastie, T., Narasimhan, B. and Chu, G. (2003). Class prediction by nearest shrunken centroids, with applications to DNA microarrays., Statist. Sci. 18 104–117.
    Mathematical Reviews (MathSciNet): MR1997067
    Digital Object Identifier: doi:10.1214/ss/1056397488
    Project Euclid: euclid.ss/1056397488
    Zentralblatt MATH: 1048.62109
  • Tibshirani, R., Saunders, M., Rosset, S., Zhu, J. and Knight, K. (2005). Sparsity and smoothness via the fused lasso., J. Roy. Statist. Soc. Ser. B 67 91–108.
    Mathematical Reviews (MathSciNet): MR2136641
    Zentralblatt MATH: 1060.62049
    Digital Object Identifier: doi:10.1111/j.1467-9868.2005.00490.x
    JSTOR: links.jstor.org
  • Wong, F., Carter, C. and Kohn, R. (2003). Efficient estimation of covariance selection models., Biometrika 90 809–830.
    Mathematical Reviews (MathSciNet): MR2024759
    Digital Object Identifier: doi:10.1093/biomet/90.4.809
  • Wu, W. B. and Pourahmadi, M. (2003). Nonparametric estimation of large covariance matrices of longitudinal data., Biometrika 90 831–844.
    Mathematical Reviews (MathSciNet): MR2024760
    Digital Object Identifier: doi:10.1093/biomet/90.4.831
  • Yuan, M. and Lin, Y. (2007). Model selection and estimation in the Gaussian graphical model., Biometrika 94 19–35.
    Mathematical Reviews (MathSciNet): MR2367824
    Zentralblatt MATH: 1142.62408
    Digital Object Identifier: doi:10.1093/biomet/asm018
  • Zhao, P., Rocha, G. and Yu, B. (2006). Grouped and hierarchical model selection through composite absolute penalties. Technical Report 703, Dept. Statistics, UC, Berkeley.

The Institute of Mathematical Statistics

The Annals of Applied Statistics

New content alerts

Email RSS ToC RSS Article
  • You have access to this content.
  • You have partial access to this content.
  • You do not have access to this content.
Turn Off MathJax

What is MathJax?

More like this

+ See more