The Annals of Statistics

Optimal classification in sparse Gaussian graphic model

Yingying Fan, Jiashun Jin, and Zhigang Yao

Full-text: Access has been disabled (more information)

Abstract

Consider a two-class classification problem where the number of features is much larger than the sample size. The features are masked by Gaussian noise with mean zero and covariance matrix $\Sigma$, where the precision matrix $\Omega=\Sigma^{-1}$ is unknown but is presumably sparse. The useful features, also unknown, are sparse and each contributes weakly (i.e., rare and weak) to the classification decision.

By obtaining a reasonably good estimate of $\Omega$, we formulate the setting as a linear regression model. We propose a two-stage classification method where we first select features by the method of Innovated Thresholding (IT), and then use the retained features and Fisher’s LDA for classification. In this approach, a crucial problem is how to set the threshold of IT. We approach this problem by adapting the recent innovation of Higher Criticism Thresholding (HCT).

We find that when useful features are rare and weak, the limiting behavior of HCT is essentially just as good as the limiting behavior of ideal threshold, the threshold one would choose if the underlying distribution of the signals is known (if only). Somewhat surprisingly, when $\Omega$ is sufficiently sparse, its off-diagonal coordinates usually do not have a major influence over the classification decision.

Compared to recent work in the case where $\Omega$ is the identity matrix [Proc. Natl. Acad. Sci. USA 105 (2008) 14790–14795; Philos. Trans. R. Soc. Lond. Ser. A Math. Phys. Eng. Sci. 367 (2009) 4449–4470], the current setting is much more general, which needs a new approach and much more sophisticated analysis. One key component of the analysis is the intimate relationship between HCT and Fisher’s separation. Another key component is the tight large-deviation bounds for empirical processes for data with unconventional correlation structures, where graph theory on vertex coloring plays an important role.

Article information

Source
Ann. Statist. Volume 41, Number 5 (2013), 2537-2571.

Dates
First available in Project Euclid: 19 November 2013

Permanent link to this document
http://projecteuclid.org/euclid.aos/1384871345

Digital Object Identifier
doi:10.1214/13-AOS1163

Mathematical Reviews number (MathSciNet)
MR3161437

Zentralblatt MATH identifier
1294.62061

Subjects
Primary: 62G05: Estimation
Secondary: 62G32: Statistics of extreme values; tail inference

Keywords
Chromatic number Fisher’s LDA Fisher’s separation phase diagram precision matrix rare and weak model sparse graph

Citation

Fan, Yingying; Jin, Jiashun; Yao, Zhigang. Optimal classification in sparse Gaussian graphic model. Ann. Statist. 41 (2013), no. 5, 2537--2571. doi:10.1214/13-AOS1163. http://projecteuclid.org/euclid.aos/1384871345.


Export citation

References

  • [1] Anderson, T. W. (2003). An Introduction to Multivariate Statistical Analysis, 3rd ed. Wiley, Hoboken, NJ.
    Mathematical Reviews (MathSciNet): MR1990662
  • [2] Arias-Castro, E., Candès, E. J. and Plan, Y. (2011). Global testing under sparse alternatives: ANOVA, multiple comparisons and the higher criticism. Ann. Statist. 39 2533–2556.
    Mathematical Reviews (MathSciNet): MR2906877
    Digital Object Identifier: doi:10.1214/11-AOS910
    Project Euclid: euclid.aos/1322663467
  • [3] Bickel, P. J. and Levina, E. (2004). Some theory of Fisher’s linear discriminant function, “naive Bayes”, and some alternatives when there are many more variables than observations. Bernoulli 10 989–1010.
    Mathematical Reviews (MathSciNet): MR2108040
    Digital Object Identifier: doi:10.3150/bj/1106314847
    Project Euclid: euclid.bj/1106314847
  • [4] Bickel, P. J. and Levina, E. (2008). Regularized estimation of large covariance matrices. Ann. Statist. 36 199–227.
    Mathematical Reviews (MathSciNet): MR2387969
    Digital Object Identifier: doi:10.1214/009053607000000758
    Project Euclid: euclid.aos/1201877299
  • [5] Bollobás, B. (1998). Modern Graph Theory. Graduate Texts in Mathematics 184. Springer, New York.
    Mathematical Reviews (MathSciNet): MR1633290
  • [6] Breiman, L. (2001). Random forests. Mach Learning 24 5–32.
  • [7] Burges, C. (1998). A tutorial on support vector machines for pattern recognition. Data Min. Knowl. Discov. 2 121–167.
  • [8] Cai, T. and Liu, W. (2011). A direct estimation approach to sparse linear discriminant analysis. J. Amer. Statist. Assoc. 106 1566–1577.
    Mathematical Reviews (MathSciNet): MR2896857
    Digital Object Identifier: doi:10.1198/jasa.2011.tm11199
  • [9] Cai, T., Liu, W. and Luo, X. (2011). A constrained $\ell_{1}$ minimization approach to sparse precision matrix estimation. J. Amer. Statist. Assoc. 106 594–607.
    Mathematical Reviews (MathSciNet): MR2847973
    Digital Object Identifier: doi:10.1198/jasa.2011.tm10155
  • [10] Cai, T. T., Jin, J. and Low, M. G. (2007). Estimation and confidence sets for sparse normal mixtures. Ann. Statist. 35 2421–2449.
    Mathematical Reviews (MathSciNet): MR2382653
    Digital Object Identifier: doi:10.1214/009053607000000334
    Project Euclid: euclid.aos/1201012967
  • [11] Candes, E. and Tao, T. (2007). The Dantzig selector: Statistical estimation when $p$ is much larger than $n$. Ann. Statist. 35 2313–2351.
    Mathematical Reviews (MathSciNet): MR2382644
    Digital Object Identifier: doi:10.1214/009053606000001523
    Project Euclid: euclid.aos/1201012958
  • [12] Cayon, L., Jin, J. and Treaster, A. (2005). Higher criticism statistic: Detecting and identifying non-Gaussianity in the WMAP first-year data. Mon. Not. R. Astron. Soc. 362 826–832.
  • [13] Dempster, A. P. (1972). Covariance selection. Biometrics 28 157–175.
  • [14] Dettling, M. and Bühlmann, P. (2003). Boosting for tumor classification with gene expression data. Bioinformatics 19 1061–1069.
  • [15] Donoho, D. and Jin, J. (2004). Higher criticism for detecting sparse heterogeneous mixtures. Ann. Statist. 32 962–994.
    Mathematical Reviews (MathSciNet): MR2065195
    Digital Object Identifier: doi:10.1214/009053604000000265
    Project Euclid: euclid.aos/1085408492
  • [16] Donoho, D. and Jin, J. (2008). Higher criticism thresholding: Optimal feature selection when useful features are rare and weak. Proc. Natl. Acad. Sci. USA 105 14790–14795.
  • [17] Donoho, D. and Jin, J. (2009). Feature selection by higher criticism thresholding achieves the optimal phase diagram. Philos. Trans. R. Soc. Lond. Ser. A Math. Phys. Eng. Sci. 367 4449–4470.
    Mathematical Reviews (MathSciNet): MR2546396
    Digital Object Identifier: doi:10.1098/rsta.2009.0129
  • [18] Donoho, D. L. and Johnstone, I. M. (1994). Minimax risk over $l_{p}$-balls for $l_{q}$-error. Probab. Theory Related Fields 99 277–303.
    Mathematical Reviews (MathSciNet): MR1278886
    Digital Object Identifier: doi:10.1007/BF01199026
  • [19] Efron, B. (2009). Empirical Bayes estimates for large-scale prediction problems. J. Amer. Statist. Assoc. 104 1015–1028.
    Mathematical Reviews (MathSciNet): MR2562003
    Digital Object Identifier: doi:10.1198/jasa.2009.tm08523
  • [20] Fan, J. and Fan, Y. (2008). High-dimensional classification using features annealed independence rules. Ann. Statist. 36 2605–2637.
    Mathematical Reviews (MathSciNet): MR2485009
    Digital Object Identifier: doi:10.1214/07-AOS504
    Project Euclid: euclid.aos/1231165181
  • [21] Fan, J., Feng, Y. and Tong, X. (2012). A road to classification in high dimensional space: The regularized optimal affine discriminant. J. R. Stat. Soc. Ser. B Stat. Methodol. 74 745–771.
    Mathematical Reviews (MathSciNet): MR2965958
    Digital Object Identifier: doi:10.1111/j.1467-9868.2012.01029.x
  • [22] Fan, J. and Li, R. (2001). Variable selection via nonconcave penalized likelihood and its oracle properties. J. Amer. Statist. Assoc. 96 1348–1360.
    Mathematical Reviews (MathSciNet): MR1946581
    Digital Object Identifier: doi:10.1198/016214501753382273
  • [23] Fan, Y., Jin, J. and Yao, Z. (2013). Supplement to “Optimal classification in sparse Gaussian graphic model.” DOI:10.1214/13-AOS1163SUPP.
  • [24] Fisher, R. A. (1936). The use of multiple measurements in taxonomic problems. Annals of Eugenics 7 179–188.
  • [25] Friedman, J., Hastie, T. and Tibshirani, R. (2008). Sparse inverse covariance estimation with the graphical lasso. Biostatistics 9 432–441.
  • [26] Genovese, C. R., Jin, J., Wasserman, L. and Yao, Z. (2012). A comparison of the lasso and marginal regression. J. Mach. Learn. Res. 13 2107–2143.
    Mathematical Reviews (MathSciNet): MR2956354
  • [27] Hall, P. and Jin, J. (2010). Innovated higher criticism for detecting sparse signals in correlated noise. Ann. Statist. 38 1686–1732.
    Mathematical Reviews (MathSciNet): MR2662357
    Digital Object Identifier: doi:10.1214/09-AOS764
    Project Euclid: euclid.aos/1269452652
  • [28] Hall, P., Pittelkow, Y. and Ghosh, M. (2008). Theoretical measures of relative performance of classifiers for high dimensional data with small sample sizes. J. R. Stat. Soc. Ser. B Stat. Methodol. 70 159–173.
    Mathematical Reviews (MathSciNet): MR2412636
    Digital Object Identifier: doi:10.1111/j.1467-9868.2007.00631.x
  • [29] He, S. and Wu, Z. (2012). Gene-based higher criticism methods for large-scale exonic SNP data. BMC Proceedings 5 S65.
  • [30] Ingster, Y. I. (1997). Some problems of hypothesis testing leading to infinitely divisible distributions. Math. Methods Statist. 6 47–69.
    Mathematical Reviews (MathSciNet): MR1456646
  • [31] Ingster, Y. I. (1999). Minimax detection of a signal for $l^{p}_{n}$-balls. Math. Methods Statist. 7 401–428.
    Mathematical Reviews (MathSciNet): MR1680087
  • [32] Ingster, Y. I., Pouet, C. and Tsybakov, A. B. (2009). Classification of sparse high-dimensional vectors. Philos. Trans. R. Soc. Lond. Ser. A Math. Phys. Eng. Sci. 367 4427–4448.
    Mathematical Reviews (MathSciNet): MR2546395
    Digital Object Identifier: doi:10.1098/rsta.2009.0156
  • [33] Jager, L. and Wellner, J. A. (2007). Goodness-of-fit tests via phi-divergences. Ann. Statist. 35 2018–2053.
    Mathematical Reviews (MathSciNet): MR2363962
    Digital Object Identifier: doi:10.1214/0009053607000000244
    Project Euclid: euclid.aos/1194461721
  • [34] Ji, P. and Jin, J. (2012). UPS delivers optimal phase diagram in high-dimensional variable selection. Ann. Statist. 40 73–103.
    Mathematical Reviews (MathSciNet): MR3013180
    Digital Object Identifier: doi:10.1214/11-AOS947
    Project Euclid: euclid.aos/1331830775
  • [35] Jin, J. (2009). Impossibility of successful classification when useful features are rare and weak. Proc. Natl. Acad. Sci. USA 106 8859–8864.
    Mathematical Reviews (MathSciNet): MR2520682
    Digital Object Identifier: doi:10.1073/pnas.0903931106
  • [36] Jin, J. and Wang, W. (2012). Optimal spectral clustering by higher criticism thresholding. Unpublished manuscript.
  • [37] Li, C. and Li, H. (2008). Network-constrained regularization and variable selection for analysis of genomic data. Bioinformatics 24 1175–1182.
  • [38] Ravikumar, P., Wainwright, M. J., Raskutti, G. and Yu, B. (2011). High-dimensional covariance estimation by minimizing $\ell_{1}$-penalized log-determinant divergence. Electron. J. Stat. 5 935–980.
    Mathematical Reviews (MathSciNet): MR2836766
    Digital Object Identifier: doi:10.1214/11-EJS631
    Project Euclid: euclid.ejs/1316092865
  • [39] Sabatti, C., Service, S. and Hartikainen, A. t. (2008). Genome-wide association analysis of metabolic traits in a birth cohort from a founder population. Nat. Genet. 41 35–46.
  • [40] Shao, J., Wang, Y., Deng, X. and Wang, S. (2011). Sparse linear discriminant analysis by thresholding for high dimensional data. Ann. Statist. 39 1241–1265.
    Mathematical Reviews (MathSciNet): MR2816353
    Digital Object Identifier: doi:10.1214/10-AOS870
    Project Euclid: euclid.aos/1304947049
  • [41] Shorack, G. R. and Wellner, J. A. (1986). Empirical Processes with Applications to Statistics. Wiley, New York.
    Mathematical Reviews (MathSciNet): MR838963
  • [42] Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. J. R. Stat. Soc. Ser. B Stat. Methodol. 58 267–288.
    Mathematical Reviews (MathSciNet): MR1379242
  • [43] Tibshirani, R., Hastie, T., Narasimhan, B. and Chu, G. (2002). Diagnosis of multiple cancer types by shrunken centroids of gene expression. Proc. Natl. Acad. Sci. USA 99 6567–6572.
  • [44] Tukey, J. W. (1976). T13 N: The higher criticism. Course notes, Stat. 411, Princeton Univ.
  • [45] Zhong, P., Chen, S. and Xu, M. (2012). Alternative tests to higher criticism for high dimensional means under sparsity and column-wise dependency. Unpublished manuscript.

Supplemental materials

  • Supplementary material: Supplementary material for “Optimal classification in sparse Gaussian graphic model”. We include all technical proofs omitted from the main text.
    Digital Object Identifier: doi:10.1214/13-AOS1163SUPP
    Supplemental files available for subscribers.

The Institute of Mathematical Statistics

The Annals of Statistics

New content alerts

Email RSS ToC RSS Article
  • You have access to this content.
  • You have partial access to this content.
  • You do not have access to this content.
Turn Off MathJax

What is MathJax?

More like this

+ See more