## Electronic Journal of Statistics

### Graphical model selection with latent variables

#### Abstract

Gaussian graphical models are commonly used to characterize the conditional dependence among variables. However, ignorance of the effect of latent variables may blur the structure of a graph and corrupt statistical inference. In this paper, we propose a method for learning $\underline{\mathrm{L}}$atent $\underline{\mathrm{V}}$ariables graphical models via $\ell_{1}$ and trace penalized $\underline{\mathrm{D}}$-trace loss (LVD), which achieves parameter estimation and model selection consistency under certain identifiability conditions. We also present an efficient ADMM algorithm to obtain the penalized estimation of the sparse precision matrix. Using simulation studies, we validate the theoretical properties of our estimator and show its superior performance over other methods. The usefulness of the proposed method is also demonstrated through its application to a yeast genetical genomic data.

#### Article information

Source
Electron. J. Statist., Volume 11, Number 2 (2017), 3485-3521.

Dates
Received: October 2016
First available in Project Euclid: 6 October 2017

Permanent link to this document
https://projecteuclid.org/euclid.ejs/1507255612

Digital Object Identifier
doi:10.1214/17-EJS1331

Mathematical Reviews number (MathSciNet)
MR3709861

Zentralblatt MATH identifier
1384.62210

#### Citation

Wu, Changjing; Zhao, Hongyu; Fang, Huaying; Deng, Minghua. Graphical model selection with latent variables. Electron. J. Statist. 11 (2017), no. 2, 3485--3521. doi:10.1214/17-EJS1331. https://projecteuclid.org/euclid.ejs/1507255612

#### References

• Agakov, F. V., Orchard, P., and Storkey, A. J. (2012). Discriminative mixtures of sparse latent fields for risk management. In, International Conference on Artificial Intelligence and Statistics, pages 10–18.
• Agarwal, A., Negahban, S., and Wainwright, M. J. (2012). Noisy matrix decomposition via convex relaxation: optimal rates in high dimensions., The Annals of Statistics, 40(2):1171–1197.
• Boyd, S., Parikh, N., Chu, E., Peleato, B., and Eckstein, J. (2011). Distributed optimization and statistical learning via the alternating direction method of multipliers., Foundations and Trends® in Machine Learning, 3(1):1–122.
• Brem, R. B. and Kruglyak, L. (2005). The landscape of genetic complexity across 5,700 gene expression traits in yeast., Proceedings of the National Academy of Sciences, 102(5):1572–1577.
• Cai, T. T., Li, H., Liu, W., and Xie, J. (2013). Covariate-adjusted precision matrix estimation with an application in genetical genomics., Biometrika, 100(1):139–156.
• Cai, T. T., Liu, W., and Luo, X. (2011). A constrained $\ell_1$ minimization approach to sparse precision matrix estimation., Journal of the American Statistical Association, 106(494):594–607.
• Cai, T. T., Ren, Z., and Zhou, H. H. (2016). Estimating structured high-dimensional covariance and precision matrices: optimal rates and adaptive estimation., Electronic Journal of Statistics, 10(1):1–59.
• Candès, E. J., Li, X., Ma, Y., and Wright, J. (2011). Robust principal component analysis?, Journal of the ACM (JACM), 58(3):1–37.
• Candès, E. J. and Tao, T. (2007). The Dantzig selector: statistical estimation when $p$ is much larger than $n$., The Annals of Statistics, 35(6):2313–2351.
• Candès, E. J. and Tao, T. (2010). The power of convex relaxation: near-optimal matrix completion., IEEE Transactions on Information Theory, 56(5):2053–2080.
• Chandrasekaran, V., Parrilo, P. A., and Willsky, A. S. (2012). Latent variable graphical model selection via convex optimization., The Annals of Statistics, 40(4):1935–1967.
• Chandrasekaran, V., Sanghavi, S., Parrilo, P. A., and Willsky, A. S. (2011). Rank-sparsity incoherence for matrix decomposition., SIAM Journal on Optimization, 21(2):572–596.
• Chen, M., Ren, Z., Zhao, H., and Zhou, H. (2016). Asymptotically normal and efficient estimation of covariate-adjusted Gaussian graphical model., Journal of the American Statistical Association, 111(513):394–406.
• Chen, S., Witten, D. M., and Shojaie, A. (2015). Selection and estimation for mixed graphical models., Biometrika, 102(1):47–64.
• Cheng, J., Li, T., Levina, E., and Zhu, J. (2016). High-dimensional mixed graphical models., Journal of Computational and Graphical Statistics, 26(2):367–378.
• Cheung, V. G. and Spielman, R. S. (2002). The genetics of variation in gene expression., Nature Genetics, 32:522–525.
• Cox, D. R. and Wermuth, N. (1996)., Multivariate Dependencies: Models, Analysis and Interpretation, volume 67. CRC Press.
• Donoho, D. L. (2006). Compressed sensing., IEEE Transactions on Information Theory, 52(4):1289–1306.
• Fan, J., Liao, Y., and Mincheva, M. (2013). Large covariance estimation by thresholding principal orthogonal complements., Journal of the Royal Statistical Society: Series B (Statistical Methodology), 75(4):603–680.
• Fan, J., Liu, H., Ning, Y., and Zou, H. (2017). High dimensional semiparametric latent graphical model for mixed data., Journal of the Royal Statistical Society: Series B (Statistical Methodology), 79(2):405–421.
• Friedman, J., Hastie, T., and Tibshirani, R. (2008). Sparse inverse covariance estimation with the graphical Lasso., Biostatistics, 9(3):432–441.
• Giraud, C. and Tsybakov, A. (2012). Discussion: latent variable graphical model selection via convex optimization., The Annals of Statistics, 40(4):1984–1988.
• Horn, R. A. and Johnson, C. R. (2012)., Matrix Analysis. Cambridge University Press.
• Hsu, D., Kakade, S. M., and Zhang, T. (2011). Robust matrix decomposition with sparse corruptions., IEEE Transactions on Information Theory, 57(11):7221–7234.
• Janková, J. and van de Geer, S. (2015). Confidence intervals for high-dimensional inverse covariance estimation., Electronic Journal of Statistics, 9:1205–1229.
• Kalaitzis, A. and Lawrence, N. (2012). Residual component analysis: generalising PCA for more flexible inference in linear-Gaussian models., arXiv preprint arXiv:1206.4560.
• Kanehisa, M., Goto, S., Furumichi, M., Tanabe, M., and Hirakawa, M. (2010). KEGG for representation and analysis of molecular networks involving diseases and drugs., Nucleic Acids Research, 38(suppl 1):D355–D360.
• Kelder, T., van Iersel, M. P., Hanspers, K., Kutmon, M., Conklin, B. R., Evelo, C. T., and Pico, A. R. (2012). Wikipathways: building research communities on biological pathways., Nucleic Acids Research, 40(D1):D1301–D1307.
• Lam, C. and Fan, J. (2009). Sparsistency and rates of convergence in large covariance matrix estimation., The Annals of Statistics, 37(6B):4254–4278.
• Lauritzen, S. and Meinshausen, N. (2012). Discussion: latent variable graphical model selection via convex optimization., The Annals of Statistics, 40(4):1973–1977.
• Lauritzen, S. L. (1996)., Graphical Models. Oxford University Press.
• Li, B., Chun, H., and Zhao, H. (2012). Sparse estimation of conditional graphical models with application to gene networks., Journal of the American Statistical Association, 107(497):152–167.
• Li, H. and Gui, J. (2006). Gradient directed regularization for sparse Gaussian concentration graphs, with applications to inference of genetic networks., Biostatistics, 7(2):302–317.
• Liu, W. (2013). Gaussian graphical model estimation with false discovery rate control., The Annals of Statistics, 41(6):2948–2978.
• Liu, W. and Luo, X. (2015). Fast and adaptive sparse precision matrix estimation in high dimensions., Journal of Multivariate Analysis, 135:153–162.
• Loh, P.-L. and Wainwright, M. J. (2013). Structure estimation for discrete graphical models: generalized covariance matrices and their inverses., The Annals of Statistics, 41(6):3022–3049.
• Ma, S., Xue, L., and Zou, H. (2013). Alternating direction methods for latent variable Gaussian graphical model selection., Neural Computation, 25(8):2172–2198.
• Meinshausen, N. and Bühlmann, P. (2006). High-dimensional graphs and variable selection with the Lasso., The Annals of Statistics, 34(3):1436–1462.
• Meinshausen, N. and Bühlmann, P. (2010). Stability selection., Journal of the Royal Statistical Society: Series B (Statistical Methodology), 72(4):417–473.
• Meng, Z., Eriksson, B., and Hero III, A. O. (2014). Learning latent variable Gaussian graphical models., arXiv preprint arXiv:1406.2721.
• Ravikumar, P., Wainwright, M. J., Lafferty, J. D., et al. (2010). High-dimensional Ising model selection using $\ell_1$-regularized logistic regression., The Annals of Statistics, 38(3):1287–1319.
• Ravikumar, P., Wainwright, M. J., Raskutti, G., and Yu, B. (2011). High-dimensional covariance estimation by minimizing $\ell_1$-penalized log-determinant divergence., Electronic Journal of Statistics, 5:935–980.
• Ren, Z., Sun, T., Zhang, C.-H., and Zhou, H. H. (2015). Asymptotic normality and optimalities in estimation of large Gaussian graphical models., The Annals of Statistics, 43(3):991–1026.
• Ren, Z. and Zhou, H. H. (2012). Discussion: latent variable graphical model selection via convex optimization., The Annals of Statistics, 40(4):1989–1996.
• Rothman, A. J., Bickel, P. J., Levina, E., and Zhu, J. (2008). Sparse permutation invariant covariance estimation., Electronic Journal of Statistics, 2:494–515.
• Segal, E., Friedman, N., Kaminski, N., Regev, A., and Koller, D. (2005). From signatures to models: understanding cancer using microarrays., Nature Genetics, 37:S38–S45.
• Städler, N. and Bühlmann, P. (2012). Missing values: sparse inverse covariance estimation and an extension to sparse regression., Statistics and Computing, 22(1):219–235.
• Sun, T. and Zhang, C.-H. (2013). Sparse matrix inversion with scaled Lasso., The Journal of Machine Learning Research, 14(1):3385–3418.
• Taeb, A. and Chandrasekaran, V. (2016). Interpreting latent variables in factor models via convex optimization., arXiv preprint arXiv:1601.00389.
• Tan, K. M., Ning, Y., Witten, D. M., and Liu, H. (2016). Replicates in high dimensions, with applications to latent variable graphical models., Biometrika, 103(4):761–777.
• Vershynin, R. (2010). Introduction to the non-asymptotic analysis of random matrices., arXiv preprint arXiv:1011.3027.
• Vershynin, R. (2012). How close is the sample covariance matrix to the actual covariance matrix?, Journal of Theoretical Probability, 25(3):655–686.
• Wainwright, M. J. (2009). Sharp thresholds for high-dimensional and noisy sparsity recovery using-constrained quadratic programming (Lasso)., IEEE Transactions on Information Theory, 55(5):2183–2202.
• Wasserman, L., Kolar, M., and Rinaldo, A. (2014). Berry-Esseen bounds for estimating undirected graphs., Electronic Journal of Statistics, 8(1):1188–1224.
• Witten, D. M., Tibshirani, R., and Hastie, T. (2009). A penalized matrix decomposition, with applications to sparse principal components and canonical correlation analysis., Biostatistics, 10(3):515–534.
• Xu, P., Ma, J., and Gu, Q. (2017). Speeding up latent variable Gaussian graphical model estimation via nonconvex optimizations., arXiv preprint arXiv:1702.08651.
• Yang, E., Ravikumar, P., Allen, G. I., and Liu, Z. (2015). Graphical models via univariate exponential family distributions., Journal of Machine Learning Research, 16(1):3813–3847.
• Yuan, M. (2010). High dimensional inverse covariance matrix estimation via linear programming., Journal of Machine Learning Research, 11(Aug):2261–2286.
• Yuan, M. (2012). Discussion: latent variable graphical model selection via convex optimization., The Annals of Statistics, 40(4):1968–1972.
• Yuan, M. and Lin, Y. (2007). Model selection and estimation in the Gaussian graphical model., Biometrika, 94(1):19–35.
• Zhang, T. and Zou, H. (2014). Sparse precision matrix estimation via Lasso penalized D-trace loss., Biometrika, 101(1):103–120.
• Zhao, P. and Yu, B. (2006). On model selection consistency of Lasso., Journal of Machine Learning Research, 7(Nov):2541–2563.