Electronic Journal of Statistics

Group symmetry and covariance regularization

Parikshit Shah and Venkat Chandrasekaran

Full-text: Open access

Abstract

Statistical models that possess symmetry arise in diverse settings such as random fields associated to geophysical phenomena, exchangeable processes in Bayesian statistics, and cyclostationary processes in engineering. We formalize the notion of a symmetric model via group invariance. We propose projection onto a group fixed point subspace as a fundamental way of regularizing covariance matrices in the high-dimensional regime. In terms of parameters associated to the group we derive precise rates of convergence of the regularized covariance matrix and demonstrate that significant statistical gains may be expected in terms of the sample complexity. We further explore the consequences of symmetry in related model-selection problems such as the learning of sparse covariance and inverse covariance matrices. We also verify our results with simulations.

Article information

Source
Electron. J. Statist., Volume 6 (2012), 1600-1640.

Dates
First available in Project Euclid: 26 September 2012

Permanent link to this document
https://projecteuclid.org/euclid.ejs/1348665230

Digital Object Identifier
doi:10.1214/12-EJS723

Mathematical Reviews number (MathSciNet)
MR2988459

Zentralblatt MATH identifier
1295.62023

Subjects
Primary: 62F12: Asymptotic properties of estimators 62H12: Estimation

Keywords
Group invariance covariance selection exchangeability high dimensional asymptotics

Citation

Shah, Parikshit; Chandrasekaran, Venkat. Group symmetry and covariance regularization. Electron. J. Statist. 6 (2012), 1600--1640. doi:10.1214/12-EJS723. https://projecteuclid.org/euclid.ejs/1348665230


Export citation

References

  • [1] S. Andersson. Invariant normal models., Annals of Statistics, 3(1):132–154, 1975.
  • [2] S. Andersson and J. Madsen. Symmetry and lattice conditional independence in a multivariate normal distribution., Annals of Statistics, 26, 1998.
  • [3] M. Artin., Algebra. Prentice Hall, 1991.
  • [4] C. Bachoc and F. Vallentin. New upper bounds for kissing numbers from semidefinite programming., Journal of the American Mathematical Society, 21:909–924, 2008.
  • [5] Y. Bai, E. de Klerk, D. V. Pasechnik, and R. Sotirov. Exploiting group symmetry in truss topology optimization., Optimization and Engineering, 10:331–349, 2009.
  • [6] J. Besag. Spatial interaction and the statistical analysis of lattice systems (with discussion)., Journal of the Royal Statistical Society: Series B, 36:192–236, 1974.
  • [7] J. Besag and P. A. Moran. On the estimation and testing of spatial interaction in Gaussian lattice processes., Biometrika, 62:555–562, 1975.
  • [8] P. J. Bickel and E. Levina. Covariance regularization by thresholding., Annals of Statistics, 36(6) :2577–2604, 2008.
  • [9] P. J. Bickel and E. Levina. Regularized estimation of large covariance matrices., Annals of Statistics, 36(1):199–227, 2008.
  • [10] S. Boyd, P. Diaconis, P. A. Parrilo, and L. Xiao. Fastest mixing Markov chain on graphs with symmetries., SIAM Journal on Optimization, 20:792, 2009.
  • [11] S. Boyd and L. Vandenberghe., Convex optimization. Cambridge University Press, 2004.
  • [12] V. V. Buldygin and Y. V. Kozachenko., Metric Characterization of Random Variables and Random Processes. American Mathematical Society, 2000.
  • [13] V. Chandrasekaran, P. A. Parrilo, and A. S. Willsky. Latent variable graphical model selection via convex optimization., Preprint available on www.arxiv.org/abs/1008.1290v1, 2010.
  • [14] A. d’Aspremont, O. Banerjee, and L. El Ghaoui. First-order methods for sparse covariance selection., SIAM Journal on Matrix Analysis and its Applications, 30:56–66, 2007.
  • [15] K. R. Davidson and S.J. Szarek. Local operator theory, random matrices and Banach spaces., Handbook of the Geometry of Banach Spaces, I:317–366, 2001.
  • [16] E. de Klerk, D. V. Pasechnik, and A. Schrijver. Reduction of symmetric semidefinite programs using the regular $*$-representation., Mathematical Programming, 109:613–624, 2007.
  • [17] P. Diaconis., Group Representations in Probability and Statistics, volume 11 of Lecture notes-monograph series. Institute of Mathematical Statistics, 1988.
  • [18] D. Dummit and R. Foote., Abstract Algebra. Wiley, 2004.
  • [19] J. Fan, Y. Fan, and J. Lv. High dimensional covariance matrix estimation using a factor model., Journal of Econometrics, 147(1):43, 2007.
  • [20] A. Fässler and E. Stiefel., Group theoretical methods and their applications. Birkhäuser, 1992.
  • [21] R. Foote, G. Mirchandani, D. Rockmore, D. Healy, and T. Olson. A wreath product group approach to signal and image processing: Part I - multiresolution analysis., IEEE Transactions on Signal Processing, 48(1):102–132, 2000.
  • [22] W. A. Gardner, A. Napolitano, and L. Paura. Cyclostationarity: Half a century of research., Signal Processing, 86(4):639–697, 2006.
  • [23] K. Gatermann and P. A. Parrilo. Symmetry groups, semidefinite programs, and sums of squares., Journal of Pure and Applied Algebra, 192(1-3):95–128, 2004.
  • [24] S. Højsgaard and S. Lauritzen. Graphical Gaussian models with edge and vertex symmetries., Journal of the Royal Statistical Society - Series B: Statistical Methodology, 70(5) :1005–1027, 2008.
  • [25] B. Hylleberg, B. Jensen, and E. Ørnbøl. Graphical symmetry models., Master’s thesis, University of Aalborg, 1993.
  • [26] Y. Kanno, M. Ohsaki, K. Murota, and N. Katoh. Group symmetry in interior point methods for semidefinite programs., Optimization and Engineering, 2:293–320, 2001.
  • [27] S. Lauritzen., Graphical Models. Clarendon Press, Oxford, 1996.
  • [28] J. Madsen. Invariant normal models with recursive graphical Markov structure., Annals of Statistics, 28 :1150–1178, 2000.
  • [29] N. Meinshausen and P. Bühlmann. High dimensional graphs and variable selection with the LASSO., Annals of Statistics, 34 :1436–1462, 2006.
  • [30] J.M.F. Moura and S. Goswami. Gauss-Markov random fields (GMrf) with continuous indices., Information Theory, IEEE Transactions on, 43(5) :1560–1573, Sep 1997.
  • [31] H. Z. Munthe-Kaas. On group Fourier analysis and symmetry preserving discretizations of PDEs., Journal of Physics A: Mathematical and General, 39(19) :5563–5584, 2006.
  • [32] I. Olkin and S. J. Press. Testing and estimation for a circular stationary model., Annals of Mathematical Statistics, 40 :1358–1373, 1969.
  • [33] M. Perlman. A review of multivariate analysis, comment: Group symmetry covariance models., Statistical Science, 2:421–425, 1987.
  • [34] P. Ravikumar, M. J. Wainwright, G. Raskutti, and B. Yu. High-dimensional covariance estimation by minimizing $\ell_1$-penalized log-determinant divergence., Electronic Journal of Statistics, 5, 2011.
  • [35] H. Rue and L. Held., Gaussian Markov Random Fields: Theory and Applications, volume 104 of Monographs on Statistics and Applied Probability. Chapman & Hall, London, 2005.
  • [36] A. Schrijver. New code upper bounds from the Terwilliger algebra and semidefinite programming., IEEE Transactions on Information Theory, 51 :2859–2866, 2005.
  • [37] J. P. Serre., Linear Representations of Finite Groups. Springer, 1977.
  • [38] T. Strohmer. Four short stories about Toeplitz matrix calculations., Linear Algebra and its Applications, 343-344(0):321–344, 2002.
  • [39] F. Vallentin. Symmetry in semidefinite programs., Preprint available on www.arxiv.org/abs/0706.4233v3, 2008.
  • [40] D. F. Votaw. Testing compound symmetry in a normal multivariate distribution., Annals of Mathematical Statistics, 19:447–473, 1948.
  • [41] P. Whittle. On stationary processes in the plane., Biometrika, 41:439–449, 1954.
  • [42] S. S. Wilks. Sample criteria for testing equality of means, equality of variances, and equality of covariances in a normal multivariate distribution., Annals of Mathematical Statistics, 17:257–281, 1946.
  • [43] A. Willsky. Multiresolution markov models for signal and image processing., Proceedings of the IEEE, 90(8) :1396–1458, 2002.