Electronic Journal of Statistics

Matrix completion via max-norm constrained optimization

T. Tony Cai and Wen-Xin Zhou

Full-text: Open access

Abstract

Matrix completion has been well studied under the uniform sampling model and the trace-norm regularized methods perform well both theoretically and numerically in such a setting. However, the uniform sampling model is unrealistic for a range of applications and the standard trace-norm relaxation can behave very poorly when the underlying sampling scheme is non-uniform.

In this paper we propose and analyze a max-norm constrained empirical risk minimization method for noisy matrix completion under a general sampling model. The optimal rate of convergence is established under the Frobenius norm loss in the context of approximately low-rank matrix reconstruction. It is shown that the max-norm constrained method is minimax rate-optimal and yields a unified and robust approximate recovery guarantee, with respect to the sampling distributions. The computational effectiveness of this method is also discussed, based on first-order algorithms for solving convex optimizations involving max-norm regularization.

Article information

Source
Electron. J. Statist., Volume 10, Number 1 (2016), 1493-1525.

Dates
Received: December 2015
First available in Project Euclid: 31 May 2016

Permanent link to this document
https://projecteuclid.org/euclid.ejs/1464710239

Digital Object Identifier
doi:10.1214/16-EJS1147

Mathematical Reviews number (MathSciNet)
MR3507371

Zentralblatt MATH identifier
1342.62091

Subjects
Primary: 62H12: Estimation 62J99: None of the above, but in this section
Secondary: 15A83: Matrix completion problems

Keywords
Compressed sensing low-rank matrix matrix completion max-norm constrained minimization minimax optimality non-uniform sampling sparsity

Citation

Cai, T. Tony; Zhou, Wen-Xin. Matrix completion via max-norm constrained optimization. Electron. J. Statist. 10 (2016), no. 1, 1493--1525. doi:10.1214/16-EJS1147. https://projecteuclid.org/euclid.ejs/1464710239


Export citation

References

  • Argyriou, A., Evgeniou, T. and Pontil, M. (2008). Convex multi-task feature learning., Mach. Learn. 73, 243–272.
  • Bartlett, P. and Mendelson, S. (2002). Rademacher and Gaussian complexities: Risk bounds and structural results., J. Mach. Learn. Res. 3, 463–482.
  • Bousquet, O. (2003). Concentration inequalities for sub-addtive functions using the entropy method. In, Stochastic Inequalities and Applications. Progress in Probability 56, 213–247. Birkhäuser, Basel.
  • Burer, S. and Choi, C. (2006). Computational enhancements in low-rank semidefinite programming., Optim. Method Softw. 21, 493–512.
  • Cai, T. T. and Zhou, W.-X. (2013). A max-norm constrained minimization approach to 1-bit matrix completion., J. Mach. Learn. Res. 13, 3619–3647.
  • Candès, E. and Plan, Y. (2010). Matrix completion with noise., Proc. IEEE 98, 925–936.
  • Candès, E. and Plan, Y. (2011). Tight oracle bounds for low-rank matrix recovery from a minimal number of random measurements., IEEE Trans. Inform. Theory 57, 2342–2359.
  • Candès, E. and Recht, B. (2009). Exact matrix completion via convex optimization., Found. Comput. Math. 9, 717–772.
  • Candès, E. and Tao, T. (2010). The power of convex relaxations: Near-optimal matrix completion., IEEE Trans. Inform. Theory 56, 2053–2080.
  • Chen, P. and Suter, D. (2004). Recovering the missing components in a large noisy low-rank matrix: Application to SFM., IEEE Trans. Pattern Anal. Mach. Intell. 26, 1051–1063.
  • Cover, T. M. and Thomas, J. A. (1991)., Elements of Information Theory. John Wiley and Sons, New York.
  • Davenport, M. A., Plan, Y., van den Berg, E. and Wootters, M. (2014). 1-bit matrix completion., Information and Inference: A Journal of the IMA 3, 189–223.
  • Fang, E. X., He, B., Liu, H. and Yuan, X. (2015a). Generalized alternating direction method of multipliers: New theoretical insights and applications., Math. Prog. Comp. 7, 149–187.
  • Fang, E. X., Liu, H., Toh, K.-C. and Zhou, W.-X. (2015b). Max-norm optimization for robust matrix recovery. Technical, Report.
  • Foygel, R., Salakhutdinov, R., Shamir, R. and Srebro, N. (2011). Learning with the weighted trace-norm under arbitrary sampling distributions., Advances in Neural Information Processing Systems 24, 2133–2141.
  • Foygel, R. and Srebro, N. (2011). Concentration-based guarantees for low-rank matrix reconstruction., JMLR: Workshop and Conference Proceedings 19, 315–339.
  • Goldberg, D., Nichols, D., Oki, B. M. and Terry, D. (1992). Using collaborative filtering to weave an information tapestry., Comm. ACM 35, 61–70.
  • Green, P. and Wind, Y. (1973)., Multiattribute Decisions in Marketing: A Measurement Approach. Dryden Press, Hinsdale, IL.
  • Gross, D. (2011). Recovering low-rank matrices from few coefficients in any basis., IEEE Trans. Inform. Theory 57, 1548–1566.
  • Jameson, G. J. O. (1987)., Summing and Nuclear Norms in Banach Space Theory. London Mathematical Society Student Texts, 8. Cambridge University Press, Cambridge.
  • Julià, C., Sappa, A. D., Lumbreras, F., Serrat, J. and López, A. (2011). Rank estimation in missing data matrix problems., J. Math. Imaging Vis. 39, 140–160.
  • Keshavan, R., Montanari, A. and Oh, S. (2010). Matrix completion from noisy entries., J. Mach. Learn. Res. 11, 2057–2078.
  • Klopp, O. (2011). Rank penalized estimators for high-dimensional matrices., Electron. J. Stat. 5, 1161–1183.
  • Klopp, O. (2014). Noisy low-rank matrix completion with general sampling distribution., Bernoulli 20, 282–303.
  • Koltchinskii, V. (2011). Von Neumann entropy penalization and low-rank matrix estimation., Ann. Statist. 39, 2936–2973.
  • Koltchinskii, V., Lounici, K. and Tsybakov, A. B. (2011). Nuclear norm penalization and optimal rates for noisy low rank matrix completion., Ann. Statist. 39, 2302–2329.
  • Ledoux, M. and Talagrand, M. (1991)., Probability in Banach Spaces: Isoperimetry and Processes. Springer-Verlag, New York.
  • Lee, J., Recht, B., Salakhutdinov, R., Srebro, N. and Tropp, J. (2010). Practical large-scale optimization for max-norm regularization., Advances in Neural Information Processing Systems 23, 1297–1305.
  • Lee, T., Shraibman, A. and Špalek, R. (2008). A direct product theorem for discrepancy. In, Proceedings of the 23rd Annual IEEE Conference on Computational Complexity, 71–80.
  • Lin, Z., Ganesh, A., Wright, J., Wu, L., Chen, M. and Ma, Y. (2009). Fast convex optimization algorithms for exact recovery of a corrupted low-rank matrix., International Workshop on Computational Advances in Multi-Sensor Adaptive Processing, Aruba, Dutch Antilles.
  • Linial, N., Mendelson, S., Schechtman, G. and Shraibman, A. (2004). Complexity measures of sign measures., Combinatorica 27, 439–463.
  • Liu, Z. and Vandenberghe, L. (2009). Interior-point method for nuclear norm approximation with application to system identification., SIAM J. Matrix Anal. Appl. 31, 1235–1256.
  • Mazumber, R., Hastie, T. and Tibshirani, R. (2010). Spectral regularization algorithms for learning large incomplete matrices., J. Mach. Learn. Res. 11, 2287–2322.
  • Negahban, S. and Wainwright, M. J. (2011). Estimation of (near) low-rank matrices with noise and high-dimensional scaling., Ann. Statist. 39, 1069–1097.
  • Negahban, S. and Wainwright, M. J. (2012). Restricted strong convexity and weighted matrix completion: Optimal bounds with noise., J. Mach. Learn. Res. 13, 1665–1697.
  • Nesterov, Y. (2013). Gradient methods for minimizing composite objective function., Math. Program. 140, 125–161.
  • Pisier, G. (1989)., The Volume of Convex Bodies and Banach Space Geometry. Cambridge University Press, Cambridge.
  • Recht, B. (2011). A simpler approach to matrix completion., J. Mach. Learn. Res. 12, 3413–3430.
  • Rohde, A. and Tsybakov, A. B. (2011). Estimation of high-dimensional low-rank matrices., Ann. Statist. 39, 887–930.
  • Salakhutdinov, R. and Srebro, N. (2010). Collaborative filtering in a non-uniform world: Learning with the weighted trace norm., Advances in Neural Information Processing Systems 23, 2056–2064.
  • Singer, A. and Cucuringu, M. (2010). Uniqueness of low-rank matrix completion by rigidity theory., SIAM J. Matrix Anal. Appl. 31, 1621–1641.
  • Srebro, N., Rennie, J. and Jaakkola. T. (2004). Maximum-margin matrix factorization., Advances in Neural Information Processing Systems 17, 1329–1336.
  • Srebro, N. and Shraibman, A. (2005). Rank, trace-norm and max-norm. In, Learning Theory, Proceedings of COLT-2005. Lecture Notes in Comput. Sci. 3559, 545–560. Springer, Berlin.
  • Srebro, N., Sridharan, K. and Tewari, A. (2010). Optimistic rates for learning with a smooth loss., Advances in Neural Information Processing Systems 23, 2199–2207.
  • Sun, T. and Zhang, C.-H. (2012). Scaled sparse linear regression., Biometrika 99, 879–898.
  • Tomasi, C. and Kanade, T. (1992). Shape and motion from image streams under orthography: A factorization method., Int. J. Comput. Vis. 9, 137–154.
  • Tropp, J. A. (2012). User-friendly tail bounds for sums of random matrices., Found. Comput. Math. 12, 389–434.
  • Vershynin, R. (2012). Introduction to the non-asymptotic analysis of random matrices. In, Compressed Sensing: Theory and Applications (Y. Eldar and G. Kutyniok, eds.) 210–268. Cambridge University Press, Cambridge.
  • Yang, Y. and Barron, A. (1999). Information-theoretic determination of minimax rates of convergence., Ann. Statist. 27, 1564–1599.
  • Yu, B. (1997). Assouad, Fano, and Le Cam. In, Festschrift for Lucien Le Cam (D. Pollard, E. Torgersen and G. L. Yang, eds.) 423–435. Springer, New York.