The Annals of Statistics

Optimal computational and statistical rates of convergence for sparse nonconvex learning problems

Zhaoran Wang, Han Liu, and Tong Zhang

Full-text: Access has been disabled (more information)

Abstract

We provide theoretical analysis of the statistical and computational properties of penalized $M$-estimators that can be formulated as the solution to a possibly nonconvex optimization problem. Many important estimators fall in this category, including least squares regression with nonconvex regularization, generalized linear models with nonconvex regularization and sparse elliptical random design regression. For these problems, it is intractable to calculate the global solution due to the nonconvex formulation. In this paper, we propose an approximate regularization path-following method for solving a variety of learning problems with nonconvex objective functions. Under a unified analytic framework, we simultaneously provide explicit statistical and computational rates of convergence for any local solution attained by the algorithm. Computationally, our algorithm attains a global geometric rate of convergence for calculating the full regularization path, which is optimal among all first-order algorithms. Unlike most existing methods that only attain geometric rates of convergence for one single regularization parameter, our algorithm calculates the full regularization path with the same iteration complexity. In particular, we provide a refined iteration complexity bound to sharply characterize the performance of each stage along the regularization path. Statistically, we provide sharp sample complexity analysis for all the approximate local solutions along the regularization path. In particular, our analysis improves upon existing results by providing a more refined sample complexity bound as well as an exact support recovery result for the final estimator. These results show that the final estimator attains an oracle statistical property due to the usage of nonconvex penalty.

Article information

Source
Ann. Statist. Volume 42, Number 6 (2014), 2164-2201.

Dates
First available in Project Euclid: 20 October 2014

Permanent link to this document
http://projecteuclid.org/euclid.aos/1413810725

Digital Object Identifier
doi:10.1214/14-AOS1238

Mathematical Reviews number (MathSciNet)
MR3269977

Zentralblatt MATH identifier
1302.62066

Subjects
Primary: 62F30: Inference under constraints 90C26: Nonconvex programming, global optimization
Secondary: 62J12: Generalized linear models 90C52: Methods of reduced gradient type

Keywords
Nonconvex regularized $M$-estimation path-following method geometric computational rate optimal statistical rate

Citation

Wang, Zhaoran; Liu, Han; Zhang, Tong. Optimal computational and statistical rates of convergence for sparse nonconvex learning problems. Ann. Statist. 42 (2014), no. 6, 2164--2201. doi:10.1214/14-AOS1238. http://projecteuclid.org/euclid.aos/1413810725.


Export citation

References

  • Agarwal, A., Negahban, S. and Wainwright, M. J. (2012). Fast global convergence of gradient methods for high-dimensional statistical recovery. Ann. Statist. 40 2452–2482.
    Mathematical Reviews (MathSciNet): MR3097609
    Digital Object Identifier: doi:10.1214/12-AOS1032
    Project Euclid: euclid.aos/1359987527
  • Bickel, P. J., Ritov, Y. and Tsybakov, A. B. (2009). Simultaneous analysis of lasso and Dantzig selector. Ann. Statist. 37 1705–1732.
    Mathematical Reviews (MathSciNet): MR2533469
    Digital Object Identifier: doi:10.1214/08-AOS620
    Project Euclid: euclid.aos/1245332830
  • Blumensath, T. and Davies, M. E. (2009). Iterative hard thresholding for compressed sensing. Appl. Comput. Harmon. Anal. 27 265–274.
    Mathematical Reviews (MathSciNet): MR2559726
    Digital Object Identifier: doi:10.1016/j.acha.2009.04.002
  • Breheny, P. and Huang, J. (2011). Coordinate descent algorithms for nonconvex penalized regression, with applications to biological feature selection. Ann. Appl. Stat. 5 232–253.
    Mathematical Reviews (MathSciNet): MR2810396
    Digital Object Identifier: doi:10.1214/10-AOAS388
    Project Euclid: euclid.aoas/1300715189
  • Candès, E. J. and Tao, T. (2005). Decoding by linear programming. IEEE Trans. Inform. Theory 51 4203–4215.
    Mathematical Reviews (MathSciNet): MR2243152
    Digital Object Identifier: doi:10.1109/TIT.2005.858979
  • Efron, B., Hastie, T., Johnstone, I. and Tibshirani, R. (2004). Least angle regression. Ann. Statist. 32 407–499.
    Mathematical Reviews (MathSciNet): MR2060166
    Digital Object Identifier: doi:10.1214/009053604000000067
    Project Euclid: euclid.aos/1083178935
  • Fan, J. and Li, R. (2001). Variable selection via nonconcave penalized likelihood and its oracle properties. J. Amer. Statist. Assoc. 96 1348–1360.
    Mathematical Reviews (MathSciNet): MR1946581
    Digital Object Identifier: doi:10.1198/016214501753382273
  • Fan, J., Xue, L. and Zou, H. (2014). Strong oracle optimality of folded concave penalized estimation. Ann. Statist. 42 819–849.
    Mathematical Reviews (MathSciNet): MR3210988
    Digital Object Identifier: doi:10.1214/13-AOS1198
    Project Euclid: euclid.aos/1400592644
  • Friedman, J., Hastie, T. and Tibshirani, R. (2010). Regularization paths for generalized linear models via coordinate descent. J. Stat. Softw. 33 1–22.
  • Hastie, T., Rosset, S., Tibshirani, R. and Zhu, J. (2004). The entire regularization path for the support vector machine. J. Mach. Learn. Res. 5 1391–1415.
    Mathematical Reviews (MathSciNet): MR2248021
  • Hunter, D. R. and Li, R. (2005). Variable selection using MM algorithms. Ann. Statist. 33 1617–1642.
    Mathematical Reviews (MathSciNet): MR2166557
    Digital Object Identifier: doi:10.1214/009053605000000200
    Project Euclid: euclid.aos/1123250224
  • Kim, Y., Choi, H. and Oh, H.-S. (2008). Smoothly clipped absolute deviation on high dimensions. J. Amer. Statist. Assoc. 103 1665–1673.
    Mathematical Reviews (MathSciNet): MR2510294
    Digital Object Identifier: doi:10.1198/016214508000001066
  • Koltchinskii, V. (2009). Sparsity in penalized empirical risk minimization. Ann. Inst. Henri Poincaré Probab. Stat. 45 7–57.
    Mathematical Reviews (MathSciNet): MR2500227
    Digital Object Identifier: doi:10.1214/07-AIHP146
    Project Euclid: euclid.aihp/1234469970
  • Liu, W. and Luo, X. (2012). High-dimensional sparse precision matrix estimation via sparse column inverse operator. Preprint. Available at arXiv:1203.3896.
  • Loh, P.-L. and Wainwright, M. J. (2013). Regularized $M$-estimators with nonconvexity: Statistical and algorithmic theory for local optima. Preprint. Available at arXiv:1305.2436.
    Mathematical Reviews (MathSciNet): MR3161456
    Digital Object Identifier: doi:10.1214/13-AOS1162
    Project Euclid: euclid.aos/1388545677
  • Mairal, J. and Yu, B. (2012). Complexity analysis of the lasso regularization path. Preprint. Available at arXiv:1205.0079.
  • Mazumder, R., Friedman, J. H. and Hastie, T. (2011). SparseNet: Coordinate descent with nonconvex penalties. J. Amer. Statist. Assoc. 106 1125–1138.
    Mathematical Reviews (MathSciNet): MR2894769
    Digital Object Identifier: doi:10.1198/jasa.2011.tm09738
  • Negahban, S. N., Ravikumar, P., Wainwright, M. J. and Yu, B. (2012). A unified framework for high-dimensional analysis of $M$-estimators with decomposable regularizers. Statist. Sci. 27 538–557.
    Mathematical Reviews (MathSciNet): MR3025133
    Digital Object Identifier: doi:10.1214/12-STS400
    Project Euclid: euclid.ss/1356098555
  • Nesterov, Y. (2004). Introductory Lectures on Convex Optimization: A Basic Course. Applied Optimization 87. Springer, New York.
    Mathematical Reviews (MathSciNet): MR2142598
  • Nesterov, Yu. (2013). Gradient methods for minimizing composite functions. Math. Program. 140 125–161.
    Mathematical Reviews (MathSciNet): MR3071865
    Digital Object Identifier: doi:10.1007/s10107-012-0629-5
  • Park, M. Y. and Hastie, T. (2007). $L_1$-regularization path algorithm for generalized linear models. J. R. Stat. Soc. Ser. B Stat. Methodol. 69 659–677.
    Mathematical Reviews (MathSciNet): MR2370074
    Digital Object Identifier: doi:10.1111/j.1467-9868.2007.00607.x
  • Raskutti, G., Wainwright, M. J. and Yu, B. (2010). Restricted eigenvalue properties for correlated Gaussian designs. J. Mach. Learn. Res. 11 2241–2259.
    Mathematical Reviews (MathSciNet): MR2719855
  • Raskutti, G., Wainwright, M. J. and Yu, B. (2011). Minimax rates of estimation for high-dimensional linear regression over $\ell_q$-balls. IEEE Trans. Inform. Theory 57 6976–6994.
    Mathematical Reviews (MathSciNet): MR2882274
    Digital Object Identifier: doi:10.1109/TIT.2011.2165799
  • Rosset, S. and Zhu, J. (2007). Piecewise linear regularized solution paths. Ann. Statist. 35 1012–1030.
    Mathematical Reviews (MathSciNet): MR2341696
    Digital Object Identifier: doi:10.1214/009053606000001370
    Project Euclid: euclid.aos/1185303996
  • Rothman, A. J., Bickel, P. J., Levina, E. and Zhu, J. (2008). Sparse permutation invariant covariance estimation. Electron. J. Stat. 2 494–515.
    Mathematical Reviews (MathSciNet): MR2417391
    Digital Object Identifier: doi:10.1214/08-EJS176
    Project Euclid: euclid.ejs/1214491853
  • She, Y. (2009). Thresholding-based iterative selection procedures for model selection and shrinkage. Electron. J. Stat. 3 384–415.
    Mathematical Reviews (MathSciNet): MR2501318
    Digital Object Identifier: doi:10.1214/08-EJS348
    Project Euclid: euclid.ejs/1241011807
  • She, Y. (2012). An iterative algorithm for fitting nonconvex penalized generalized linear models with grouped predictors. Comput. Statist. Data Anal. 56 2976–2990.
    Mathematical Reviews (MathSciNet): MR2929353
  • Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. J. R. Stat. Soc. Ser. B Stat. Methodol. 58 267–288.
    Mathematical Reviews (MathSciNet): MR1379242
  • van de Geer, S. (2000). Empirical Processes in $M$-Estimation 45. Cambridge Univ. Press, Cambridge.
  • van de Geer, S. A. (2008). High-dimensional generalized linear models and the lasso. Ann. Statist. 36 614–645.
    Mathematical Reviews (MathSciNet): MR2396809
    Digital Object Identifier: doi:10.1214/009053607000000929
    Project Euclid: euclid.aos/1205420513
  • Wainwright, M. J. (2009). Sharp thresholds for high-dimensional and noisy sparsity recovery using $\ell_1$-constrained quadratic programming (Lasso). IEEE Trans. Inform. Theory 55 2183–2202.
    Mathematical Reviews (MathSciNet): MR2729873
    Digital Object Identifier: doi:10.1109/TIT.2009.2016018
  • Wang, L., Kim, Y. and Li, R. (2013). Calibrating nonconvex penalized regression in ultra-high dimension. Ann. Statist. 41 2505–2536.
    Mathematical Reviews (MathSciNet): MR3127873
    Digital Object Identifier: doi:10.1214/13-AOS1159
    Project Euclid: euclid.aos/1383661271
  • Wang, Z., Liu, H. and Zhang, T. (2014). Supplement to “Optimal computational and statistical rates of convergence for sparse nonconvex learning problems.” DOI:10.1214/14-AOS1238SUPP.
    Mathematical Reviews (MathSciNet): MR3269977
    Digital Object Identifier: doi:10.1214/14-AOS1238
    Project Euclid: euclid.aos/1413810725
  • Wang, L., Wu, Y. and Li, R. (2012). Quantile regression for analyzing heterogeneity in ultra-high dimension. J. Amer. Statist. Assoc. 107 214–222.
    Mathematical Reviews (MathSciNet): MR2949353
    Digital Object Identifier: doi:10.1080/01621459.2012.656014
  • Wright, S. J., Nowak, R. D. and Figueiredo, M. A. T. (2009). Sparse reconstruction by separable approximation. IEEE Trans. Signal Process. 57 2479–2493.
    Mathematical Reviews (MathSciNet): MR2650165
    Digital Object Identifier: doi:10.1109/TSP.2009.2016892
  • Xiao, L. and Zhang, T. (2013). A proximal-gradient homotopy method for the sparse least-squares problem. SIAM J. Optim. 23 1062–1091.
    Mathematical Reviews (MathSciNet): MR3057332
    Digital Object Identifier: doi:10.1137/120869997
  • Zhang, T. (2009). Some sharp performance bounds for least squares regression with $L_1$ regularization. Ann. Statist. 37 2109–2144.
    Mathematical Reviews (MathSciNet): MR2543687
    Digital Object Identifier: doi:10.1214/08-AOS659
    Project Euclid: euclid.aos/1247663750
  • Zhang, C.-H. (2010a). Nearly unbiased variable selection under minimax concave penalty. Ann. Statist. 38 894–942.
    Mathematical Reviews (MathSciNet): MR2604701
    Digital Object Identifier: doi:10.1214/09-AOS729
    Project Euclid: euclid.aos/1266586618
  • Zhang, T. (2010b). Analysis of multi-stage convex relaxation for sparse regularization. J. Mach. Learn. Res. 11 1081–1107.
    Mathematical Reviews (MathSciNet): MR2629825
  • Zhang, T. (2013). Multi-stage convex relaxation for feature selection. Bernoulli 19 2277–2293.
    Mathematical Reviews (MathSciNet): MR3160554
    Digital Object Identifier: doi:10.3150/12-BEJ452
    Project Euclid: euclid.bj/1386078603
  • Zhang, C.-H. and Huang, J. (2008). The sparsity and bias of the LASSO selection in high-dimensional linear regression. Ann. Statist. 36 1567–1594.
    Mathematical Reviews (MathSciNet): MR2435448
    Digital Object Identifier: doi:10.1214/07-AOS520
    Project Euclid: euclid.aos/1216237292
  • Zhang, C.-H. and Zhang, T. (2012). A general theory of concave regularization for high-dimensional sparse estimation problems. Statist. Sci. 27 576–593.
    Mathematical Reviews (MathSciNet): MR3025135
    Digital Object Identifier: doi:10.1214/12-STS399
    Project Euclid: euclid.ss/1356098557
  • Zhao, P. and Yu, B. (2007). Stagewise lasso. J. Mach. Learn. Res. 8 2701–2726.
    Mathematical Reviews (MathSciNet): MR2383572
  • Zou, H. and Li, R. (2008). One-step sparse estimates in nonconcave penalized likelihood models. Ann. Statist. 36 1509–1533.
    Mathematical Reviews (MathSciNet): MR2435443
    Digital Object Identifier: doi:10.1214/009053607000000802
    Project Euclid: euclid.aos/1216237287

Supplemental materials

  • Supplementary material: Optimal computational and statistical rates of convergence for sparse nonconvex learning problems. We provide the detailed proof in the supplement [Wang, Liu and Zhang (2014)].
    Digital Object Identifier: doi:10.1214/14-AOS1238SUPP
    Supplemental files available for subscribers.

The Institute of Mathematical Statistics

The Annals of Statistics

New content alerts

Email RSS ToC RSS Article
  • You have access to this content.
  • You have partial access to this content.
  • You do not have access to this content.
Turn Off MathJax

What is MathJax?

More like this

+ See more