Open Access
2010 Majorization-Minimization algorithms for nonsmoothly penalized objective functions
Elizabeth D. Schifano, Robert L. Strawderman, Martin T. Wells
Electron. J. Statist. 4: 1258-1299 (2010). DOI: 10.1214/10-EJS582


The use of penalization, or regularization, has become common in high-dimensional statistical analysis, where an increasingly frequent goal is to simultaneously select important variables and estimate their effects. It has been shown by several authors that these goals can be achieved by minimizing some parameter-dependent “goodness-of-fit” function (e.g., a negative loglikelihood) subject to a penalization that promotes sparsity. Penalty functions that are singular at the origin have received substantial attention, arguably beginning with the Lasso penalty [62].

The current literature tends to focus on specific combinations of differentiable goodness-of-fit functions and penalty functions singular at the origin. One result of this combined specificity has been a proliferation in the number of computational algorithms designed to solve fairly narrow classes of optimization problems involving objective functions that are not everywhere continuously differentiable. In this paper, we propose a general class of algorithms for optimizing an extensive variety of nonsmoothly penalized objective functions that satisfy certain regularity conditions. The proposed framework utilizes the majorization-minimization (MM) algorithm as its core optimization engine. In the case of penalized regression models, the resulting algorithms employ iterated soft-thresholding, implemented componentwise, allowing for fast and stable updating that avoids the need for inverting high-dimensional matrices. We establish convergence theory under weaker assumptions than previously considered in the statistical literature. We also demonstrate the exceptional effectiveness of new acceleration methods, originally proposed for the EM algorithm, in this class of problems. Simulation results and a microarray data example are provided to demonstrate the algorithm’s capabilities and versatility.


Download Citation

Elizabeth D. Schifano. Robert L. Strawderman. Martin T. Wells. "Majorization-Minimization algorithms for nonsmoothly penalized objective functions." Electron. J. Statist. 4 1258 - 1299, 2010.


Published: 2010
First available in Project Euclid: 12 November 2010

zbMATH: 1267.65009
MathSciNet: MR2738533
Digital Object Identifier: 10.1214/10-EJS582

Primary: 62J07 , 65C60
Secondary: 62J05 , 62J12

Keywords: Convex optimization , iterative soft thresholding , lasso penalty , minimax concave penalty , non-convex optimization , smoothly clipped absolute deviation penalty

Rights: Copyright © 2010 The Institute of Mathematical Statistics and the Bernoulli Society

Back to Top