Simultaneous factors selection and fusion of their levels in penalized logistic regression

Abstract

Nowadays, several data analysis problems are high-dimensional, requiring a complexity reduction for their modeling. Under the sparsity assumption, variable selection is feasible, removing the non-influential explanatory variables. When factors are present, with their levels being dummy coded, the number of parameters included in the model grows rapidly, leading to high-dimensional problems even in cases with moderate number of factors. This fact emphasizes the need for a drastical parameter reduction, not only through variable selection but also through fusion of levels of factors. The levels fused are those not differentiating significantly in terms of their influence on the response variable. Such fusions, beyond reducing the dimension of the model, propose scale adjustments for categorical predictors. In this work a new regularization technique is introduced, called $L_0$-fused group lasso ($L_0$-FGL) for binary logistic regression. It uses a group lasso penalty for factor selection and for the fusion part it applies a $L_0$ penalty on the differences among the levels’ parameters of a categorical predictor. Using adaptive weights, the adaptive version of the $L_0$-FGL method is derived. Theoretical properties, such as existence, $\sqrt{n}$ consistency and oracle properties under certain conditions, are established. In addition, it is shown that even in the diverging case where the number of parameters $p_n$ grows with the sample size n, $\sqrt{n}$ consistency and a consistency in variable selection result are achieved, as well as a respective result on asymptotic normality for an approximate $L_0$-FGL solution. Two computational methods, the penalized iteratively reweighted least squares (PIRLS) and a block coordinate descent (BCD) approach using quasi Newton, are developed and implemented. A simulation study supports the outstanding performance of $L_0$-FGL, especially in cases with a large number of factors. Finally, we apply our method on a real dataset corresponding to breast cancer recurrence events.