Electronic Journal of Statistics

Delete or merge regressors for linear model selection

Aleksandra Maj-Kańska, Piotr Pokarowski, and Agnieszka Prochenka

Full-text: Open access

Enhanced PDF (373 KB)

Abstract

We consider a problem of linear model selection in the presence of both continuous and categorical predictors. Feasible models consist of subsets of numerical variables and partitions of levels of factors. A new algorithm called delete or merge regressors (DMR) is presented which is a stepwise backward procedure involving ranking the predictors according to squared t-statistics and choosing the final model minimizing BIC. We prove consistency of DMR when the number of predictors tends to infinity with the sample size and describe a simulation study using a pertaining R package. The results indicate significant advantage in time complexity and selection accuracy of our algorithm over Lasso-based methods described in the literature. Moreover, a version of DMR for generalized linear models is proposed.

Article information

Source
Electron. J. Statist., Volume 9, Number 2 (2015), 1749-1778.

Dates
Received: May 2015
First available in Project Euclid: 25 August 2015

Permanent link to this document
https://projecteuclid.org/euclid.ejs/1440507392

Digital Object Identifier
doi:10.1214/15-EJS1050

Mathematical Reviews number (MathSciNet)
MR3391118

Zentralblatt MATH identifier
1323.62025

Subjects
Primary: 62F07: Ranking and selection
Secondary: 62J07: Ridge regression; shrinkage estimators

Keywords
ANOVA consistency BIC merging levels t-statistic variable selection

Citation

Maj-Kańska, Aleksandra; Pokarowski, Piotr; Prochenka, Agnieszka. Delete or merge regressors for linear model selection. Electron. J. Statist. 9 (2015), no. 2, 1749--1778. doi:10.1214/15-EJS1050. https://projecteuclid.org/euclid.ejs/1440507392


Export citation

References

  • Bondell, H. D. and Reich, B. J. (2009). Simultaneous factor selection and collapsing levels in ANOVA., Biometrics 65 169–177.
    Mathematical Reviews (MathSciNet): MR2665858
    Digital Object Identifier: doi:10.1111/j.1541-0420.2008.01061.x
  • Caliński, T. and Corsten, L. (1985). Clustering means in ANOVA by simultaneous testing., Biometrics 39–48.
  • Chen, J. and Chen, Z. (2008). Extended Bayesian information criteria for model selection with large model spaces., Biometrika 95 759–771.
    Mathematical Reviews (MathSciNet): MR2443189
    Zentralblatt MATH: 05609546
    Digital Object Identifier: doi:10.1093/biomet/asn034
  • Ciampi, A., Lechevallier, Y., Limas, M. C. and Marcos, A. G. (2008). Hierarchical clustering of subpopulations with a dissimilarity based on the likelihood ratio statistic: application to clustering massive data sets., Pattern Analysis and Applications 11 199–220.
    Mathematical Reviews (MathSciNet): MR2411393
    Digital Object Identifier: doi:10.1007/s10044-007-0088-4
  • Dayton, C. M. (2003). Information criteria for pairwise comparisons., Psychological Methods 8 61–71.
  • Efron, B., Hastie, T., Johnstone, I. and Tibshirani, R. (2004). Least angle regression., Annals of Statistics 32 407–499.
    Mathematical Reviews (MathSciNet): MR2060166
    Zentralblatt MATH: 1091.62054
    Digital Object Identifier: doi:10.1214/009053604000000067
    Project Euclid: euclid.aos/1083178935
  • Gertheiss, J. and Tutz, G. (2010). Sparse modeling of categorial explanatory variables., Annals of Applied Statistics 4 2150–2180.
    Mathematical Reviews (MathSciNet): MR2829951
    Zentralblatt MATH: 1220.62092
    Digital Object Identifier: doi:10.1214/10-AOAS355
    Project Euclid: euclid.aoas/1294167814
  • Oelker, M.-R., Gertheiss, J. and Tutz, G. (2014). Regularization and model selection with categorical predictors and effect modifiers in generalized linear models., Statistical Modelling 14 157–177.
    Mathematical Reviews (MathSciNet): MR3192557
    Digital Object Identifier: doi:10.1177/1471082X13503452
  • Pokarowski, P. and Mielniczuk, J. (2015). Combined l_1 and greedy l_0 penalized least squares for linear model selection., Journal of Machine Learning Research 16(5).
    Mathematical Reviews (MathSciNet): MR3417774
  • Porreca, R. and Ferrari-Trecate, G. (2010). Partitioning datasets based on equalities among parameters., Automatica 46 460–465.
    Mathematical Reviews (MathSciNet): MR2877094
    Digital Object Identifier: doi:10.1016/j.automatica.2009.12.006
  • Scott, A. and Knott, M. (1974). A cluster analysis method for grouping means in the analysis of variance., Biometrics 507–512.
  • Tibshirani, R. (1996). Regression shrinkage and selection via the lasso., Journal of the Royal Statistical Society. Series B (Methodological) 267–288.
    Mathematical Reviews (MathSciNet): MR1379242
  • Tibshirani, R., Saunders, M., Rosset, S., Zhu, J. and Knight, K. (2004). Sparsity and smoothness via the fused lasso., Journal of the Royal Statistical Society: Series B (Statistical Methodology) 67 91–108.
    Mathematical Reviews (MathSciNet): MR2136641
    Zentralblatt MATH: 1060.62049
    Digital Object Identifier: doi:10.1111/j.1467-9868.2005.00490.x
  • Tukey, J. W. (1949). Comparing individual means in the analysis of variance., Biometrics 99–114.
    Mathematical Reviews (MathSciNet): MR30734
    Digital Object Identifier: doi:10.2307/3001913
  • Zhang, T. (2010). Analysis of multi-stage convex relaxation for sparse regularization., Journal of Machine Learning Research 11 1081–1107.
    Mathematical Reviews (MathSciNet): MR2629825
    Zentralblatt MATH: 1242.68262
  • Zheng, X. and Loh, W.-Y. (1995). Consistent variable selection in linear models., Journal of the American Statistical Association 90 151–156.
    Mathematical Reviews (MathSciNet): MR1325122
    Zentralblatt MATH: 0818.62060
    Digital Object Identifier: doi:10.1080/01621459.1995.10476497
  • Zou, H. and Li, R. (2008). One-step sparse estimates in nonconcave penalized likelihood models., Annals of Statistics 36 1509.
    Mathematical Reviews (MathSciNet): MR2435443
    Digital Object Identifier: doi:10.1214/009053607000000802
    Project Euclid: euclid.aos/1216237287

The Institute of Mathematical Statistics and the Bernoulli Society

Electronic Journal of Statistics

New content alerts

Email RSS ToC RSS Article
  • You have access to this content.
  • You have partial access to this content.
  • You do not have access to this content.
Turn Off MathJax

What is MathJax?

More like this

+ See more