Bayesian Analysis

Bayesian Effect Fusion for Categorical Predictors

Daniela Pauger and Helga Wagner

Full-text: Open access

Abstract

We propose a Bayesian approach to obtain a sparse representation of the effect of a categorical predictor in regression type models. As this effect is captured by a group of level effects, sparsity cannot only be achieved by excluding single irrelevant level effects or the whole group of effects associated to this predictor but also by fusing levels which have essentially the same effect on the response. To achieve this goal, we propose a prior which allows for almost perfect as well as almost zero dependence between level effects a priori. This prior can alternatively be obtained by specifying spike and slab prior distributions on all effect differences associated to this categorical predictor. We show how restricted fusion can be implemented and develop an efficient MCMC (Markov chain Monte Carlo) method for posterior computation. The performance of the proposed method is investigated on simulated data and we illustrate its application on real data from EU-SILC (European Union Statistics on Income and Living Conditions).

Article information

Source
Bayesian Anal., Volume 14, Number 2 (2019), 341-369.

Dates
First available in Project Euclid: 25 May 2018

Permanent link to this document
https://projecteuclid.org/euclid.ba/1527213627

Digital Object Identifier
doi:10.1214/18-BA1096

Mathematical Reviews number (MathSciNet)
MR3934089

Zentralblatt MATH identifier
07045434

Keywords
spike and slab prior sparsity nominal and ordinal predictor regression model MCMC Gibbs sampler

Rights
Creative Commons Attribution 4.0 International License.

Citation

Pauger, Daniela; Wagner, Helga. Bayesian Effect Fusion for Categorical Predictors. Bayesian Anal. 14 (2019), no. 2, 341--369. doi:10.1214/18-BA1096. https://projecteuclid.org/euclid.ba/1527213627


Export citation

References

  • Binder, D. A. (1978). “Bayesian Cluster Analysis.” Biometrika, 65: 31–38.
  • Bondell, H. D. and Reich, B. J. (2009). “Simultaneous Factor Selection and Collapsing Levels in ANOVA.” Biometrics, 65: 169–177.
  • Chen, R.-B., Chu, C.-H., Yuan, S., and Wu, Y. N. (2016). “Bayesian Sparse Group Selection.” Journal of Computational and Graphical Statistics, 25: 665–683.
  • Chipman, H. (1996). “Bayesian Variable Selection with Related Predictors.” The Canadian Journal of Statistics, 1: 17–36.
  • Dellaportas, P. and Tarantola, C. (2005). “Model Determination for Categorical Data with Factor Level Merging.” Journal of Royal Statistical Society, Series B, 67: 269–283.
  • Fahrmeir, L., Kneib, T., and Konrath, S. (2010). “Bayesian Regularisation in Structured Additive Regression: A Unifying Perspective on Shrinkage, Smoothing and Predictor Selection.” Statistics and Computing, 20: 203–219.
  • Fritsch, A. and Ickstadt, K. (2009). “Improved Criteria for Clustering Based on the Posterior Similarity Matrix.” Bayesian Analysis, 4: 367–392.
  • George, E. and McCulloch, R. (1997). “Approaches for Bayesian Variable Selection.” Statistica Sinica, 7: 339–373.
  • Gertheiss, J. and Tutz, G. (2009). “Penalized Regression with Ordinal Predictors.” International Statistical Review, 345–365.
  • Gertheiss, J. and Tutz, G. (2010). “Sparse Modelling of Categorical Explanatory Variables.” The Annals of Applied Statistics, 4: 2150–2180.
  • Griffin, J. and Brown, P. J. (2010). “Inference with Normal-Gamma Prior Distributions in Regression Problems.” Bayesian Analysis, 5: 171–188.
  • Ishwaran, H. and Rao, S. J. (2005). “Spike and Slab Variable Selection; Frequentist and Bayesian Strategies.” Annals of Statistics, 33: 730–773.
  • Kyung, M., Gill, J., Ghosh, M., and Casella, G. (2010). “Penalized Regression, Standard Errors, and Bayesian Lassos.” Bayesian Analysis, 5(2): 369–412.
  • Lau, J. W. and Green, P. J. (2007). “Bayesian Model Based Clustering Procedures.” Journal of Computational and Graphical Statistics, 16: 526–558.
  • Lee, K.-J. and Chen, R.-B. (2015). “BSGS: Bayesian Sparse Group Selection.” The R Journal, 7(2): 122–132.
  • Liu, F., Chakraborty, S., Li, F., Liu, Y., and Lozano, A. C. (2014). “Bayesian Regularization via Graph Laplacian.” Bayesian Analysis, 9(2): 449–474.
  • Malsiner-Walli, G., Pauger, D., and Wagner, H. (2018). “Effect Fusion Using Model-Based Clustering.” Statistical Modelling, 18(2): 175–196
  • Mitchell, T. and Beauchamp, J. J. (1988). “Bayesian Variable Selection in Linear Regression.” Journal of the American Statistical Association, 83: 1023–1032.
  • Park, T. and Casella, G. (2008). “The Bayesian Lasso.” Journal of the American Statistical Association, 103(482): 681–686.
  • Pauger, D. and Wagner, H. (2018). “Supplementary Material of “Bayesian Effect Fusion for Categorical Predictors”.” Bayesian Analysis.
  • Pauger, D., Wagner, H., and Malsiner-Walli, G. (2016). “effectFusion: Bayesian Effect Fusion for Categorical Predictors.” http://www.R-project.org/.
  • Raman, S., Fuchs, T. J., Wild, P. J., Dahl, E., and Roth, V. (2009). “The Bayesian Group-Lasso for Analyzing Contingency Tables.” In Proceedings of the 26th Annual International Conference on Machine Learning. ICML 2009, Montreal.
  • Scheipl, F., Fahrmeir, L., and Kneib, T. (2012). “Spike-and-Slab Priors for Function Selection in Structured Additive Regression Models.” Journal of the American Statistical Association, 107(500): 1518–1532.
  • Simon, N., Friedman, J., Hastie, T., and Tibshirani, R. (2013). “A Sparse-Group Lasso.” Journal of Computational and Graphical Statistics, 22(2): 231–245.
  • Sun, D., Tsutakawa, R. K., and He, Z. (2001). “Propriety of Posteriors with Improper Priors in Hierarchical Linear Mixed Models.” Statistica Sinica, 11: 77–95.
  • Tibshirani, R. (1996). “Regression Shrinkage and Selection via the Lasso.” Journal of Royal Statistical Society, Series B, 58(1): 267–288.
  • Tibshirani, R., Saunders, M., Rosset, S., Zhu, J., and Kneight, K. (2005). “Sparsity and Smoothness via the Fused Lasso.” Journal of Royal Statistical Society, Series B, 67(1): 91–108.
  • Tutz, G. and Berger, M. (2018). “Tree-Structured Clustering in Fixed Effects Models.” Journal of Computational and Graphical Statistics, published online.
  • Tutz, G. and Gertheiss, J. (2016). “Regularized Regression for Categorical Data.” Statistical Modelling, 16(3): 161–200.
  • Yuan, M. and Lin, Y. (2006). “Model Selection and Estimation in Regression with Grouped Variables.” Journal of Royal Statistical Society, Series B, 68: 49–67.
  • Zou, H. and Hastie, T. (2005). “Regularization and Variable Selection via the Elastic Net.” Journal of Royal Statistical Society, Series B, 67: 301–320.

Supplemental materials