Bayesian Analysis
- Bayesian Anal.
- Volume 12, Number 3 (2017), 807-829.
A Bayes Interpretation of Stacking for $\mathcal{M}$-Complete and $\mathcal{M}$-Open Settings
Abstract
In ${\mathcal{M}}$-open problems where no true model can be conceptualized, it is common to back off from modeling and merely seek good prediction. Even in ${\mathcal{M}}$-complete problems, taking a predictive approach can be very useful. Stacking is a model averaging procedure that gives a composite predictor by combining individual predictors from a list of models using weights that optimize a cross-validation criterion. We show that the stacking weights also asymptotically minimize a posterior expected loss. Hence we formally provide a Bayesian justification for cross-validation. Often the weights are constrained to be positive and sum to one. For greater generality, we omit the positivity constraint and relax the ‘sum to one’ constraint.
A key question is ‘What predictors should be in the average?’ We first verify that the stacking error depends only on the span of the models. Then we propose using bootstrap samples from the data to generate empirical basis elements that can be used to form models. We use this in two computed examples to give stacking predictors that are (i) data driven, (ii) optimal with respect to the number of component predictors, and (iii) optimal with respect to the weight each predictor gets.
Article information
Source
Bayesian Anal. Volume 12, Number 3 (2017), 807-829.
Dates
First available in Project Euclid: 7 September 2016
Permanent link to this document
http://projecteuclid.org/euclid.ba/1473276261
Digital Object Identifier
doi:10.1214/16-BA1023
Keywords
stacking cross-validation Bayes action prediction problem classes optimization constrains
Rights
Creative Commons Attribution 4.0 International License.
Citation
Le, Tri; Clarke, Bertrand. A Bayes Interpretation of Stacking for M -Complete and M -Open Settings. Bayesian Anal. 12 (2017), no. 3, 807--829. doi:10.1214/16-BA1023. http://projecteuclid.org/euclid.ba/1473276261.
Supplemental materials
- Supplementary Material: Supplementary Appendices of “A Bayes interpretation of stacking for M-complete and M-open settings”. Digital Object Identifier: doi:10.1214/16-BA1023SUPPSupplemental files available for subscribers.

