Hierarchical selection of variables in sparse high-dimensional regression

Abstract

We study a regression model with a huge number of interacting variables. We consider a specific approximation of the regression function under two assumptions: (i) there exists a sparse representation of the regression function in a suggested basis, (ii) there are no interactions outside of the set of the corresponding main effects. We suggest an hierarchical randomized search procedure for selection of variables and of their interactions. We show that given an initial estimator, an estimator with a similar prediction loss but with a smaller number of non-zero coordinates can be found.