Estimating a regression function in exponential families by model selection

Abstract

Let $(W_1,Y_1),\dots ,(W_n,Y_n)$ be n pairs of independent random variables. We assume that, for each $i\in \{1,\dots ,n\}$, the conditional distribution of $Y_i$ given $W_i$ belongs to a one-parameter exponential family with parameter ${\mathit{\gamma }}^{\star }(W_i)\in \mathbb{R}$. The statistical goal is to estimate these conditional distributions. We consider a model selection procedure which works based on a general assumption that each of the model is VC-subgraph. We establish a non-asymptotic risk bound for the resulting estimator with respect to a Hellinger-type distance. By leveraging this result, we extend several findings previously explored in Gaussian regression to the regression in exponential families. Specifically, we address the curse of dimensionality by imposing structural assumptions, such as general additive and multiple index structures, on ${\mathit{\gamma }}^{\star }$. We also study model selection for ReLU neural networks, and provide a concrete example of how ReLU neural networks can achieve a significantly faster convergence rate than traditional models. When ${\mathit{\gamma }}^{\star }$ is close to a composition of several Hölder functions, we show that under a suitable parametrization of the exponential family, our estimator achieves the same rate of convergence as in the Gaussian case. Combining with a lower bound, the rate is minimax optimal up to a logarithmic term. Finally, we apply the model selection procedure to address adaptation and variable selection problems in exponential families.