Adaptive estimation in multivariate response regression with hidden variables

Abstract

A prominent concern of scientific investigators is the presence of unobserved hidden variables in association analysis. Ignoring hidden variables often yields biased statistical results and misleading scientific conclusions. Motivated by this practical issue, this paper studies the multivariate response regression with hidden variables, $\mathit{Y}={({\mathrm{\Psi }}^{\ast })}^{\mathit{T}}\mathit{X}+{({\mathit{B}}^{\ast })}^{\mathit{T}}\mathit{Z}+\mathit{E}$, where $\mathit{Y}\in {\mathbb{R}}^{\mathit{m}}$ is the response vector, $\mathit{X}\in {\mathbb{R}}^{\mathit{p}}$ is the observable feature, $\mathit{Z}\in {\mathbb{R}}^{\mathit{K}}$ represents the vector of unobserved hidden variables, possibly correlated with X, and E is an independent error. The number of hidden variables K is unknown and both m and p are allowed, but not required, to grow with the sample size n.

Though ${\mathrm{\Psi }}^{\ast }$ is shown to be nonidentifiable due to the presence of hidden variables, we propose to identify the projection of ${\mathrm{\Psi }}^{\ast }$ onto the orthogonal complement of the row space of ${\mathit{B}}^{\ast }$, denoted by ${\mathrm{\Theta }}^{\ast }$. The quantity ${({\mathrm{\Theta }}^{\ast })}^{\mathit{T}}\mathit{X}$ measures the effect of X on Y that cannot be explained through the hidden variables, and thus ${\mathrm{\Theta }}^{\ast }$ is treated as the parameter of interest. Motivated by the identifiability proof, we propose a novel estimation algorithm for ${\mathrm{\Theta }}^{\ast }$, called HIVE, under homoscedastic errors. The first step of the algorithm estimates the best linear prediction of Y given X, in which the unknown coefficient matrix exhibits an additive decomposition of ${\mathrm{\Psi }}^{\ast }$ and a dense matrix due to the correlation between X and Z. Under the sparsity assumption on ${\mathrm{\Psi }}^{\ast }$, we propose to minimize a penalized least squares loss by regularizing ${\mathrm{\Psi }}^{\ast }$ and the dense matrix via group-lasso and multivariate ridge, respectively. Nonasymptotic deviation bounds of the in-sample prediction error are established. Our second step estimates the row space of ${\mathit{B}}^{\ast }$ by leveraging the covariance structure of the residual vector from the first step. In the last step, we estimate ${\mathrm{\Theta }}^{\ast }$ via projecting Y onto the orthogonal complement of the estimated row space of ${\mathit{B}}^{\ast }$ to remove the effect of hidden variables. Nonasymptotic error bounds of our final estimator of ${\mathrm{\Theta }}^{\ast }$, which are valid for any $\mathit{m},\mathit{p},\mathit{K}$ and n, are established. We further show that, under mild assumptions, the rate of our estimator matches the best possible rate with known ${\mathit{B}}^{\ast }$ and is adaptive to the unknown sparsity of ${\mathrm{\Theta }}^{\ast }$ induced by the sparsity of ${\mathrm{\Psi }}^{\ast }$. The model identifiability, estimation algorithm and statistical guarantees are further extended to the setting with heteroscedastic errors. Thorough numerical simulations and two real data examples are provided to back up our theoretical results.