Multivariate regression through affinely weighted penalized least squares

Abstract

Stein’s [In Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability 1 (1956) 197–206 University of California Press] asymptotically superior shrinkage estimator of $p$ univariate means may be rederived through adaptive penalized least squares (PLS) estimation in a regression model with one nominal covariate. This paper treats adaptive PLS estimators for $p$ unknown $d$-dimensional mean vectors, each of which depends on $k_{0}$ scalar covariates that may be nominal or ordinal. The initial focus is on complete regression designs, not necessarily balanced: for every $k_{0}$-tuple of possible covariate values, at least one observation is made on the corresponding mean vector. The results include definition of suitable candidate classes of PLS estimators in multivariate regression problems, comparison of these candidate estimators through their estimated quadratic risks under a general model that makes no assumptions about the regression function, and supporting asymptotic theory as the number $p$ of covariate-value combinations observed tends to infinity. Empirical process theory establishes that: (a) estimated risks converge to their intended targets uniformly over large classes of candidate PLS estimators; (b) a candidate PLS estimator that minimizes estimated risk within such a class has risk that converges to the minimal possible risk within the class. Extension of the results to incomplete regression designs is outlined. The Efron-Morris [ Journal of the American Statistical Association 68 (1973) 117–130] and Beran [ Annals of the Institute of Statistical Mathematics 60 (2008) 843–864] estimators for multivariate means in balanced complete designs are seen as special cases.