In many important statistical applications, the number of variables or parameters p is much larger than the number of observations n. Suppose then that we have observations y=Xβ+z, where β∈R^p is a parameter vector of interest, X is a data matrix with possibly far fewer rows than columns, n≪p, and the z_i’s are i.i.d. N(0, σ²). Is it possible to estimate β reliably based on the noisy data y?

To estimate β, we introduce a new estimator—we call it the Dantzig selector—which is a solution to the ℓ₁-regularization problem $$\min_{\tilde{\beta}\in\mathbf{R}^{p}}\|\tilde{\beta}\|_{\ell_{1}}\quad\mbox{subject to}\quad \|X^{*}r\|_{\ell_{\infty}}\leq(1+t^{-1})\sqrt{2\log p}\cdot\sigma,$$ where r is the residual vector y−Xβ̃ and t is a positive scalar. We show that if X obeys a uniform uncertainty principle (with unit-normed columns) and if the true parameter vector β is sufficiently sparse (which here roughly guarantees that the model is identifiable), then with very large probability,

‖β̂−β‖_ℓ2²≤C²⋅2log p⋅(σ²+∑_imin(β_i², σ²)).

Our results are nonasymptotic and we give values for the constant C. Even though n may be much smaller than p, our estimator achieves a loss within a logarithmic factor of the ideal mean squared error one would achieve with an oracle which would supply perfect information about which coordinates are nonzero, and which were above the noise level.

In multivariate regression and from a model selection viewpoint, our result says that it is possible nearly to select the best subset of variables by solving a very simple convex program, which, in fact, can easily be recast as a convenient linear program (LP).

In many applications of multiple hypothesis testing where more than one false rejection can be tolerated, procedures controlling error rates measuring at least k false rejections, instead of at least one, for some fixed k≥1 can potentially increase the ability of a procedure to detect false null hypotheses. The k-FWER, a generalized version of the usual familywise error rate (FWER), is such an error rate that has recently been introduced in the literature and procedures controlling it have been proposed. A further generalization of a result on the k-FWER is provided in this article. In addition, an alternative and less conservative notion of error rate, the k-FDR, is introduced in the same spirit as the k-FWER by generalizing the usual false discovery rate (FDR). A k-FWER procedure is constructed given any set of increasing constants by utilizing the kth order joint null distributions of the p-values without assuming any specific form of dependence among all the p-values. Procedures controlling the k-FDR are also developed by using the kth order joint null distributions of the p-values, first assuming that the sets of null and nonnull p-values are mutually independent or they are jointly positively dependent in the sense of being multivariate totally positive of order two (MTP₂) and then discarding that assumption about the overall dependence among the p-values.

For high dimensional statistical models, researchers have begun to focus on situations which can be described as having relatively few moderately large coefficients. Such situations lead to some very subtle statistical problems. In particular, Ingster and Donoho and Jin have considered a sparse normal means testing problem, in which they described the precise demarcation or detection boundary. Meinshausen and Rice have shown that it is even possible to estimate consistently the fraction of nonzero coordinates on a subset of the detectable region, but leave unanswered the question of exactly in which parts of the detectable region consistent estimation is possible.

In the present paper we develop a new approach for estimating the fraction of nonzero means for problems where the nonzero means are moderately large. We show that the detection region described by Ingster and Donoho and Jin turns out to be the region where it is possible to consistently estimate the expected fraction of nonzero coordinates. This theory is developed further and minimax rates of convergence are derived. A procedure is constructed which attains the optimal rate of convergence in this setting. Furthermore, the procedure also provides an honest lower bound for confidence intervals while minimizing the expected length of such an interval. Simulations are used to enable comparison with the work of Meinshausen and Rice, where a procedure is given but where rates of convergence have not been discussed. Extensions to more general Gaussian mixture models are also given.

Theoretical developments on cross validation (CV) have mainly focused on selecting one among a list of finite-dimensional models (e.g., subset or order selection in linear regression) or selecting a smoothing parameter (e.g., bandwidth for kernel smoothing). However, little is known about consistency of cross validation when applied to compare between parametric and nonparametric methods or within nonparametric methods. We show that under some conditions, with an appropriate choice of data splitting ratio, cross validation is consistent in the sense of selecting the better procedure with probability approaching 1.

Our results reveal interesting behavior of cross validation. When comparing two models (procedures) converging at the same nonparametric rate, in contrast to the parametric case, it turns out that the proportion of data used for evaluation in CV does not need to be dominating in size. Furthermore, it can even be of a smaller order than the proportion for estimation while not affecting the consistency property.

Application of nonparametric and semiparametric regression techniques to high-dimensional time series data has been hampered due to the lack of effective tools to address the “curse of dimensionality.” Under rather weak conditions, we propose spline-backfitted kernel estimators of the component functions for the nonlinear additive time series data that are both computationally expedient so they are usable for analyzing very high-dimensional time series, and theoretically reliable so inference can be made on the component functions with confidence. Simulation experiments have provided strong evidence that corroborates the asymptotic theory.

Regression problems are traditionally analyzed via univariate characteristics like the regression function, scale function and marginal density of regression errors. These characteristics are useful and informative whenever the association between the predictor and the response is relatively simple. More detailed information about the association can be provided by the conditional density of the response given the predictor. For the first time in the literature, this article develops the theory of minimax estimation of the conditional density for regression settings with fixed and random designs of predictors, bounded and unbounded responses and a vast set of anisotropic classes of conditional densities. The study of fixed design regression is of special interest and novelty because the known literature is devoted to the case of random predictors. For the aforementioned models, the paper suggests a universal adaptive estimator which (i) matches performance of an oracle that knows both an underlying model and an estimated conditional density; (ii) is sharp minimax over a vast class of anisotropic conditional densities; (iii) is at least rate minimax when the response is independent of the predictor and thus a bivariate conditional density becomes a univariate density; (iv) is adaptive to an underlying design (fixed or random) of predictors.

We study the asymptotic behavior of the Maximum Likelihood and Least Squares Estimators of a k-monotone density g₀ at a fixed point x₀ when k>2. We find that the jth derivative of the estimators at x₀ converges at the rate n^{−(k−j)/(2k+1)} for j=0, …, k−1. The limiting distribution depends on an almost surely uniquely defined stochastic process H_k that stays above (below) the k-fold integral of Brownian motion plus a deterministic drift when k is even (odd). Both the MLE and LSE are known to be splines of degree k−1 with simple knots. Establishing the order of the random gap τ_n⁺−τ_n⁻, where τ_n^± denote two successive knots, is a key ingredient of the proof of the main results. We show that this “gap problem” can be solved if a conjecture about the upper bound on the error in a particular Hermite interpolation via odd-degree splines holds.

Cook’s [J. Roy. Statist. Soc. Ser. B 48 (1986) 133–169] local influence approach based on normal curvature is an important diagnostic tool for assessing local influence of minor perturbations to a statistical model. However, no rigorous approach has been developed to address two fundamental issues: the selection of an appropriate perturbation and the development of influence measures for objective functions at a point with a nonzero first derivative. The aim of this paper is to develop a differential–geometrical framework of a perturbation model (called the perturbation manifold) and utilize associated metric tensor and affine curvatures to resolve these issues. We will show that the metric tensor of the perturbation manifold provides important information about selecting an appropriate perturbation of a model. Moreover, we will introduce new influence measures that are applicable to objective functions at any point. Examples including linear regression models and linear mixed models are examined to demonstrate the effectiveness of using new influence measures for the identification of influential observations.

This paper discusses a nonparametric regression model that naturally generalizes neural network models. The model is based on a finite number of one-dimensional transformations and can be estimated with a one-dimensional rate of convergence. The model contains the generalized additive model with unknown link function as a special case. For this case, it is shown that the additive components and link function can be estimated with the optimal rate by a smoothing spline that is the solution of a penalized least squares criterion.

A low-degree polynomial model for a response curve is used commonly in practice. It generally incorporates a linear or quadratic function of the covariate. In this paper we suggest methods for testing the goodness of fit of a general polynomial model when there are errors in the covariates. There, the true covariates are not directly observed, and conventional bootstrap methods for testing are not applicable. We develop a new approach, in which deconvolution methods are used to estimate the distribution of the covariates under the null hypothesis, and a “wild” or moment-matching bootstrap argument is employed to estimate the distribution of the experimental errors (distinct from the distribution of the errors in covariates). Most of our attention is directed at the case where the distribution of the errors in covariates is known, although we also discuss methods for estimation and testing when the covariate error distribution is estimated. No assumptions are made about the distribution of experimental error, and, in particular, we depart substantially from conventional parametric models for errors-in-variables problems.

We consider nonparametric estimation of a regression function for a situation where precisely measured predictors are used to estimate the regression curve for coarsened, that is, less precise or contaminated predictors. Specifically, while one has available a sample (W₁, Y₁), …, (W_n, Y_n) of independent and identically distributed data, representing observations with precisely measured predictors, where E(Y_i|W_i)=g(W_i), instead of the smooth regression function g, the target of interest is another smooth regression function m that pertains to predictors X_i that are noisy versions of the W_i. Our target is then the regression function m(x)=E(Y|X=x), where X is a contaminated version of W, that is, X=W+δ. It is assumed that either the density of the errors is known, or replicated data are available resembling, but not necessarily the same as, the variables X. In either case, and under suitable conditions, we obtain $\sqrt{n}$-rates of convergence of the proposed estimator and its derivatives, and establish a functional limit theorem. Weak convergence to a Gaussian limit process implies pointwise and uniform confidence intervals and $\sqrt{n}$-consistent estimators of extrema and zeros of m. It is shown that these results are preserved under more general models in which X is determined by an explanatory variable. Finite sample performance is investigated in simulations and illustrated by a real data example.

In this paper we propose two new methods to estimate the dimension-reduction directions of the central subspace (CS) by constructing a regression model such that the directions are all captured in the regression mean. Compared with the inverse regression estimation methods [e.g., J. Amer. Statist. Assoc. 86 (1991) 328–332, J. Amer. Statist. Assoc. 86 (1991) 316–342, J. Amer. Statist. Assoc. 87 (1992) 1025–1039], the new methods require no strong assumptions on the design of covariates or the functional relation between regressors and the response variable, and have better performance than the inverse regression estimation methods for finite samples. Compared with the direct regression estimation methods [e.g., J. Amer. Statist. Assoc. 84 (1989) 986–995, Ann. Statist. 29 (2001) 1537–1566, J. R. Stat. Soc. Ser. B Stat. Methodol. 64 (2002) 363–410], which can only estimate the directions of CS in the regression mean, the new methods can detect the directions of CS exhaustively. Consistency of the estimators and the convergence of corresponding algorithms are proved.

This article contains two main theoretical results on neural spike train models, using the counting or point process on the real line as a model for the spike train. The first part of this article considers template matching of multiple spike trains. P-values for the occurrences of a given template or pattern in a set of spike trains are computed using a general scoring system. By identifying the pattern with an experimental stimulus, multiple spike trains can be deciphered to provide useful information.

The second part of the article assumes that the counting process has a conditional intensity function that is a product of a free firing rate function s, which depends only on the stimulus, and a recovery function r, which depends only on the time since the last spike. If s and r belong to a q-smooth class of functions, it is proved that sieve maximum likelihood estimators for s and r achieve the optimal convergence rate (except for a logarithmic factor) under L₁ loss.

We introduce a useful tool for analyzing boosting algorithms called the “smooth margin function,” a differentiable approximation of the usual margin for boosting algorithms. We present two boosting algorithms based on this smooth margin, “coordinate ascent boosting” and “approximate coordinate ascent boosting,” which are similar to Freund and Schapire’s AdaBoost algorithm and Breiman’s arc-gv algorithm. We give convergence rates to the maximum margin solution for both of our algorithms and for arc-gv. We then study AdaBoost’s convergence properties using the smooth margin function. We precisely bound the margin attained by AdaBoost when the edges of the weak classifiers fall within a specified range. This shows that a previous bound proved by Rätsch and Warmuth is exactly tight. Furthermore, we use the smooth margin to capture explicit properties of AdaBoost in cases where cyclic behavior occurs.

Distance correlation is a new measure of dependence between random vectors. Distance covariance and distance correlation are analogous to product-moment covariance and correlation, but unlike the classical definition of correlation, distance correlation is zero only if the random vectors are independent. The empirical distance dependence measures are based on certain Euclidean distances between sample elements rather than sample moments, yet have a compact representation analogous to the classical covariance and correlation. Asymptotic properties and applications in testing independence are discussed. Implementation of the test and Monte Carlo results are also presented.

Longitudinal studies are often conducted to explore the cohort and age effects in many scientific areas. The within cluster correlation structure plays a very important role in longitudinal data analysis. This is because not only can an estimator be improved by incorporating the within cluster correlation structure into the estimation procedure, but also the within cluster correlation structure can sometimes provide valuable insights in practical problems. For example, it can reveal the correlation strengths among the impacts of various factors. Motivated by data typified by a set from Bangladesh pertinent to the use of contraceptives, we propose a random effect varying-coefficient model, and an estimation procedure for the within cluster correlation structure of the proposed model. The estimation procedure is optimization-free and the proposed estimators enjoy asymptotic normality under mild conditions. Simulations suggest that the proposed estimation is practicable for finite samples and resistent against mild forms of model misspecification. Finally, we analyze the data mentioned above with the new random effect varying-coefficient model together with the proposed estimation procedure, which reveals some interesting sociological dynamics.