Covariate-adaptive randomization (CAR) is commonly implemented in clinical trials to balance observed covariates. Recent studies have demonstrated the advantages of CAR procedures in balancing covariates and improving the subsequent statistical analysis. Covariate balance is crucial, but it is not a panacea for the valid statistical inferences. If the response to a treatment interacts with some unobserved covariates, the conclusion drawn from a CAR experiment may be affected, and thus, be inconsistent with other evidence. This paper aims to demonstrate the relationships between unobserved covariates and the analysis of treatment and covariate effects in CAR experiments. We first derive the asymptotic properties of the statistical methods based on a linear model framework with interactions between the treatment and an unobserved covariate. We also provide sufficient conditions for the identifiability of the treatment and covariate effects. Our results theoretically explain how inconsistent estimations are generated in CAR experiments when some important covariates are unobserved. Under these sufficient conditions, we show that the tests for the treatment and covariate effects can have reduced Type I errors under CAR procedures. A residual-based adjusted test is proposed to recover the Type I error when the effect can be correctly estimated. Numerical studies are conducted to evaluate the performance of our proposed procedure and theoretical findings.

This paper considers one-sample testing of a high-dimensional covariance matrix by deriving the detection boundary as a function of the signal sparsity and signal strength under the sparse alternative hypotheses. It first shows that the optimal detection boundary for testing sparse means is the minimax detection lower boundary for testing the covariance matrix. A multilevel thresholding test is proposed and is shown to be able to attain the detection lower boundary over a substantial range of the sparsity parameter, implying that the multilevel thresholding test is sharp optimal in the minimax sense over the range. The asymptotic distribution of the multilevel thresholding statistic for covariance matrices is derived under both Gaussian and non-Gaussian distributions by developing a novel U-statistic decomposition in conjunction with the matrix blocking and the coupling techniques to handle the complex dependence among the elements of the sample covariance matrix. The superiority in the detection boundary of the multilevel thresholding test over the existing tests is also demonstrated.

The linear fractional stable motion generalizes two prominent classes of stochastic processes, namely stable Lévy processes, and fractional Brownian motion. For this reason, it may be regarded as a basic building block for continuous time models. We study a stylized model consisting of a superposition of independent linear fractional stable motions and our focus is on parameter estimation of the model. Applying an estimating equations approach, we construct estimators for the whole set of parameters and derive their asymptotic normality in a high-frequency regime. The conditions for consistency turn out to be sharp for two prominent special cases: (i) for Lévy processes, that is, for the estimation of the successive Blumenthal–Getoor indices and (ii) for the mixed fractional Brownian motion introduced by Cheridito. In the remaining cases, our results reveal a delicate interplay between the Hurst parameters and the indices of stability. Our asymptotic theory is based on new limit theorems for multiscale moving average processes.

This paper develops a new approach to post-selection inference for screening high-dimensional predictors of survival outcomes. Post-selection inference for right-censored outcome data has been investigated in the literature, but much remains to be done to make the methods both reliable and computationally-scalable in high dimensions. Machine learning tools are commonly used to provide predictions of survival outcomes, but the estimated effect of a selected predictor suffers from confirmation bias unless the selection is taken into account. The new approach involves the construction of semiparametrically efficient estimators of the linear association between the predictors and the survival outcome, which are used to build a test statistic for detecting the presence of an association between any of the predictors and the outcome. Further, a stabilization technique reminiscent of bagging allows a normal calibration for the resulting test statistic, which enables the construction of confidence intervals for the maximal association between predictors and the outcome and also greatly reduces computational cost. Theoretical results show that this testing procedure is valid even when the number of predictors grows superpolynomially with sample size, and our simulations support this asymptotic guarantee at moderate sample sizes. The new approach is applied to the problem of identifying patterns in viral gene expression associated with the potency of an antiviral drug.

Topic models provide a useful tool to organize and understand the structure of large corpora of text documents, in particular, to discover hidden thematic structure. Clustering documents from big unstructured corpora into topics is an important task in various fields, such as image analysis, e-commerce, social networks and population genetics. Since the number of topics is typically substantially smaller than the size of the corpus and of the dictionary, the methods of topic modeling can lead to a dramatic dimension reduction. We study the problem of estimating the topic-document matrix, which gives the topics distribution for each document in a given corpus, that is, we focus on the clustering aspect of the problem. We introduce an algorithm that we call Successive Projection Overlapping Clustering (SPOC) inspired by the successive projection algorithm for separable matrix factorization. This algorithm is simple to implement and computationally fast. We establish upper bounds on the performance of the SPOC algorithm for estimation of the topic-document matrix, as well as near matching minimax lower bounds. We also propose a method that achieves analogous results when the number of topics is unknown and provides an estimate of the number of topics. Our theoretical results are complemented with a numerical study on synthetic and semisynthetic data.

We study the multitask learning problem that aims to simultaneously analyze multiple data sets collected from different sources and learn one model for each of them. We propose a family of adaptive methods that automatically utilize possible similarities among those tasks while carefully handling their differences. We derive sharp statistical guarantees for the methods and prove their robustness against outlier tasks. Numerical experiments on synthetic and real data sets demonstrate the efficacy of our new methods.

Nonparametric inference on tail conditional quantiles and their least squares analogs, expectiles, remains limited to i.i.d. data. We develop a fully operational inferential theory for extreme conditional quantiles and expectiles in the challenging framework of α-mixing, conditional heavy-tailed data whose tail index may vary with covariate values. This requires a dedicated treatment to deal with data sparsity in the far tail of the response, in addition to handling difficulties inherent to mixing, smoothing and sparsity associated to covariate localization. We prove the pointwise asymptotic normality of our estimators and obtain optimal rates of convergence reminiscent of those found in the i.i.d. regression setting, but which had not been established in the conditional extreme value literature. Our assumptions hold in a wide range of models. We propose full bias and variance reduction procedures, and simple but effective data-based rules for selecting tuning hyperparameters. Our inference strategy is shown to perform well in finite samples and is showcased in applications to stock returns and tornado loss data.

We propose a general optimization-based framework for computing differentially private M-estimators and a new method for constructing differentially private confidence regions. First, we show that robust statistics can be used in conjunction with noisy gradient descent or noisy Newton methods in order to obtain optimal private estimators with global linear or quadratic convergence, respectively. We establish local and global convergence guarantees, under both local strong convexity and self-concordance, showing that our private estimators converge with high probability to a small neighborhood of the nonprivate M-estimators. Second, we tackle the problem of parametric inference by constructing differentially private estimators of the asymptotic variance of our private M-estimators. This naturally leads to approximate pivotal statistics for constructing confidence regions and conducting hypothesis testing. We demonstrate the effectiveness of a bias correction that leads to enhanced small-sample empirical performance in simulations. We illustrate the benefits of our methods in several numerical examples.

A stylized feature of high-dimensional data is that many variables have heavy tails, and robust statistical inference is critical for valid large-scale statistical inference. Yet, the existing developments such as Winsorization, Huberization and median of means require the bounded second moments and involve variable-dependent tuning parameters, which hamper their fidelity in applications to large-scale problems. To liberate these constraints, this paper revisits the celebrated Hodges–Lehmann (HL) estimator for estimating location parameters in both the one- and two-sample problems, from a nonasymptotic perspective. Our study develops Berry–Esseen inequality and Cramér-type moderate deviation for the HL estimator based on newly developed nonasymptotic Bahadur representation and builds data-driven confidence intervals via a weighted bootstrap approach. These results allow us to extend the HL estimator to large-scale studies and propose tuning-free and moment-free high-dimensional inference procedures for testing global null and for large-scale multiple testing with false discovery proportion control. It is convincingly shown that the resulting tuning-free and moment-free methods control false discovery proportion at a prescribed level. The simulation studies lend further support to our developed theory.

Conditional local independence is an asymmetric independence relation among continuous time stochastic processes. It describes whether the evolution of one process is directly influenced by another process given the histories of additional processes, and it is important for the description and learning of causal relations among processes. We develop a model-free framework for testing the hypothesis that a counting process is conditionally locally independent of another process. To this end, we introduce a new functional parameter called the Local Covariance Measure (LCM), which quantifies deviations from the hypothesis. Following the principles of double machine learning, we propose an estimator of the LCM and a test of the hypothesis using nonparametric estimators and sample splitting or cross-fitting. We call this test the (cross-fitted) Local Covariance Test ((X)-LCT), and we show that its level and power can be controlled uniformly, provided that the nonparametric estimators are consistent with modest rates. We illustrate the theory by an example based on a marginalized Cox model with time-dependent covariates, and we show in simulations that when double machine learning is used in combination with cross-fitting, then the test works well without restrictive parametric assumptions.

In the context of state-space models, skeleton-based smoothing algorithms rely on a backward sampling step, which by default, has a $\mathcal{O}({\mathit{N}}^2)$ complexity (where N is the number of particles). Existing improvements in the literature are unsatisfactory: a popular rejection sampling-based approach, as we shall show, might lead to badly behaved execution time; another rejection sampler with stopping lacks complexity analysis; yet another MCMC-inspired algorithm comes with no stability guarantee. We provide several results that close these gaps. In particular, we prove a novel nonasymptotic stability theorem, thus enabling smoothing with truly linear complexity and adequate theoretical justification. We propose a general framework, which unites most skeleton-based smoothing algorithms in the literature and allows to simultaneously prove their convergence and stability, both in online and offline contexts. Furthermore, we derive, as a special case of that framework, a new coupling-based smoothing algorithm applicable to models with intractable transition densities. We elaborate practical recommendations and confirm those with numerical experiments.

Given a set of incomplete observations, we study the nonparametric problem of testing whether data are Missing Completely At Random (MCAR). Our first contribution is to characterise precisely the set of alternatives that can be distinguished from the MCAR null hypothesis. This reveals interesting and novel links to the theory of Fréchet classes (in particular, compatible distributions) and linear programming, that allow us to propose MCAR tests that are consistent against all detectable alternatives. We define an incompatibility index as a natural measure of ease of detectability, establish its key properties and show how it can be computed exactly in some cases and bounded in others. Moreover, we prove that our tests can attain the minimax separation rate according to this measure, up to logarithmic factors. Our methodology does not require any complete cases to be effective, and is available in the $\mathtt{R}$ package $\mathtt{MCARtest}$.

The Lasso is a method for high-dimensional regression, which is now commonly used when the number of covariates p is of the same order or larger than the number of observations n. Classical asymptotic normality theory does not apply to this model due to two fundamental reasons: $(1)$ The regularized risk is nonsmooth; $(2)$ The distance between the estimator $\hat{\mathit{\theta }}$ and the true parameters vector ${\mathit{\theta }}^{\ast }$ cannot be neglected. As a consequence, standard perturbative arguments that are the traditional basis for asymptotic normality fail.

On the other hand, the Lasso estimator can be precisely characterized in the regime in which both n and p are large and $\mathit{n}/\mathit{p}$ is of order one. This characterization was first obtained in the case of Gaussian designs with i.i.d. covariates: here we generalize it to Gaussian correlated designs with non-singular covariance structure. This is expressed in terms of a simpler “fixed-design” model. We establish nonasymptotic bounds on the distance between the distribution of various quantities in the two models, which hold uniformly over signals ${\mathit{\theta }}^{\ast }$ in a suitable sparsity class and over values of the regularization parameter.

As an application, we study the distribution of the debiased Lasso and show that a degrees-of-freedom correction is necessary for computing valid confidence intervals.

We propose a concept of quantiles for probability measures on the unit hypersphere ${\mathcal{S}}^{\mathit{d}-1}$ of ${\mathbb{R}}^{\mathit{d}}$. The innermost quantile is the Fréchet median, that is, the ${\mathit{L}}_1$-analog of the Fréchet mean. The proposed quantiles ${\mathit{\mu }}_{\mathit{\alpha },\mathit{u}}^{\mathit{m}}$ are directional in nature: they are indexed by a scalar order $\mathit{\alpha }\in [0,1]$ and a unit vector u in the tangent space ${\mathit{T}}_{\mathit{m}}{\mathcal{S}}^{\mathit{d}-1}$ to ${\mathcal{S}}^{\mathit{d}-1}$ at m. To ensure computability in any dimension d, our quantiles are essentially obtained by considering the Euclidean (Chaudhuri (J. Amer. Statist. Assoc. 91 (1996) 862–872)) spatial quantiles in a suitable stereographic projection of ${\mathcal{S}}^{\mathit{d}-1}$ onto ${\mathit{T}}_{\mathit{m}}{\mathcal{S}}^{\mathit{d}-1}$. Despite this link with Euclidean spatial quantiles, studying the proposed spherical quantiles requires understanding the nature of the (Chaudhuri (1996)) quantiles in a version of the projective space where all points at infinity are identified. We thoroughly investigate the structural properties of our quantiles and we further study the asymptotic behavior of their sample versions, which requires controlling the impact of estimating m. Our spherical quantile concept also allows for companion concepts of ranks and depth on the hypersphere. We illustrate the relevance of our construction by considering two inferential applications, related to supervised classification and to testing for rotational symmetry.

In this paper, we establish the central limit theorem (CLT) for linear spectral statistics (LSSs) of a large-dimensional sample covariance matrix when the population covariance matrices are involved with diverging spikes. This constitutes a nontrivial extension of the Bai–Silverstein theorem (BST) (Ann. Probab. 32 (2004) 553–605), a theorem that has strongly influenced the development of high-dimensional statistics, especially in the applications of random matrix theory to statistics. Recently, there has been a growing realization that the assumption of uniform boundedness of the population covariance matrices in the BST is not satisfied in some fields, such as economics, where the variances of principal components may diverge as the dimension tends to infinity. Therefore, in this paper, we aim to eliminate this obstacle to applications of the BST. Our new CLT accommodates spiked eigenvalues, which may either be bounded or tend to infinity. A distinguishing feature of our result is that the variance in the new CLT is related to both spiked eigenvalues and bulk eigenvalues, with dominance being determined by the divergence rate of the largest spiked eigenvalues. The new CLT for LSS is then applied to test the hypothesis that the population covariance matrix is the identity matrix or a generalized spiked model. The asymptotic distributions of the corrected likelihood ratio test statistic and the corrected Nagao’s trace test statistic are derived under the alternative hypothesis. Moreover, we present power comparisons between these two LSSs and Roy’s largest root test. In particular, we demonstrate that except for the case in which there is only one spike, the LSSs could exhibit higher asymptotic power than Roy’s largest root test.

The expected Euler characteristic (EEC) of excursion sets of a smooth Gaussian-related random field over a compact manifold approximates the distribution of its supremum for high thresholds. Viewed as a function of the excursion threshold, the EEC of a Gaussian-related field is expressed by the Gaussian kinematic formula (GKF) as a finite sum of known functions multiplied by the Lipschitz–Killing curvatures (LKCs) of the generating Gaussian field. This paper proposes consistent estimators of the LKCs as linear projections of “pinned” Euler characteristic (EC) curves obtained from realizations of zero-mean, unit variance Gaussian processes. As observed, data seldom is Gaussian and the exact mean and variance is unknown, yet the statistic of interest often satisfies a CLT with a Gaussian limit process; we adapt our LKC estimators to this scenario using a Gaussian multiplier bootstrap approach. This yields consistent estimates of the LKCs of the possibly nonstationary Gaussian limiting field that have low variance and are computationally efficient for complex underlying manifolds. For the EEC of the limiting field, a parametric plug-in estimator is presented, which is more efficient than the nonparametric average of EC curves. The proposed methods are evaluated using simulations of 2D fields, and illustrated on cosmological observations and simulations on the 2-sphere and 3D fMRI volumes.