Estimation of the number of components (or order) of a finite mixture model is a long standing and challenging problem in statistics. We propose the Group-Sort-Fuse (GSF) procedure—a new penalized likelihood approach for simultaneous estimation of the order and mixing measure in multidimensional finite mixture models. Unlike methods which fit and compare mixtures with varying orders using criteria involving model complexity, our approach directly penalizes a continuous function of the model parameters. More specifically, given a conservative upper bound on the order, the GSF groups and sorts mixture component parameters to fuse those which are redundant. For a wide range of finite mixture models, we show that the GSF is consistent in estimating the true mixture order and achieves the ${\mathit{n}}^{-1/2}$ convergence rate for parameter estimation up to polylogarithmic factors. The GSF is implemented for several univariate and multivariate mixture models in the R package $\mathtt{GroupSortFuse}$. Its finite sample performance is supported by a thorough simulation study, and its application is illustrated on two real data examples.

We propose a coefficient of conditional dependence between two random variables Y and Z given a set of other variables ${\mathit{X}}_1,\dots ,{\mathit{X}}_{\mathit{p}}$, based on an i.i.d. sample. The coefficient has a long list of desirable properties, the most important of which is that under absolutely no distributional assumptions, it converges to a limit in $[0,1]$, where the limit is 0 if and only if Y and Z are conditionally independent given ${\mathit{X}}_1,\dots ,{\mathit{X}}_{\mathit{p}}$, and is 1 if and only if Y is equal to a measurable function of Z given ${\mathit{X}}_1,\dots ,{\mathit{X}}_{\mathit{p}}$. Moreover, it has a natural interpretation as a nonlinear generalization of the familiar partial ${\mathit{R}}^2$ statistic for measuring conditional dependence by regression. Using this statistic, we devise a new variable selection algorithm, called Feature Ordering by Conditional Independence (FOCI), which is model-free, has no tuning parameters, and is provably consistent under sparsity assumptions. A number of applications to synthetic and real data sets are worked out.

Let ${({\mathit{Y}}_{\mathit{t}})}_{\mathit{t}\ge 1}$ be a sequence of i.i.d. observations and $\{{\mathit{f}}_{\mathit{\theta }},\mathit{\theta }\in {\mathbb{R}}^{\mathit{d}}\}$ be a parametric model. We introduce a new online algorithm for computing a sequence ${({\hat{\mathit{\theta }}}_{\mathit{t}})}_{\mathit{t}\ge 1}$, which is shown to converge almost surely to ${argmax}_{\mathit{\theta }\in {\mathbb{R}}^{\mathit{d}}}\mathbb{E}[log{\mathit{f}}_{\mathit{\theta }}({\mathit{Y}}_1)]$ at rate $\mathcal{O}(log{(\mathit{t})}^{(1+\mathit{\epsilon })/2}{\mathit{t}}^{-1/2})$, with $\mathit{\epsilon }>0$ a user specified parameter. This convergence result is obtained under standard conditions on the statistical model and, most notably, we allow the mapping $\mathit{\theta }\mapsto \mathbb{E}[log{\mathit{f}}_{\mathit{\theta }}({\mathit{Y}}_1)]$ to be multi-modal. However, the computational cost to process each observation grows exponentially with the dimension of θ, which makes the proposed approach applicable to low or moderate dimensional problems only. We also derive a version of the estimator ${\hat{\mathit{\theta }}}_{\mathit{t}}$, which is well suited to student-t linear regression models that are popular tools for robust linear regression. As shown by experiments on simulated and real data, the corresponding estimator of the regression coefficients is, as expected, robust to the presence of outliers and thus, as a by-product, we obtain a new adaptive and robust online estimation procedure for linear regression models.

A method is proposed for estimating the microergodic parameters (including the smoothness parameter) of stationary Gaussian random fields on ${\mathbb{R}}^{\mathit{d}}$ with isotropic Matérn covariance functions using irregularly spaced data. This approach uses higher-order quadratic variations and is applied to three designs, namely stratified sampling design, randomized sampling design and deformed lattice design. Microergodic parameter estimators are constructed for each of the designs. Under mild conditions, these estimators are shown to be consistent with respect to fixed-domain asymptotics. Upper bounds to the convergence rate of the estimators are also established. A simulation study is conducted to gauge the accuracy of the proposed estimators.

We derive improved regression and classification rates for support vector machines using Gaussian kernels under the assumption that the data has some low-dimensional intrinsic structure that is described by the box-counting dimension. Under some standard regularity assumptions for regression and classification, we prove learning rates, in which the dimension of the ambient space is replaced by the box-counting dimension of the support of the data generating distribution. In the regression case, our rates are in some cases minimax optimal up to logarithmic factors, whereas in the classification case our rates are minimax optimal up to logarithmic factors in a certain range of our assumptions and otherwise of the form of the best known rates. Furthermore, we show that a training validation approach for choosing the hyperparameters of a SVM in a data dependent way achieves the same rates adaptively, that is, without any knowledge on the data generating distribution.

We study the problem of community detection in multilayer networks, where pairs of nodes can be related in multiple modalities. We introduce a general framework, that is, mixture multilayer stochastic block model (MMSBM), which includes many earlier models as special cases. We propose a tensor-based algorithm (TWIST) to reveal both global/local memberships of nodes, and memberships of layers. We show that the TWIST procedure can accurately detect the communities with small misclassification error as the number of nodes and/or number of layers increases. Numerical studies confirm our theoretical findings. To our best knowledge, this is the first systematic study on the mixture multilayer networks using tensor decomposition. The method is applied to two real datasets: worldwide trading networks and malaria parasite genes networks, yielding new and interesting findings.

Many statistical estimands can expressed as continuous linear functionals of a conditional expectation function. This includes the average treatment effect under unconfoundedness and generalizations for continuous-valued and personalized treatments. In this paper, we discuss a general approach to estimating such quantities: we begin with a simple plug-in estimator based on an estimate of the conditional expectation function, and then correct the plug-in estimator by subtracting a minimax linear estimate of its error. We show that our method is semiparametrically efficient under weak conditions and observe promising performance on both real and simulated data.

The empirical likelihood inference is extended to a class of semiparametric models for stationary, weakly dependent series. A partially linear single-index regression is used for the conditional mean of the series given its past, and the present and past values of a vector of covariates. A parametric model for the conditional variance of the series is added to capture further nonlinear effects. We propose suitable moment equations which characterize the mean and variance model. We derive an empirical log-likelihood ratio which includes nonparametric estimators of several functions, and we show that this ratio behaves asymptotically as if the functions were given.

Bayesian inference and uncertainty quantification in a general class of nonlinear inverse regression models is considered. Analytic conditions on the regression model $\{\mathcal{G}(\mathit{\theta }):\mathit{\theta }\in \mathrm{\Theta }\}$ and on Gaussian process priors for θ are provided such that semiparametrically efficient inference is possible for a large class of linear functionals of θ. A general Bernstein–von Mises theorem is proved that shows that the (non-Gaussian) posterior distributions are approximated by certain Gaussian measures centred at the posterior mean. As a consequence, posterior-based credible sets are valid and optimal from a frequentist point of view. The theory is illustrated with two applications with PDEs that arise in nonlinear tomography problems: an elliptic inverse problem for a Schrödinger equation, and inversion of non-Abelian X-ray transforms. New analytical techniques are deployed to show that the relevant Fisher information operators are invertible between suitable function spaces.

Transfer Learning addresses common situations in Machine Leaning where little or no labeled data is available for a target prediction problem—corresponding to a distribution Q, but much labeled data is available from some related but different data distribution P. This work is concerned with the fundamental limits of transfer, that is, the limits in target performance in terms of (1) sample sizes from P and Q, and (2) differences in data distributions P, Q. In particular, we aim to address practical questions such as how much target data from Q is sufficient given a certain amount of related data from P, and how to optimally sample such target data for labeling.

We present new minimax results for transfer in nonparametric classification (i.e., for situations where little is known about the target classifier), under the common assumption that the marginal distributions of covariates differ between P and Q (often termed covariate-shift). Our results are first to concisely capture the relative benefits of source and target labeled data in these settings through information-theoretic limits. Namely, we show that the benefits of target labels are tightly controlled by a transfer-exponent γ that encodes how singular Q is locally with respect to P, and interestingly paints a more favorable picture of transfer than what might be believed from insights from previous work. In fact, while previous work rely largely on refinements of traditional metrics and divergences between distributions, and often only yield a coarse view of when transfer is possible or not, our analysis—in terms of γ—reveals a continuum of new regimes ranging from easy to hard transfer.

We then address the practical question of how to efficiently sample target data to label, by showing that a recently proposed semi-supervised procedure—based on k-NN classification, can be refined to adapt to unknown γ and, therefore, requests target labels only when beneficial, while achieving nearly minimax-optimal transfer rates without knowledge of distributional parameters. Of independent interest, we obtain new minimax-optimality results for vanilla k-NN classification in regimes with nonuniform marginals.

Manski’s celebrated maximum score estimator for the discrete choice model, which is an optimal linear discriminator, has been the focus of much investigation in both the econometrics and statistics literatures, but its behavior under growing dimension scenarios largely remains unknown. This paper addresses that gap. Two different cases are considered: p grows with n but at a slow rate, that is, $\mathit{p}/\mathit{n}\to 0$; and $\mathit{p}\gg \mathit{n}$ (fast growth). In the binary response model, we recast Manski’s score estimation as empirical risk minimization for a classification problem, and derive the ${\ell }_2$ rate of convergence of the score estimator under a new transition condition in terms of a margin parameter that calibrates the level of difficulty of the estimation problem. We also establish upper and lower bounds for the minimax ${\ell }_2$ error in the binary choice model that differ by a logarithmic factor, and construct a minimax-optimal estimator in the slow growth regime. Some extensions to the multinomial choice model are also considered.

Expectiles define a least squares analogue of quantiles. They have been the focus of a substantial quantity of research in the context of actuarial and financial risk assessment over the last decade. The behaviour and estimation of unconditional extreme expectiles using independent and identically distributed heavy-tailed observations have been investigated in a recent series of papers. We build here a general theory for the estimation of extreme conditional expectiles in heteroscedastic regression models with heavy-tailed noise; our approach is supported by general results of independent interest on residual-based extreme value estimators in heavy-tailed regression models, and is intended to cope with covariates having a large but fixed dimension. We demonstrate how our results can be applied to a wide class of important examples, among which are linear models, single-index models as well as ARMA and GARCH time series models. Our estimators are showcased on a numerical simulation study and on real sets of actuarial and financial data.

This paper develops a general theory on rates of convergence of penalized spline estimators for function estimation when the likelihood functional is concave in candidate functions, where the likelihood is interpreted in a broad sense that includes conditional likelihood, quasi-likelihood and pseudo-likelihood. The theory allows all feasible combinations of the spline degree, the penalty order and the smoothness of the unknown functions. According to this theory, the asymptotic behaviors of the penalized spline estimators depends on interplay between the spline knot number and the penalty parameter. The general theory is applied to obtain results in a variety of contexts, including regression, generalized regression such as logistic regression and Poisson regression, density estimation, conditional hazard function estimation for censored data, quantile regression, diffusion function estimation for a diffusion type process and estimation of spectral density function of a stationary time series. For multidimensional function estimation, the theory (presented in the Supplementary Material) covers both penalized tensor product splines and penalized bivariate splines on triangulations.

Given a symmetric social network, we are interested in testing whether it has only one community or multiple communities. The desired tests should (a) accommodate severe degree heterogeneity, (b) accommodate mixed memberships, (c) have a tractable null distribution and (d) adapt automatically to different levels of sparsity, and achieve the optimal phase diagram. How to find such a test is a challenging problem.

We propose the Signed Polygon as a class of new tests. Fixing $\mathit{m}\ge 3$, for each m-gon in the network, define a score using the centered adjacency matrix. The sum of such scores is then the mth order Signed Polygon statistic. The Signed Triangle (SgnT) and the Signed Quadrilateral (SgnQ) are special examples of the Signed Polygon.

We show that both the SgnT and SgnQ tests satisfy (a)–(d), and especially, they work well for both very sparse and less sparse networks. Our proposed tests compare favorably with existing tests. For example, the EZ and GC tests behave unsatisfactorily in the less sparse case and do not achieve the optimal phase diagram. Also, many existing tests do not allow for severe heterogeneity or mixed memberships, and they behave unsatisfactorily in our settings.

The analysis of the SgnT and SgnQ tests is delicate and extremely tedious, and the main reason is that we need a unified proof that covers a wide range of sparsity levels and a wide range of degree heterogeneity. For lower bound theory, we use a phase transition framework, which includes the standard minimax argument, but is more informative. The proof uses classical theorems on matrix scaling.

Modern statistical applications often involve minimizing an objective function that may be nonsmooth and/or nonconvex. This paper focuses on a broad Bregman-surrogate algorithm framework including the local linear approximation, mirror descent, iterative thresholding, DC programming and many others as particular instances. The recharacterization via generalized Bregman functions enables us to construct suitable error measures and establish global convergence rates for nonconvex and nonsmooth objectives in possibly high dimensions. For sparse learning problems with a composite objective, under some regularity conditions, the obtained estimators as the surrogate’s fixed points, though not necessarily local minimizers, enjoy provable statistical guarantees, and the sequence of iterates can be shown to approach the statistical truth within the desired accuracy geometrically fast. The paper also studies how to design adaptive momentum based accelerations without assuming convexity or smoothness by carefully controlling stepsize and relaxation parameters.

We present a novel approach to test for heteroscedasticity of a nonstationary time series that is based on Gini’s mean difference of logarithmic local sample variances. In order to analyse the large sample behaviour of our test statistic, we establish new limit theorems for U-statistics of dependent triangular arrays. We derive the asymptotic distribution of the test statistic under the null hypothesis of a constant variance and show that the test is consistent against a large class of alternatives, including multiple structural breaks in the variance. Our test is applicable even in the case of nonstationary processes, assuming a locally stationary mean function. The performance of the test and its comparatively low computation time are illustrated in an extensive simulation study. As an application, we analyse Google Trends data, monitoring the relative search interest for the topic “global warming.”

This work affords new insights into Bayesian CART in the context of structured wavelet shrinkage. The main thrust is to develop a formal inferential framework for Bayesian tree-based regression. We reframe Bayesian CART as a g-type prior which departs from the typical wavelet product priors by harnessing correlation induced by the tree topology. The practically used Bayesian CART priors are shown to attain adaptive near rate-minimax posterior concentration in the supremum norm in regression models. For the fundamental goal of uncertainty quantification, we construct adaptive confidence bands for the regression function with uniform coverage under self-similarity. In addition, we show that tree-posteriors enable optimal inference in the form of efficient confidence sets for smooth functionals of the regression function.

Non-Euclidean data that are indexed with a scalar predictor such as time are increasingly encountered in data applications, while statistical methodology and theory for such random objects are not well developed yet. To address the need for new methodology in this area, we develop a total variation regularization technique for nonparametric Fréchet regression, which refers to a regression setting where a response residing in a metric space is paired with a scalar predictor and the target is a conditional Fréchet mean. Specifically, we seek to approximate an unknown metric-space valued function by an estimator that minimizes the Fréchet version of least squares and at the same time has small total variation, appropriately defined for metric-space valued objects. We show that the resulting estimator is representable by a piece-wise constant function and establish the minimax convergence rate of the proposed estimator for metric data objects that reside in Hadamard spaces. We illustrate the numerical performance of the proposed method for both simulated and real data, including metric spaces of symmetric positive-definite matrices with the affine-invariant distance, of probability distributions on the real line with the Wasserstein distance, and of phylogenetic trees with the Billera–Holmes–Vogtmann metric.

Estimation of the population size n from k i.i.d. binomial observations with unknown success probability p is relevant to a multitude of applications and has a long history. Without additional prior information this is a notoriously difficult task when p becomes small, and the Bayesian approach becomes particularly useful. For a large class of priors, we establish posterior contraction and a Bernstein-von Mises type theorem in a setting where $\mathit{p}\to 0$ and $\mathit{n}\to \infty$ as $\mathit{k}\to \infty$. Furthermore, we suggest a new class of Bayesian estimators for n and provide a comprehensive simulation study in which we investigate their performance. To showcase the advantages of a Bayesian approach on real data, we also benchmark our estimators in a novel application from super-resolution microscopy.

We consider Bayesian nonparametric inference in the right-censoring survival model, where modeling is made at the level of the hazard rate. We derive posterior limiting distributions for linear functionals of the hazard, and then for ‘many’ functionals simultaneously in appropriate multiscale spaces. As an application, we derive Bernstein–von Mises theorems for the cumulative hazard and survival functions, which lead to asymptotically efficient confidence bands for these quantities. Further, we show optimal posterior contraction rates for the hazard in terms of the supremum norm. In medical studies, a popular approach is to model hazards a priori as random histograms with possibly dependent heights. This and more general classes of arbitrarily smooth prior distributions are considered as applications of our theory. A sampler is provided for possibly dependent histogram posteriors. Its finite sample properties are investigated on both simulated and real data experiments.

Classical change point analysis aims at (1) detecting abrupt changes in the mean of a possibly nonstationary time series and at (2) identifying regions where the mean exhibits a piecewise constant behavior. In many applications however, it is more reasonable to assume that the mean changes gradually in a smooth way. Those gradual changes may either be nonrelevant (i.e., small), or relevant for a specific problem at hand, and the present paper presents statistical methodology to detect the latter. More precisely, we consider the common nonparametric regression model ${\mathit{X}}_{\mathit{i}}=\mathit{\mu }(\mathit{i}/\mathit{n})+{\mathit{\epsilon }}_{\mathit{i}}$ with centered errors and propose a test for the null hypothesis that the maximum absolute deviation of the regression function μ from a functional $\mathit{g}(\mathit{\mu })$ (such as the value $\mathit{\mu }(0)$ or the integral ${\int }_0^1\mathit{\mu }(\mathit{t})\mathrm{d}\mathit{t}$) is smaller than a given threshold on a given interval $[{\mathit{x}}_0,{\mathit{x}}_1]\subseteq [0,1]$. A test for this type of hypotheses is developed using an appropriate estimator, say ${\hat{\mathit{d}}}_{\infty ,\mathit{n}}$, for the maximum deviation ${\mathit{d}}_{\infty }={sup}_{\mathit{t}\in [{\mathit{x}}_0,{\mathit{x}}_1]}|\mathit{\mu }(\mathit{t})-\mathit{g}(\mathit{\mu })|$. We derive the limiting distribution of an appropriately standardized version of ${\hat{\mathit{d}}}_{\infty ,\mathit{n}}$, where the standardization depends on the Lebesgue measure of the set of extremal points of the function $\mathit{\mu }(·)-\mathit{g}(\mathit{\mu })$. A refined procedure based on an estimate of this set is developed and its consistency is proved. The results are illustrated by means of a simulation study and a data example.

In transfer learning, we wish to make inference about a target population when we have access to data both from the distribution itself, and from a different but related source distribution. We introduce a flexible framework for transfer learning in the context of binary classification, allowing for covariate-dependent relationships between the source and target distributions that are not required to preserve the Bayes decision boundary. Our main contributions are to derive the minimax optimal rates of convergence (up to poly-logarithmic factors) in this problem, and show that the optimal rate can be achieved by an algorithm that adapts to key aspects of the unknown transfer relationship, as well as the smoothness and tail parameters of our distributional classes. This optimal rate turns out to have several regimes, depending on the interplay between the relative sample sizes and the strength of the transfer relationship, and our algorithm achieves optimality by careful, decision tree-based calibration of local nearest-neighbour procedures.