We study a general class of quasi-maximum likelihood estimators for observation-driven time series models. Our main focus is on models related to the exponential family of distributions like Poisson based models for count time series or duration models. However, the proposed approach is more general and covers a variety of time series models including the ordinary GARCH model which has been studied extensively in the literature. We provide general conditions under which quasi-maximum likelihood estimators can be analyzed for this class of time series models and we prove that these estimators are consistent and asymptotically normally distributed regardless of the true data generating process. We illustrate our results using classical examples of quasi-maximum likelihood estimation including standard GARCH models, duration models, Poisson type autoregressions and ARMA models with GARCH errors. Our contribution unifies the existing theory and gives conditions for proving consistency and asymptotic normality in a variety of situations.

In this paper a robust version of the classical Wald test statistics for linear hypothesis in the logistic regression model is introduced and its properties are explored. We study the problem under the assumption of random covariates although some ideas with non random covariates are also considered. A family of robust Wald type tests are considered here, where the minimum density power divergence estimator is used instead of the maximum likelihood estimator. We obtain the asymptotic distribution and also study the robustness properties of these Wald type test statistics. The robustness of the tests is investigated theoretically through the influence function analysis as well as suitable practical examples. It is theoretically established that the level as well as the power of the Wald-type tests are stable against contamination, while the classical Wald type test breaks down in this scenario. Some classical examples are presented which numerically substantiate the theory developed. Finally a simulation study is included to provide further confirmation of the validity of the theoretical results established in the paper.

It is widely pointed out in the literature that misspecification of a parametric model can lead to inconsistent estimators and wrong inference. However, even a misspecified model can provide some valuable information about the phenomena under study. This is the main idea behind the development of an approach known, in the literature, as parametrically guided nonparametric estimation. Due to its promising bias reduction property, this approach has been investigated in different frameworks such as density estimation, least squares regression and local quasi-likelihood. Our contribution is concerned with parametrically guided local quasi-likelihood estimation adapted to randomly right censored data. The generalization to censored data involves synthetic data and local linear fitting. The asymptotic properties of the guided estimator as well as its finite sample performance are studied and compared with the unguided local quasi-likelihood estimator. The results confirm the bias reduction property and show that, using an appropriate guide and an appropriate bandwidth, the proposed estimator outperforms the classical local quasi-likelihood estimator.

There is a growing demand for nonparametric conditional density estimators (CDEs) in fields such as astronomy and economics. In astronomy, for example, one can dramatically improve estimates of the parameters that dictate the evolution of the Universe by working with full conditional densities instead of regression (i.e., conditional mean) estimates. More generally, standard regression falls short in any prediction problem where the distribution of the response is more complex with multi-modality, asymmetry or heteroscedastic noise. Nevertheless, much of the work on high-dimensional inference concerns regression and classification only, whereas research on density estimation has lagged behind. Here we propose FlexCode, a fully nonparametric approach to conditional density estimation that reformulates CDE as a non-parametric orthogonal series problem where the expansion coefficients are estimated by regression. By taking such an approach, one can efficiently estimate conditional densities and not just expectations in high dimensions by drawing upon the success in high-dimensional regression. Depending on the choice of regression procedure, our method can adapt to a variety of challenging high-dimensional settings with different structures in the data (e.g., a large number of irrelevant components and nonlinear manifold structure) as well as different data types (e.g., functional data, mixed data types and sample sets). We study the theoretical and empirical performance of our proposed method, and we compare our approach with traditional conditional density estimators on simulated as well as real-world data, such as photometric galaxy data, Twitter data, and line-of-sight velocities in a galaxy cluster.

In this paper we investigate error bounds for convex loss functions for the Lasso in linear models, by first establishing a gap in the theory with respect to the existing error bounds. Then, under the compatibility condition, we recover bounds for the absolute value estimation error and the squared prediction error under mild conditions, which appear to be far more appropriate than the existing bounds for the convex loss Lasso. Interestingly, asymptotically the only difference between the new bounds of the convex loss Lasso and the classical Lasso is a term solely depending on a well-known expression in the robust statistics literature appearing multiplicatively in the bounds. We show that this result holds whether or not the scale parameter needs to be estimated jointly with the regression coefficients. Finally, we use the ratio to optimize our bounds in terms of minimaxity.

This paper considers a nonparametric functional autoregression model of order one. Existing contributions addressing the problem of functional time series prediction have focused on the linear model and literatures are rather lacking in the context of nonlinear functional time series. In our nonparametric setting, we define the functional version of kernel estimator for the autoregressive operator and develop its asymptotic theory under the assumption of a strong mixing condition on the sample. The results are general in the sense that high-order autoregression can be naturally written as a first-order AR model. In addition, a component-wise bootstrap procedure is proposed that can be used for estimating the distribution of the kernel estimation and its asymptotic validity is theoretically justified. The bootstrap procedure is implemented to construct prediction regions that achieve good coverage rate. A supporting simulation study is presented in the end to illustrate the theoretical advances in the paper.

Partially linear models (PLMs) are important generalizations of linear models and are very useful for analyzing high-dimensional data. Compared to linear models, the PLMs possess desirable flexibility of non-parametric regression models because they have both linear and non-linear components. Variable selection for PLMs plays an important role in practical applications and has been extensively studied with respect to the linear component. However, for the non-linear component, variable selection has been well developed only for PLMs with extra structural assumptions such as additive PLMs and generalized additive PLMs. There is currently an unmet need for variable selection methods applicable to general PLMs without structural assumptions on the non-linear component. In this paper, we propose a new variable selection method based on learning gradients for general PLMs without any assumption on the structure of the non-linear component. The proposed method utilizes the reproducing-kernel-Hilbert-space tool to learn the gradients and the group-lasso penalty to select variables. In addition, a block-coordinate descent algorithm is suggested and some theoretical properties are established including selection consistency and estimation consistency. The performance of the proposed method is further evaluated via simulation studies and illustrated using real data.

Recent experimental techniques produce thousands of data of single cell growth, consequently stochastic models of growth can be validated on true data and used to understand the main mechanisms that control the cell cycle. A sequence of growing cells is usually modeled by a suitable Markov chain. In this framework, the most interesting goal is to infer the distribution of the doubling time (or of the added size) of a cell given its initial size and its elongation rate. In the literature, these distributions are described in terms of the corresponding conditional hazard function, referred as division hazard rate. In this work we propose a simple but effective way to estimate the division hazard by using extended Cox modeling. We investigate the convergence to the stationary distribution of the Markov chain describing the sequence of growing cells and we prove that, under reasonable conditions, the proposed estimators of the division hazard rates are asymptotically consistent. Finally, we apply our model to study some published datasets of E-Coli cells.

We consider maximum likelihood estimation with data from a bivariate Gaussian process with a separable exponential covariance model under fixed domain asymptotics. We first characterize the equivalence of Gaussian measures under this model. Then consistency and asymptotic normality for the maximum likelihood estimator of the microergodic parameters are established. A simulation study is presented in order to compare the finite sample behavior of the maximum likelihood estimator with the given asymptotic distribution.

Ordinary least squares regression focuses on the expected response and strongly depends on the assumption of normally distributed errors for inferences. An approach to overcome these restrictions is expectile regression, where no distributional assumption is made but rather the whole distribution of the response is described in terms of covariates. This is similar to quantile regression, but expectiles provide a convenient generalization of the arithmetic mean while quantiles are a generalization of the median. To analyze more complex data structures where purely linear predictors are no longer sufficient, semiparametric regression methods have been introduced for both ordinary least squares and expectile regression. However, with increasing complexity of the data and the regression structure, the selection of the true covariates and their effects becomes even more important than in standard regression models. Therefore we introduce several approaches depending on selection criteria and shrinkage methods to perform model selection in semiparametric expectile regression. Moreover, we propose a joint approach for model selection based on several asymmetries simultaneously to deal with the special feature that expectile regression estimates the complete distribution of the response. Furthermore, to distinguish between linear and smooth predictors, we split nonlinear effects into the purely linear trend and the deviation from this trend. All selection methods are compared with the benchmark of functional gradient descent boosting in a simulation study and applied to determine the relevant covariates when studying childhood malnutrition in Peru.

To make statistical inference about a group of parameters on high-dimensional data, we develop the method of estimator augmentation for the block lasso, which is defined via block norm regularization. By augmenting a block lasso estimator $\hat{\beta }$ with the subgradient $S$ of the block norm evaluated at $\hat{\beta }$, we derive a closed-form density for the joint distribution of $(\hat{\beta },S)$ under a high-dimensional setting. This allows us to draw from an estimated sampling distribution of $\hat{\beta }$, or more generally any function of $(\hat{\beta },S)$, by Monte Carlo algorithms. We demonstrate the application of estimator augmentation in group inference with the group lasso and a de-biased group lasso constructed as a function of $(\hat{\beta },S)$. Our numerical results show that importance sampling via estimator augmentation can be orders of magnitude more efficient than parametric bootstrap in estimating tail probabilities for significance tests. This work also brings new insights into the geometry of the sample space and the solution uniqueness of the block lasso. To broaden its application, we generalize our method to a scaled block lasso, which estimates the error variance simultaneously.

The development of global sensitivity analysis of numerical model outputs has recently raised new issues on 1-dimensional Poincaré inequalities. Typically two kinds of sensitivity indices are linked by a Poincaré type inequality, which provides upper bounds of the most interpretable index by using the other one, cheaper to compute. This allows performing a low-cost screening of unessential variables. The efficiency of this screening then highly depends on the accuracy of the upper bounds in Poincaré inequalities.

The novelty in the questions concern the wide range of probability distributions involved, which are often truncated on intervals. After providing an overview of the existing knowledge and techniques, we add some theory about Poincaré constants on intervals, with improvements for symmetric intervals. Then we exploit the spectral interpretation for computing exact value of Poincaré constants of any admissible distribution on a given interval. We give semi-analytical results for some frequent distributions (truncated exponential, triangular, truncated normal), and present a numerical method in the general case.

Finally, an application is made to a hydrological problem, showing the benefits of the new results in Poincaré inequalities to sensitivity analysis.

Consider a quite arbitrary (semi)parametric model for i.i.d. observations with a Euclidean parameter of interest and assume that an asymptotically (semi)parametrically efficient estimator of it is given. If the parameter of interest is known to lie on a general surface (image of a continuously differentiable vector valued function), we have a submodel in which this constrained Euclidean parameter may be rewritten in terms of a lower-dimensional Euclidean parameter of interest. An estimator of this underlying parameter is constructed based on the given estimator of the original Euclidean parameter, and it is shown to be (semi)parametrically efficient. It is proved that the efficient score function for the underlying parameter is determined by the efficient score function for the original parameter and the Jacobian of the function defining the general surface, via a chain rule for score functions. Efficient estimation of the constrained Euclidean parameter itself is considered as well.

Our general estimation method is applied to location-scale, Gaussian copula and semiparametric regression models, and to parametric models.

We address the question of estimating Kullback-Leibler losses rather than squared losses in recovery problems where the noise is distributed within the exponential family. Inspired by Stein unbiased risk estimator (SURE), we exhibit conditions under which these losses can be unbiasedly estimated or estimated with a controlled bias. Simulations on parameter selection problems in applications to image denoising and variable selection with Gamma and Poisson noises illustrate the interest of Kullback-Leibler losses and the proposed estimators.

We study asymptotically minimax predictive distributions in infinite sequence models. First, we discuss the connection between prediction in an infinite sequence model and prediction in a function model. Second, we construct an asymptotically minimax predictive distribution for the setting in which the parameter space is a known ellipsoid. We show that the Bayesian predictive distribution based on the Gaussian prior distribution is asymptotically minimax in the ellipsoid. Third, we construct an asymptotically minimax predictive distribution for any Sobolev ellipsoid. We show that the Bayesian predictive distribution based on the product of Stein’s priors is asymptotically minimax for any Sobolev ellipsoid. Finally, we present an efficient sampling method from the proposed Bayesian predictive distribution.

We investigate the frequentist properties of Bayesian procedures for estimation based on the horseshoe prior in the sparse multivariate normal means model. Previous theoretical results assumed that the sparsity level, that is, the number of signals, was known. We drop this assumption and characterize the behavior of the maximum marginal likelihood estimator (MMLE) of a key parameter of the horseshoe prior. We prove that the MMLE is an effective estimator of the sparsity level, in the sense that it leads to (near) minimax optimal estimation of the underlying mean vector generating the data. Besides this empirical Bayes procedure, we consider the hierarchical Bayes method of putting a prior on the unknown sparsity level as well. We show that both Bayesian techniques lead to rate-adaptive optimal posterior contraction, which implies that the horseshoe posterior is a good candidate for generating rate-adaptive credible sets.

How to recover weak signals (i.e., small nonzero regression coefficients) is a difficult task in high dimensional feature selection problems. Both convex and nonconvex regularization methods fail to fully recover the true model whenever there exist strong columnwise correlations in design matrices or small nonzero coefficients below some threshold. To address the two challenges, we propose a procedure, Perturbed LASSO (PLA), that weakens correlations in the design matrix and strengthens signals by adding random perturbations to the design matrix. Moreover, a quantitative relationship between the selection accuracy and computing cost of PLA is derived. We theoretically prove and demonstrate using simulations that PLA substantially improves the chance of recovering weak signals and outperforms comparable methods at a limited cost of computation.

The concept of P-value was proposed by Fisher to measure inconsistency of data with a specified null hypothesis, and it plays a central role in statistical inference. For classical linear regression analysis, it is a standard procedure to calculate P-values for regression coefficients based on least squares estimator (LSE) to determine their significance. However, for high dimensional data when the number of predictors exceeds the sample size, ordinary least squares are no longer proper and there is not a valid definition for P-values based on LSE. It is also challenging to define sensible P-values for other high dimensional regression methods such as penalization and resampling methods. In this paper, we introduce a new concept called oracle P-value to generalize traditional P-values based on LSE to high dimensional sparse regression models. Then we propose several estimation procedures to approximate oracle P-values for real data analysis. We show that the oracle P-value framework is useful for developing new and powerful tools to enhance high dimensional data analysis, including variable ranking, variable selection, and screening procedures with false discovery rate (FDR) control. Numerical examples are then presented to demonstrate performance of the proposed methods.

A collection of quantile curves provides a complete picture of conditional distributions. A properly centered and scaled version of the estimated curves at various quantile levels gives rise to the so-called quantile regression process (QRP). In this paper, we establish weak convergence of QRP in a general series approximation framework, which includes linear models with increasing dimension, nonparametric models and partial linear models. An interesting consequence is obtained in the last class of models, where parametric and non-parametric estimators are shown to be asymptotically independent. Applications of our general process convergence results include the construction of non-crossing quantile curves and the estimation of conditional distribution functions. As a result of independent interest, we obtain a series of Bahadur representations with exponential bounds for tail probabilities of all remainder terms. Bounds of this kind are potentially useful in analyzing statistical inference procedures under the divide-and-conquer setup.

In this paper we present novel results on the asymptotic behavior of the so-called Ibragimov minimum contrast estimates. The case of tapered data for various models of Gaussian random fields is investigated. The CLT for quadratic forms with tapered data is presented.

Existing Bayesian nonparametric methodologies for bandit problems focus on exact observations, leaving a gap in those bandit applications where censored observations are crucial. We address this gap by extending a Bayesian nonparametric two-armed bandit problem to right-censored data, where each arm is generated from a beta-Stacy process as defined by Walker and Muliere (1997). We first show some properties of the expected advantage of choosing one arm over the other, namely the monotonicity in the arm response and, limited to the case of continuous state space, the continuity in the right-censored arm response. We partially characterize optimal strategies by proving the existence of stay-with-a-winner and stay-with-a-winner/switch-on-a-loser break-even points, under non-restrictive conditions that include the special cases of the simple homogeneous process and the Dirichlet process. Numerical estimations and simulations for a variety of discrete and continuous state space settings are presented to illustrate the performance and flexibility of our framework.

The present paper deals with the robustness of estimators and tests for ordinal response models. In this context, gross-errors in the response variable, specific deviations due to some respondents’ behavior, and outlying covariates can strongly affect the reliability of the maximum likelihood estimators and that of the related test procedures.

The paper highlights that the choice of the link function can affect the robustness of inferential methods, and presents a comparison among the most frequently used links. Subsequently robust $M$-estimators are proposed as an alternative to maximum likelihood estimators. Their asymptotic properties are derived analytically, while their performance in finite samples is investigated through extensive numerical experiments either at the model or when data contaminations occur. Wald and $t$-tests for comparing nested models, derived from $M$-estimators, are also proposed. $M$ based inference is shown to outperform maximum likelihood inference, producing more reliable results when robustness is a concern.

It has been proved that direct bootstrapping of the nonparametric maximum likelihood estimator (MLE) of the distribution function in the current status model leads to inconsistent confidence intervals. We show that bootstrapping of functionals of the MLE can however be used to produce valid intervals. To this end, we prove that the bootstrapped MLE converges at the right rate in the $L_{p}$-distance. We also discuss applications of this result to the current status regression model.

Gaussian graphical models are commonly used to characterize the conditional dependence among variables. However, ignorance of the effect of latent variables may blur the structure of a graph and corrupt statistical inference. In this paper, we propose a method for learning $\underline{\mathrm{L}}$atent $\underline{\mathrm{V}}$ariables graphical models via $\ell_{1}$ and trace penalized $\underline{\mathrm{D}}$-trace loss (LVD), which achieves parameter estimation and model selection consistency under certain identifiability conditions. We also present an efficient ADMM algorithm to obtain the penalized estimation of the sparse precision matrix. Using simulation studies, we validate the theoretical properties of our estimator and show its superior performance over other methods. The usefulness of the proposed method is also demonstrated through its application to a yeast genetical genomic data.

The single index model is an intuitive extension of the linear regression model. It has become increasingly popular due to its flexibility in modeling. Similar to the linear regression model, the set of predictors for the single index model can contain a large number of irrelevant variables. Therefore, it is important to select the relevant variables when fitting the single index model. However, the problem of variable selection for high-dimensional single index model is not well settled in the literature. In this work, we combine the idea of applying cubic B-splines for estimating the single index model with the idea of using the family of the smooth integration of counting and absolute deviation (SICA) penalty functions for variable selection. We propose a new method to simultaneously perform parameter estimation and model selection for the single index model. This method is referred to as the B-spline and SICA method for the single index model, or in short, BS-SIM. We develop a coordinate descent algorithm to efficiently implement BS-SIM. We also show that under certain conditions, the proposed method can consistently estimate the true index and select the true model. Simulations with various settings and a real data analysis are conducted to demonstrate the estimation accuracy, selection consistency and computational efficiency of BS-SIM.

We develop methodology and theory for a mean field variational Bayes approximation to a linear model with a spike and slab prior on the regression coefficients. In particular we show how our method forces a subset of regression coefficients to be numerically indistinguishable from zero; under mild regularity conditions estimators based on our method consistently estimate the model parameters with easily obtainable and (asymptotically) appropriately sized standard error estimates; and select the true model at an exponential rate in the sample size. We also develop a practical method for simultaneously choosing reasonable initial parameter values and tuning the main tuning parameter of our algorithms which is both computationally efficient and empirically performs as well or better than some popular variable selection approaches. Our method is also faster and highly accurate when compared to MCMC.

We estimate the quantum state of a light beam from results of quantum homodyne tomography noisy measurements performed on identically prepared quantum systems. We propose two Bayesian nonparametric approaches. The first approach is based on mixture models and is illustrated through simulation examples. The second approach is based on random basis expansions. We study the theoretical performance of the second approach by quantifying the rate of contraction of the posterior distribution around the true quantum state in the $L^{2}$ metric.

We consider an estimator for the location of a shift in the mean of long-range dependent sequences. The estimation is based on the two-sample Wilcoxon statistic. Consistency and the rate of convergence for the estimated change point are established. In the case of a constant shift height, the $1/n$ convergence rate (with $n$ denoting the number of observations), which is typical under the assumption of independent observations, is also achieved for long memory sequences. It is proved that if the change point height decreases to $0$ with a certain rate, the suitably standardized estimator converges in distribution to a functional of a fractional Brownian motion. The estimator is tested on two well-known data sets. Finite sample behaviors are investigated in a Monte Carlo simulation study.

The wild bootstrap is the resampling method of choice in survival analytic applications. Theoretic justifications typically rely on the assumption of existing intensity functions which is equivalent to an exclusion of ties among the event times. However, such ties are omnipresent in practical studies. It turns out that the wild bootstrap should only be applied in a modified manner that corrects for altered limit variances and emerging dependencies. This again ensures the asymptotic exactness of inferential procedures. An analogous necessity is the use of the Greenwood-type variance estimator for Nelson-Aalen estimators which is particularly preferred in tied data regimes. All theoretic arguments are transferred to bootstrapping Aalen-Johansen estimators for cumulative incidence functions in competing risks. An extensive simulation study as well as an application to real competing risks data of male intensive care unit patients suffering from pneumonia illustrate the practicability of the proposed technique.

Many applications involve estimation of a signal matrix from a noisy data matrix. In such cases, it has been observed that estimators that shrink or truncate the singular values of the data matrix perform well when the signal matrix has approximately low rank. In this article, we generalize this approach to the estimation of a tensor of parameters from noisy tensor data. We develop new classes of estimators that shrink or threshold the mode-specific singular values from the higher-order singular value decomposition. These classes of estimators are indexed by tuning parameters, which we adaptively choose from the data by minimizing Stein’s unbiased risk estimate. In particular, this procedure provides a way to estimate the multilinear rank of the underlying signal tensor. Using simulation studies under a variety of conditions, we show that our estimators perform well when the mean tensor has approximately low multilinear rank, and perform competitively when the signal tensor does not have approximately low multilinear rank. We illustrate the use of these methods in an application to multivariate relational data.

Preferential attachment is an appealing mechanism for modeling power-law behavior of the degree distributions in directed social networks. In this paper, we consider methods for fitting a 5-parameter linear preferential model to network data under two data scenarios. In the case where full history of the network formation is given, we derive the maximum likelihood estimator of the parameters and show that it is strongly consistent and asymptotically normal. In the case where only a single-time snapshot of the network is available, we propose an estimation method which combines method of moments with an approximation to the likelihood. The resulting estimator is also strongly consistent and performs quite well compared to the MLE estimator. We illustrate both estimation procedures through simulated data and explore the usage of this model in a real data example.

Prediction polling is an increasingly popular form of crowdsourcing in which multiple participants estimate the probability or magnitude of some future event. These estimates are then aggregated into a single forecast. Historically, randomness in scientific estimation has been generally assumed to arise from unmeasured factors which are viewed as measurement noise. However, when combining subjective estimates, heterogeneity stemming from differences in the participants’ information is often more important than measurement noise. This paper formalizes information diversity as an alternative source of such heterogeneity and introduces a novel modeling framework that is particularly well-suited for prediction polls. A practical specification of this framework is proposed and applied to the task of aggregating probability and point estimates from two real-world prediction polls. In both cases our model outperforms standard measurement-error-based aggregators, hence providing evidence in favor of information diversity being the more important source of heterogeneity.

In this paper, we generalize the metric-based permutation test for the equality of covariance operators proposed by Pigoli et al. (2014) to the case of multiple samples of functional data. To this end, the non-parametric combination methodology of Pesarin and Salmaso (2010) is used to combine all the pairwise comparisons between samples into a global test. Different combining functions and permutation strategies are reviewed and analyzed in detail. The resulting test allows to make inference on the equality of the covariance operators of multiple groups and, if there is evidence to reject the null hypothesis, to identify the pairs of groups having different covariances. It is shown that, for some combining functions, step-down adjusting procedures are available to control for the multiple testing problem in this setting. The empirical power of this new test is then explored via simulations and compared with those of existing alternative approaches in different scenarios. Finally, the proposed methodology is applied to data from wheel running activity experiments, that used selective breeding to study the evolution of locomotor behavior in mice.

We study the asymptotic behavior of the least concave majorant of an estimator of a concave distribution function under general conditions. The true concave distribution function is permitted to violate strict concavity, so that the empirical distribution function and its least concave majorant are not asymptotically equivalent. Our results are proved by demonstrating the Hadamard directional differentiability of the least concave majorant operator. Standard approaches to bootstrapping fail to deliver valid inference when the true distribution function is not strictly concave. While the rescaled bootstrap of Dümbgen delivers asymptotically valid inference, its performance in small samples can be poor, and depends upon the selection of a tuning parameter. We show that two alternative bootstrap procedures—one obtained by approximating a conservative upper bound, the other by resampling from the Grenander estimator—can be used to construct reliable confidence bands for the true distribution. Some related results on isotonic regression are provided.

We consider the estimation of the transition matrix in the high-dimensional time-varying vector autoregression (TV-VAR) models. Our model builds on a general class of locally stationary VAR processes that evolve smoothly in time. We propose a hybridized kernel smoothing and $\ell^{1}$-regularized method to directly estimate the sequence of time-varying transition matrices. Under the sparsity assumption on the transition matrix, we establish the rate of convergence of the proposed estimator and show that the convergence rate depends on the smoothness of the locally stationary VAR processes only through the smoothness of the transition matrix function. In addition, for our estimator followed by thresholding, we prove that the false positive rate (type I error) and false negative rate (type II error) in the pattern recovery can asymptotically vanish in the presence of weak signals without assuming the minimum nonzero signal strength condition. Favorable finite sample performances over the $\ell^{2}$-penalized least-squares estimator and the unstructured maximum likelihood estimator are shown on simulated data. We also provide two real examples on estimating the dependence structures on financial stock prices and economic exchange rates datasets.

This is a study of principal component analysis performed on a statistical sample. We assume that this data sample is made of independent copies of some random variable ranging in a separable real Hilbert space. This covers data in function spaces as well as data represented in reproducing kernel Hilbert spaces. Based on some new inequalities about the perturbation of nonnegative self-adjoint operators, we provide new bounds for the statistical fluctuations of the principal component representation with the draw of the statistical sample.

We suggest two kinds of improvements to decrease these fluctuations: the first is to use a robust estimate of the covariance operator, for which non-asymptotic bounds of the estimation error are available under weak polynomial moment assumptions. The second improvement is to use some modification of the projection on the principal components based on functional calculus applied to the covariance operator. Using this modified projection, we can obtain bounds that do not depend on the spectral gap but on some more favorable factor.

In appendix, we provide a new approach to the analysis of the relative positions of two orthogonal projections that is useful for our proofs and that has an interest of its own.

Estimating individualized treatment rules is a central task for personalized medicine. [25] and [23] proposed outcome weighted learning to estimate individualized treatment rules directly through maximizing the expected outcome without modeling the response directly. In this paper, we extend the outcome weighted learning to right censored survival data without requiring either inverse probability of censoring weighting or semiparametric modeling of the censoring and failure times as done in [26]. To accomplish this, we take advantage of the tree based approach proposed in [29] to nonparametrically impute the survival time in two different ways. The first approach replaces the reward of each individual by the expected survival time, while in the second approach only the censored observations are imputed by their conditional expected failure times. We establish consistency and convergence rates for both estimators. In simulation studies, our estimators demonstrate improved performance compared to existing methods. We also illustrate the proposed method on a phase III clinical trial of non-small cell lung cancer.

We present an adaptation of the Kato–Temple inequality for bounding perturbations of eigenvalues with applications to statistical inference for random graphs, specifically hypothesis testing and change-point detection. We obtain explicit high-probability bounds for the individual distances between certain signal eigenvalues of a graph’s adjacency matrix and the corresponding eigenvalues of the model’s edge probability matrix, even when the latter eigenvalues have multiplicity. Our results extend more broadly to the perturbation of singular values in the presence of quite general random matrix noise.

We propose an empirical estimator of the preferential attachment function $f$ in the setting of general sublinear preferential attachment trees. Using a supercritical continuous-time branching process framework, we prove the almost sure consistency of the proposed estimator. We perform simulations to study the empirical properties of our estimators.

In this paper we investigate the task of parameter learning of Bayesian networks and, in particular, we deal with the prior uncertainty of learning using a Bayesian framework. Parameter learning is explored in the context of Bayesian inference and we subsequently introduce Bayes, constrained Bayes and robust Bayes parameter learning methods. Bayes and constrained Bayes estimates of parameters are obtained to meet the twin objective of simultaneous estimation and closeness between the histogram of the estimates and the posterior estimates of the parameter histogram. Treating the prior uncertainty, we consider some classes of prior distributions and derive simultaneous Posterior Regret Gamma Minimax estimates of parameters. Evaluation of the merits of the various procedures was done using synthetic data and a real clinical dataset.

We consider three Bayesian penalized regression models and show that the respective deterministic scan Gibbs samplers are geometrically ergodic regardless of the dimension of the regression problem. We prove geometric ergodicity of the Gibbs samplers for the Bayesian fused lasso, the Bayesian group lasso, and the Bayesian sparse group lasso. Geometric ergodicity along with a moment condition results in the existence of a Markov chain central limit theorem for Monte Carlo averages and ensures reliable output analysis. Our results of geometric ergodicity allow us to also provide default starting values for the Gibbs samplers.

Previous works on location and location-scale mixtures of normals have shown different upper bounds on the posterior rates of contraction, either in a density estimation context or in nonlinear regression. In both cases, the observations were assumed not too spread by considering either the true density has light tails or the regression function has compact support. It has been conjectured that in a situation where the data are diffuse, location-scale mixtures may benefit from allowing a spatially varying order of approximation. Here we test the argument on the mean regression with normal errors and random design model. Although we cannot invalidate the conjecture due to the lack of lower bound, we find slower upper bounds for location-scale mixtures, even under heavy tails assumptions on the design distribution. However, the proofs suggest to introduce hybrid location-scale mixtures for which faster upper bounds are derived. Finally, we show that all tails assumptions on the design distribution can be released at the price of making the prior distribution covariate dependent.

In this paper we estimate both the Hurst and the stability indices of a $H$-self-similar stable process. More precisely, let $X$ be a $H$-sssi (self-similar stationary increments) symmetric $\alpha$-stable process. The process $X$ is observed at points $\frac{k}{n}$, $k=0,\ldots,n$. Our estimate is based on $\beta$-negative power variations with $-\frac{1}{2}<\beta<0$. We obtain consistent estimators, with rate of convergence, for several classical $H$-sssi $\alpha$-stable processes (fractional Brownian motion, well-balanced linear fractional stable motion, Takenaka’s process, Lévy motion). Moreover, we obtain asymptotic normality of our estimators for fractional Brownian motion and Lévy motion.

Modeling and analysis of spectroscopy data is an active area of research with applications to chemistry and biology. This paper focuses on modelling high-dimensional spectra for the purpose of noise reduction and prediction in problems where the spectra can be used as covariates. We propose a functional representation of the spectra as well as functional regression model that accommodates multiple spatial dimensions. Both steps emphasize sparsity to reduce the number of parameters and mitigate over-fitting. The motivating application for these models, discussed in some detail, is predicting bone-mineral-density (BMD), an important indicator of fracture healing, from Raman spectra, in both the in vivo and ex vivo settings of a bone fracture healing experiment. To illustrate the general applicability of the method, we also use it to predict lipoprotein concentrations from spectra obtained by nuclear magnetic resonance (NMR) spectroscopy.

In this paper we provide new methodology for inference of the geometric features of a multivariate density in deconvolution. Our approach is based on multiscale tests to detect significant directional derivatives of the unknown density at arbitrary points in arbitrary directions. The multiscale method is used to identify regions of monotonicity and to construct a general procedure for the detection of modes of the multivariate density. Moreover, as an important application a significance test for the presence of a local maximum at a pre-specified point is proposed. The performance of the new methods is investigated from a theoretical point of view and the finite sample properties are illustrated by means of a small simulation study.

Many data sets consist of variables with an inherent group structure. The problem of group selection has been well studied, but in this paper, we seek to do the opposite: our goal is to select at least one variable from each group in the context of predictive regression modeling. This problem is NP-hard, but we propose the tightest convex relaxation: a composite penalty that is a combination of the $\ell_{1}$ and $\ell_{2}$ norms. Our so-called Exclusive Lasso method performs structured variable selection by ensuring that at least one variable is selected from each group. We study our method’s statistical properties and develop computationally scalable algorithms for fitting the Exclusive Lasso. We study the effectiveness of our method via simulations as well as using NMR spectroscopy data. Here, we use the Exclusive Lasso to select the appropriate chemical shift from a dictionary of possible chemical shifts for each molecule in the biological sample.

This paper presents a variational Bayes approach to a semiparametric regression model that consists of parametric and nonparametric components. The assumed univariate nonparametric component is represented with a cosine series based on a spectral analysis of Gaussian process priors. Here, we develop fast variational methods for fitting the semiparametric regression model that reduce the computation time by an order of magnitude over Markov chain Monte Carlo methods. Further, we explore the possible use of the variational lower bound and variational information criteria for model choice of a parametric regression model against a semiparametric alternative. In addition, variational methods are developed for estimating univariate shape-restricted regression functions that are monotonic, monotonic convex or monotonic concave. Since these variational methods are approximate, we explore some of the trade-offs involved in using them in terms of speed, accuracy and automation of the implementation in comparison with Markov chain Monte Carlo methods and discuss their potential and limitations.

For a general class of stationary random fields we study asymptotic properties of the discrete Fourier transform (DFT), periodogram, parametric and nonparametric spectral density estimators under an easily verifiable short-range dependence condition expressed in terms of functional dependence measures. We allow irregularly spaced data which is indexed by a subset $\Gamma $ of $\mathbb{Z}^{d}$. Our asymptotic theory requires minimal restriction on the index set $\Gamma $. Asymptotic normality is derived for kernel spectral density estimators and the Whittle estimator of a parameterized spectral density function. We also develop asymptotic results for a covariance matrix estimate.

We propose estimating equations whose unknown parameters are the values taken by a circular density and its derivatives at a point. Specifically, we solve equations which relate local versions of population trigonometric moments with their sample counterparts. Major advantages of our approach are: higher order bias without asymptotic variance inflation, closed form for the estimators, and absence of numerical tasks. We also investigate situations where the observed data are dependent. Theoretical results along with simulation experiments are provided.

In many applications, linear models fit the data poorly. This article studies an appealing alternative, the generalized regression model. This model only assumes that there exists an unknown monotonically increasing link function connecting the response $Y$ to a single index $\boldsymbol{X} ^{\mathsf{T}}\boldsymbol{\beta } ^{*}$ of explanatory variables $\boldsymbol{X} \in{\mathbb{R}} ^{d}$. The generalized regression model is flexible and covers many widely used statistical models. It fits the data generating mechanisms well in many real problems, which makes it useful in a variety of applications where regression models are regularly employed. In low dimensions, rank-based M-estimators are recommended to deal with the generalized regression model, giving root-$n$ consistent estimators of $\boldsymbol{\beta } ^{*}$. Applications of these estimators to high dimensional data, however, are questionable. This article studies, both theoretically and practically, a simple yet powerful smoothing approach to handle the high dimensional generalized regression model. Theoretically, a family of smoothing functions is provided, and the amount of smoothing necessary for efficient inference is carefully calculated. Practically, our study is motivated by an important and challenging scientific problem: decoding gene regulation by predicting transcription factors that bind to cis-regulatory elements. Applying our proposed method to this problem shows substantial improvement over the state-of-the-art alternative in real data.

We propose a new method to perform approximate likelihood inference in latent variable models. Our approach provides an approximation of the integrals involved in the likelihood function through a reduction of their dimension that makes the computation feasible in situations in which classical and adaptive quadrature based methods are not applicable. We derive new theoretical results on the accuracy of the obtained estimators. We show that the proposed approximation outperforms several existing methods in simulations, and it can be successfully applied in presence of multidimensional longitudinal data when standard techniques are not applicable or feasible.

Spatiotemporal point processes (STPPs) are important in modeling randomly appeared events developed in space and time. Statistical methods of STPPs have been widely used in applications. In all of these methods, evaluations and inferences of intensity functions are the primary issues. The present article proposes a new method, which attempts to evaluate angles of gradient vectors of intensity functions rather than the intensity functions themselves. According to the nature of many natural and human phenomena, the evaluation of angle patterns of the gradient vectors is more important than the evaluation of their magnitude patterns because changes of angle patterns often indicate global changes of these phenomena. This issue is investigated by simulation studies, where significant variations of gradient angle patterns are identified only when modes of intensity functions are changed. To study these phenomena, the article proposes an analysis method for gradient angles of the first-order intensity function of STPPs. The proposed method is used to analyze aftershock earthquake activities caused by great mainshock earthquakes occurred in Japan 2011 and Indian Ocean 2004, respectively, where a significant global change in the second case is identified.

A distribution whose normalization constant is an $A$-hypergeometric polynomial is called an $A$-hypergeometric distribution. Such a distribution is in turn a generalization of the generalized hypergeometric distribution on the contingency tables with fixed marginal sums. In this paper, we will see that an $A$-hypergeometric distribution with a homogeneous matrix of two rows, especially, that associated with the rational normal curve, appears in inferences involving exchangeable partition structures. An exact sampling algorithm is presented for the general (any number of rows) $A$-hypergeometric distributions. Then, the maximum likelihood estimation of the $A$-hypergeometric distribution associated with the rational normal curve, which is an algebraic exponential family, is discussed. The information geometry of the Newton polytope is useful for analyzing the full and the curved exponential family. Algebraic methods are provided for evaluating the $A$-hypergeometric polynomials.

Isotonic regression offers a flexible modeling approach under monotonicity assumptions, which are natural in many applications. Despite this attractive setting and extensive theoretical research, isotonic regression has enjoyed limited interest in practical modeling primarily due to its tendency to suffer significant overfitting, even in moderate dimension, as the monotonicity constraints do not offer sufficient complexity control. Here we propose to regularize isotonic regression by penalizing or constraining the range of the fitted model (i.e., the difference between the maximal and minimal predictions). We show that the optimal solution to this problem is obtained by constraining the non-penalized isotonic regression model to lie in the required range, and hence can be found easily given this non-penalized solution. This makes our approach applicable to large datasets and to generalized loss functions such as Huber’s loss or exponential family log-likelihoods. We also show how the problem can be reformulated as a Lasso problem in a very high dimensional basis of upper sets. Hence, range regularization inherits some of the statistical properties of Lasso, notably its degrees of freedom estimation. We demonstrate the favorable empirical performance of our approach compared to various relevant alternatives.

In the problem of testing for signal in Gaussian white noise, over a smoothness class with an $L_{2}$-ball removed, minimax rates of convergences (separation rates) are well known (Ingster [24]); they are expressed in the rate of the ball radius tending to zero along with noise intensity, such that a nontrivial asymptotic power is possible. It is also known that, if the smoothness class is a Sobolev type ellipsoid of degree $\beta$ and size $M$, the optimal rate result can be sharpened towards a Pinsker type asymptotics for the critical radius (Ermakov [9]). The minimax optimal tests in that setting depend on $\beta$ and $M$; but whereas in nonparametric estimation with squared $L_{2}$-loss, adaptive estimators attaining the Pinsker constant are known, the analogous problem in testing is open. First, for adaptation to $M$ only, we establish that it is not possible at the critical separation rate, but is possible in the sense of the asymptotics of tail error probabilities at slightly slower rates. For full adaptation to $(\beta,M)$, it is well known that a $\log\log n$-penalty over the separation rate is incurred. We extend a preliminary result of Ingster and Suslina [25] relating to fixed $M$ and unknown $\beta$, and establish that sharp minimax adaptation to both parameters is possible. Thus a complete solution is obtained, in the basic $L_{2}$-case, to the problem of adaptive nonparametric testing at the level of asymptotic minimax constants.

Location mixture models, resulting in shifting a common distribution with some probability, have been widely used to account for existence of clusters in the data. Assuming only symmetry of this common distribution allows for great flexibility, especially when the traditional normality assumption is violated. This semi-parametric model has been studied in several papers, where the mixture parameters are first estimated before constructing an estimator for the non-parametric component. The plug-in method suggested by Hunter et al. (2007) has the merit to be easily implementable and fast to compute. However, no result is available on the limit distribution of the obtained estimator, hindering for instance construction of asymptotic confidence intervals. In this paper, we give sufficient conditions on the symmetric distribution for asymptotic normality to hold. In case the symmetric distribution admits a log-concave density, our assumptions are automatically satisfied. The obtained result has to be used with caution in case the mixture location are too close or the mixing probability is close to $0$ or $1$. Three examples are considered where we show that the estimator is not to be advocated when the mixture components are not well separated.

This work is closely related to the theories of set estimation and manifold estimation. Our object of interest is a, possibly lower-dimensional, compact set $S\subset{\mathbb{R}}^{d}$. The general aim is to identify (via stochastic procedures) some qualitative or quantitative features of $S$, of geometric or topological character. The available information is just a random sample of points drawn on $S$. The term “to identify” means here to achieve a correct answer almost surely (a.s.) when the sample size tends to infinity. More specifically the paper aims at giving some partial answers to the following questions: is $S$ full dimensional? Is $S$ “close to a lower dimensional set” $\mathcal{M}$? If so, can we estimate $\mathcal{M}$ or some functionals of $\mathcal{M}$ (in particular, the Minkowski content of $\mathcal{M}$)? As an important auxiliary tool in the answers of these questions, a denoising procedure is proposed in order to partially remove the noise in the original data. The theoretical results are complemented with some simulations and graphical illustrations.

Rao and Teh (2012, 2013) introduced an efficient MCMC algorithm for sampling from the posterior distribution of a hidden Markov jump process. The algorithm is based on the idea of sampling virtual jumps. In the present paper we show that the Markov chain generated by Rao and Teh’s algorithm is geometrically ergodic. To this end we establish a geometric drift condition towards a small set. A similar result is also proved for a special version of the algorithm, used for probabilistic inference in Continuous Time Bayesian Networks.

Controlling false discovery rate (FDR) is a powerful approach to multiple testing. In many applications, the tested hypotheses have an inherent hierarchical structure. In this paper, we focus on the fixed sequence structure where the testing order of the hypotheses has been strictly specified in advance. We are motivated to study such a structure, since it is the most basic of hierarchical structure, yet it is often seen in real applications such as statistical process control and streaming data analysis. We first consider a conventional fixed sequence method that stops testing once an acceptance occurs, and develop such a method controlling FDR under both arbitrary and negative dependencies. The method under arbitrary dependency is shown to be unimprovable without losing control of FDR and, unlike existing FDR methods; it cannot be improved even by restricting to the usual positive regression dependence on subset (PRDS) condition. To account for any potential mistakes in the ordering of the tests, we extend the conventional fixed sequence method to one that allows more but a given number of acceptances. Simulation studies show that the proposed procedures can be powerful alternatives to existing FDR controlling procedures. The proposed procedures are illustrated through a real data set from a microarray experiment.

Assuming that one-step transition kernel of a discrete time, time-homogenous Markov chain model is parameterized by a parameter $\theta\in\boldsymbol{\Theta}$, we derive a recursive (in time) construction of confidence regions for the unknown parameter of interest, say $\theta^{*}\in\boldsymbol{\Theta}$. It is supposed that the observed data used in the construction of the confidence regions is generated by a Markov chain whose transition kernel corresponds to $\theta^{*}$. The key step in our construction is the derivation of a recursive scheme for an appropriate point estimator of $\theta^{*}$. To achieve this, we start by what we call the base recursive point estimator, using which we design a quasi-asymptotically linear recursive point estimator (a concept introduced in this paper). For the latter estimator we prove its weak consistency and asymptotic normality. The recursive construction of confidence regions is needed not only for the purpose of speeding up the computation of the successive confidence regions, but, primarily, for the ability to apply the dynamic programming principle in the context of robust adaptive stochastic control methodology.

Efficient recovery of a low-dimensional structure from high-dimensional data has been pursued in various settings including wavelet denoising, generalized linear models and low-rank matrix estimation. By thresholding some parameters to zero, estimators such as lasso, elastic net and subset selection perform variable selection. One crucial step challenges all these estimators: the amount of thresholding governed by a threshold parameter $\lambda $. If too large, important features are missing; if too small, incorrect features are included. Within a unified framework, we propose a selection of $\lambda $ at the detection edge. To that aim, we introduce the concept of a zero-thresholding function and a null-thresholding statistic, that we explicitly derive for a large class of estimators. The new approach has the great advantage of transforming the selection of $\lambda $ from an unknown scale to a probabilistic scale. Numerical results show the effectiveness of our approach in terms of model selection and prediction.

It is well-known that the Kullback–Leibler support condition implies posterior consistency in the weak topology, but is not sufficient for consistency in the total variation distance. There is a counter–example. Since then many authors have proposed sufficient conditions for strong consistency; and the aim of the present paper is to introduce new conditions with specific application to nonparametric mixture models with heavy–tailed components, such as the Student-$t$. The key is a more focused result on sets of densities where if strong consistency fails then it fails on such densities. This allows us to move away from the traditional types of sieves currently employed.

We study the problem of estimating an unknown vector $\theta $ from an observation $X$ drawn according to the normal distribution with mean $\theta $ and identity covariance matrix under the knowledge that $\theta $ belongs to a known closed convex set $\Theta $. In this general setting, Chatterjee (2014) proved that the natural constrained least squares estimator is “approximately admissible” for every $\Theta $. We extend this result by proving that the same property holds for all convex penalized estimators as well. Moreover, we simplify and shorten the original proof considerably. We also provide explicit upper and lower bounds for the universal constant underlying the notion of approximate admissibility.

Respondent-driven sampling (RDS) is a method of chain referral sampling popular for sampling hidden and/or marginalized populations. As such, even under the ideal sampling assumptions, the performance of RDS is restricted by the underlying social network: if the network is divided into communities that are weakly connected to each other, then RDS is likely to oversample one of these communities. In order to diminish the “referral bottlenecks” between communities, we propose anti-cluster RDS (AC-RDS), an adjustment to the standard RDS implementation. Using a standard model in the RDS literature, namely, a Markov process on the social network that is indexed by a tree, we construct and study the Markov transition matrix for AC-RDS. We show that if the underlying network is generated from the Stochastic Blockmodel with equal block sizes, then the transition matrix for AC-RDS has a larger spectral gap and consequently faster mixing properties than the standard random walk model for RDS. In addition, we show that AC-RDS reduces the covariance of the samples in the referral tree compared to the standard RDS and consequently leads to a smaller variance and design effect. We confirm the effectiveness of the new design using both the Add-Health networks and simulated networks.

Statistical modeling of multivariate and spatial extreme events has attracted broad attention in various areas of science. Max-stable distributions and processes are the natural class of models for this purpose, and many parametric families have been developed and successfully applied. Due to complicated likelihoods, the efficient statistical inference is still an active area of research, and usually composite likelihood methods based on bivariate densities only are used. Thibaud et al. (2016) use a Bayesian approach to fit a Brown–Resnick process to extreme temperatures. In this paper, we extend this idea to a methodology that is applicable to general max-stable distributions and that uses full likelihoods. We further provide simple conditions for the asymptotic normality of the median of the posterior distribution and verify them for the commonly used models in multivariate and spatial extreme value statistics. A simulation study shows that this point estimator is considerably more efficient than the composite likelihood estimator in a frequentist framework. From a Bayesian perspective, our approach opens the way for new techniques such as Bayesian model comparison in multivariate and spatial extremes.

The paper “Nonparametric Laguerre estimation in the multiplicative censoring model”, Electronic Journal of Statistics, 2016, 10, 3114–3152, contains a wrong statement. We localize the place of the error and give a correct proof.

Following recent success on the analysis of the Slope estimator, we provide a sharp oracle inequality in term of prediction error for Graph-Slope, a generalization of Slope to signals observed over a graph. In addition to improving upon best results obtained so far for the Total Variation denoiser (also referred to as Graph-Lasso or Generalized Lasso), we propose an efficient algorithm to compute Graph-Slope. The proposed algorithm is obtained by applying the forward-backward method to the dual formulation of the Graph-Slope optimization problem. We also provide experiments showing the practical applicability of the method.

Respondent-Driven Sampling is a popular technique for sampling hidden populations. This paper models Respondent-Driven Sampling as a Markov process indexed by a tree. Our main results show that the Volz-Heckathorn estimator is asymptotically normal below a critical threshold. The key technical difficulties stem from (i) the dependence between samples and (ii) the tree structure which characterizes the dependence. The theorems allow the growth rate of the tree to exceed one and suggest that this growth rate should not be too large.

Sketching techniques scale up machine learning algorithms by reducing the sample size or dimensionality of massive data sets, without sacrificing their statistical properties. In this paper, we study sketching from an optimization point of view. We first show that the iterative Hessian sketch is an optimization process with preconditioning and develop an accelerated version using this insight together with conjugate gradient descent. Next, we establish a primal-dual connection between the Hessian sketch and dual random projection, which allows us to develop an accelerated iterative dual random projection method by applying the preconditioned conjugate gradient descent on the dual problem. Finally, we tackle the problems of large sample size and high-dimensionality in massive data sets by developing the primal-dual sketch. The primal-dual sketch iteratively sketches the primal and dual formulations and requires only a logarithmic number of calls to solvers of small sub-problems to recover the optimum of the original problem up to arbitrary precision. Our iterative sketching techniques can also be applied for solving distributed optimization problems where data are partitioned by samples or features. Experiments on synthetic and real data sets complement our theoretical results.

Bayesian optimization has become a successful tool for optimizing the hyperparameters of machine learning algorithms, such as support vector machines or deep neural networks. Despite its success, for large datasets, training and validating a single configuration often takes hours, days, or even weeks, which limits the achievable performance. To accelerate hyperparameter optimization, we propose a generative model for the validation error as a function of training set size, which is learned during the optimization process and allows exploration of preliminary configurations on small subsets, by extrapolating to the full dataset. We construct a Bayesian optimization procedure, dubbed FABOLAS, which models loss and training time as a function of dataset size and automatically trades off high information gain about the global optimum against computational cost. Experiments optimizing support vector machines and deep neural networks show that FABOLAS often finds high-quality solutions 10 to 100 times faster than other state-of-the-art Bayesian optimization methods or the recently proposed bandit strategy Hyperband.

We consider the Bayesian active learning and experimental design problem, where the goal is to learn the value of some unknown target variable through a sequence of informative, noisy tests. In contrast to prior work, we focus on the challenging, yet practically relevant setting where test outcomes can be conditionally dependent given the hidden target variable. Under such assumptions, common heuristics, such as greedily performing tests that maximize the reduction in uncertainty of the target, often perform poorly.

We propose ECED, a novel, efficient active learning algorithm, and prove strong theoretical guarantees that hold with correlated, noisy tests. Rather than directly optimizing the prediction error, at each step, ECED picks the test that maximizes the gain in a surrogate objective, which takes into account the dependencies between tests. Our analysis relies on an information-theoretic auxiliary function to track the progress of ECED, and utilizes adaptive submodularity to attain the approximation bound. We demonstrate strong empirical performance of ECED on three problem instances, including a Bayesian experimental design task intended to distinguish among economic theories of how people make risky decisions, an active preference learning task via pairwise comparisons, and a third application on pool-based active learning.

The horseshoe prior has proven to be a noteworthy alternative for sparse Bayesian estimation, but has previously suffered from two problems. First, there has been no systematic way of specifying a prior for the global shrinkage hyperparameter based on the prior information about the degree of sparsity in the parameter vector. Second, the horseshoe prior has the undesired property that there is no possibility of specifying separately information about sparsity and the amount of regularization for the largest coefficients, which can be problematic with weakly identified parameters, such as the logistic regression coefficients in the case of data separation. This paper proposes solutions to both of these problems. We introduce a concept of effective number of nonzero parameters, show an intuitive way of formulating the prior for the global hyperparameter based on the sparsity assumptions, and argue that the previous default choices are dubious based on their tendency to favor solutions with more unshrunk parameters than we typically expect a priori. Moreover, we introduce a generalization to the horseshoe prior, called the regularized horseshoe, that allows us to specify a minimum level of regularization to the largest values. We show that the new prior can be considered as the continuous counterpart of the spike-and-slab prior with a finite slab width, whereas the original horseshoe resembles the spike-and-slab with an infinitely wide slab. Numerical experiments on synthetic and real world data illustrate the benefit of both of these theoretical advances.

For optimal decision making under variable class distributions and misclassification costs a classifier needs to produce well-calibrated estimates of the posterior probability. Isotonic calibration is a powerful non-parametric method that is however prone to overfitting on smaller datasets; hence a parametric method based on the logistic sigmoidal curve is commonly used. While logistic calibration is designed for normally distributed per-class scores, we demonstrate experimentally that many classifiers including Naive Bayes and Adaboost suffer from a particular distortion where these score distributions are heavily skewed. In such cases logistic calibration can easily yield probability estimates that are worse than the original scores. Moreover, the logistic curve family does not include the identity function, and hence logistic calibration can easily uncalibrate a perfectly calibrated classifier.

In this paper we solve all these problems with a richer class of parametric calibration maps based on the beta distribution. We derive the method from first principles and show that fitting it is as easy as fitting a logistic curve. Extensive experiments show that beta calibration is superior to logistic calibration for a wide range of classifiers: Naive Bayes, Adaboost, random forest, logistic regression, support vector machine and multi-layer perceptron. If the original classifier is already calibrated, then beta calibration learns a function close to the identity. On this we build a statistical test to recognise if the model deviates from being well-calibrated.

Despite the fundamental nature of the inhomogeneous Poisson process in the theory and application of stochastic processes, and its attractive generalizations (e.g. Cox process), few tractable nonparametric modeling approaches of intensity functions exist, especially when observed points lie in a high-dimensional space. In this paper we develop a new, computationally tractable Reproducing Kernel Hilbert Space (RKHS) formulation for the inhomogeneous Poisson process. We model the square root of the intensity as an RKHS function. Whereas RKHS models used in supervised learning rely on the so-called representer theorem, the form of the inhomogeneous Poisson process likelihood means that the representer theorem does not apply. However, we prove that the representer theorem does hold in an appropriately transformed RKHS, guaranteeing that the optimization of the penalized likelihood can be cast as a tractable finite-dimensional problem. The resulting approach is simple to implement, and readily scales to high dimensions and large-scale datasets.

Many generative models can be expressed as a differentiable function applied to input variables sampled from a known probability distribution. This framework includes both the generative component of learned parametric models such as variational autoencoders and generative adversarial networks, and also procedurally defined simulator models which involve only differentiable operations. Though the distribution on the input variables to such models is known, often the distribution on the output variables is only implicitly defined. We present a method for performing efficient Markov chain Monte Carlo inference in such models when conditioning on observations of the model output. For some models this offers an asymptotically exact inference method where approximate Bayesian computation might otherwise be employed. We use the intuition that computing conditional expectations is equivalent to integrating over a density defined on the manifold corresponding to the set of inputs consistent with the observed outputs. This motivates the use of a constrained variant of Hamiltonian Monte Carlo which leverages the smooth geometry of the manifold to move between inputs exactly consistent with observations. We validate the method by performing inference experiments in a diverse set of models.

We derive an alternative proof for the regret of Thompson sampling (TS) in the stochastic linear bandit setting. While we obtain a regret bound of order $\widetilde{O}(d^{3/2}\sqrt{T})$ as in previous results, the proof sheds new light on the functioning of the TS. We leverage the structure of the problem to show how the regret is related to the sensitivity (i.e., the gradient) of the objective function and how selecting optimal arms associated to optimistic parameters does control it. Thus we show that TS can be seen as a generic randomized algorithm where the sampling distribution is designed to have a fixed probability of being optimistic, at the cost of an additional $\sqrt{d}$ regret factor compared to a UCB-like approach. Furthermore, we show that our proof can be readily applied to regularized linear optimization and generalized linear model problems.

We study the interpretability of conditional probability estimates for binary classification under the agnostic setting or scenario. Under the agnostic setting, conditional probability estimates do not necessarily reflect the true conditional probabilities. Instead, they have a certain calibration property: among all data points that the classifier has predicted $\mathcal{P}(Y=1|X)=p$, $p$ portion of them actually have label $Y=1$. For cost-sensitive decision problems, this calibration property provides adequate support for us to use Bayes Decision Rule. In this paper, we define a novel measure for the calibration property together with its empirical counterpart, and prove a uniform convergence result between them. This new measure enables us to formally justify the calibration property of conditional probability estimations. It also provides new insights on the problem of estimating and calibrating conditional probabilities, and allows us to reliably estimate the expected cost of decision rules when applied to an unlabeled dataset.

This paper presents R2PCA, a random consensus method for robust principal component analysis. R2PCA takes RANSAC’s principle of using as little data as possible one step further. It iteratively selects small subsets of the data to identify pieces of the principal components, to then stitch them together. We show that if the principal components are in general position and the errors are sufficiently sparse, R2PCA will exactly recover the principal components with probability $1$, in lieu of assumptions on coherence or the distribution of the sparse errors, and even under adversarial settings. R2PCA enjoys many advantages: it works well under noise, its computational complexity scales linearly in the ambient dimension, it is easily parallelizable, and due to its low sample complexity, it can be used in settings where data is so large it cannot even be stored in memory. We complement our theoretical findings with synthetic and real data experiments showing that R2PCA outperforms state-of-the-art methods in a broad range of settings.

We propose a flexible convex relaxation for the phase retrieval problem that operates in the natural domain of the signal. Therefore, we avoid the prohibitive computational cost associated with “lifting” and semidefinite programming (SDP) in methods such as PhaseLift and compete with recently developed non-convex techniques for phase retrieval. We relax the quadratic equations for phaseless measurements to inequality constraints each of which representing a symmetric “slab”. Through a simple convex program, our proposed estimator finds an extreme point of the intersection of these slabs that is best aligned with a given anchor vector. We characterize geometric conditions that certify success of the proposed estimator. Furthermore, using classic results in statistical learning theory, we show that for random measurements the geometric certificates hold with high probability at an optimal sample complexity. We demonstrate the effectiveness of the proposed method through numerical simulations using both independent random measurements and coded diffraction patterns.

We also extend this formulation to include sparsity constraints on the target vector. With this additional constraint, we show that, considering “nested” measurements, the number of phaseless measurements needed to recover the sparse vector is essentially the same (to within a logarithmic factor) as the number of linear measurements needed by standard compressive sensing techniques.

A key challenge in online learning is that classical algorithms can be slow to adapt to changing environments. Recent studies have proposed “meta” algorithms that convert any online learning algorithm to one that is adaptive to changing environments, where the adaptivity is analyzed in a quantity called the strongly-adaptive regret. This paper describes a new meta algorithm that has a strongly-adaptive regret bound that is a factor of $\sqrt{\log (T)}$ better than other algorithms with the same time complexity, where $T$ is the time horizon. We also extend our algorithm to achieve a first-order (i.e., dependent on the observed losses) strongly-adaptive regret bound for the first time, to our knowledge. At its heart is a new parameter-free algorithm for the learning with expert advice (LEA) problem in which experts sometimes do not output advice for consecutive time steps (i.e., sleeping experts). This algorithm is derived by a reduction from optimal algorithms for the so-called coin betting problem. Empirical results show that our algorithm outperforms state-of-the-art methods in both learning with expert advice and metric learning scenarios.

In this paper we describe an algorithm for estimating the provenance of hacks on websites. That is, given properties of sites and the temporal occurrence of attacks, we are able to attribute individual attacks to joint causes and vulnerabilities, as well as estimate the evolution of these vulnerabilities over time. Specifically, we use hazard regression with a time-varying additive hazard function parameterized in a generalized linear form. The activation coefficients on each feature are continuous-time functions over time. We formulate the problem of learning these functions as a constrained variational maximum likelihood estimation problem with total variation penalty and show that the optimal solution is a $0$th order spline (a piecewise constant function) with a finite number of adaptively chosen knots. This allows the inference problem to be solved efficiently and at scale by solving a finite dimensional optimization problem. Extensive experiments on real data sets show that our method significantly outperforms Cox’s proportional hazard model. We also conduct case studies and verify that the fitted functions of the features respond to real-life campaigns.

We consider the problem of structure learning for bow-free acyclic path diagrams (BAPs). BAPs can be viewed as a generalization of linear Gaussian DAG models that allow for certain hidden variables. We present a first method for this problem using a greedy score-based search algorithm. We also prove some necessary and some sufficient conditions for distributional equivalence of BAPs which are used in an algorithmic approach to compute (nearly) equivalent model structures. This allows us to infer lower bounds of causal effects. We also present applications to real and simulated datasets using our publicly available R-package.

The field of discrete event simulation and optimization techniques motivates researchers to adjust classic ranking and selection (R&S) procedures to the settings where the number of populations is large. We use insights from extreme value theory in order to reveal the asymptotic properties of R&S procedures. Namely, we generalize the asymptotic result of Robbins and Siegmund regarding selection from independent Gaussian populations with known constant variance by their means to the case of selecting a subset of varying size out of a given set of populations. In addition, we revisit the problem of selecting the population with the highest mean among independent Gaussian populations with unknown and possibly different variances. Particularly, we derive the relative asymptotic efficiency of Dudewicz and Dalal’s and Rinott’s procedures, showing that the former can be asymptotically superior by a multiplicative factor which is larger than one, but this factor may be reduced by proper choice of parameters. We also use our asymptotic results to suggest that the sample size in the first stage of the two procedures should be logarithmic in the number of populations.

We study exchangeable, Markov survival processes – stochastic processes giving rise to infinitely exchangeable non-negative sequences $(T_{1},T_{2},\ldots)$. We show how these are determined by their characteristic index $\{\zeta_{n}\}_{n=1}^{\infty}$. We identify the harmonic process as the family of exchangeable, Markov survival processes that compose the natural set of statistical models for time-to-event data. In particular, this two-dimensional family comprises the set of exchangeable, Markov survival processes with weakly continuous predictive distributions. The harmonic process is easy to generate sequentially, and a simple expression exists for both the joint probability distribution and multivariate survivor function. We show a close connection with the Kaplan-Meier estimator of the survival distribution. Embedded within the process is an infinitely exchangeable ordered partition. Aspects of the process, such as the distribution of the number of blocks, are investigated.