Recent work has shown that combining multiple versions of unstable classifiers such as trees or neural nets results in reduced test set error. One of the more effective is bagging. Here, modified training sets are formed by resampling from the original training set, classifiers constructed using these training sets and then combined by voting. Freund and Schapire propose an algorithm the basis of which is to adaptively resample and combine (hence the acronym “arcing”) so that the weights in the resampling are increased for those cases most often misclassified and the combining is done by weighted voting. Arcing is more successful than bagging in test set error reduction. We explore two arcing algorithms, compare them to each other and to bagging, and try to understand how arcing works. We introduce the definitions of bias and variance for a classifier as components of the test set error. Unstable classifiers can have low bias on a large range of data sets. Their problem is high variance. Combining multiple versions either through bagging or arcing reduces variance significantly.

The finite-sample risk of the $k$ nearest neighbor classifier that uses a weighted $L^p$-metric as a measure of class similarity is examined. For a family of classification problems with smooth distributions in $mathbb{R}^n$, an asymptotic expansion for the risk is obtained in decreasing fractional powers of the reference sample size. An analysis of the leading expansion coefficients reveals that the optimal weighted $L^p$-metric, that is, the metric that minimizes the finite-sample risk, tends to a weighted Euclidean (i.e., $L^2$) metric as the sample size is increased. Numerical simulations corroborate this finding for a pattern recognition problem with normal class-conditional densities.

We attempt to recover an unknown function from noisy, sampled data. Using orthonormal bases of compactly supported wavelets, we develop a nonlinear method which works in the wavelet domain by simple nonlinear shrinkage of the empirical wavelet coefficients. The shrinkage can be tuned to be nearly minimax over any member of a wide range of Triebel- and Besov-type smoothness constraints and asymptotically mini-max over Besov bodies with $p \leq q$. Linear estimates cannot achieve even the minimax rates over Triebel and Besov classes with $p<2$, so the method can significantly outperform every linear method (e.g., kernel, smoothing spline, sieve in a minimax sense). Variants of our method based on simple threshold nonlinear estimators are nearly minimax. Our method possesses the interpretation of spatial adaptivity; it reconstructs using a kernel which may vary in shape and bandwidth from point to point, depending on the data. Least favorable distributions for certain of the Triebel and Besov scales generate objects with sparse wavelet transforms. Many real objects have similarly sparse transforms, which suggests that these minimax results are relevant for practical problems. Sequels to this paper, which was first drafted in November 1990, discuss practical implementation, spatial adaptation properties, universal near minimaxity and applications to inverse problems.

Motivated by recently developed threshold rules for wavelet estimators, we suggest threshold methods for general kernel density estimators, including those of classical Rosenblatt–Parzen type. Thresholding makes kernel methods competitive in terms of their adaptivity to a wide variety of aberrations in complex signals. It is argued that term-by-term thresholding does not always produce optimal performance, since individual coefficients cannot be estimated sufficiently accurately for reliable decisions to be made. Therefore, we suggest grouping coefficients into blocks and making simultaneous threshold decisions about all coefficients within a given block. It is argued that block thresholding has a number of advantages, including that it produces adaptive estimators which achieve minimax-optimal convergence rates without the logarithmic penalty that is sometimes associated with term-by-term thresholding. More than this, the convergence rates are achieved over large classes of functions with discontinuities, indeed with a number of discontinuities that diverges polynomially fast with sample size. These results are also established for block thresholded wavelet estimators, which, although they can be interpreted within the kernel framework, are often most conveniently constructed in a slightly different way.

Additive regression models have turned out to be a useful statistical tool in analyses of high-dimensional data sets. Recently, an estimator of additive components has been introduced by Linton and Nielsen which is based on marginal integration. The explicit definition of this estimator makes possible a fast computation and allows an asymptotic distribution theory. In this paper an asymptotic treatment of this estimate is offered for several models. A modification of this procedure is introduced. We consider weighted marginal integration for local linear fits and we show that this estimate has the following advantages.

(i) With an appropriate choice of the weight function, the additive components can be efficiently estimated: An additive component can be estimated with the same asymptotic bias and variance as if the other components were known.

(ii) Application of local linear fits reduces the design related bias.

For analyzing samples of curves, Kneip and Gasser proposed a “structural analysis” for estimating an average curve which gives the average amplitude and dynamics of the sample curves. An important step of their method is the estimation of the warping functions in order to eliminate the differences in dynamics between different curves. It is of interest to compute confidence bounds for the structural average curve. First, we derive the necessary asymptotic results to obtain confidence intervals based on asymptotic normality. Due to the complex form of the asymptotic formulas and due to their asymptotic nature, bootstrap procedures are studied in a second step. A small archive SACI (Structural Averaging and Confidence Intervals), written in Fortran, can be obtained from the Web site “http://www.unizh.ch/biostat” to compute the structural average curve, to construct confidence regions at a structural point and to compute confidence bars at other points.

The random truncation model is defined by the conditional probability distribution $H (x, y) =P[X\leq x,Y\leq y |X \geq Y] where $X$ and $Y$ are independent random variables. A problem of interest is the estimation of the distribution function $F$ of $X$ with data from the distribution $H$. Under random truncation, $F$ need not be fully identifiable from $H$ and only a part of it, say $F_0$ , is. We show that the nonparametric MLE $F_n$ of $F_0$ obeys the strong law of large numbers in the sense that for any nonnegative, measurable function $\phi(x)$, the integrals $\int\phi(x)dF_n(x)\to\int\phi(x)dF_0(x)$ almost surely as $n$ tends to infinity. Similar results were first obtained by Stute and Wang for the right censoring model. The results are useful in establishing the strong consistency of various estimates. Some of our results are derived from the weak consistency of $F_n$ obtained by Woodroofe.

Under random truncation, a pair of independent random variables $X$ and $Y$ is observable only if $X$ is larger than $Y$. The resulting model is the conditional probability distribution $H( x, y) =P[X \leq x,Y \leq y|X \geq Y]$. For the truncation probability $\alpha=P[X \geq Y]$, a proper estimate is not the sample proportion but $\alpha_n=\int G_n (s)dF_n(s)$ where $F_n$ and $G_n$ are product limit estimates of the distribution functions $F$ and$G$ of $X$ and$Y$, respectively. We obtain a much simpler representation $\hat {\alpha}_n$ for $\alpha_n$. With this, the strong consistency, an iid representation (and hence asymptotic normality), and a LIL for the estimate are established. The results are true for arbitrary$F$ and $G$. The continuity restriction on $F$ and $G$ often imposed in the literature is not necessary. Furthermore, the representation $\hat {\alpha}_n$ of $\alpha_n$ facilitates the establishment of the strong law for the product limit estimates $F_n$ and $G_n$.

Fan, Heckman and Wand proposed locally weighted kernel polynomial regression methods for generalized linear models and quasilikelihood functions. When the covariate variables are missing at random, we propose a weighted estimator based on the inverse selection probability weights. Distribution theory is derived when the selection probabilities are estimated nonparametrically. We show that the asymptotic variance of the resulting nonparametric estimator of the mean function in the main regression model is the same as that when the selection probabilities are known, while the biases are generally different. This is different from results in parametric problems, where it is known that estimating weights actually decreases asymptotic variance. To reconcile the difference between the parametric and nonparametric problems, we obtain a second-order variance result for the nonparametric case. We generalize this result to local estimating equations. Finite-sample performance is examined via simulation studies. The proposed method is demonstrated via an analysis of data from a case-control study.

In this paper we generalize the major results of Andersson and Perlman on LCI models to the setting of symmetric cones and give an explicit closed form formula for the estimate of the covariance matrix in the generalized LCI models that we define.

To this end, we replace the cone $H_I^+(\mathbb{R})$ sitting inside the Jordan algebra of symmetric real $I \times I$-matrices by the symmetric cone $\Omega$ of an Euclidean Jordan algebra $V$. We introduce the Andersson-Perlman cone $\Omega(\mathscr{K}\subseteq\Omega$ which generalizes $\mathscr{P}(\mathscr{K})\subseteq H_I^+(\mathscr{R})$. We prove several characterizations and properties of $\Omega(\mathscr{K})$ which allows us to recover, though with different proofs, the main results of Andersson and Perlman regarding $\mathscr{P}(\mathscr{K})$. The new lattice conditional independence models are defined, assuming that the Euclidean Jordan algebra $V$ has a symmetric representation. Using standard results from the theory of Jordan algebras, we can reduce the general model to the case where $V$ is the Jordan algebra of Hermitian matrices over the real, complex or quaternionic numbers, and $\Omega$ is the corresponding cone of positive-definite matrices. Our main statistical result is a closed-form formula for the estimate of the covariance matrix in the generalized LCI model. We also give the likelihood ratio test for testing a given model versus another one, nested within the first.

This paper develops nonparametric deconvolution density estimation over $SO(N)$, the group of $N \times N$ orthogonal matrices of determinant 1. The methodology is to use the group and manifold structures to adapt the Euclidean deconvolution techniques to this Lie group environment. This is achieved by employing the theory of group representations explicit to $SO(N)$. General consistency results are obtained with specific rates of convergence achieved under sufficient smoothness conditions. Application to empirical Bayes prior estimation and inference is also discussed.

This article gives two constructions of a weighted mean which has a large domain, is affinely equivariant, has a locally high breakdown point and is locally uniformly linearizable. One construction is based on $M$-functionals with smooth defining $\psi$- and $\chi$ -functions which are used to control the weighting. The second construction involves a locally uniformly linearizable reduction of the data to a finite set of points. This construction has the advantage of computational speed and opens up the possibility of allowing the weighting to take the shape of the original data set into account. Its disadvantage lies in its inability to deal with large atoms. The aim of the locally uniform linearizability is to provide a stable analysis based on uniform asymptotics or uniform bootstrapping. The stability of the first construction is exhibited using different stochastic models and different data sets. Its performance is compared with three other functionals which are not locally uniformly linearizable.

Robust tests for linear models are derived via Wald-type tests that are based on asymptotically linear estimators. For a robustness criterion, the maximum asymptotic bias of the level of the test for distributions in a shrinking contamination neighborhood is used. By also regarding the asymptotic power of the test, admissible robust tests and most-efficient robust tests are derived. For the greatest efficiency, the determinant of the covariance matrix of the underlying estimator is minimized. Also, most-robust tests are derived. It is shown that at the classical $D$-optimal designs, the most-robust tests and the most-efficient robust tests have a very simple form. Moreover, the $D$-optimal designs provide the highest robustness and the highest efficiency under robustness constraints across all designs. So, $D$-optimal designs are also the optimal designs for robust testing. Two examples are considered for which the most-robust tests and the most-efficient robust tests are given.

This paper provides a comparative sensitivity analysis of one-step Newton–Raphson estimators for linear regression. Such estimators have been proposed as a way to combine the global stability of high breakdown estimators with the local stability of generalized maximum likelihood estimators. We analyze this strategy, obtaining upper bounds for the maximum bias induced by $\varepsilon$-contamination of the model. These bounds yield break-down points and local rates of convergence of the bias as $\varepsilon$decreases to zero. We treat a unified class of Newton–Raphson estimators, including one-step versions of the well-known Schweppe, Mallows and Hill–Ryan GM estimators. Of the three well-known types, the Hill–Ryan form emerges as the most stable in terms of one-step estimation. The Schweppe form is susceptible to a breakdown of the Hessian matrix. For this reason it fails to improve on the local stability of the initial estimator, and it may lead to falsely optimistic estimates of precision.

In this article, we consider the asymptotic behavior of three kinds of sample breakdown points. It is shown that for the location $M$-estimator with bounded objective function, both the addition sample breakdown point and the simplified replacement sample breakdown point strongly converge to the gross-error asymptotic breakdown point, whereas the replacement sample breakdown point strongly converges to a smaller value. In addition, it is proved that under some regularity conditions these sample breakdown points are asymptotically normal. The addition sample breakdown point has a smaller asymptotic variance than the simplified replacement sample breakdown point. For the commonly used redescending $M$-estimators of location, numerical results indicate that among the three kinds of sample breakdown points, the replacement sample breakdown point has the largest asymptotic variance.

All classical “prophet inequalities” for independent random variables hold also in the case where only a noise-corrupted version of those variables is observable. That is, if the pairs $(X_1, Z_1),\ldots,(X_n, Z_n)$ are independent with arbitrary, known joint distributions, and only the sequence $Z_1 ,\ldots,Z_n$ is observable, then all prophet inequalities which would 1 n hold if the $X$’s were directly observable still hold, even though the expected $X$-values (i.e., the payoffs) for both the prophet and statistician, will be different. Our model includes, for example, the case when $Z_i=X_i + Y_i$, where the $Y$’s are any sequence of independent random variables.

Surveillance can be based, in some change-point detection problems, on a sequence of invariant statistics. Gordon and Pollak prove that, under certain conditions, the average run length (ARL) to false alarm of invariance- based Shiryayev–Roberts detection schemes is asymptotically the same as that of the dual classical scheme that is based on the original sequence of observations. In this paper we give alternative conditions under which the two ARLs coincide and demonstrate that these conditions are satisfied in cases where Gordon and Pollak’s conditions are difficult to check.