One of the surprising recurring phenomena observed in experiments with boosting is that the test error of the generated classifier usually does not increase as its size becomes very large, and often is observed to decrease even after the training error reaches zero. In this paper, we show that this phenomenon is related to the distribution of margins of the training examples with respect to the generated voting classification rule, where the margin of an example is simply the difference between the number of correct votes and the maximum number of votes received by any incorrect label. We show that techniques used in the analysis of Vapnik’s support vector classifiers and of neural networks with small weights can be applied to voting methods to relate the margin distribution to the test error. We also show theoretically and experimentally that boosting is especially effective at increasing the margins of the training examples. Finally, we compare our explanation to those based on the bias-variance decomposition.

In the problem of regions, we wish to know which one of a discrete set of possibilities applies to a continuous parameter vector. This problem arises in the following way: we compute a descriptive statistic from a set of data, notice an interesting feature and wish to assign a confidence level to that feature. For example, we compute a density estimate and notice that the estimate is bimodal. What confidence can we assign to bimodality? A natural way to measure confidence is via the bootstrap: we compute our descriptive statistic on a large number of bootstrap data sets and record the proportion of times that the feature appears. This seems like a plausible measure of confidence for the feature. The paper studies the construction of such confidence values and examines to what extent they approximate frequentist $p$-values and Bayesian a posteriori probabilities. We derive more accurate confidence levels using both frequentist and objective Bayesian approaches. The methods are illustrated with a number of examples, including polynomial model selection and estimating the number of modes of a density.

A general formula for computing the breakdown point in robustness for the $t$th bootstrap quantile of a statistic $T_n$ is obtained. The answer depends on $t$ and the breakdown point of $T_n$. Since the bootstrap quantiles are vital ingredients of bootstrap confidence intervals, the theory has implications pertaining to robustness of bootstrap confidence intervals. For certain $L$ and $M$ estimators, a robustification of bootstrap is suggested via the notion of Winsorization.

A computational scheme for fitting smoothing spline ANOVA models to large data sets with a (near) tensor product design is proposed. Such data sets are common in spatial-temporal analyses. The proposed scheme uses the backfitting algorithm to take advantage of the tensor product design to save both computational memory and time. Several ways to further speed up the backfitting algorithm, such as collapsing component functions and successive over-relaxation, are discussed. An iterative imputation procedure is used to handle the cases of near tensor product designs. An application to a global historical surface air temperature data set, which motivated this work, is used to illustrate the scheme proposed.

In this paper, we study the local behavior of regression splines. In particular, explicit expressions for the asymptotic pointwise bias and variance of regression splines are obtained. In addition, asymptotic normality for regression splines is established, leading to the construction of approximate confidence intervals and confidence bands for the regression function.

Standard wavelet shrinkage procedures for nonparametric regression are restricted to equispaced samples. There, data are transformed into empirical wavelet coefficients and threshold rules are applied to the coefficients. The estimators are obtained via the inverse transform of the denoised wavelet coefficients. In many applications, however, the samples are nonequispaced. It can be shown that these procedures would produce suboptimal estimators if they were applied directly to nonequispaced samples.

We propose a wavelet shrinkage procedure for nonequispaced samples. We show that the estimate is adaptive and near optimal. For global estimation, the estimate is within a logarithmic factor of the minimax risk over a wide range of piecewise Hölder classes, indeed with a number of discontinuities that grows polynomially fast with the sample size. For estimating a target function at a point, the estimate is optimally adaptive to unknown degree of smoothness within a constant. In addition, the estimate enjoys a smoothness property: if the target function is the zero function, then with probability tending to 1 the estimate is also the zero function.

The asymptotic risk of efficient estimators with Kullback–Leibler loss in smoothly parametrized statistical models is $k/2_n$, where $k$ is the parameter dimension and $n$ is the sample size. Under fairly general conditions, we given a simple information-theoretic proof that the set of parameter values where any arbitrary estimator is superefficient is negligible. The proof is based on a result of Rissanen that codes have asymptotic redundancy not smaller than $(k/2)\log n$, except in a set of measure 0.

An unknown signal plus white noise is observed at $n$ discrete time points. Within a large convex class of linear estimators of $\xi$, we choose the estimator $\hat{\xi}$ that minimizes estimated quadratic risk. By construction, $\hat{\xi}$ is nonlinear. This estimation is done after orthogonal transformation of the data to a reasonable coordinate system. The procedure adaptively tapers the coefficients of the transformed data. If the class of candidate estimators satisfies a uniform entropy condition, then $\hat{\xi}$ is asymptotically minimax in Pinsker’s sense over certain ellipsoids in the parameter space and shares one such asymptotic minimax property with the James–Stein estimator. We describe computational algorithms for $\hat{\xi}$ and construct confidence sets for the unknown signal. These confidence sets are centered at $\hat{\xi}$, have correct asymptotic coverage probability and have relatively small risk as set-valued estimators of $\xi$.

Based on empirical Lévy-type concentration functions, a new graphical representation of the ML-density estimator under order restrictions is given. This representation generalizes the well-known representation of the Grenander estimator of a monotone density as the slope of the least concave majorant of the empirical distribution function to higher dimensions and arbitrary order restrictions. From the given representation it follows that a density estimator called silhouette, which arises naturally out of the excess mass approach, is the ML-density estimator under order restrictions. This fact provides a new point of view to ML-density estimation from which one gains additional insight to this problem, as demonstrated in the present paper.

The purpose of this article is to derive and illustrate a method for fitting models involving both convex and log-convex constraints on the probability vector(s) of a (product) multinomial distribution. We give a two-step algorithm to obtain maximum likelihood estimates of the probability vector(s) and show that it is guaranteed to converge to the true solution. Some examples are discussed which illustrate the procedure.

This paper outlines a theoretical framework for finite population models with unequal sample probabilities, along with sampling schemes for drawing random samples from these models. We first present four exact weighted sampling schemes that can be used for any finite population model to satisfy such requirements as ordered/ unordered samples, with/without replacement, and fixed/nonfixed sample size. We then introduce a new class of finite population models called weighted polynomial models or, in short, WPM. The probability density of a WPM is defined through a symmetric polynomial of the weights of the units in the sample. The WPM is shown to have been applied in many statistical analyses including survey sampling, logistic regression, case-control studies, lottery, DNA sequence alignment and MCMC simulations. We provide general strategies that can help improve the efficiency of the exact weighted sampling schemes for any given WPM. We show that under a mild condition, sampling from any WPM can be implemented within polynomial time. A Metropolis-Hasting-type scheme is proposed for approximate weighted sampling when the exact sampling schemes become intractable for moderate population and sample sizes. We show that under a mild condition, the average acceptance rate of the approximate sampling scheme for any WPM can be expressed in closed form using only the inclusion probabilities.

In the context of regression analysis it is known that the residual cusum process may serve as a basis for the construction of various omnibus, smooth and directional goodness-of-fit tests. Since a deeper analysis requires the decomposition of the cusums into their principal components and this is difficult to obtain, we propose to replace this process by its innovation martingale. It turns out that the resulting tests are (asymptotically) distribution free under composite null models and may be readily performed. A simulation study is included which indicates that the distributional approximations already work for small to moderate sample sizes.

Consider a counting process $N(t), t\inT}$ with compensator process ${A(t),t\in T}$, where $A(t)=\int_0^t Y(s) ds, {Y(t), t\in T}$ is an observable predictable process, and $\lambda_0(\dot)$ is an unknown hazard rate function. A general procedure for extending Neyman’s smooth goodness­of­fit test for the composite null hypothesis $H_0: \lambda_0(\dot)\inC ={\lambda_0(\dot;\eta):\eta\in\Gamma\subseteq\Re^q}$ is proposed and developed. The extension is obtained by embedding $C$ in the class $A_ k$ whose members are of the form $\lambda_0(\dot;\eta)\exp{\theta^t\psi(\dot;\eta)}, (\eta,\theta) \in\Gamma\times\Re^k$, where $\psi(\dot;\eta)=(\psi_1(\dot;\eta,\ldots,\psi_k(\dot;\eta))^t$ is a vector of observable random processes satisfying certain regularity conditions. The tests are based on quadratic forms of the statistic $\int_0^\tau\psi(s;\hat{\eta})dM(s;\hat{\eta})$, where $M(t;\eta) = N(t) - \int_0^t Y(s)\lambda_0(s;\eta) ds$ and $\hat {\eta}$ is a restricted maximum likelihood estimator of $\eta$. Asymptotic properties of the test statistics are obtained under a sequence of local alternatives, and the asymptotic local powers of the tests are examined. The effect of estimating $\eta$ by $\hat{\eta}$ is ascertained, and the problem of choosing the $\lambda$­process is discussed. The procedure is illustrated by developing tests for testing that $\lambda_0(\dot)$ belongs to (i) the class of constant hazard rates and ii the class of Weibull hazard rates, with particular emphasis on the random censorship model. Simulation results concerning the achieved levels and powers of the tests are presented, and the procedures are applied to three data sets that have been considered in the literature.

In this paper, a method for multivariate testing based on low-dimensional, data-dependent, linear scores is proposed. The new approach reduces the dimensionality of observations and increases the stability of the solutions. The method is reliable, even if there are many redundant variables. As a key feature, the score coefficients are chosen such that a left-spherical distribution of the scores is reached under the null hypothesis. Therefore, well-known tests become applicable in high-dimensional situations, too. The presented strategy is an alternative to least squares and maximum likelihood approaches. In a natural way, standard problems of multivariate analysis thus induce the occurrence of left-spherical, nonnormal distributions. Hence, new fields of application are opened up to the generalized multivariate analysis. The proposed methodology is not restricted to normally distributed data, but can also be extended to any left-spherically distributed observations.

Let $\mathrm{y}~ N(Bx_i,\Sigma),i=1,2\ldots,N$, and $\mathrm{y}~N(B\theta, \Sigma)$ be independent multivariate observations, where the $x_i$'s are known vectors, $B,\theta$ and $\Sigma$ are unknown, $\Sigma$ being a positive definite matrix. The calibration problem deals with statistical inference concerning $\theta$ and the problem that we have addressed is the construction of confidence regions. In this article, we have constructed a region for $\theta$ based on a criterion similar to that satisfied by a tolerance region. The situation where $\theta$ is possibly a nonlinear function, say $\mathrm{h}(\xi)$ of fewer unknown parameters denoted by the vector $(\xi)$, is also considered. The problem addressed in this context is the construction of a region for $\xi$. The numerical computations required for the practical implementation of our region are explained in detail and illustrated using an example. Limited numerical results indicate that our regions satisfy the coverage probability requirements of multiple­use confidence regions.

We derive an approximation of a density estimator based on weakly dependent random vectors by a density estimator built from independent random vectors. We construct, on a sufficiently rich probability space, such a pairing of the random variables of both experiments that the set of observations $X_1,\ldots,X_n}$ from the time series model is nearly the same as the set of observations $Y_1,\ldots,Y_n}$ from the i.i.d. model. With a high probability, all sets of the form $({X_1,\ldots,X_n}\\Delta{Y_1,\ldots,Y_n})\bigcap([a_1,b_1]\times\ldots\times[a_d,b_d])$ contain no more than $O({[n^1/2 \prod(b_i-a_i)]+ 1} \log(n))$ elements, respectively. Although this does not imply very much for parametric problems, it has important implications in nonparametric statistics. It yields a strong approximation of a kernel estimator of the stationary density by a kernel density estimator in the i.i.d. model. Moreover, it is shown that such a strong approximation is also valid for the standard bootstrap and the smoothed bootstrap. Using these results we derive simultaneous confidence bands as well as supremum­type nonparametric tests based on reasoning for the i.i.d. model.

We study the sample ACVF and ACF of a general stationary sequence under a weak mixing condition and in the case that the marginal distributions are regularly varying. This includes linear and bilinear processes with regularly varying noise and ARCH processes, their squares and absolute values. We show that the distributional limits of the sample ACF can be random, provided that the variance of the marginal distribution is infinite and the process is nonlinear. This is in contrast to infinite variance linear processes. If the process has a finite second but infinite fourth moment, then the sample ACF is consistent with scaling rates that grow at a slower rate than the standard $\sqrt{n}$. Consequently, asymptotic confidence bands are wider than those constructed in the classical theory. We demonstrate the theory in full detail for an ARCH (1) process.