Boosting is one of the most important recent developments in classification methodology. Boosting works by sequentially applying a classification algorithm to reweighted versions of the training data and then taking a weighted majority vote of the sequence of classifiers thus produced. For many classification algorithms, this simple strategy results in dramatic improvements in performance. We show that this seemingly mysterious phenomenon can be understood in terms of well-known statistical principles, namely additive modeling and maximum likelihood. For the two-class problem, boosting can be viewed as an approximation to additive modeling on the logistic scale using maximum Bernoulli likelihood as a criterion. We develop more direct approximations and show that they exhibit nearly identical results to boosting. Direct multiclass generalizations based on multinomial likelihood are derived that exhibit performance comparable to other recently proposed multiclass generalizations of boosting in most situations, and far superior in some. We suggest a minor modification to boosting that can reduce computation, often by factors of 10 to 50. Finally, we apply these insights to produce an alternative formulation of boosting decision trees. This approach, based on best-first truncated tree induction, often leads to better performance, and can provide interpretable descriptions of the aggregate decision rule. It is also much faster computationally, making it more suitable to large-scale data mining applications.

Scale space theory from computer vision leads to an interesting and novel approach to nonparametric curve estimation. The family of smooth curve estimates indexed by the smoothing parameter can be represented as a surface called the scale space surface. The smoothing parameter here plays the same role as that played by the scale of resolution in a visual system. In this paper, we study in detail various features of that surface from a statistical viewpoint. Weak convergence of the empirical scale space surface to its theoretical counterpart and some related asymptotic results have been established under appropriate regularity conditions. Our theoretical analysis provides new insights into nonparametric smoothing procedures and yields useful techniques for statistical exploration of features in the data. In particular, we have used the scale space approach for the development of an effective exploratory data analytic tool called SiZer.

Generalized linear models (GLM) include many useful models. This paper studies simultaneous confidence regions for the mean response function in these models. The coverage probabilities of these regions are related to tail probabilities of maxima of Gaussian random fields, asymptotically, and hence, the so-called tube formula is applicable without any modification. However, in the generalized linear models, the errors are often nonadditive and non-Gaussian and may be discrete. This poses a challenge to the accuracy of the approximation by the tube formula in the moderate sample situation. Here two alternative approaches are considered. These approaches are based on an Edgeworth expansion for the distribution of a maximum likelihood estimator and a version of Skorohod’s representation theorem, which are used to convert an error term (which is of order $n^{-1 /2}$ in one-sided confidence regions and of $n^{-1} in two-sided confidence regions) from the Edgeworth expansion to a “bias” term. The bias is then estimated and corrected in two ways to adjust the approximation formula. Examples and simulations show that our methods are viable and complementary to existing methods. An application to insect data is provided. Code for implementing our procedures is available via the software parfit

Statistical depth functions are being formulated ad hoc with increasing popularity in nonparametric inference for multivariate data. Here we introduce several general structures for depth functions, classify many existing examples as special cases, and establish results on the possession, or lack thereof, of four key properties desirable for depth functions in general. Roughly speaking, these properties may be described as: affine invariance, maximality at center, monotonicity relative to deepest point, and vanishing at infinity. This provides a more systematic basis for selection of a depth function. In particular, from these and other considerations it is found that the halfspace depth behaves very well overall in comparison with various competitors.

Statistical depth functions have become increasingly used in nonparametric inference for multivariate data. Here the contours of such functions are studied. Structural properties of the regions enclosed by contours, such as affine equivariance, nestedness, connectedness and compactness, and almost sure convergence results for sample depth contours, are established. Also, specialized results are established for some popular depth functions, includinghalfspace depth, and for the case of elliptical distributions. Finally, some needed foundational results on almost sure convergence of sample depth functions are provided.

We consider the asymptotic behavior of posterior distributions and Bayes estimators for infinite-dimensional statistical models. We give general results on the rate of convergence of the posterior measure. These are applied to several examples, including priors on finite sieves, log-spline models, Dirichlet processes and interval censoring.

We study the Bayesian approach to nonparametric function estimation problems such as nonparametric regression and signal estimation. We consider the asymptotic properties of Bayes procedures for conjugate (= Gaussian) priors.

We show that so long as the prior puts nonzero measure on the very large parameter set of interest then the Bayes estimators are not satisfactory. More specifically, we show that these estimators do not achieve the correct minimax rate over norm bounded sets in the parameter space. Thus all Bayes estimators for proper Gaussian priors have zero asymptotic efficiency in this minimax sense.

We then present a class of priors whose Bayes procedures attain the optimal minimax rate of convergence. These priors may be viewed as compound, or hierarchical, mixtures of suitable Gaussian distributions.

Designs for sets of experimental units with many blocking factors are studied. It is shown that if the set of blocking factors satisfies a certain simple condition then the information matrix for the design has a simple form. In consequence, a design is optimal if it is optimal with respect to one particular blocking factor and regular with respect to all the rest, in a sense which is made precise in the paper. This encompasses several previous results for optimal designs with more than one blocking factor, and applications to many other situations are given.

For mixture models on the simplex, we discuss the improvement of a given design in terms of increasing symmetry, as well as obtaining a larger moment matrix under the Loewner ordering. The two criteria together define the Kiefer design ordering. For the second-degree mixture model, we show that the set of weighted centroid designs constitutes a convex complete class for the Kiefer ordering. For four ingredients, the class is minimal complete. Of essential importance for the derivation is a certain moment polytope, which is studied in detail.

Let $\{Y_j : j = 1,\ldots,n\}$ be independent observations in $\mathbb{R}^m, m \geq 1$ with common distribution $Q$. Suppose that $Y_j = X_j + \xi_j, j = 1,\ldots,n$, where $\{X_j, \xi_j, j = 1,\ldots,n\}$ are independent, $X_ j, j = 1,\ldots,n$ have common distribution $P$ and $\xi_ j, j = 1, \ldots,n$ have common distribution $\mu$, so that $Q = P * \mu$. The problem is to recover hidden geometric structure of the support of $P$ based on the independent observations $Y_j$. Assuming that the distribution of the errors $\mu$ is known, deconvolution statistical estimates of the fractal dimension and the hierarchical cluster tree of the support that converge with exponential rates are suggested. Moreover, the exponential rates of convergence hold for adaptive versions of the estimates even in the case of normal noise $\xi_ j$ with unknown covariance. In the case of the dimension estimation, though, the exponential rate holds only when the set of all possible values of the dimension is finite (e.g., when the dimension is known to be integer). If this set is infinite, the optimal convergence rate of the estimator becomes very slow (typically, logarithmic), even when there is no noise in the observations.

The aim of tomography is to reconstruct a multidimensional function from observations of its integrals over hyperplanes. We consider the model that corresponds to the case of positron emission tomography. We have $n$ i.i.d.observations from a probability density proportional to $Rf$, where $Rf$ stands for the Radon transform of the density $f$.We assume that $f$ is an $N$-dimensional density such that its Fourier transform is exponentially decreasing. We find an estimator of $f$ which is asymptotically efficient; it achieves the optimal rate of convergence and also the best constant for the minimax risk.

We exhibit an empirical Bayes test $\delta_n^*$ for the normal mean testing problem using a linear error loss. Under the condition that the critical point of a Bayes test is within some known compact interval, $\delta_n^*$ is shown to be asymptotically optimal and its associated regret Bayes risk converges to zero at a rate $O(n^{-1}(\ln n)^{1.5})$, where $n$ is the number of past experiences available when the current component decision problem is considered. Under the same condition this rate is faster than the optimal rate of convergence claimed by Karunamuni.