Recently, Rissanen proposed a new model selection criterion PLS that selects the model that minimizes the accumulated squares of prediction errors. Usually, the information-based criteria, such as AIC and BIC, select the model that minimizes a loss function which can be expressed as a sum of two terms. One measures the goodness of fit and the other penalizes the complexity of the selected model. In this paper we provide such an interpretation for PLS. Using this relationship, we give sufficient conditions for PLS to be strongly consistent in stochastic regression models. The asymptotic equivalence between PLS and BIC for ergodic models is then studied. Finally, based on the Fisher information, a new criterion FIC is proposed. This criterion shares most asymptotic properties with PLS while removing some of the difficulties encountered by PLS in a finite-sample situation.

Improvements to the classical inclusion-exclusion identity are developed. There are two main results: an abstract combinatoric result and a concrete geometric result. In the abstract result conditions are given which guarantee the existence of a depth $d + 1$ identity or inequality for the indicator function of a union of a finite collection of events, that is, an expression which is a linear combination of indicator functions of at most $(d + 1)$-fold intersections of the events. Such an identity or inequality can be integrated with respect to any probability measure to yield a probability identity or inequality. Connections are given to previous work on Bonferroni-type inequalities. The concrete result says that there is a depth $d + 1$ identity for the union of finitely many balls in $d$-dimensional Euclidean space. With a single correction term this result also holds in the $d$-dimensional sphere. These results form the basis for a discrete theory of tubes, which up to now has been continuous in nature. The spherical result is used to give a simulation method for finding critical probabilities for multiple-comparisons procedures, and a computer program implementing the method is described. Numerical results are presented which demonstrate that in the tails of the distribution probability estimates based on the method tend to exhibit less variability than estimates based on naive simulation.

Consider a stationary time series $(\mathbf{X}_t, Y_t), t = 0, \pm 1,\ldots,$ with $\mathbf{X}_t$ being $\mathbb{R}^d$-valued and $Y_t$ real-valued. The conditional mean function is given by $\theta(\mathbf{X}_0) = E(Y_0\mid\mathbf{X}_0)$. Under appropriate regularity conditions, a local average estimator of this function based on a finite realization $(\mathbf{X}_1, Y_1),\ldots,(\mathbf{X}_n, Y_n)$ can be chosen to achieve the optimal rate of convergence $n^{-1/(2 + d)}$ both pointwise and in $L_2$ norms restricted to a compact; and it can also be chosen to achieve the optimal rate of convergence $(n^{-1} \log(n))^{1/(2 + d)}$ in $L_\infty$ norm restricted to a compact. Similar results hold for local median estimators of the conditional median function, which is given by $\theta(\mathbf{X}_0) = \operatorname{med}(Y_0\mid\mathbf{X}_0)$.

For $i = 1,2,\ldots$, let $X_i$ and $Y_i$ be $\mathbb{R}^d$-valued ($d \geq 1$ integer) and $\mathbb{R}$-valued, respectively, random variables, and let $\{(X_i, Y_i)\}, i \geq 1$, be a strictly stationary and $\alpha$-mixing stochastic process. Set $m(x) = \mathscr{E}(Y_1\mid X_1 = x), x \in \mathbb{R}^d$, and let $\hat{m}_n(x)$ be a certain recursive kernel estimate of $m(x)$. Under suitable regularity conditions and as $n \rightarrow \infty$, it is shown that $\hat{m}_n(x)$, properly normalized, is asymptotically normal with mean 0 and a specified variance. This result is established, first under almost sure boundedness of the $Y_i$'s, and then by replacing boundedness by continuity of certain truncated moments. It is also shown that, for distinct points $x_1,\ldots,x_N$ in $\mathbb{R}^d (N \geq 2$ integer), the joint distribution of the random vector, $(\hat{m}_n(x_1),\ldots,\hat{m}_n(x_N))$, properly normalized, is asymptotically $N$-dimensional normal with mean vector 0 and a specified covariance function.

An approach to bootstrapping kernel spectral density estimates is described which is based on resampling from the periodogram of the original data. We show that it is asymptotically valid under suitable conditions, and we illustrate its performance for a medium-sized time series sample with a small simulation study.

Ordinary smoothing splines have an integrated mean squared error which is dominated by bias contributions at the boundaries. When the estimated function has additional derivatives, the boundary contribution to the bias affects the asymptotic rate of convergence unless the derivatives of the estimated function meet the natural boundary conditions. This paper introduces relaxed boundary smoothing splines and shows that they obtain the optimal asymptotic rate of convergence without conditions on the boundary derivatives of the estimated function.

The problem of estimating the integral of a stochastic process from observations at a finite number of sampling points is considered. Sacks and Ylvisaker found a sequence of asymptotically optimal sampling designs for general processes with exactly 0 and 1 quadratic mean (q.m.) derivatives using optimal-coefficient estimators, which depend on the process covariance. These results were extended to a restricted class of processes with exactly $K$ q.m. derivatives, for all $K = 0,1,2,\ldots$, by Eubank, Smith and Smith. The asymptotic performance of these optimal-coefficient estimators is determined here for regular sequences of sampling designs and general processes with exactly $K$ q.m. derivatives, $K \geq 0$. More significantly, simple nonparametric estimators based on an adjusted trapezoidal rule using regular sampling designs are introduced whose asymptotic performance is identical to that of the optimal-coefficient estimators for general processes with exactly $K$ q.m. derivatives for all $K = 0,1,2,\ldots$.

In this paper a method of identifying stationary and invertible vector autoregressive moving-average time series is proposed. The models are presumed to be represented in (reversed) echelon canonical form. Consideration is given to both parameter estimation and the determination of structural indices, the evaluations being based on the use of closed form least squares calculations. Consistency of the technique is shown and the operational characteristics of the procedure when employed as a means of approximating more general processes is discussed.

Bayes $A$-optimal designs for the one-way and the two-way elimination of heterogeneity models and optimal $\Gamma$-minimax designs for the one-way elimination of heterogeneity model for experiments to compare test treatments with a control are given. Properties such as robustness, of these designs are studied. Optimality results are derived in the exact theory setup.

In the class of polynomials of odd (or even) degree up to the order $2r - 1 (2r)$ optimal designs are determined which minimize a product of the variances of the estimates for the highest coefficients weighted with a prior $\gamma = (\gamma_1,\ldots,\gamma_r)$, where the numbers $\gamma_j$ correspond to the models of degree $2j - 1 (2j)$ for $j = 1,\ldots,r$. For a special class of priors, optimal designs of a very simple structure are calculated generalizing the $D_1$-optimal design for polynomial regression of degree $2r - 1 (2r)$. The support of these designs splits up in three sets and the masses of the optimal design at the support points of every set are all equal. The results are derived in a general context using the theory of canonical moments and continued fractions. Some applications are given to the $D$-optimal design problem for polynomial regression with vanishing coefficients of odd (or even) powers.

Product partition models assume that observations in different components of a random partition of the data are independent. If the probability distribution of random partitions is in a certain product form prior to making the observations, it is also in product form given the observations. The product model thus provides a convenient machinery for allowing the data to weight the partitions likely to hold; and inference about particular future observations may then be made by first conditioning on the partition and then averaging over all partitions. These models apply with special computational simplicity to change point problems, where the partitions divide the sequence of observations into components within which different regimes hold. We show, with appropriate selection of prior product models, that the observations can eventually determine approximately the true partition.

Guess, Hollander and Proschan proposed tests for exponentiality versus IDMRL (increasing initially and then decreasing mean residual life) distributions when the change point, or corresponding quantile, is known. In this paper we propose two tests which do not require such knowledge of the change point. The tests are based on estimates of functionals of the $\operatorname{cdf}$ which discriminate between the exponential and IDMRL families.

Suppose we wish to construct a variable $k$-cell histogram based on an independent identically distributed sample of size $n - 1$ from an unknown density $f$ on the interval of finite length. A variable cell histogram requires cutpoints and heights of all of its cells to be specified. We propose the following procedure: (i) choose from the order statistics corresponding to the sample a set of $k + 1$ cutpoints that maximize a criterion, a function of the sample spacings; (ii) compute heights of the $k$ cells according to a formula. The resulting histogram estimates a $k$-cell theoretical histogram that stays constant within a cell and that minimizes the Hellinger distance to the density $f$. The histogram tends to estimate low density regions accurately and is easy to compute. We find the number of cells of order $n^{1/3}$ minimizes the mean Hellinger distance between the density $f$ and a class of histograms whose cutpoints are chosen from the order statistics.

We show that regression quantiles, which could be computed as solutions of a linear programming problem, and the solutions of the corresponding dual problem, which we call the regression rank-scores, generalize the duality of order statistics and of ranks from the location to the linear model. Noting this fact, we study the regression quantile and regression rank-score processes in the heteroscedastic linear regression model, obtaining some new estimators and interesting comparisons with existing estimators.

P. J. Bickel's approach to and results on estimating the parameter vector $\beta$ of a conditionally contaminated linear regression model by asymptotically linear (AL) estimators $\hat\beta^\ast$ which have minimum trace of the asymptotic covariance matrix among all AL estimators with a given bound $b$ on their asymptotic bias (MT-AL estimators with bias bound $b$) is here extended to conditionally contaminated general linear models and in particular for estimating arbitrary linear aspects $\varphi(\beta) = C\beta$ of $\beta$ which are of actual interest in applications. Admitting that $\beta$ itself is not identifiable in the model (also a practically important situation), a complete characterization of MT-AL estimators with bias bound $b$ including the case where $b$ is smallest possible is presented here, which extends and sharpens H. Rieder's characterization of all AL estimators with minimum asymptotic bias. These characterizations (Theorem 1) represent generalizations (in different directions) of those which define Hampel-Krasker estimators for $\beta$ in linear regression models and admit (Theorem 2) explicit constructions of MT-AL estimators under generally applicable model assumption. Obviously, even in linear regression models, $\hat\varphi^\ast = C\hat\beta^\ast$ is not an MT-AL estimator for $\varphi$ if $\hat\beta^\ast$ is one for $\beta$ (there does not even exist an AL estimator nor an $M$ estimator for $\beta$, if $\beta$ is not identifiable in the model). Examples such as quadratic regression illustrate the not at all obvious relation between $\hat\beta^\ast$ and $\hat\varphi^\ast$, demonstrate the applicability of the general results and show explicitly the influence of the parametrization and the underlying design of the linear model.

We relate various measures of the stability of estimates in general parametric families and consider their application to direction estimates on spheres. We show that constructions such as the SB-robustness of Ko and Guttorp and the information-standardized gross-error sensitivity of Hampel, Ronchetti, Rousseeuw and Stahel fit into a general framework in which one measures the effect of model contamination by the Kullback-Leibler discrepancy. We also define a breakdown point appropriate for a compact parameter space. Specific results concerning direction estimation include the optimal robustness of the circular median, the optimal breakdown point of the least median of squares on the sphere, the SB-robustness of certain scale-adjusted $M$-estimators and the SB-robustness in arbitrary dimensions of a class of estimators including the $L_1$-estimator and the hyperspherical median. The latter estimators avoid the need for simultaneous scale estimates, and they have breakdown points approaching 1/2 as the model becomes concentrated. A slight modification in their definition yields the same theoretical breakdown point as the least median of squares.

It is known that the standard delete-1 jackknife and the classical bootstrap are in general equally efficient for estimating the mean-square-error of a statistic in the i.i.d. setting. However, this equivalence no longer holds in the linear regression model. It turns out that the bootstrap is more efficient when error variables are homogeneous and the jackknife is more robust when they are heterogeneous. In fact, we can divide all the commonly used resampling procedures for linear regression models into two types: the E-type (the efficient ones like the bootstrap) and the R-type (the robust ones like the jackknife). Thus the theory presented here provides a unified view of all the known resampling procedures in linear regression.

This paper discusses robust estimation for structural errors-in-variables (EV) linear regression models. Such models have important applications in many areas. Under certain assumptions, including normality, the maximum likelihood estimates for the EV model are provided by orthogonal regression (OR) which minimizes the orthogonal distance from the regression line to the data points instead of the vertical distance used in ordinary regression. OR is very sensitive to contamination and thus efficient robust procedures are needed. This paper examines the theoretical properties of bounded influence estimators for univariate Gaussian EV models using a generalized $M$-estimate approach. The results include Fisher consistency, most $B$-robust estimators and the OR version of Hampel's optimality problem.

We propose an affine equivariant estimator of multivariate location that combines a high breakdown point and a bounded influence function with high asymptotic efficiency. This proposal is basically a location $M$-estimator based on the observations obtained after scaling with an affine equivariant high breakdown covariance estimator. The resulting location estimator will inherit the breakdown point of the initial covariance estimator and within the location-covariance model only the $M$-estimator will determine the type of influence function and the asymptotic behaviour. We prove consistency and asymptotic normality and obtain the breakdown point and the influence function.

An adaptive maximum likelihood estimator based on the estimation of the log-density by $B$-splines is introduced. A data-driven method of selecting the smoothing parameter involved in the consequent density estimation is demonstrated. A Monte Carlo study is conducted to evaluate the small sample performance of the estimator in a location and a regression problem. The adaptive estimator is seen to compare favorably to some standard estimates. We show that the estimator is asymptotically efficient.

In this paper stochastic integrals with respect to the basic martingale are approximated by Gaussian processes. The probability inequalities governing this approximation are used to study goodness-of-fit tests based on sublinear functionals of weighted versions of these stochastic integrals. As special cases of these tests, generalized rank and supremum-type tests are considered.

We prove strong consistency of a class of maximum objective estimators for exponential parametric families of Markov random fields on $\mathbb{Z}^d$, including both maximum likelihood and pseudolikelihood estimators, using large deviation estimates. We also obtain the optimality property for the maximum likelihood estimator in the sense of Bahadur.

Suppose $\hat{\theta}_n$ is a strongly consistent estimator for $\theta_0$ in some i.i.d. situation. Let $N_\varepsilon$ and $Q_\varepsilon$ be, respectively, the last $n$ and the total number of $n$ for which $\hat{\theta}_n$ is at least $\varepsilon$ away from $\theta_0$. The limit distributions for $\varepsilon^2N_\varepsilon$ and $\varepsilon^2Q_\varepsilon$ as $\varepsilon$ goes to zero are obtained under natural and weak conditions. The theory covers both parametric and nonparametric cases, multidimensional parameters and general distance functions. Our results are of probabilistic interest, and, on the statistical side, suggest ways in which competing estimators can be compared. In particular several new optimality properties for the maximum likelihood estimator sequence in parametric families are established. Another use of our results is ways of constructing sequential fixed-volume or shrinking-volume confidence sets, as well as sequential tests with power 1. The paper also includes limit distribution results for the last $n$ and the number of $n$ for which the supremum distance $\|F_n - F\| \geq \varepsilon$, where $F_n$ is the empirical distribution function. Other results are reached for $\varepsilon^{5/2}N_\varepsilon$ and $\varepsilon^{5/2}Q_\varepsilon$ in the context of nonparametric density estimation, referring to the last time and the number of times where $|f_n(x) - f(x)| \geq \varepsilon$. Finally, it is shown that our results extend to several non-i.i.d. situations.

Consider the problem of estimating the total of some finite population. Suppose the labels attached to the units of the population are such that members of the population whose labels are close together are more alike than units whose labels are far apart. For such a population a sensible estimator of the population total is one that interpolates linearly between successive members of the sample. The admissibility of this estimator will be demonstrated. The related interval estimators will be studied as well.

Normal approximations, as provided by permutational central limit theorems, conditionally can be arbitrarily bad. Such approximations therefore are poorly suited to the construction of critical values for Pitman (permutation) tests. A classical remedy consists in substituting a beta approximation (over the appropriate conditional interval range) for the normal one. Whereas deriving permutational extreme values for usual, nonserial statistics is generally straightforward, the corresponding problem for serial statistics (e.g., autocorrelation coefficients), however, appears somewhat more difficult. This problem, which is shown to reduce to a particular case of the well-known travelling salesman problem, is explicitly solved here for the autocorrelation coefficient of order one, allowing for a simple computation of permutational critical values for Pitman tests against serial dependence. The case of higher order autocorrelations is, however, of a different nature and requires another approach.

In this paper we consider the product-limit estimator of the survival distribution function in the context of independent but nonidentically distributed censoring times. An upper bound on the mean square increment of the stopped Kaplan-Meier process is obtained. Also, a representation is given for the ratio of the survival distribution function to the product-limit estimator as the product of a bounded process and a martingale. From this representation bounds on the mean square of the ratio and on the tail probability of the sup norm of the ratio are derived.

White noise models often renormalize exactly yielding optimal rates of convergence for pointwise nonparametric functional estimation problems. Similar rescaling ideas lead to a sequence of experiments appropriate for pointwise density estimation problems.

Let $X_i$ be i.i.d. $X_i \sim F_\theta$. For some parametric families $\{F_\theta\}$, we describe a monotonicity property of Bayes sequential procedures for the decision problem $H_0: \theta = 0$ versus $H_1: \theta \neq 0$. A surprising counterexample is given in the case where $F_\theta$ is $N(\theta, 1)$.

A sequential procedure for estimating the mean of a normal distribution is proposed. The procedure is shown to have a negative regret at the origin and the same regret as the usual procedure at the other point asymptotically.

In the intuitive approach, a distribution function $F$ is said to be not more heavily tailed than $G$ if $\lim \sup_{x \rightarrow \infty} \bar{F}/\bar{G} < \infty$. An alternative is to consider the behavior of the ratio $F^{-1}(u)/G^{-1}(u)$, in a neighborhood of one. The present paper examines the relationship between these two criteria and concludes that the intuitive approach gives a more thorough comparison of distribution functions than the ratio of the quantile functions approach in the case $F$ or $G$ have tails that decrease faster than, or at, an exponential rate. If $F$ or $G$ have slowly varying tails, the intuitive approach gives a less thorough comparison of distributions. When $F$ or $G$ have polynomial tails, the approaches agree.

Simple conditions on the observed information ensure asymptotic normality of the conditional distributions of the randomly normed score statistic and maximum likelihood estimator given a suitable asymptotically ancillary statistic. In particular, asymptotic normality holds conditional on any asymptotically ancillary statistic asymptotically equivalent to observed information. The results apply to inference from a general stochastic process and are of particular relevance in the case of nonergodic models.

Given a sequence of independent, identically distributed random biased bits, von Neumann's simple procedure extracts independent unbiased bits. In this note we show that the number of unbiased bits produced by iterating this procedure is arbitrarily close to the entropy bound.

A strong adaptive criteria is defined for density estimation problems. In a particular case it is shown that there is no strongly adaptive sequence of estimators. In contrast Woodroofe has shown that a weakly adaptive result holds.

This paper is concerned with the size of confidence intervals for parameters of stochastic processes based on limit laws with two competing normalizations, one producing asymptotic normality and the other asymptotic mixed normality. It is shown that, in a certain sense, the interval based on asymptotic normality is preferable on average. Applications to estimation of parameters in nonergodic stochastic processes and to estimation of steady-state parameters in a simulation are given to illustrate the theory.

A general convergence criterion for certain iterative sequences in Hilbert space is presented. For an important subclass of these sequences, estimates of the rate of convergence are given. Under very mild assumptions these results establish an $O(1/ \sqrt n)$ nonsampling convergence rate for projection pursuit regression and neural network training; where $n$ represents the number of ridge functions, neurons or coefficients in a greedy basis expansion.

In this note we study comparison of experiments via the positive dependence of normal variables with a common univariate marginal distribution. We show that positive dependence has an adverse effect on the information concerning the common mean $\theta$, and give a partial ordering of the information via a majorization ordering of the correlation matrices. In the special case when the random variables are equally correlated, the main theorem yields a result for the comparison of experiments for permutation symmetric normal variables.