We present an approach to the general, non-Markovian dynamic allocation (or multiarmed bandit) problem, formulated in continuous time as a problem of stochastic control for multiparameter processes in the manner of Mandelbaum. This approach is based on a direct, martingale study of auxiliary questions in optimal stopping. Using a methodology similar to that of Whittle and relying on simple time-change arguments, we construct Gittins-index-type strategies, verify their optimality, provide explicit expressions for the values of dynamic allocation and associated optimal stopping problems, explore interesting dualities and derive various characterizations of Gittins indices. This paper extends results of our recent work on discrete-parameter dynamic allocation to the continuous time setup; it can be read independently of that work.

The main problem considered in this paper is the construction of numerical methods and proofs of their convergence for the problem of "shape from shading." In the first part of the paper, it is assumed that the height function that describes the surface to be reconstructed is known at all local minima (or maxima). These points are a subset of the singular points, which are the brightest points in the image. A pair of optimal control problems are defined that provide representations for the height function. Numerical schemes based on these representations are then constructed. While both schemes lead to the same approximation, one yields a more efficient algorithm, while the other is more convenient in the convergence analysis. The proof of convergence is based on a representation of the approximation to the height as a functional of a controlled Markov chain. In a later part of the paper the assumption that the height must be known at all local minima (or maxima) is dropped. An extension of the algorithm is described that is capable of reconstruction without this information. Numerical experiments for both algorithms on synthetic and real data are included.

In this paper we obtain bounds on the spectral gap of the transition probability matrix of Markov chains associated with the Metropolis algorithm and with the Gibbs sampler. In both cases we prove that, for small values of $T,$ the spectral gap is equal to $1 - \lambda_2,$ where $\lambda_2$ is the second largest eigenvalue of $P$. In the case of the Metropolis algorithm we give also two examples in which the spectral gap is equal to $1 - \lambda_{\min}$, where $\lambda_{\min}$ is the smallest eigenvalue of $P$. Furthermore we prove that random updating dynamics on sites based on the Metropolis algorithm and on the Gibbs sampler have the same rate of convergence at low temperatures. The obtained bounds are discussed and compared with those obtained with a different approach.

This paper applies the earlier work of Barlow, Rogers and Williams on the Wiener-Hopf factorization of finite Markov chains to a number of questions in the theory of fluid models of queues. Specifically, the invariant distribution for an infinite-buffer model and for a finite-buffer model are derived. The laws of other functionals of the fluid models can be easily derived and compactly expressed in terms of the fundamental Wiener-Hopf factorization.

Consider a queueing network with customers arriving according to a rate-1 Poisson process. Each customer proceeds along the same prescribed route, waiting at the different queues until exiting from the system. The service times are assumed to be independent and exponentially distributed. Individual queues may be visited more than once by a customer, with the mean service time perhaps depending on the stage along the route. The network is assumed to be first-in, first-out. An obvious necessary condition for such a queueing network to have an equilibrium distribution is that the sum of the mean service times at each queue be less than 1. We show by means of a class of examples that this condition does not suffice, these networks being unstable. Each such network possesses two queues, the first with one slow and one quick stage, and the other with one slow and numerous quick stages.

Three results on hitting a rare set by the increments of an $\mathbb{R}^d$-valued random process with stationary independent increments are presented: the first time that it occurs, the duration of such a segment and the typical trajectory during the segment.

We study a closed, three-station queueing network with general service time distributions and balanced workloads (that is, each station has the same relative traffic intensity). If the customer population is large, then the queue length process of such a network can be approximated by driftless reflected Brownian motion (RBM) in a simplex. Building on earlier work by Harrison, Landau ands Shepp, we develop explicit formulas for various quantities associated with the stationary distribution of RBM in a general triangle and use them to derive approximate performance measures for the closed queueing network. In particular, we develop approximations for the throughput rate and for moments and tail fractiles of the throughput time distribution. Also, crude bounds on the throughput rate and mean throughput time are proposed. Finally, we present three examples that test the accuracy of both the Brownian approximation and our performance estimates.

The asymptotic distribution of the area $V_n$ outside the convex hull of $n$ i.i.d. points uniformly distributed on the two-dimensional unit disk is studied. The asymptotic variance of $V_n$ is found to be of the order $n^{-5/3}$, and the asymptotic distribution of $V_n$ is shown to be normal. The results are obtained by carefully analyzing the strength of dependence between sample points at different locations close to the boundary of the unit disk.

A model for the activities of a finite number of agents in an economy is presented as the solution to a system of stochastic differential equations driven by general semimartingales and displaying an extended form of strong local interaction. We demonstrate a law of large numbers for the systems of processes as the number of agents goes to infinity under a weak convergence hypothesis on the triangular array of starting values and driving semimartingales which induces the systems of equations. Further, it is shown that the limit can be uniquely characterized by the distributions of the coordinate processes of the solution to an associated infinite-dimensional stochastic differential equation. Finally, an explicit example describing a currency market is discussed.

In this paper the extremal behaviour of real-valued, stationary Markov chains is studied under fairly general assumptions. Conditions are obtained for convergence in distribution of multilevel exceedance point processes associated with suitable families of high levels. Although applicable to general stationary sequences, these conditions are tailored for Markov chains and are seen to hold for a large class of chains. The extra assumptions used are that the marginal distributions belong to the domain of attraction of some extreme value law together with rather weak conditions on the transition probabilities. Also, a complete convergence result is given. The results are applied to an AR(1) process with uniform margins and to solutions of a first order stochastic difference equation with random coefficients.

The asymptotic behavior of the solutions of linear equations with random coefficients, random external forces and with affine boundary conditions is studied, motivated by a transmission-reflection problem for a one-dimensional wave equation in a random slab. The fluctuations of the coefficients are on a small scale in such a way that our problem is a diffusion-approximation problem except that we impose boundary conditions which force the solution to be anticipating. In the limit we obtain linear stochastic differential equations with affine boundary conditions, studied by Ocone and Pardoux. Our main tools are diffusion approximation results (Papanicolaou, Stroock and Varadhan or Ethier and Kurtz) and the properties of the limiting equations involving generalized Stratonovich integrals (Ocone and Pardoux). As an application, the transmission-reflection problem is discussed. We prove that the solution has a density with respect to the Lebesgue measure and satisfies the Markov field property.

Let $T = \inf\{n \geq 1: S_n > 0\}$ and $H = S_T$ be ladder variables for a random walk $\{S_n\}_{n \geq 1}$ with nonnegative drift. Integral formulas for generating functions and moments of $T, H$ and related quantities are developed. These formulas are suitable for numerical quadrature and should be easier to implement than formulas based on Spitzer's identity when the distribution of $S_n$ is complicated. The approach used makes key use of the Hilbert transform and the main regularity assumption is that some power of the characteristic function for steps of the random walk is integrable.

Assume that $k$ jobs circulate clockwise through a cyclic network of $n$ single-server queues, where at each integer time instant the job at the head of each queue moves with probability $p$ to the next queue, independent of the other jobs. The equilibrium distribution for the associated Markov chain is determined, and an exact expression for the expected number of busy servers is obtained. If $n$ and $k$ are large, a simple approximation for the proportion of busy servers is derived. In a second model, where the queues have no waiting room and where movement of a job occurs with probability $p$ only if the next queue was empty, a similar, simple asymptotic representation for the proportion of busy servers is deduced. This representation readily yields a simple expression for the asymptotic cycle time for a single job.