In this paper we consider approximate dynamic programming methods for ambulance redeployment. We first demonstrate through simple examples how typical value function fitting techniques, such as approximate policy iteration and linear programming, may not be able to locate a high-quality policy even when the value function approximation architecture is rich enough to provide the optimal policy. To make up for this potential shortcoming, we show how to use direct search methods to tune the parameters in a value function approximation architecture so as to obtain high-quality policies. Direct search is computationally intensive. We therefore use a post-decision state dynamic programming formulation of ambulance redeployment that, together with direct search, requires far less computation with no noticeable performance loss. We provide further theoretical support for the post-decision state formulation of the ambulance-deployment problem by showing that this formulation can be obtained through a limiting argument on the original dynamic programming formulation.

We consider the structure learning problem for graphical models that we call loosely connected Markov random fields, in which the number of short paths between any pair of nodes is small, and present a new conditional independence test based algorithm for learning the underlying graph structure. The novel maximization step in our algorithm ensures that the true edges are detected correctly even when there are short cycles in the graph. The number of samples required by our algorithm is $C\log p$, where $p$ is the size of the graph and the constant $C$ depends on the parameters of the model. We show that several previously studied models are examples of loosely connected Markov random fields, and our algorithm achieves the same or lower computational complexity than the previously designed algorithms for individual cases. We also get new results for more general graphical models, in particular, our algorithm learns general Ising models on the Erdős-Rényi random graph $\mathcal{G}(p,\frac{c}{p})$ correctly with running time $O(np^{5})$.

A supermarket game is considered with $N$ FCFS queues with unit exponential service rate and global Poisson arrival rate $N\lambda$. Upon arrival each customer chooses a number of queues to be sampled uniformly at random and joins the least loaded sampled queue. Customers are assumed to have cost for both waiting and sampling, and they want to minimize their own expected total cost.

We study the supermarket game in a mean field model that corresponds to the limit as $N$ converges to infinity in the sense that (i) for a fixed symmetric customer strategy, the joint equilibrium distribution of any fixed number of queues converges as $N\to \infty$ to a product distribution determined by the mean field model and (ii) a Nash equilibrium for the mean field model is an $\epsilon$-Nash equilibrium for the finite $N$ model with $N$ sufficiently large. It is shown that there always exists a Nash equilibrium for $\lambda <1$ and the Nash equilibrium is unique with homogeneous waiting cost for $\lambda \le1/\sqrt{2}$. Furthermore, we find that the action of sampling more queues by some customers has a positive externality on the other customers in the mean field model, but can have a negative externality for finite $N$.

We consider an inventory system in which inventory level fluctuates as a Brownian motion in the absence of control. The inventory continuously accumulates cost at a rate that is a general convex function of the inventory level, which can be negative when there is a backlog. At any time, the inventory level can be adjusted by a positive or negative amount, which incurs a fixed cost and a proportional cost. The challenge is to find an adjustment policy that balances the holding cost and adjustment cost to minimize the long-run average cost. When both upward and downward fixed costs are positive, our problem is an impulse control problem. When both fixed costs are zero, our problem is a singular or instantaneous control problem. For the impulse control problem, we prove that a four-parameter control band policy is optimal among all feasible policies. For the singular control problem, we prove that a two-parameter control band policy is optimal.

We use a lower-bound approach, widely known as the “verification theorem”, to prove the optimality of a control band policy for both the impulse and singular control problems. Our major contribution is to prove the existence of a “smooth” solution to the free boundary problem under some mild assumptions on the holding cost function. The existence proof leads naturally to a numerical algorithm to compute the optimal control band parameters. We demonstrate that the lower-bound approach also works for a Brownian inventory model which prohibits an inventory backlog. A companion paper (“Brownian inventory models with convex holding cost, part 2: discount-optimal controls”, Stochastic Systems, Vol. 3, No. 2, pp. 500–573) explains how to adapt the lower-bound approach to study a Brownian inventory model under a discounted cost criterion.

We consider an inventory system in which inventory level fluctuates as a Brownian motion in the absence of control. The inventory continuously accumulates cost at a rate that is a general convex function of the inventory level, which can be negative when there is a backlog. At any time, the inventory level can be adjusted by a positive or negative amount, which incurs a fixed positive cost and a proportional cost. The challenge is to find an adjustment policy that balances the inventory cost and adjustment cost to minimize the expected total discounted cost. We provide a tutorial on using a three-step lower-bound approach to solving the optimal control problem under a discounted cost criterion. In addition, we prove that a four-parameter control band policy is optimal among all feasible policies. A key step is the constructive proof of the existence of a unique solution to the free boundary problem. The proof leads naturally to an algorithm to compute the four parameters of the optimal control band policy.

We consider a pair of coupled queues driven by independent spectrally-positive Lévy processes. With respect to the bi-variate workload process this framework includes both the coupled processor model and the two-server fluid network with independent Lévy inputs. We identify the joint transform of the stationary workload distribution in terms of Wiener-Hopf factors corresponding to two auxiliary Lévy processes with explicit Laplace exponents. We reinterpret and extend the ideas of Cohen and Boxma (1983) to provide a general and uniform result with a neat transform expression.

In this paper we consider a compound Poisson risk model with regularly varying claim sizes. For this model in [4] an asymptotic formula for the finite time ruin probability is provided when the time is scaled by the mean excess function. In this paper we derive the rate of convergence for this finite time ruin probability when the claims are regularly varying with a finite second moment.