On Transient Markov Processes with a Countable Number of States and Stationary Transition Probabilities

Abstract

We consider a Markov process $x_0, x_1, \cdots$ with a countable set $S$ of states and stationary transition probabilities $p(t \mid s) = P\{x_{n+1} = t \mid x_n = s\}$. Call a set $C$ of states almost closed if (a) $P\{x_n \varepsilon C$ for an infinite number of $n\} > 0$ and (b) $x_n \varepsilon C$ infinitely often implies $x_n \varepsilon C$ for all sufficiently large $n$, with probability one. It is shown that there is a set $(C_1, C_2, \cdots)$ essentially unique, of disjoint almost closed sets such that (a) all except at most one of the $C_i$ are atomic, that is, $C_i$ does not contain two disjoint almost closed subsets, (b) the non-atomic $C_i$, if present, contains no atomic subsets, (c) the process is certain to enter and remain in some set $C_i$. A relation between the sets $C_i$ and the bounded solutions of the system of equations \begin{equation*}\tag{1} \alpha(s) = \sum_t \alpha(t)p(t \mid s)\end{equation*} is obtained; in particular there is only one atomic $C_i$ and no non-atomic $C_i$ if and only if the only bounded solution of (1) is $\alpha(t)$ = constant. This condition is shown to hold if the process is the sum of independent identical (numerical or vector) variables; whence, for such a process, the probability of entering a set $J$ infinitely often is zero or one. The results are new only if the process has transient components. The main tool is the martingale convergence theorem.