Open Access
February 2018 Ergodic theory for controlled Markov chains with stationary inputs
Yue Chen, Ana Bušić, Sean Meyn
Ann. Appl. Probab. 28(1): 79-111 (February 2018). DOI: 10.1214/17-AAP1300


Consider a stochastic process $\boldsymbol{X}$ on a finite state space $\mathsf{X}=\{1,\dots,d\}$. It is conditionally Markov, given a real-valued “input process” $\boldsymbol{\zeta}$. This is assumed to be small, which is modeled through the scaling, \[\zeta_{t}=\varepsilon\zeta^{1}_{t},\qquad0\le\varepsilon\le1,\] where $\boldsymbol{\zeta}^{1}$ is a bounded stationary process. The following conclusions are obtained, subject to smoothness assumptions on the controlled transition matrix and a mixing condition on $\boldsymbol{\zeta}$:

(i) A stationary version of the process is constructed, that is coupled with a stationary version of the Markov chain $\boldsymbol{X}^{\bullet}$ obtained with $\boldsymbol{\zeta} \equiv0$. The triple $(\boldsymbol{X} ,\boldsymbol{X}^{\bullet},\boldsymbol{\zeta} )$ is a jointly stationary process satisfying

\[\mathsf{P}\{X(t)\neq X^{\bullet}(t)\}=O(\varepsilon).\] Moreover, a second-order Taylor-series approximation is obtained:

\[\mathsf{P}\{X(t)=i\}=\mathsf{P}\{X^{\bullet}(t)=i\}+\varepsilon^{2}\pi^{(2)}(i)+o(\varepsilon^{2}),\qquad1\le i\le d,\] with an explicit formula for the vector $\pi^{(2)}\in\mathbb{R}^{d}$.

(ii) For any $m\ge1$ and any function $f:\{1,\dots,d\}\times\mathbb{R}\to\mathbb{R}^{m}$, the stationary stochastic process $Y(t)=f(X(t),\zeta(t))$ has a power spectral density $\mathrm{S}_{f}$ that admits a second-order Taylor series expansion: A function $\mathrm{S}^{(2)}_{f}:[-\pi,\pi]\to\mathbb{C}^{m\times m}$ is constructed such that

\[\mathrm{S}_{f}(\theta)=\mathrm{S}^{\bullet}_{f}(\theta)+\varepsilon^{2}\mathrm{S}^{(2)}_{f}(\theta)+o(\varepsilon^{2}),\qquad\theta\in[-\pi,\pi ]\] in which the first term is the power spectral density obtained with $\varepsilon=0$. An explicit formula for the function $\mathrm{S}^{(2)}_{f}$ is obtained, based in part on the bounds in (i).

The results are illustrated with two general examples: mean field games, and a version of the timing channel of Anantharam and Verdu.


Download Citation

Yue Chen. Ana Bušić. Sean Meyn. "Ergodic theory for controlled Markov chains with stationary inputs." Ann. Appl. Probab. 28 (1) 79 - 111, February 2018.


Received: 1 June 2016; Revised: 1 April 2017; Published: February 2018
First available in Project Euclid: 3 March 2018

zbMATH: 06873680
MathSciNet: MR3770873
Digital Object Identifier: 10.1214/17-AAP1300

Primary: 60J20
Secondary: 60G10 , 68M20 , 94A15

Keywords: Controlled Markov chain , ergodic theory , information theory

Rights: Copyright © 2018 Institute of Mathematical Statistics

Vol.28 • No. 1 • February 2018
Back to Top