Translator Disclaimer
February 2018 Ergodic theory for controlled Markov chains with stationary inputs
Yue Chen, Ana Bušić, Sean Meyn
Ann. Appl. Probab. 28(1): 79-111 (February 2018). DOI: 10.1214/17-AAP1300

## Abstract

Consider a stochastic process $\boldsymbol{X}$ on a finite state space $\mathsf{X}=\{1,\dots,d\}$. It is conditionally Markov, given a real-valued “input process” $\boldsymbol{\zeta}$. This is assumed to be small, which is modeled through the scaling, $\zeta_{t}=\varepsilon\zeta^{1}_{t},\qquad0\le\varepsilon\le1,$ where $\boldsymbol{\zeta}^{1}$ is a bounded stationary process. The following conclusions are obtained, subject to smoothness assumptions on the controlled transition matrix and a mixing condition on $\boldsymbol{\zeta}$:

(i) A stationary version of the process is constructed, that is coupled with a stationary version of the Markov chain $\boldsymbol{X}^{\bullet}$ obtained with $\boldsymbol{\zeta} \equiv0$. The triple $(\boldsymbol{X} ,\boldsymbol{X}^{\bullet},\boldsymbol{\zeta} )$ is a jointly stationary process satisfying

$\mathsf{P}\{X(t)\neq X^{\bullet}(t)\}=O(\varepsilon).$ Moreover, a second-order Taylor-series approximation is obtained:

$\mathsf{P}\{X(t)=i\}=\mathsf{P}\{X^{\bullet}(t)=i\}+\varepsilon^{2}\pi^{(2)}(i)+o(\varepsilon^{2}),\qquad1\le i\le d,$ with an explicit formula for the vector $\pi^{(2)}\in\mathbb{R}^{d}$.

(ii) For any $m\ge1$ and any function $f:\{1,\dots,d\}\times\mathbb{R}\to\mathbb{R}^{m}$, the stationary stochastic process $Y(t)=f(X(t),\zeta(t))$ has a power spectral density $\mathrm{S}_{f}$ that admits a second-order Taylor series expansion: A function $\mathrm{S}^{(2)}_{f}:[-\pi,\pi]\to\mathbb{C}^{m\times m}$ is constructed such that

$\mathrm{S}_{f}(\theta)=\mathrm{S}^{\bullet}_{f}(\theta)+\varepsilon^{2}\mathrm{S}^{(2)}_{f}(\theta)+o(\varepsilon^{2}),\qquad\theta\in[-\pi,\pi ]$ in which the first term is the power spectral density obtained with $\varepsilon=0$. An explicit formula for the function $\mathrm{S}^{(2)}_{f}$ is obtained, based in part on the bounds in (i).

The results are illustrated with two general examples: mean field games, and a version of the timing channel of Anantharam and Verdu.

## Citation

Yue Chen. Ana Bušić. Sean Meyn. "Ergodic theory for controlled Markov chains with stationary inputs." Ann. Appl. Probab. 28 (1) 79 - 111, February 2018. https://doi.org/10.1214/17-AAP1300

## Information

Received: 1 June 2016; Revised: 1 April 2017; Published: February 2018
First available in Project Euclid: 3 March 2018

zbMATH: 06873680
MathSciNet: MR3770873
Digital Object Identifier: 10.1214/17-AAP1300

Subjects:
Primary: 60J20
Secondary: 60G10 , 68M20 , 94A15

Keywords: Controlled Markov chain , ergodic theory , information theory  