Toward theoretical understandings of robust Markov decision processes: Sample complexity and asymptotics

Abstract

In this paper, we study the nonasymptotic and asymptotic performances of the optimal robust policy and value function of robust Markov Decision Processes (MDPs), where the optimal robust policy and value function are estimated from a generative model. While prior work focusing on nonasymptotic performances of robust MDPs is restricted in the setting of the KL uncertainty set and $(\mathit{s},\mathit{a})$-rectangular assumption, we improve their results and also consider other uncertainty sets, including the ${\mathit{L}}_1$ and ${\mathit{\chi }}^2$ balls. Our results show that when we assume $(\mathit{s},\mathit{a})$-rectangular on uncertainty sets, the sample complexity is about $\tilde{\mathcal{O}}(\frac{|\mathcal{S}{|}^2|\mathcal{A}|}{{\mathit{\epsilon }}^2{\mathit{\rho }}^2{(1-\mathit{\gamma })}^4})$. In addition, we extend our results from the $(\mathit{s},\mathit{a})$-rectangular assumption to the s-rectangular assumption. In this scenario, the sample complexity varies with the choice of uncertainty sets and is generally larger than the case under the $(\mathit{s},\mathit{a})$-rectangular assumption. Moreover, we also show that the optimal robust value function is asymptotically normal with a typical rate $\sqrt{\mathit{n}}$ under the $(\mathit{s},\mathit{a})$ and s-rectangular assumptions from both theoretical and empirical perspectives.