August 2023 Off-policy evaluation in partially observed Markov decision processes under sequential ignorability
Yuchen Hu, Stefan Wager
Author Affiliations +
Ann. Statist. 51(4): 1561-1585 (August 2023). DOI: 10.1214/23-AOS2287


We consider off-policy evaluation of dynamic treatment rules under sequential ignorability, given an assumption that the underlying system can be modeled as a partially observed Markov decision process (POMDP). We propose an estimator, partial history importance weighting, and show that it can consistently estimate the stationary mean rewards of a target policy, given long enough draws from the behavior policy. We provide an upper bound on its error that decays polynomially in the number of observations (i.e., the number of trajectories times their length) with an exponent that depends on the overlap of the target and behavior policies as well as the mixing time of the underlying system. Furthermore, we show that this rate of convergence is minimax, given only our assumptions on mixing and overlap. Our results establish that off-policy evaluation in POMDPs is strictly harder than off-policy evaluation in (fully observed) Markov decision processes but strictly easier than model-free off-policy evaluation.


We are grateful for helpful comments and suggestions from Emma Brunskill, Ramesh Johari, Nathan Kallus, Michael Kosorok, Johan Ugander, and seminar participants at a number of venues. We would also like to thank the Editors and the reviewers for their valuable feedback and constructive comments that helped improve the quality of this manuscript.


Download Citation

Yuchen Hu. Stefan Wager. "Off-policy evaluation in partially observed Markov decision processes under sequential ignorability." Ann. Statist. 51 (4) 1561 - 1585, August 2023.


Received: 1 April 2022; Revised: 1 January 2023; Published: August 2023
First available in Project Euclid: 19 October 2023

Digital Object Identifier: 10.1214/23-AOS2287

Primary: 62D20 , 62M09

Keywords: Causal inference , Importance weighting , Lepski’s method , mixing time , sequential ignorability

Rights: Copyright © 2023 Institute of Mathematical Statistics


This article is only available to subscribers.
It is not available for individual sale.

Vol.51 • No. 4 • August 2023
Back to Top