Extension of Wald's first lemma to Markov processes

Abstract

Let ξ₀,ξ₁,ξ₂,... be a homogeneous Markov process and let S_n denote the partial sum S_n = θ(ξ₁) + ... + θ(ξ_n), where θ(ξ) is a scalar nonlinearity. If N is a stopping time with EN < ∞ and the Markov process {ξ_n}^∞_n=0 satisfies certain ergodicity properties, we then show that ES_N = [lim_n→∞Eθ(ξ_n)]EN + Eω(ξ₀) - Eω(ξ_N). The function ω(ξ) is a well defined scalar nonlinearity directly related to θ(ξ) through a Poisson integral equation, with the characteristic that ω(ξ) becomes zero in the i.i.d. case. Consequently our result constitutes an extension to Wald's first lemma for the case of Markov processes. We also show that, when EN → ∞, the correction term is negligible as compared to EN in the sense that Eω(ξ₀) - Eω(ξ_N) = o(EN).