For controlled Markov processes taking values in a Polish space, control problems with ergodic cost, infinite-horizon discounted cost and finite-horizon cost are studied. Each is posed as a convex optimization problem wherein one tries to minimize a linear functional on a closed convex set of appropriately defined occupation measures for the problem. These are characterized as solutions of a linear equation asssociated with the problem. This characterization is used to establish the existence of optimal Markov controls. The dual convex optimization problem is also studied.
"Occupation measures for controlled Markov processes: characterization and optimality." Ann. Probab. 24 (3) 1531 - 1562, July 1996. https://doi.org/10.1214/aop/1065725192