A dynamic analytic method for risk-aware controlled martingale problems

Abstract

We present a new, tractable method for solving risk-aware problems over finite and infinite, discounted time-horizons where the dynamics of the controlled process are described using the martingale method. Supposing general Polish state and action spaces, and using the martingale characterization, we state a risk-aware dynamic optimal control problem of minimizing risk of costs described by a generic risk function. From this, we construct an alternative formulation of the optimization problem that takes the form of a nonlinear programming problem, constrained by the dynamic, that is, time-dependent and linear Kolmogorov forward equation describing the time-dependent distribution of the state and running costs. This formulation is similar to the convex analytic method, in that the control problem is recast into a form where the objective is optimized over distributions representing the state space visitation frequencies. However, in our approach, the distributions are dynamic and also encode the cost distribution. As our main results, we prove the equivalence of the original martingale and dynamic analytic problems, in the sense that both have the same optimal values, and that the solution of either problem yields a solution of the other. Moreover, we find an optimal control process can be taken to be Markov in the controlled process state, running costs, and time. We further show that under additional assumptions the optimal value is attained. An example numeric problem is presented and solved.