Risk-sensitive optimal stopping with unbounded terminal cost function

In this paper we consider an infinite time horizon risk-sensitive optimal stopping problem for a Feller--Markov process with an unbounded terminal cost function. We show that in the unbounded case an associated Bellman equation may have multiple solutions and we give a probabilistic interpretation for the minimal and the maximal one. Also, we show how to approximate them using finite time horizon problems. The analysis, covering both discrete and continuous time case, is supported with illustrative examples.


Introduction
Many practical optimal control problems could be expressed in terms of optimal stopping. This includes examples in mathematical finance (American options theory, optimal asset liquidation), statistics (sequential testing), operations research, ecology; see e.g. Shiryaev (1978); Bensoussan and Lions (1984); Carmona and Touzi (2008); Bäuerle and Rieder (2011) for details.
Typically, a characterisation of the optimal stopping time is obtained through the study of the corresponding Snell envelope of the value process; see e.g. El Karoui (1981) for details and Kobylanski and Quenez (2012) for more recent contribution. Also, in the Markovian case this could be done with the help of a specific optimality Wald-Bellman equation; see e.g. Shiryaev (1978) for a classical contribution. The existence of a solution to this equation could be obtained e.g. by value iteration argument or penalty approach, see Stettner (2011). Also, it may result from the use of viscosity techniques applied to variational inequalities; see e.g. Bensoussan and Lions (1984) and Dai and Menoukeu-Pamen (2018).
Risk-sensitive problems constitute a special class of general stochastic control problems (in particular, optimal stopping problems). In this case, a decisionmaker tries to optimise the certainty equivalent of the exponential utility function; see Howard and Matheson (1972) and Whittle (1990). This criterion may be seen as a non-linear extension of the mean-variance (Markowitz) approach which facilitates more robust control strategies; see e.g. Bielecki and Pliska (2003) for a comprehensive overview. However, using risk-sensitive criterion results in multiplicative control problems that are usually more difficult to solve than their classic risk-neutral (additive) counterparts; see Nagai (2007) and Bäuerle and Popp (2018).
In this paper we consider the infinite time horizon risk-sensitive optimal stopping problems u(x) := inf τ ln E x exp τ 0 g(X s )ds + G(X τ ) , x ∈ E; (1.1) where X is a standard Feller-Markov process starting at x from the state space E, while g and G are continuous and non-negative running cost function and terminal cost function, respectively. The function g is assumed to be bounded while G may be unbounded from above. The map u describes the value of a standard risk-sensitive optimal stopping problem. As we show in this paper, the map w emerges naturally as a limit of finite horizon stopping problems. Also, the map w may be seen as a version of u, when a decision-maker is allowed to choose only bounded stopping times. Arguably, the main contribution of this paper is the proof that both functions u and w are solutions to the associated optimal stopping Bellman equation. In fact, we show that u and w are minimal and maximal solutions to this equation, respectively, and in general we do not have an equality between u and w.
This paper extends the results from Jelito et al. (2021), where the function G is assumed to be bounded. In that case, it can be shown that the Bellman equation admits a unique solution, which can be used to prove continuity of the function u ≡ w. This result was one of the main building blocks used in Jelito et al. (2020), where the long-run impulse control problem was analysed. In the present paper we show a more general sufficient condition for the identity u ≡ w. This may be used to generalise the results from Jelito et al. (2020) to the unbounded case.
In the literature, regularity properties of the optimal stopping value function were mostly studied in the context of risk-neutral (additive) stopping problems; see e.g. Bassan and Ceci (2002). In particular, this applies to non-uniqueness of a solution to the Bellman equation; see Section 2.11 in Shiryaev (1978) and Theorem 1.13 in Peskir and Shiryaev (2006) for classic contributions. However, the risk-sensitive case is mostly unexplored; see Nagai (2007) and Jelito et al. (2021). Also, it should be noted that many approximative solutions to optimal stopping problems are based on numerical solutions to the Bellman equation; see e.g. Kushner and Dupuis (2013) for a comprehensive overview. Thus, the study on regularity properties of optimality equation is important both from theoretical and practical point of view.
The structure of this paper is as follows. In Section 2 we introduce notation and assumptions used throughout this paper. Next, in Section 3 we study discrete time version of the problem. The main contribution of this part is Theorem 3.3, where we link the discrete time Bellman equation with the limits of suitable finite horizon stopping value functions. In Section 4 we study a continuous time version of the problem. This is used in Section 5, where we give a characterisation of solutions to the continuous time Bellman equation; see Theorem 5.2 for details. Also, in Theorem 5.9 we show a condition for the uniqueness of a solution to the Bellman equation. Our results are illustrated by the examples presented in Section 6. In particular, in Example 6.4 we show explicit formulae for distinct solutions to the Bellman equation. Finally, in Appendix A we present some deferred proofs.

Preliminaries
Let X = (X t ) t≥0 be a time-homogeneous continuous time standard Markov process on a filtered measurable space (Ω, F , (F t )) with values in a locally compact separable metric space E. With any x ∈ E we associate a probability measure P x describing the dynamics of the process starting from X 0 = x; see Definition 4 in (Shiryaev, 1978, Section 1.4) for details. We assume that X satisfies the C 0 -Feller property, i.e.
where P t is the corresponding transition semigroup and C 0 (E) denotes the family of real-valued continuous functions defined on E, vanishing at infinity. This is a standard assumption in the stochastic control theory. In particular, it is satisfied by Lévy processes and solutions to stochastic differential equations driven by Lévy processes; see Theorem 3.1.9 and Theorem 6.7.2 in Applebaum (2009) for details. In addition to the C 0 -Feller property of the Markov process, we assume several properties of the cost functions. To ease the notation, for any T ≥ 0, let us define ζ T := sup t∈[0,T ] e G(Xt) . Throughout this paper we make the following Assumptions: (A1) (Cost functions constraints). The map G : E → [0, ∞) is continuous and the map g : E → [0, ∞) is continuous and bounded. Also, the map g is bounded away from zero, i.e. for some c > 0 we get g(·) ≥ c > 0. (A2) (Integrability). For any T ≥ 0 and x ∈ E we get (A3) (Continuity). For any T ≥ 0 and a continuous function h satisfying 0 ≤ h(·) ≤ G(·), we get that the map is continuous.
Let us now comment on these conditions. Assumption (A2) requires several regularity properties for the cost functions. First, it should be highlighted that while g is assumed to be bounded, we allow G to be unbounded from above. Also, note that the non-negativity assumption for G is merely a technical normalisation. Indeed, for a generic continuous mapG : E → R which is bounded from below, we may subtract the quantity inf y∈EG (y) from the both sides of (1.1) and (1.2) and set G(·) :=G(·) − inf y∈EG (y). Finally, note that the assumption g(·) ≥ c > 0 could be used to show that stopping at infinity cannot be optimal for our problems as this leads to infinite cost.
Assumption (A2) requires integrability for the finite time horizon and is a standard condition in the optimal stopping literature.
Assumption (A3) requires continuity of the specific semigroup for unbounded functions h. Note that from the Feller property and monotone convergence theorem we get that x → E x exp T 0 f (X s )ds + h(X T ) is lower semicontinuous for any T ≥ 0 and a continuous function h : E → [0, ∞). Thus, in assumption (A3) we additionally require upper semicontinuity.
Further comments on Assumptions (A2) and (A3) could be found in Section 6.1. More specifically, we show that Assumptions (A2) and (A3) could be deduced from a more general condition related to the integrability of the tail of ζ T , T ≥ 0; see (B1) and the following discussion for details. Now, let us comment on the specific forms of (1.1) and (1.2). Setting from quasi-left continuity of Z and Fatou Lemma, for any x ∈ E and P x -almost surely finite stopping time τ , we get (2.1) Some of the results in this paper are related to the situation when there is an equality in (2.1). Let us now provide a useful characterisation of this property.
Lemma 2.1. Let x ∈ E and let τ be a stopping time satisfying Then, the following are equivalent (1) We get Proof. Note that the equivalence of (1) and (2) follows from the standard result; see e.g. Theorem 16.14 in Billingsley (1995) for details. Thus, it is enough to show that (1) is equivalent to (3). Using the identity and noting that T → 1 {τ ≤T } Z τ is increasing, by monotone convergence theorem and quasi-left continuity of (Z t ) we get Thus, letting T → ∞ in (2.2), we conclude the proof. Now, observe that from (2.1), for any x ∈ E, we get where u and w are given by (1.1) and (1.2), respectively. In the following lemma we show that w may be seen as a value of the optimal stopping problem with infimum over the family of bounded stopping times. This provides an additional explanation for (2.3).
Lemma 2.2. Let w be given by (1.2) and let T b denote the family of bounded stopping times. Then, we get Proof. First, note that using boundedness of τ ∈ T b , we get Second, let x ∈ E, ε > 0, and τ ε be an ε-optimal stopping time for w(x). Then, there exists a sequence (T n ) ⊂ R + such that T n → ∞ as n → ∞ and , which concludes the proof.

Discrete time optimal stopping
In this section we consider a discrete-time version of the problems (1.1) and (1.2). By X we denote a standard discrete-time Markov process with values in E and for simplicity we write X = (X n ) n∈N , where N := {0, 1, 2, . . .} denotes the set of nonnegative integers. It should be noted that the results in this section do not require continuity assumptions from Section 2.
By analogy to (1.1) and (1.2), we define where T 0 denotes the family of stopping times with values in N and we follow the convention −1 i=0 (·) = 0. Also, let us define the Bellman operator where h : E → R + is a non-negative measurable function. In this section we characterise solutions to the Bellman equation, i.e. measurable functions v : (3.3) More explicitly, in Theorem 3.3 we show that u and w are minimal and maximal solutions to (3.3), respectively.
We start with finding the minimal and maximal solutions to (3.3). Recalling non-negativity of the functions g and G and (2.3), we get Based on these inequalities, to get the extremal solutions to (3.3) we iterate the lower and upper bounds for u and w. Thus, we define recursively the families of functions (3.5) In the following proposition we show the probabilistic characterisation of the sequences (w n ) and (w n ). The proof is similar to the proof of Proposition 3 from Jelito et al. (2021), where G is assumed to be bounded from above, and therefore is omitted for brevity.
Proposition 3.1. Let the sequences of functions (w n ) and (w n ) be given by (3.4) and (3.5), respectively. Then, (1) For any x ∈ E, the sequence (w n (x)) is non-decreasing. Moreover, we get and the optimal stopping time for w n is given by (3.6) (2) For any x ∈ E, the sequence (w n (x)) is non-increasing. Moreover we get and the optimal stopping time for w n is given by (3.8) Using monotone convergence theorem we get that both w and w satisfy the Bellman equation (3.3). Also, for any measurable function v solving (3.3) and satisfying 0 ≤ v(x) ≤ G(x), we iteratively get w n (x) ≤ v(x) ≤ w n (x), x ∈ E, and consequently Thus, the maps w and w are minimal and maximal solutions to the Bellman equation (3.3), respectively. For bounded G one may show that w ≡ w; see Proposition 5 and Corollary 6 in Jelito et al. (2021) for details. However, for unbounded G this may no longer be true; see Example 6.4. Thus, it is interesting to characterise the structure of solutions to (3.3). We start with the following lemma giving a martingale characterisation of solutions to the Bellman equation.
Lemma 3.2. Let v be a non-negative measurable solution to (3.3) and let τ v := inf{n ∈ N : v(X n ) ≥ G(X n )}. Define the process (3.10) Then, for any stopping time τ we get that (z v (τ ∧ n)), n ∈ N, is a submartingale. Also, (z v (τ v ∧ n)), n ∈ N, is a martingale.
Proof. First, using the inequality e g(x) E x e v(X1) ≥ e v(x) , x ∈ E, and Markov property, for any x ∈ E and n ∈ N, we get and the process (z v (n)), n ∈ N, is a submartingale. Thus, using Doob optional stopping theorem, we get that for any stopping time τ the process (z v (τ ∧ n)), n ∈ N, is also a submartingale. Second, note that on the set {τ v > n}, we get e v(Xn) = e g(Xn) E Xn e v(X1) . Thus, for any x ∈ E and n ∈ N, we get  we get u(·) ≤ v(·) ≤ w(·).
Proof. Recalling (3.9) and the successive discussion we get that (2) and (3) follow directly from (1). Thus, it is enough to show (1). For transparency, we split the rest of the proof into two parts: (1) proof of u ≡ w; (2) proof of w ≡ w.
Part 1. We show that u ≡ w. Recalling w n from (3.4) and Proposition 3.1, for any n ∈ N and x ∈ E, we get where the inequality follows from non-negativity of g and G. Letting n → ∞ we get w ≤ u. Now, let us define z(n) := exp n−1 i=0 g(X i ) + w(X n ) , n ∈ N; (3.11) τ := inf{n ∈ N : w(X n ) ≥ G(X n )}; (3.12) and note that by Lemma 3.2 the process (z(τ ∧ n)), n ∈ N, is a martingale. Also, recalling that g(·) ≥ c > 0 and w(·) ≥ 0, and using Fatou Lemma, for any x ∈ E, we get In particular, we get P x [τ < ∞] = 1. Thus, noting that w(X τ ) = G(X τ ), we get hence u ≡ w, which concludes the proof of this part.
Part 2. We show that w ≡ w. Recalling Proposition 3.1 and the maps w k from (3.5), for any k ∈ N and x ∈ E, we get Thus, letting k → ∞, we get w ≤ w. Also, for any n ∈ N andτ ∈ T 0 we get Thus, letting n → ∞ and taking infimum overτ ∈ T 0 , we get w ≤ w, which concludes the proof.
Remark 3.4. From Theorem 3.3 we deduce that in the unbounded case the family of finite time horizon stopping problems may not converge to their infinite horizon version. More specifically, from Proposition 3.1 we get that the function w n may be seen as a finite horizon counterpart of u, with stopping times bounded by n ∈ N. Thus, one might conjecture that w n converges to u as n → ∞. However, from Theorem 3.3 we get w n → w as n → ∞ and from Examples 6.3 and 6.4 we see that in general u = w. Also, note that Theorem 3.3 provides a finite horizon approximation scheme for u; this can be done with the help of the family w n .
From the proof of Theorem 3.3 we get a useful corollary about the optimal stopping time for u.
Corollary 3.5. Let u be given by (3.1). Then, the stopping time is optimal for u. Also, the process (z(n ∧ τ )), n ∈ N, with z given by (3.11), is a uniformly integrable martingale.
Proof. Optimality of τ follows directly from (3.13). Also, martingale property of (z(n ∧ τ )), n ∈ N, follows from Lemma 3.2. Finally, uniform integrability follows from (3.13). Now we formulate a sufficient condition for the identity u ≡ w. To ease the notation, we define the process Theorem 3.6. Let u and w be given by (3.1) and (3.2), respectively. Also, let Proof. Recall that by Corollary 3.5 the stopping time τ is optimal for u. Thus, using uniform integrability of (Z n∧τ ), n ≥ 0, for any x ∈ E, we get Recalling that we always get u ≤ w, we conclude the proof.
Remark 3.7. By analogy to (3.14), let us define τ := inf{t ≥ 0 : w(X t ) ≥ G(X t )}. Since w ≤ w, we get τ ≤ τ , where is given by (3.14). Based on the condition from Theorem 3.6 it is natural to ask whether uniform integrability of (Z τ ∧n ) is also sufficient for u ≡ w. However, as discussed in Remark 6.5, this is not the case.

Continuous time optimal stopping
In this section, by analogy to (3.1) and (3.2), we consider the continuous time optimal stopping problems (4.1) Assuming (A1)-(A3), we prove several regularity properties of the maps u and w. Also, we show various approximation results, including finite time horizon limits. These results extend the analysis from Jelito et al. (2021) to the case when G is unbounded from above. First, by analogy to Proposition 3.1 we consider the finite time horizon optimal stopping problems. For any T ≥ 0, let us define In Proposition 4.1 we summarise the properties of the maps (T, Proposition 4.1. Let the maps (w T ) and (w T ) be given by (4.3) and (4.4), respectively. Then, is jointly continuous and, for any x ∈ E, the map T → w T (x) is non-decreasing. Also, for any T ≥ 0, an optimal stopping time for w T is given by Moreover, the process Also, for any T ≥ 0, an optimal stopping time for w T is given by Moreover, the process Based on Proposition 4.1 we may define the limits Let us now link the functions w and w with (4.1) and (4.2).
Theorem 4.2. Let the functions u and w be given by (4.1) and (4.2), respectively. Also, let w and w be given by (4.7). Then we get u ≡ w and w ≡ w. Also, u is lower semicontinuous and w is upper semicontinuous.
Proof. The proof for w ≡ w follows the lines of the second step in the proof of Theorem 3.3 and is omitted for brevity. Now we show that u ≡ w. The proof is partially based on Theorem 15 in Jelito et al. (2021). For transparency, we present it in detail. First, recalling non-negativity of g and G, for any T ≥ 0 and x ∈ E we get Thus, letting T → ∞, we get w ≤ u. Let us now show the reverse inequality. For any T > 0, let τ T be an optimal stopping time for w T , given by the formula (4.5). Defineτ and observe that τ T =τ T ∧ T . By monotonicity of the sequence (w n (x)) n∈N , we getτ n+1 ≤τ n . Thus, for any n ∈ N, on the set {τ n < n}, we getτ n = τ n , thuŝ τ n+1 = τ n+1 , and consequently τ n+1 ≤ τ n . Moreover, recalling that g(·) ≥ c > 0 and G(·) ≥ 0, for any x ∈ E, we get Consequently, for any x ∈ E, we get e cn < ∞. Hence, by Borel-Cantelli Lemma, for any x ∈ E, we get P x [ ∞ n=1 {τ n < n}] = 1, and consequently the stopping timeτ is well defined. Also, we get that P x [τ < ∞] = 1, x ∈ E. This follows from the fact that for P x almost all ω ∈ Ω, starting from some n (depending on ω), the sequence (τ n (ω)) is non-increasing. Thus, using right continuity of (X t ) and Fatou Lemma, for any x ∈ E, we get which concludes the proof of u ≡ w. Finally, recalling that by Proposition 4.1 the map u is an increasing limit of continuous functions, we get that u is lower semicontinuous. Using similar argument for w we get upper semicontinuity.
Remark 4.3. From the proof of Theorem 4.2 we get that the stopping timeτ given by (4.9) is optimal for u ≡ w; see (4.10). Also, note that in the proof we showed that P x [τ < ∞] = 1, x ∈ E; see the discussion following (4.9).
In Theorem 4.2 we showed that the function u given by (1.1) may be seen as a limit of finite horizon stopping problems w T . Let us now show that u may also be approximated by stopping problems with truncated terminal cost function. More explicitly, for any n ∈ N, we define (4.11) for any x ∈ E and n ∈ N. In Theorem 4.4 we link the functions u and u n .
Theorem 4.4. Let the functions u and u n be given by (4.1) and (4.11), respectively. Then, for any x ∈ E, we get u(x) = lim n→∞ u n (x).
Proof. Let us define the sequence of events A n := {G(X τn ) ≤ n}, n ∈ N, where Using Theorem 15 from Jelito et al. (2021) we get that τ n is an optimal stopping time for u n (x), x ∈ E, n ∈ N. Also, recalling that g(·) ≥ 0, for any x ∈ E and k ∈ N, we get Thus Let us fix n ∈ N and note that on the set ∩ ∞ k=n A k , for any j ≥ 0, we get Thus, on the set ∩ ∞ k=n A k , for any j ≥ 0, we get τ n+j+1 ≤ τ n+j . Combining this with (4.12), we may define the stopping timeτ := lim n→∞ τ n . Moreover, we get thatτ is almost surely finite since, for any n ∈ N, the stopping time τ n is almost surely finite; see Remark 16 in Jelito et al. (2021) for details. Thus, using right continuity of X and Fatou Lemma, for any x ∈ E, we get Remark 4.5. By analogy to Theorem 4.4 one could try to approximate the function w from (4.2) by the family However, since for any n ∈ N the map G(·) ∧ n is bounded, using Theorem 15 from Jelito et al. (2021) we get u n ≡ w n and by Theorem 4.4 we get w n → u. In fact, the identity u n ≡ w n may also be deduced from Corollary 5.11 in this paper.

Continuous time Bellman equation
In this section we extend the results from Section 3 to the continuous time case. We consider the continuous time Bellman equation which takes the form of optimal stopping dynamic programming principle (5.1) By analogy to Section 3 we show that the maps u and w are minimal and maximal solutions to this equation, respectively. First, note that (5.1) may be expressed in the operator form as where, for any t ≥ 0, the operator Φ t is given by and h : E → R + is a non-negative measurable function. To characterise the solutions to (5.1), for any t ≥ 0, let us define recursively We start with linking v t n and v t n with the functions w T and w T . Proposition 5.1. For any t ≥ 0 and n ∈ N, let the maps v t n and v t n be given by (5.3) and (5.4), respectively. Then, (1) For any t ≥ 0 and n ∈ N, we get v t n ≡ w nt and v t n ≡ w nt , where the functions w T and w T are given by (4.3) and (4.4), respectively.
(2) For any x ∈ E and t ≥ 0, we get where the functions u and w be given by (4.1) and (4.2), respectively. In particular, the limits lim n→∞ v t n (x) and lim n→∞ v t n (x) are well-defined and independent of t ≥ 0.
Proof. For transparency, we split the proof into two parts.
Proof of (1). We present the proof only for v t n ; the argument for v t n is similar and is omitted for brevity. Also, for the notational convenience we set t = 1; the general case follows the same logic.
Proof of (2). Recalling (1) and Theorem 4.2, for any x ∈ E and t ≥ 0, we get Using similar argument we get lim n→∞ v t n (x) = w(x), t ≥ 0, x ∈ E, which concludes the proof.
In the following theorem we characterise the solutions to the Bellman equation (5.1). In particular, we get that u and w are minimal and maximal solutions to (5.1), respectively. This may be seen as a continuous time version of Theorem 3.3.
Theorem 5.2. Let the functions u and w be given by (4.1) and (4.2), respectively. Then, (1) The functions u and w are solutions to (5.1).
Proof. For transparency, we split the proof into two parts.
Proof of (1). First, we prove that u satisfies (5.1). Let us define the process where w is given by (4.7). We show that (z(t)), t ≥ 0, is a submartingale. From Proposition 4.1, using submartingale property of z T , for any T, t, h ≥ 0 and x ∈ E, we get Thus, recalling monotonicity of T → w T (x), x ∈ E, and letting T → ∞, for any t, h ≥ 0 and x ∈ E, we get which concludes the proof of submartingale property of (z(t)), t ≥ 0. Next, using submartingale property of (z(t)), t ≥ 0, Doob optional stopping theorem, and the fact that w ≤ G, for any t ≥ 0 and x ∈ E, we get (5.8) To conclude the proof we show that for any t ≥ 0 and x ∈ E, we get where the stopping timeτ is given by (4.9). From Proposition 4.1, using martingale property of (z T (t ∧ τ T )), for any t ≥ 0, T ≥ t, and x ∈ E, we get Thus, using right-continuity of X, recalling Assumption (A2), and letting T → ∞, we get (5.9). Combining this with (5.8), for any t ≥ 0 and x ∈ E, we get Recalling that by Theorem 4.2 we get u ≡ w, we conclude the proof that u satisfies (5.1). Second, we prove that w is also a solution to (5.1). Noting that for any x ∈ E the sequence (v t n (x)) n∈N is non-increasing, using Proposition 5.1 and monotone convergence theorem, for any t ≥ 0 and x ∈ E, we get thus w is a solution to (5.1).
Proof of (2). Recall that if v is a solution to (5.1), then Φ t v = v, for any t ≥ 0. Thus, recalling (5.3) and (5.4), inductively we get v t n (x) ≤ v(x) ≤ v t n (x) for any t ≥ 0, n ∈ N, and x ∈ E. Hence, letting n → ∞ and using Proposition 5.1 we get (2).
Remark 5.3. It should be noted that combining Theorem 4.2 with Theorem 5.2 we get a possible numerical approximation scheme for extremal solutions to the Bellman equation. More specifically, we get that the map u, which is the smallest solution to (5.1), could be approximated by finite horizon optimal stopping value functions w T as T → ∞. Also, note that in the Step 2 of the proof of Proposition 4.1 we discuss a possible iterative procedure to approximate w T . Similar relations hold for the map w, which could be approximated by w T as T → ∞.
Based on Theorem 5.2 we get the following corollary.
Corollary 5.4. Let u and w be given by (4.1) and (4.2), respectively. Then, the following are equivalent (1) We get u ≡ w; (2) There is a unique solution to the Bellman equation (5.1) in the class of measurable functions v satisfying 0 ≤ v(·) ≤ G(·).
In the next proposition we study the properties of continuous solutions to the Bellman equation (5.1). This may be seen as a continuous time analogue of Lemma 3.2. Note that, in contrast to the discrete time case, here we additionally require continuity of v.
Proof. For any T ≥ 0 let us define and note that by (5.1) in fact we have v T ≡ v for any T ≥ 0. In particular, we get that the map (T, x) → v T (x) is continuous. Hence, using Lemma A.3, we get that the stopping time is optimal for e vT . Thus, for any x ∈ E and T ≥ 0, we get and (5.10) holds. Finally, using Lemma A.3 again we also get the submartingale property of z v (t), t ≥ 0, and the martingale property of z v (t ∧ τ v ), t ≥ 0.
In the following lemma we show when a continuous solution to the Bellman equation may be expressed as an expectation of the stopped value process.
Lemma 5.6. Let v be a continuous solution to (5.1) such that 0 ≤ v(·) ≤ G(·). Also, let τ v be as in Proposition 5.5. Then, we get Proof. Let v be a continuous solution to the Bellman equation 5.1 satisfying 0 ≤ v(·) ≤ G(·). Using Proposition 5.5, for any x ∈ E and T ≥ 0, we get Thus, recalling that g(·) ≥ c > 0 and using Fatou Lemma, we get and, in particular, we get P x [τ v < ∞] = 1. Thus, letting T → ∞ in (5.13), we get where the second equality follows from monotone convergence theorem. This concludes the proof.
Using Proposition 5.5 we get the closed-form formula for an optimal stopping time for the function u under the continuity assumption.
Proposition 5.7. Let the function u be given by (4.1). Assume that u is continuous. Then the stopping time is optimal for u.
Proof. By Theorem 5.2 we know that u satisfies the Bellman equation (5.1). Also, as in (5.14), we may show that P x [τ u < ∞] = 1 for any x ∈ E. Thus, using Proposition 5.5, continuity of u, Fatou Lemma, and martingale property of the process (z u (t ∧ τ u )), we get which concludes the proof.
Remark 5.8. Recall that by Remark 4.3 we get thatτ from (4.9) is also optimal for u. However, the stopping time τ u from (5.15) is smaller thanτ . Indeed, noting that u ≡ w ≥ w T for any T ≥ 0, we get τ u ≤τ . Now, let us define the process By analogy to Theorem 3.6 we may formulate a sufficient condition for u ≡ w. In particular, this gives uniqueness of a solution to (5.1).
(2) The stopping time is optimal for u. Also, we get τ u = lim T →∞ τ T , where τ T is given by (4.6). (3) The stopping time τ u given by (5.17) is also optimal for w, i.e. we get (5.18) Proof. For transparency, we prove the claims point by point.
Proof of (1). Recalling that by Remark 4.3 the stopping timeτ given by (4.9) is optimal for u and using uniform integrability of (Z(t ∧τ )), t ≥ 0, for any x ∈ E, we get Recalling that we always get u ≤ w, we conclude the proof of u ≡ w. Continuity follows from lower semicontinuity of u and upper semicontinuity of w; see Theorem 4.2 for details.
Proof of (2). Note that optimality of τ u follows from Proposition 5.7 and the fact that u ≡ w. Let us now show that Recalling Proposition 4.1, we get that the map T → τ T is increasing, hence the limit τ := lim T →∞ τ T is well-defined. Also, recalling that from (5.14) we get P x [τ u < ∞] = 1 and using the fact that u ≡ w ≡ w ≤ w T for any T ≥ 0, on the event {τ u ≤ T } we get Thus, we get τ T ≤ τ u ∧ T , hence, letting T → ∞, we get τ ≤ τ u . In particular, we get P x [τ < ∞] = 1, x ∈ E. Also, recalling joint continuity of (T, x) → w T (x), we get w T −τ T (X τ T ) = G(X τ T ).
(5.20) We show that this implies u(X τ ) = G(X τ ) and consequently τ u ≤ τ . First, note that from a.s. finiteness of τ , we get (T − τ T ) → ∞ as T → ∞. Second, note that for any T n → ∞ and x n → x, we get this follows from Dini's theorem combined with the fact that (w Tn ) is a sequence of continuous functions converging monotonically to the continuous function w. Thus, letting T → ∞ in (5.20), we get w(X τ ) = G(X τ ), which combined with the fact that u ≡ w ≡ w concludes the proof of this part.
Remark 5.10. In Theorem 5.9 continuity of u was a consequence of the identity u ≡ w. However, if we know in advance that u is continuous, we may obtain the results of Theorem 5.9 under weaker conditions. Namely, following the proof of Theorem 5.9, we can see that, assuming continuity of u, one may replace uniform integrability of Z(t ∧τ ), t ≥ 0, by uniform integrability of Z(t ∧ τ u ), t ≥ 0, where τ u := inf{t ≥ 0 : u(X t ) ≥ G(X t )}. Note that by Remark 5.8 the latter condition is less restrictive as τ u ≤τ .
If the function G is bounded, using Theorem 5.9 we may recover the results from Jelito et al. (2021); see Theorem 15 therein.
Corollary 5.11. If G is bounded, then u ≡ w and this function is continuous.
Proof. Recalling (4.9) and the following discussion, for any x ∈ E, we get that for P x almost all ω ∈ Ω, starting from some n ∈ N (depending on ω), the sequence (τ n (ω)) is non-increasing. Thus, using right-continuity of X, we get G(Xτ ) = lim n→∞ 1 {τ n <n} G(X τ n ). Consequently, recalling non-negativity of G, Proposition 4.1, and using Fatou Lemma, for any x ∈ E, we get Combining this with the inequality Z(t ∧τ ) ≤ e τ 0 g(Xs)ds e G , t ≥ 0, we get that by bounded convergence theorem the process (Z(t ∧τ )), t ≥ 0, is uniformly integrable. Consequently, using Theorem 5.9 we conclude the proof.

Reference examples
In this section we provide a series of examples illustrating our assumptions and results. In particular, we provide a more general criterion for Assumptions (A2)-(A3). Also, we show explicit formulae for multiple solutions to the Bellman equation.
6.1. Examples for Assumptions (A2)-(A3). In this section we comment on Assumptions (A2) and (A3). We show that they may be deduced from a more general condition: (B1) For any T ≥ 0 and a compact set K ⊆ E we get Condition (B1) may be seen as a stronger form of integrability for ζ T . Namely, it requires that the tail of ζ T is P x -integrable uniformly in x from compact set. Exemplary dynamics satisfying (B1) is shown in Example 6.2.
Proof. For (A2), it is enough to note that for any T ≥ 0, x ∈ E, and sufficiently large m ∈ N, we get For (A3), let T ≥ 0, x ∈ E, (x n ) → x, and h : E → R + be continuous and such that h(·) ≤ G(·). Let Γ ⊆ E be a compact set satisfying x ∈ Γ and (x n ) ⊂ Γ. We get Also, combining Lemma 4 from (Gikhman and Skorokhod, 1975, Section II.5) and Corollary 2.2 from Palczewski and Stettner (2010), we get that the map x → Thus, we get (6.1), which concludes the proof.
Let us now show the exemplary dynamics satisfying Condition (B1).
Example 6.2. Let E = R, G(x) = |x|, and the process (X t ) be a Brownian motion. Also, let K ⊆ E be a compact set and L K := sup x∈K |x|. Note that under P x we get X t = x + W t , where W is a standard Brownian motion (starting from 0). For the notational convenience, for any T ≥ 0, we set ζ T := sup t∈[0,T ] e |Xt| and S T := sup t∈[0,T ] |W t |, T ≥ 0. We show that for any T ≥ 0 we get Note that, for any x ∈ E, T ≥ 0, and n ∈ N, we get Thus, noting that sup x∈K e |x| < ∞, to conclude the proof of (6.2), it is enough to show Let us now show (6.4). Recalling that (−W ) is also a Brownian motion, we get Recall that by reflection principle the distribution of sup t∈[0,T ] W t is equal to the distribution of |W T |; see e.g. Proposition 3.7 in (Revuz and Yor, 1999, Chapter III) for details. Thus, we get which concludes the proof.
6.2. Examples for the Bellman equation. In this section we provide a series of computable examples related to the Bellman equation. In particular, we show a dynamics with a non-unique solution to this equation. First, we show an example, where there is a strict inequality between the maps u and w given by (3.1) and (3.2). Recall that we already showed u ≤ w. Example 6.3. Let E = {1, 2, 3, . . .}, g ≡ c > 0 and G(x) = x, x ∈ E. Let (X n ) n∈N be an i.i.d. sequence of discrete Pareto random variables, i.e.
First, we show that w(x) = x, x ∈ E. Recalling Theorem 3.3, we get where w n (x) := inf τ ≤n ln E x e cτ +Xτ , x ∈ E. Also, using Proposition 3.1, for any n ∈ N and x ∈ E, we get e w n+1 (x) = Se w n (x), where the operator S is given by inductively we get w n (x) = x, x ∈ E, and consequently w(x) = x, x ∈ E. Second, note that for x ∈ E \ {1}, τ 1 := inf{n ≥ 0 : X n = 1}, p 1 := P[X 1 = 1], and c > 0 satisfying c < − ln(1 − p 1 ) ≈ 0.94, we get Consequently, for x > ln B, we get thus, there is a strict inequality between u and w.
To better explain this situation, we directly show that the process is not uniformly integrable, cf. Lemma 2.1. It is enough to show that L := lim n→∞ sup T ∈N E e cτ1∧T +Xτ 1 ∧T 1 {e cτ 1 ∧T +X τ 1 ∧T ≥e n } = +∞.
In the next example we show explicit formulae for distinct solutions to the Bellman equation in the discrete time setting.
Example 6.4. Let E = [0, +∞) ⊂ R, g ≡ c > 0 and G(x) = x, x ∈ E. Let α ∈ [0, 1] and (X n ) n∈N be a time-homogeneous Markov process with a transition probability Recalling (3.1) and (3.2), let us consider Also, let K := ln αe c 1−(1−α)e c ; note that this constant is well-defined if (1−α)e c < 1. We show that within this model In particular, recalling Theorem 3.3, for α ∈ (1 − e −c , 1 − e −c−1 ] we get two distinct solutions to the Bellman equation (6.5) Namely, we get that both u and w satisfy (6.5), but u(x) < w(x) for x > K. In fact, in this case we may construct infinitely many solutions to (6.5); see Remark 6.6. Also, it should be noted that for α ∈ (1 − e −c−1 , 1] both functions u and w are bounded despite the fact that G is unbounded from above. Note that u(x) = x corresponds to the situation when instantaneous stopping is optimal; similar relation holds for w. Thus, we can see that for α small enough (relative to c), immediate stopping is optimal. However, for sufficiently big α it is optimal to wait until the process returns to zero; see the argument below for details.
Let us now show that e w n+1 (x) = e x ∧ e cn+1 , n ∈ N, x ∈ E, (6.6) where e cn := n−1 k=1 α(1 − α) k−1 e kc + (1 − α) n−1 e cn , n = 1, 2, . . . (6.7) First, note that by direct calculation we get e cn+1 ≥ e cn and recalling that (1 − α)e c < 1, we get e cn → e K as n → ∞. To show (6.6), we proceed by induction. For n = 1, we get e w 1 (x) = e x ∧ e c = e x ∧ e c1 , x ∈ E. Let us now assume that the claim holds for some n ≥ 1. Then, for x + 1 ≥ c n , by direct calculation, we get Thus, to conclude the proof it is enough to show e c (α which concludes the proof of (6.6). Letting n → ∞ in (6.6) and recalling Theorem 3.3 we get u(x) = x ∧ K.
(6.8) Using (6.8), inductively we get w n (x) = x for any x ∈ E. Thus lim n→∞ w n (x) = x, x ∈ E, and recalling Theorem 3.3 we get w(x) = x, x ∈ E.
Thus, for x > K and n ≥ 1, we get which concludes the proof of (6.9).
Using the argument leading to (6.9) we may show that the process (Z n∧τ ), n ∈ N, is uniformly integrable if and only if α ∈ [0, 1 − e −c ] ∪ (1 − e −c−1 , 1]. Thus, in this case the condition from Theorem 3.6 is also necessary for the equality u ≡ w. Next, let τ := inf{n ∈ N : w(X n ) = G(X n )}. One may show that the process (Z n∧τ ), n ∈ N, is uniformly integrable for any α ∈ [0, 1]. In particular, for α ∈ (1 − e −c , 1 − e −c−1 ], we get that uniform integrability of (Z n∧τ ), n ∈ N, does not imply the equality of u and w; see Remark 3.7.
Remark 6.6. Consider the model from Example 6.
We show that v is also a solution to the Bellman equation (6.5). Indeed, noting that v(x) = w(x) for x ∈ N, where w(x) := x, x ∈ E, and recalling that w is a solution to (6.5), we get Consequently, v is a solution to (6.5) and it is different from u and w; cf. Theorem 3.3. Also, note that v is discontinuous. In fact, using similar logic we may construct infinitely many (discontinuous) solutions to (6.5).
We conclude this section with the example for the non-uniqueness of a solution to the continuous time Bellman equation.
Example 6.7. In this example we use the dynamics from Example 6.4 to get a piecewise deterministic (piecewise constant) continuous time Markov process X on the state space E := [0, +∞). In a nutshell, under the measure P x , the process X starts at x ∈ E and stays at this state up to the exponentially distributed time τ 1 . At τ 1 , the process is subject to the immediate jump, with after jump state equals to 0 with probability α and equals to x + 1 with probability (1 − α). Then, the process stays at the new state with independent exponentially distributed time and the procedure repeats.
Let us now provide more details on the process construction. First, let (Y n ) be a discrete time Markov process with dynamics studied in Example 6.4, i.e.
for some α ∈ [0, 1]. Also, let (τ n ) ∞ n=1 be an increasing sequence of non-negative random variables. We assume that under any P x , x ∈ E, the increments (τ n+1 −τ n ), n ∈ N, are exponentially distributed with (common) parameter λ > 0; note that here we follow the convention τ 0 ≡ 0. Also, we assume that under any P x , x ∈ E, jump times (τ n ) are independent of (Y n ). Finally, we define the process X as X t := Y n for t ∈ [τ n , τ n+1 ). We refer to Davis (1993) for a more detailed discussion on the piecewise deterministic Markov processes.
By analogy to Example 6.4, we set g ≡ d with d ∈ (0, λ) and G(x) = x, x ∈ E. Also, we consider the continuous time optimal stopping problems (6.10) Due to the non-negativity of d, it is optimal to stop the process only at the times when the process is subject to a jump. Thus, the problem may be embedded in the discrete-time setting with the corresponding Bellman equation of the form Using independence of (Y n ) and (τ n ) and the fact that τ 1 is exponentially distributed, for any x ∈ E, we get Thus, Equation (6.12) could be rewritten as Note that setting c := ln λ − ln(λ − d), we get which coincides with (6.5). Thus, recalling the discussion in Example 6.4, we get the continuous time dynamics with multiple solutions to the corresponding Bellman equation. More specifically, using a suitable embedding, it can be shown that solutions to (6.13) satisfy which is a version of (5.1) corresponding to (6.10) and (6.11). Since by Example 6.4 we get multiple solutions to (6.13), we also get multiple solutions to (6.14).

Appendix A. Deferred proofs
In this section we present the proof of Proposition 4.1. This is an extension of the results from Jelito et al. (2021), where the function G is assumed to be bounded from above; see Propositions 10 and 11 therein. Throughout this section we assume (A1)-(A3).
For any n ∈ N and T ≥ 0 let us define bounded versions of (4.3) and (4.4) by We summarise the properties of v n T and v n T in the following lemma. For the proof, see Proposition 11 and Remark 12 from Jelito et al. (2021). (1) The function (T, x) → v n T (x) is jointly continuous. Moreover, is an optimal stopping time for v n T .
is an optimal stopping time for v n T . Let us now link the functions v n T and v n T with w T and w T . Lemma A.2. Let the functions w T and w T be given by (4.3) and (4.4), respectively. Also, let the sequences (v n T ) and (v n T ) be given by (A.1) and (A.2), respectively. Then, for any x ∈ E and T ≥ 0, we get Proof. We present the proof only for w T ; the proof for w T is analogous and omitted for brevity. Let us fix T ≥ 0 and x ∈ E. Also, let us define the family of events A n := {sup t∈[0,T ] G(X t ) ≤ n}, n ∈ N. For any n ∈ N we get A n ⊂ A n+1 . Moreover, using càdlàg property of X, continuity of G, and the fact that T < ∞, we get P x [∪ ∞ n=1 A n ] = 1.
Recalling Lemma A.1 and using right continuity of X, on the event {τ n T < T }, In fact, we get τ n+1 T ≤ τ n T on A n ; this follows from the fact that on A n ∩ {τ n T = T } directly from (A.3) we get τ n+1 T ≤ T = τ n T . Thus, acting inductively, for any k ≥ 0 we get τ n+k+1 T ≤ τ n+k T on A n . Thus, the limitτ T := lim n→∞ τ n T is well defined. Then, using right continuity of X, finiteness of G, and Fatou Lemma we get which concludes the proof.
Let us now show a useful result characterising an optimal stopping time for the finite horizon stopping problem with possible discontinuity at the terminal point.
Assume that the map (T, x) → v T (x) is jointly continuous. Then, for any T ≥ 0 the stopping time Moreover, for any T ≥ 0 and x ∈ E, the process z T (t) := e t∧T 0 g(Xs)ds+vT −t∧T (Xt∧T ) , t ≥ 0 is a P x -submartingale and (z T (t ∧ τ T )), t ≥ 0, is a P x -martingale.
Proof. The argument is partially based on the third step of the proof of Proposition 11 in Jelito et al. (2021). For transparency, we present it in detail.
We Using argument from Fakeev (1971) one can show that z T is the Snell envelope of y T . In particular, from Theorem 2 in Fakeev (1970) we get that (z T (t)), t ≥ 0, is a submartingale. Also, using Theorem 4 from Fakeev (1970), we get that τ ε T := inf {t ≥ 0 : z T (t) ≥ −ε + y T (t)} is an ε-optimal stopping time for e vT (x) , for any ε > 0, T ≥ 0, and x ∈ E. Thus, settingτ ε T := inf t ≥ 0 : e vT −t (Xt) ≥ (−ε) · e − t 0 g(Xs)ds + e G(Xt) , (A.5) we get τ ε T =τ ε T ∧ T . Now, noting thatτ ε1 T ≥τ ε2 T , whenever 0 ≤ ε 1 ≤ ε 2 , we may defineτ Let us now show thatτ T = τ T . For any ǫ > 0, on the event {τ ε T < T }, recalling (A.5), continuity of (T, x) → v T (x) and x → G(x), and right-continuity of (X t ), we get Since v T (x) ≤ G(x), for any x ∈ E and T ≥ 0, on the event {τ T < T }, we also get v T −τT (Xτ T ) = G(Xτ T ). Recalling definition of τ T , we get τ T ≤τ T . Noting that τ ε T ≤ τ T , for any ε > 0, and letting ε → 0, we get τ T =τ T . Now we show that τ T =τ T is optimal for v T . Using Fatou Lemma we get Similarly, using quasi-left continuity of X and recalling that G ≥ h, on the event A c = {∀ε : τ ε T < T }, we get Thus, from (A.7), we get and τ T =τ T is optimal for v T . Finally, let us show martingale property of (z T (t ∧ τ T )), t ≥ 0. Noting that for any t ≥ 0 we get z T (t ∧ τ T ) ≤ e T g sup t∈[0,T ] e G(Xt) and using (A2), we get that the process (z T (t ∧ τ T )), t ≥ 0, is uniformly integrable. In particular, recalling that τ T is optimal for v T (x), x ∈ E, we get (A.8) Also, using submartingale property of (z T (t)), t ≥ 0, and Doob optional stopping theorem, for any t, h ≥ 0 and x ∈ E, we get for any t, h ≥ 0. Thus, we have equality in (A.9), which concludes the proof.
Now we are ready to show the proof of Proposition 4.1.
Proof of Proposition 4.1. We present the proof only for w T (x); the argument for w T (x) is similar and is omitted for brevity. For transparency, we split the argument into three steps: (1) proof of monotonicity and continuity of T → w T (x) for fixed x ∈ E; (2) proof of continuity of x → w T (x) for fixed T ≥ 0; (3) proof of joint continuity of (T, x) → w T (x), optimality of τ T and martingale characterisation.
Step 1. Monotonicity and continuity of T → w T (x) for fixed x ∈ E. First, we prove monotonicity property of T → w T (x). Let T, u ≥ 0 and let τ ε ≤ T be an ε-optimal stopping time for e w T (x) . Then, using the fact that g, G ≥ 0 we get Letting ε → 0, we conclude that T → w T (x) is non-decreasing. Second, we show continuity of T → w T (x). Recalling that by Lemma A.1 and Lemma A.2, for any x ∈ E, the function T → w T (x) is an increasing limit of continuous functions T → v n T (x), we get that T → w T (x) is lower semicontinuous. This, together with the fact that T → w T (x) is non-decreasing, shows left continuity of T → w T (x). For the right continuity, let τ ǫ ≤ T be an ε-optimal stopping time for e w T (x) . Using monotonicity of w T , boundedness of g and (A2), we get e w T (x) ≤ lim = E x e τε 0 g(Xs)ds+1 {τε<T } G(Xτ ε ) ≤ e w T (x) + ε; (A.11) note in the second line we used bounded convergence theorem and the fact that (X t ) is right continuous. Letting ε → 0 we get right continuity of T → w T (x), for any x ∈ E.
Step 2. Continuity of x → w T (x) for fixed T ≥ 0. As in the first step, recalling Lemma A.1 and Lemma A.2, we get that, for any T ≥ 0, the function x → w T (x) is monotonicity of T → w T (x), from Dini's theorem we get that the convergence of w Tn (x) to w T (x) is uniform in x from compact sets; see Theorem 7.13 in Rudin (1976) for details. Thus, we get |w Tn (x n ) − w T (x)| → 0, n → ∞, (A.15) which shows continuity of the map (T, x) → w T (x). Thus, using Lemma A.3 we get that, for any T ≥ 0 and x ∈ E, the stopping time τ T is optimal for w T (x), the process z T (t) is a P x -submartingale and z T (t ∧ τ T ) is a P x -martingale, which concludes the proof.