Control and optimal stopping Mean Field Games: a linear programming approach

We develop the linear programming approach to mean-field games in a general setting. This relaxed control approach allows to prove existence results under weak assumptions, and lends itself well to numerical implementation. We consider mean-field game problems where the representative agent chooses both the optimal control and the optimal time to exit the game, where the instantaneous reward function and the coefficients of the state process may depend on the distribution of the other agents. Furthermore, we establish the equivalence between mean-field games equilibria obtained by the linear programming approach and the ones obtained via the controlled/stopped martingale approach, another relaxation method used in a few previous papers in the case when there is only control.

to present the linear programming approach in a much wider context of control and stopping MFG, with coefficients depending on the measure and under weaker assumptions than in [7], as well as to establish the equivalence of the linear programming approach with the other control relaxation approaches.
Our aim is to study MFGs in a general setting, including optimal stopping, continuous control and absorption. To explain the concept, assume that we have a 'large' number N ∈ N * of players. Each agent k ∈ {1, . . . , N } has a private state process X k,N = (X k,N t ) t∈[0,T ] , whose dynamics are given by the stochastic differential equation (SDE) where W 1 , . . . , W N are independent Brownian motions, α k = (α k t ) t∈[0,T ] is the control process taking values in a closed subset A ⊂ R, chosen by agent k and m N t is the empirical occupation measure of the players still present in the game and their controls: Here τ k is the stopping time, valued in [0, T ], chosen by player k, and denotes the first exit time of agent k from an open subset O ⊂ R, with the convention inf ∅ = ∞. Each agent k seeks to choose an optimal stopping time τ k and an optimal control α k to maximize the reward functional defined as follows: where µ N is the empirical joint distribution of the stopping time and the state process at the time of stopping: The objective functionals and the dynamics of the agents are coupled through the empirical measures (m N t ) t∈[0,T ] and µ N , so that it is natural to look for a Nash equilibrium. When the number of players N goes to infinity, we expect by a "propagation of chaos" type result that the empirical occupation measures converge to a deterministic flow of subprobability measures (m t ), while the empirical joint distributions of the stopping time/state process when each player exits the game (via discretionary stopping or absorption), converges to a deterministic limiting probability measure µ. In our setting the MFG problem therefore reads as follows:  In the literature on MFGs, there are two main approaches to prove existence of an MFG Nash equilibrium. The first approach, developed by Lasry and Lions [34], is an analytic one and consists in finding a Nash equilibrium by solving a coupled system of nonlinear partial differential equations: a Hamilton-Jacobi-Bellman equation (backward in time) satisfied by the value function of the representative agent for a given distribution and a Fokker-Planck-Kolmogorov equation (forward in time) describing the evolution of the density when the optimal control is used. The second approach, introduced by Carmona and Delarue [11,12], is based on the stochastic maximum principle which reduces the problem to a system of coupled forward-backward stochastic differential equations of McKean-Vlasov type.
In the standard stochastic control theory, the controlled martingale problem approach (see e.g. [19], [23] and [20] 1 ) is a powerful tool allowing to simplify the existence proofs, by compactification of the stochastic control problem. In the original MFG framework (regular control, without optimal stopping), the controlled martingale problem approach was first used in [31] to show the existence of a mean field game equilibrium under general assumptions. Further developments have been made in the case of mean field games with branching ( [15]) or mean field games with singular controls ( [22]). Another relaxation technique used in the classical stochastic control theory is based on the linear programming formulation (see e.g. [29,14,30]). In the context of mean-field games, this method has only been used in the case of optimal stopping in [7].
Mean field games of optimal stopping are a very recent trend in the MFG literature. More generally, only a few papers study mean field games with possible exit of the players leading to a decrease of the total mass of the players still in the game. We refer here to the MFGs with absorption (see e.g. [10]) and the MFGs of optimal stopping, introduced in the case of bank run models in [36,13], studied using an analytic approach in [5], and in a more general framework in [7].
In this paper, we extend the linear programming approach initiated in [7] to a more general setting including mixed optimal stopping/control, allowing for measure depending coefficients, and involving weaker assumptions. Furthermore, we clarify the relationship between linear programming MFG equilibria and MFG equilibria in the controlled/stopped martingale problem approach (also known as the weak formulation), and state precise conditions of equivalence of the two approaches. In the pure control case, this equivalence enables us to recover directly the result of existence of a Markovian equilibrium shown in [31] by using the Markovian projection technique. In addition, our method allows us to establish the existence of mixed solutions in the sense of [5], under a more general framework (in particular, with coefficients depending on both the control and the measure which was not the case in [5]).
The existence theorems of MFG equilibria obtained through the controlled martingale problem approach are in general rather abstract and provide little insight into the computation of MFG solutions. However, the linear programming approach we develop leads to a tractable method of computing the MFG equilibria, which has been instrumental in several concrete applications (see e.g. [3,8]).
The paper is organized as follows. In Section 2, we first study the single-agent problem under the linear programming formulation: we show the existence of a solution and prove its equivalence with the various weak formulations, as well as with PDEs. In Section 3, we solve the MFG problem and relate the notions of linear programming equilibria, weak equilibria and mixed solutions. In the Appendix we give some technical results and in particular we make the connection between the linear programming and the weak formulations, extending some of the existing results in the literature to less regular coefficients (see e.g. [29,14,17,30]).
Notation. For any topological space E we denote by B(E) the Borel σ-algebra, by P(E) the set of probability measures on (E, B(E)) and by M(E) the set of positive finite measures on (E, B(E)). We endow P(E) and M(E) with the topology of weak convergence and the associated Borel σalgebra. We denote by C(E) the set of continuous functions from E to R and by C b (E) the space of continuous and bounded functions from E to R which is endowed with the supremum norm with the convention inf ∅ = ∞. Let V 0 be the space of flows of measures onŌ × A, (m t ) t∈[0,T ] , such that: for every t ∈ [0, T ], m t is a Borel finite signed measure onŌ × A, for every B ∈ B(Ō × A), the mapping t → m t (B) is measurable, and T 0 |m t |(Ō × A)dt < ∞, where |m t | is the variation of m t . We define V 1 as the quotient space given by V 0 and the almost everywhere equivalence relation on [0, T ], that is, if, dt-almost everywhere on [0, T ], the measures m 1 t and m 2 t coincide, the measure flows (m 1 t ) t∈[0,T ] and (m 2 t ) t∈[0,T ] are considered equivalent. V 1 endowed with the usual sum and scalar multiplication is a vector space, where the zero vector is given by the family of null measures (0) t∈[0,T ] . To each (m t ) t∈[0,T ] ∈ V 1 we associate a Borel finite signed measure on [0, T ] ×Ō × A defined by m t (dx, da)dt and we endow V 1 with the topology of weak convergence of the associated measures. We denote by V the set of measure flows (m t ) t∈[0,T ] ∈ V 1 such that dt-a.e. m t is a positive measure. We note that V 1 is a Hausdorff locally convex topological vector space and V is metrizable (we refer to Appendix A for more details).
Let W = (W t ) t∈[0,T ] be a standard Brownian motion on a complete probability space (Ω, F, P). We denote by F W the filtration given by F t = σ (W s , 0 ≤ s ≤ t) ∨ N , where N denotes the P-null sets of F. Denote by T the set of stopping times with respect to this filtration with values in [0, T ]. Let A be the set of F W -progressively measurable control processes taking values in A.
In the paper we adopt the following terminology: controls of the type α t with values in A are called strict controls; controls of the form α t = α(t, X t ), with α a given measurable function are called Markovian strict controls; controls of the form ν t (respectively ν t,Xt for some kernel (ν t,x )) with values in P(A) are called relaxed controls (respectively Markovian relaxed controls). Relaxed controls are related to mixed strategies in game theory and consist in randomizing the action, which allows to embed the controls in a well behaved space. More precisely, instead of choosing an action valued in A, the agent chooses an action in P(A).

Single agent problem
In this section, we study the linear programming formulation of the mixed optimal stopping/stochastic control problem in the case when there is no interaction. In the following section, these results will be used in the MFG setting. We adopt here the following definitions and assumptions.
Definition 2.1. We denote by S the set of bounded measurable functions h : Throughout this section, unless specified otherwise, we will impose the following assumption.
bounded and Lipschitz in x uniformly on (t, a).
is upper semicontinuous and the function g : [0, T ]×Ō → R is upper semicontinuous and bounded from above.
(4) m * 0 ∈ P(O) satisfies O ln(1 + |x|)m * 0 (dx) < ∞. Consider the classical mixed stochastic control/optimal stopping problem which will be called the strong problem for the single agent.
We shall now provide the linear programming formulation which consists in introducing the occupation measures and the forward equation satisfied by them. where Define now the map Γ : R → R ∪ {−∞} as follows: The linear programming optimization problem takes the form The value for the LP formulation is defined by Remark 2.5. Throughout the paper, solutions of the LP problem taking the form m t (dx, da) = δ α(t,x) (da)m t (dx, A) for some measurable function α : [0, T ] ×Ō → A are called strict control LP solutions.

Existence of a solution for the linear programming problem
Let us first study some preliminary properties of the set R.
Preliminary properties of the set of constraints R. We start by showing the following admissibility result.
Proposition 2.6 (Admissibility of the occupation measures). Let (Ω, F, F, P) be a filtered probability space, τ an F-stopping time such that τ ≤ T P-a.s., ν an F-progressively measurable process with values in P(A), M a continuous F-martingale measure such that M τ has intensity ν t (da)1 t≤τ dt, X an F-adapted process such that Define now the measures Then (µ, m) ∈ R.
We refer to Appendix B.1 for the definition of martingale measures and their properties.
Now taking the expectation and using the fact that σ∂ x u is bounded, we get (µ, m) ∈ R.
We now show that from the forward equation (2.2), we can deduce that for almost every t ∈ [0, T ], m t is a subprobability measure. Proof. For every test function u(t, x) = T t f (s)ds with f a non-negative bounded continuous function, we have Since for all n ≥ 1, we conclude that (f n ) n≥1 converges to f in L 1 ([0, T ]). Up to taking a subsequence, we suppose without loss of generality that (f n ) n≥1 converges to f t-almost everywhere on [0, T ]. On the other hand, for all t ∈ [0, T ] and all n ≥ 1, By dominated convergence, The next Lemma extends Lemma 3.3.ii. in [7] to our general framework and since the proof is different, we give it in detail. Before presenting this result, we first recall the definition of the space of functions of bounded variation.
Proof. We consider the test function We have u ∈ C 1,2 b ([0, T ]×Ō). Now, using the constraint (2.2), the fact that µ belongs to P([0, T ]×Ō) and bounding h, its derivatives, the diffusion coefficients and the measures (m t ) (Lemma 2.7) by constants, we get The estimate on the BV-norm comes from Lemma 2.7 and taking the supremum in (2.5) over the set of ψ ∈ C 1 c (]0, T [) such that ψ ∞ ≤ 1.
We now provide the following convergence result. We recall that m n → m in V if m n t (dx, da)dt converges weakly to m t (dx, da)dt.
Proof. It is sufficient to show that given an arbitrary subsequence we can extract a subsubsequence converging to the above limit in L 1 ([0, T ]). Consider a subsequence (µ n k , m n k ) k≥1 . For all k ≥ 1, by Lemma 2.9, t → m n k t (Ō × A) ∈ BV(]0, T [) and By Theorem 3.23 in [2], up to a subsequence still denoted with n k , the sequence of mappings , we can use the same argument as before and conclude that up to another subsequence still denoted with n k , there exists k 0 ≥ 1 such that for all k ≥ k 0 , From the above estimates, we obtain for all k ≥ k 0 Since the elements of V are identified with measures whose marginals with respect to the time variable are absolutely continuous with respect to the Lebesgue measure, we can expect less regularity on the time component of the test functions, as it can be seen in the following lemma.
Proof. We are going to use Corollary 2.9 of [25]. We already know by definition of the convergence in V that m n t (dx, da)dt ⇀ m t (dx, da)dt, where we use the standard notation ⇀ for the weak convergence. We need to prove that (m n t (Ō × A)dt) n≥1 is relatively compact in M([0, T ]) endowed with the weak topology generated by the bounded and measurable functions from [0, T ] to R. Since B([0, T ]) is countably generated, by Proposition 2.10 in [25], this topology is metrizable, hence it is sufficient to show that for every subsequence of (m n (Ō × A)dt) n≥1 , there exists a subsubsequence converging for the weak topology generated by the bounded and measurable functions from [0, T ] to R. Let (m n k (Ō × A)dt) k≥1 be a subsequence of (m n (Ō × A)dt) n≥1 . Then (m n k ) k≥1 converges also to m in V. By Lemma 2.10, m n k · (Ō × A) k≥1 converges in L 1 ([0, T ]) to m · (Ō × A). Finally, for any function φ : [0, T ] → R bounded and measurable, We now prove the compactness of the set of constraints R, which extends Lemma 3.5. in [7] to our setting. The proof is more involved and we present it here for sake of clarity.
Theorem 2.12. The set R is compact.
x, a)m n t (dx, da)dt.
One can show that there exists a constant C ≥ 0 independent from k such that φ ′ k and φ ′′ k are bounded by C. By Lemma 2.7, for all n ≥ 1, m n t (Ō × A) ≤ 1 t-a.e. on [0, T ], which implies that there exists a constant C ′ ≥ 0 independent from n and k such that for all n ≥ 1 and k ≥ 1, Now, since (φ k ) k≥1 is a non-decreasing sequence converging to φ(x) = ln(1 + |x| 3 ), by monotone convergence theorem, we get for all n ≥ 1 Since φ is non-negative and for all r ≥ 0, the set is compact, we conclude that (ν n ) n≥1 is tight. Since by Lemma 2.7, ν n ([0, T ] ×Ō × A) ≤ T , by Prokhorov's Theorem (Theorem 8.6.2 in [6] (Volume 2)), there exists ν ∈ M([0, T ] ×Ō × A) such that, up to a subsequence, ν n ⇀ ν. Using the test function u(t, x) = T t ϕ(t)dt with ϕ a non-negative continuous function, for all n ≥ 1 Taking n → ∞, we conclude that Ō ×A ν(dt, dx, da) is absolutely continuous with respect to the Lebesgue measure on [0, T ], which allows the disintegration ν(dt, dx, da) = m t (dx, da)dt for some m ∈ V . We conclude that m n → m in V . Now, using the same test function u k , By the monotone convergence theorem and using that u which proves that (µ n ) n≥1 is tight. By Prokhorov's theorem there exists µ ∈ P([0, T ] ×Ō) such that, up to another subsequence, µ n ⇀ µ. Let u ∈ C 1,2 b ([0, T ] ×Ō). Taking limits in x, a)m n t (dx, da)dt, and using that, ∂u ∂t + Lu ∈ S, we get by Lemma 2.11 which shows that (µ, m) ∈ R and hence R is compact.
The existence result. We now give the main result of this subsection, which consists in showing that there exists an admissible maximizer (µ ⋆ , m ⋆ ) ∈ R for Γ. Proof. Let (µ n , m n ) n≥1 ⊂ R be a maximizing sequence, that is By Theorem 2.12, we get that up to a subsequence, (µ n , m n ) n≥1 converges to some (µ ⋆ , m ⋆ ) ∈ R.
On the other hand, since µ n ⇀ µ ⋆ and g is upper semicontinuous and bounded above, then Portmanteau theorem implies lim sup We conclude that Remark 2.14. In the case when there is no control and only optimal stopping, the above existence result holds under weaker assumptions on the coefficients and reward functions compared to [7].
The following result is well known in the literature (see [19,23,17,31]) but we give a proof for sake of completeness.
Proof. Let (µ ⋆ , m ⋆ ) be a maximizer of the LP problem which exists by Theorem 2.13. Let As in Proposition 3.5 of [23] one can prove that K(t, x) is closed. Now, by Theorem I.6.13 (p. 145) in [39], By definition of K(t, x) and Theorem A.9 in [23] there exists a measurable function (t,

Relation with the weak formulation
Following the literature on the linear programming formulation of stochastic control problems for Markov processes, we now prove prove that solving the linear program allows to construct a solution to the weak problem. The terminology weak is introduced in analogy to the notion of weak solution of an SDE, the idea being to consider the probabilistic set-up as part of the solution. The weak formulation is of two types, depending on the type of control, either strict control (valued in A) or relaxed control (valued in P(A)).

Assumption 2.
We assume here that one of the following statements holds: (1) Unattainable boundary: b, σ and O are such that, for every filtered probability space (Ω, F, F, P), F-stopping time τ such that τ ≤ T P-a.s., F-progressively measurable process ν with values in P(A), continuous F-martingale measure M such that M τ has intensity ν t (da)1 t≤τ dt, and F-adapted process X such that (2) Attainable boundary: σ does not depend on the control a and there exists c σ > 0 such that for We now give the weak formulations (with strict optimal stopping/control, resp. with strict optimal stopping and relaxed control) of the single agent problem. Definition 2.16 (Weak formulation with strict optimal stopping/control). Define U W as the set of tuples U = (Ω, F, F, P, W, α, τ, X) such that (Ω, F, F, P) is a filtered probability space, W is an F-Brownian motion, α is an F-progressively measurable process with values in A, τ is an F-stopping time such that τ ≤ T P-a.s., X is an F-adapted process such that The value for the weak formulation with strict control/optimal stopping is defined by Moreover, U ⋆ ∈ U W is a solution of the weak problem with strict optimal stopping/control if Definition 2.17 (Weak formulation with strict optimal stopping and relaxed control). Define U R as the set of tuples U = (Ω, F, F, P, M, ν, τ, X) such that (Ω, F, F, P) is a filtered probability space, τ is an F-stopping time such that τ ≤ T P-a.s., ν is an F-progressively measurable process with values in P(A), M is a continuous F-martingale measure such that M τ has intensity ν t (da)1 t≤τ dt, X is an F-adapted process such that The value for the weak formulation with strict optimal stopping and relaxed control is defined by Moreover, U ⋆ ∈ U R is a solution of the weak problem with strict optimal stopping and relaxed control if Theorem 2.18 (Existence of a weak solution with Markovian relaxed control). Suppose that Assumption 2 is also in force. Then there exists a solution to the weak problem with Markovian relaxed control.
Proof. Let (µ ⋆ , m ⋆ ) be a maximizer of the LP problem which exists by Theorem 2.13. Let Corollary 2.19 (Existence of a weak solution with markovian strict control). Suppose that for all (t, x) ∈ [0, T ] ×Ō, the subset of R × R + × R is convex and Assumption 2 is in force. Then there exists a weak solution with markovian strict control.
Proof. The proof follows by Proposition 2.15, Theorem C.6 and the same argument as in Theorem 2.18.

Equivalence of different formulations of the controlled/stopped diffusion processes problem and relation with PDEs
In this part, we aim to show the equivalence between the different formulations. The values for the linear programming and weak formulations are already defined, so we define now the value for the strong formulation.
Definition 2.20 (Strong formulation). Let t ∈ [0, T ], we denote by F t the filtration given by Denote by T t the set of stopping times with respect to this filtration with values in [t, T ]. Let A t be the set of F t -progressively measurable control processes taking values in A. The value function for the strong formulation is given by T ] is the unique strong solution of the following stochastic differential equation: We also define which represents the value for the strong formulation.
The case O = R. We show that the values at time zero associated to the different formulations (LP, weak and strong) are equal. In this paragraph, instead of Assumption 1, we impose the following assumption: (2) The functions b, σ and f are in S.
(3) The final payoff function g is bounded, measurable and continuous in x for each t.
We give now the definition of the strong formulation of the mixed stochastic control/optimal stopping problem.
Proof. The proof is organized in two steps.
Step 1. We first show that V R = V LP . Note that since O = R, Assumption 2 is satisfied. By Proposition 2.6, for each U ∈ U R , there exists (µ, m) ∈ R such that H R (U ) = Γ(µ, m). Therefore, we get Step 2. We prove that V S = V W = V R . This result follows by Theorem 4.5. in [20], which uses an equivalent formulation (see p. 18 in [20]), consisting in fixing a canonical space 2 and optimizing on a set of probability measures. To apply Theorem 4.5. in [20], we check that the assumptions are satisfied. Define for (t, y, a) ∈ For In fact, one can find a strong solution for the first component using the assumptions on b and σ, and since the second component is fully determined by the first one, we get the existence. We denote by X the first component and by Z the second component. Therefore, the associated controlled/stopped martingale problem has a solution. Note that the coefficients are continuous in the control variable for any (t, y) Moreover, for each t ∈ [0, ∞], y → Φ(t, y t∧· ) is continuous (C(R + , R 2 ) is endowed with the topology of uniform convergence on compact subsets of R + ). Since f and g are bounded, the last assumption of Theorem 4.5. in [20] is satisfied. Then applying Theorem 4.5 in [20] and integrating at time t = 0 with respect to m * 0 (see Theorem 3.1 (ii) in [20]), we get The result follows.
The case O bounded. In this paragraph, instead of Assumption 1, we impose the following assumption: (2) σ does not depend on the control a and is continuous on Lipschitz in x uniformly on the other variables.
(4) f is measurable, bounded and continuous onŌ, uniformly with respect to t and a.
and a → f (t, x, a) are continuous.  (7) in the above assumption is satisfied.
Let us recall an existence theorem for the strong formulation. The theorem is a particular case of Theorem 3.2, Chapter 4, in [4]. (2.10) Moreover, optimal controls are given by (2.12) Remark 2.24. Observe that in [4], they suppose that b and f are continuous on t. This assumption is used in their proof to establish continuity of the Hamiltonian, however we need only measurability on the Hamiltonian to use the measurable selection theorem.
The next Theorem is a slight extension of Theorem 5.2 in [7]. For sake of clarity we give the proof in Appendix E.
Theorem 2.25. Suppose Assumption 4 is in force. Then, the following are true (2) Let (µ ⋆ , m ⋆ ) be a maximizer of the LP program. Then m ⋆ satisfies Proposition 2.26. Let Assumption 4 hold true, and assume that for each (t, x) ∈ [0, T ] ×Ō, the subset Proof. Follows by Theorem 2.25 and a similar argument as in the proof of Proposition 2.15.

MFG problem
Throughout this section, we let the following assumptions hold true.
(5) One of the following statements is true: (a) The coefficients b and σ do not depend on the measure.
(b) Unattainable boundary: b, σ and O are such that, for every filtered probability space (Ω, F, F, P), F-stopping time τ such that τ ≤ T P-a.s., F-progressive measurable process ν with values in P(A), F-martingale measure M such that M τ has intensity ν t (da)1 t≤τ dt, m ∈ V and F-adapted process X such that  1. First step: fix µ ∈ P([0, T ] ×Ō) and m ∈ V and find the solution to the mixed control problem 2. Given the mixed optimal stopping-control (τ µ,m , α µ,m ) (solution of the problem (3.1)) for the agent with initial distribution m * 0 facing a mean-field (µ, m), the second step consists in finding µ ∈ P([0, T ] ×Ō) and the family of distributions m ∈ V such that We now give the formulation of the linear programming MFG problem. To this end, we first provide a preliminary definition.
Remark 3.4. Note that for allm ∈ V , R[m] has the same structure as R of the previous section, thus it satisfies the same properties. Moreover, the set R 0 has been introduced in order to be able to apply the fixed point arguments specific to the MFG setting; more precisely, it satisfies all properties as the set R (see theorem below) and contains all the sets R[m] for m ∈ V . Proof. The same proofs of Section 2.1 can be applied.
Definition 3.6. Define the set valued mapping R ⋆ : Define Θ : Remark 3.7. Note that the set of LP MFG Nash equilibria coincides with the set of fixed points of Θ.

Existence of LP MFG Nash equilibria
We shall first provide some convergence results, which will be useful in the proof of existence of LP MFG Nash equilibria.
Lemma 3.8. Let (Ω, F, F, P) be a filtered probability space. Let τ be a bounded F-stopping time and let M be an F-martingale measure with intensity q t (da)1 t≤τ dt, where (q t ) t∈[0,T ] is an F-predictable process with values in P(A). Consider (μ n ,m n ) n≥1 ⊂ R 0 such thatm n →m in V and let X and (X n ) n≥1 be F-adapted processes satisfying, Then, up to a subsequence, Proof. We will denote by C > 0 any constant independent from n. To simplify the formulas, in this proof we shall use the following shorthand Using Burkholder-Davis-Gundy inequality, we get From the above estimates, Now, by the Lipschitz assumption on b, Similarly, t∧τ 0 A (σ n (r, X n r , a) − σ 0 (r, X r , a)) 2 q r (da)dr We get finally, (σ n (r, X r , a) − σ 0 (r, X r , a)) 2 q r (da)dr .
By Gronwall's inequality, Let us show that B n → 0 as n → 0. We fix ω ∈ Ω. We are going to use Lemma F.2 for this fixed ω and then use dominated convergence for the expectation. We set Θ = [0, T ], X = A, η(dr) = dr, By Theorem 3.5 and Lemma 2.10, ψ n converges to ψ in L 1 ([0, T ]; R d ). Since the hypothesis of Lemma F.2 are satisfied, we get for all ω ∈ Ω, Sinceb is bounded and q r are probabilities, we get by the dominated convergence theorem B n −→ n→∞ 0.
The convergence of S n to 0 follows by the same arguments. Taking n → ∞ in (3.2) we get the result.
We now prove the continuity of the set R ⋆ in the sense of set-valued mappings.
x,m n t , a)m n t (dx, da)dt.
By Theorem 3.5 and Lemma 2.11, we get the stable convergence of m n t (dx, da)dt to m t (dx, da)dt. In particular, By Theorem 3.5 and Lemma 2.10, x,m t , a)m t (dx, da)dt.
By the same argument, The above results, together with the convergence which means that (µ, m) ∈ R[m] = R ⋆ (μ,m).
Step 2. We now prove the lower hemicontinuity (in the sense of Definition G.3). Consider a sequence (μ n ,m n ) n≥1 ⊂ R 0 such that (μ n ,m n ) → (μ,m) and let (µ, m) ∈ R ⋆ (μ,m) = R[m]. We need to prove that up to a subsequence, we can find (µ n , m n ) n≥1 ⊂ R 0 such that (µ n , m n ) ∈ R ⋆ (μ n ,m n ) = R[m n ] and (µ n , m n ) → (µ, m). This result is trivial if Assumption 5 (5)(a) holds true, therefore consider in the sequel the cases (5)(b) or (5)(c). Let ν t,x (da) be such that By Theorem C.6, there exists a filtered probability space (Ω, F, F, P), an F-adapted process X, an F-stopping time τ such that τ ≤ T ∧ τ X O P-a.s., an F-martingale measure M with intensity ν t,Xt (da)1 t≤τ dt, such that On the same filtered probability space, define where X n denotes the unique strong solution of dX n t = A b(t, X n t ,m n t , a)ν t,Xt (da)dt + A σ(t, X n t ,m n t , a)M (dt, da), X n 0 = X 0 .
Note that existence and uniqueness follow by the Lipschitz and boundedness condition on the coefficients and the square integrability of m * 0 . We have that (µ n , m n ) ∈ R[m n ] = R ⋆ (μ n ,m n ) by a similar argument as in Proposition 2.6. Let us now prove that m n → m in V . By Remark 8.3.1 and Exercise 8.10.71 in [6] (Volume 2), it is sufficient to use bounded and Lipschitz functions as test functions. Consider a bounded and Lipschitz function φ : [0, T ] ×Ō × A → R and denote by C the maximum between φ ∞ and the Lipschitz constant of φ.
Now, by Lemma 3.8, we get the convergence of the first term. The convergence of the second one is trivial under the condition (5)(b) of Assumption 5. Suppose now condition (5)(c) of Assumption 5 holds. Then, by Theorem C.6, the martingale measure M is replaced by an F-Brownian motion W and we get Define X 0 as the unique strong solution to By pathwise uniqueness type arguments, we get that X 0 t = X t on t ≤ τ , which implies that τ X 0 O ≥ τ P-a.s. We have that for all δ > 0 and C > 0, there exists n 0 ≥ 1 such that for all n ≥ n 0 , P sup We have also that, for all δ > 0, there exists R > 0 such that, Using these two last properties, we get by Theorem 5.1 and Remark 5.4 in [38] To be more precise, by assumption, O =]c 1 , c 2 [, c 1 < c 2 , then one can choose for the assumptions in [38] the function and by the continuous mapping theorem, Since this sequence is uniformly bounded by T we get the convergence in L 1 . Finally, we can conclude that m n → m in V . Now, by the convergence of m n towards m in V and since (µ n , m n ) ∈ R[m n ], we get that, µ n ⇀ µ (using the same results as for the upper hemicontinuity).

Nash value and selection of Nash equilibria
Case of measure independent coefficients In the case where the coefficients b and σ do not depend on the measure, we can prove uniqueness of the Nash value, which holds under the well known anti-monotonicity conditions on f and g. When the coefficients do not depend on the measure, an LP Nash equilibrium is a pair (µ ⋆ , m ⋆ ) ∈ R such that for all (µ, m) ∈ R, Theorem 3.11 (Uniqueness of the Nash value). Suppose that the coefficients do not depend on the measure. Suppose also that f and g take the following form where f 1 , f 2 , f 3 , g 1 , g 2 , g 3 are bounded and measurable, f 2 is non-increasing in the second argument and g 2 is non-increasing. Let (µ 1 , m 1 ) and (µ 2 , m 2 ) be two LP Nash equilibria. Then, In particular they lead to the same Nash value, that is Proof. The proof is a slight modification of the one of Theorem 4.4 in [7], therefore we omit it.
Case of measure dependent coefficients When the coefficients depend on the measure, we do not prove the uniqueness of the Nash value, but instead we can show that there exists a maximal Nash value. Let N ⋆ be the set of Nash equilibria. As in Theorem 3.10, we can show that v is continuous. By compactness of N ⋆ and continuity of v, we conclude the existence of a maximizer.
Selection of equilibria In both cases we have not proved uniqueness of Nash equilibria, we study only the Nash value. The natural question arising in this context is how to select the equilibria. In [16] the authors propose several ways of choosing equilibria in a particular model of MFGs, one of them is to choose the equilibria by maximizing the Nash value. We have shown in Proposition 3.12 that this method is always possible under our assumptions.

Relation with MFG equilibria in the weak formulation
In this section we show the equivalence between linear programming MFG equilibria and MFGs in the weak formulation as defined below.
Definition 3.13 (Weak MFG solution with strict optimal stopping/control).
as the set of tuples U = (Ω, F, F, P, W, α, τ, X) such that (Ω, F, F, P) is a filtered probability space, τ is an F-stopping time such that τ ≤ T P-a.s., α is an F-progressively measurable process with values in A, W is an F-Brownian motion, X is an F-adapted process such that The value of the optimization problem in the weak formulation with strict optimal stopping/control associated to (µ, m) is defined by Moreover, we say that U ⋆ = (Ω, F, F, P, W, α, τ, X) is a weak MFG Nash equilibrium with strict and Definition 3.14 (Weak MFG solution with strict optimal stopping and relaxed control).
as the set of tuples U = (Ω, F, F, P, M, ν, τ, X) such that (Ω, F, F, P) is a filtered probability space, τ is an F-stopping time such that τ ≤ T P-a.s., ν is an F-progressively measurable process with values in P(A), M is a continuous F-martingale measure such that M τ has intensity ν t (da)1 t≤τ dt, X is an F-adapted process such that for all U = (Ω, F, F, P, M, ν, τ, X) ∈ U R [µ, m]. The value of the optimization problem in the weak formulation with strict optimal stopping and relaxed control associated to (µ, m) is defined by Moreover, we say that U ⋆ = (Ω, F, F, P, M, ν, τ, X) is a weak MFG Nash equilibrium with relaxed and The above definition is equivalent to the following formulation of MFG equilibrium via the controlled/stopped martingale problem.
Definition 3.15 (MFG equilibrium via the controlled/stopped martingale problem). Find a filtered probability space (Ω, F, F, P), an F-stopping time τ such that τ ≤ T P-a.s., an F-progressively measurable process (ν t (da)) t≥0 with values in P(A) and an adapted process X such that (3) If (Ω ′ , F ′ , F ′ , P ′ ) is another filtered probability space, τ ′ an F ′ -stopping time such that τ ′ ≤ T P ′ -a.s., (ν ′ t (da)) t≥0 an F ′ -progressively measurable process with values in P(A), and an adapted process X ′ such that P ′ • (X ′ 0 ) −1 = m * 0 and for all ϕ ∈ C 2 b (R), the process (M ′ t∧τ ′ (ϕ)) t≥0 is an then, Remark 3.16. This definition is also equivalent to the problem of finding an MFG equilibrium via the controlled/stopped martingale problem on the canonical space (see [31]), where the optimization is considered over the set of probabilities on the canonical space instead of all the tuples (Ω, F, F, P, τ, ν, X). We refer to [20], p. 18, for more details on this equivalence.
Proof. Considering measure dependent coefficients, the equivalence follows from Proposition 2.6 and Theorem C.6. Proof. By Theorem 3.10 we get the existence of LP MFG Nash equilibrium, which implies by Theorem 3.17 the existence of a weak Nash equilibrium (with Markovian relaxed control).
Remark 3.19. In the case when there is only control, we recover the existence result of Markovian relaxed controls of [31,Corollary 3.8]. In that paper, this result is shown by using the Mimicking Theorem (or Markovian projection theorem) from Corollary 3.7. in [9], while in our case this result follows naturally by the disintegration T ] is the unique strong solution of the following stochastic differential equation: We have the following equality: Proof. Since (µ ⋆ , m ⋆ ) is fixed in the functions b, σ, f and g, we can apply Theorem 2.21 noticing that Assumption 3 is satisfied.
Proposition 3.21. Suppose that Assumption 5 with either (5)(b) or (5)(c) holding true and that for all (t, x, (µ, m)) ∈ [0, T ] ×Ō × R 0 , the subset Then there exist a strict control LP Nash equilibrium and a weak Nash equilibrium with Markovian strict control.
Proof. The proof is almost the same as that of Proposition 2.15; it relies on the fact that the dependence of b, σ 2 and f in the measure is of the form for some function h, which is independent of the control.

Relation with mixed solutions
In this subsection, to establish the link with PDE formulation, we shall need the following assumptions: (4) f is measurable, bounded and continuous in x onŌ, uniformly with respect to t, m and a.   (1) Relation with the strong formulation: (2) Relation with mixed solutions: (a) (c) For all C ∞ functions φ such that supp(φ) ⊂ C ⋆ , the following holds Note that (2)(c) holds true if and only if µ ⋆ (C ⋆ ) = 0, which is also equivalent to µ ⋆ (S ⋆ ∪([0, T ]× ∂O)) = 1.
Proof. The proof follows by applying Theorem 2.25 taking into account that the inputs (b, σ, f, g) depend now on (m ⋆ , µ ⋆ ) but still satisfy the required assumptions. of R × R + × R is convex. Let (µ ⋆ , m ⋆ ) an LP MFG equilibrium, then, there exists α ⋆ (t, x) such thatm t (dx) ≡ m ⋆ t (dx, A) satisfies the following system: x))m t (dx)dt = 0, for all C ∞ functions φ such that supp(φ) ⊂ C.
Remark 3.24. The above result gives the link with the notion of mixed solution in the case of optimal stopping/continuous control introduced in [5] in a less general framework (in particular, the author considers the drift to be zero and the volatility √ 2).
the projective topology τ 1 := π −1 (σ(M s 1 , F s 1 )) (that is the topology of weak convergence of the associated measures on [0, T ] ×Ō × A). With this definition, π is an isomorphim between the topological vector spaces, which implies that (V 1 , τ 1 ) is a Hausdorff locally convex topological vector space.
The relative topology on V is metrizable since π(V ) ⊂ M([0, T ] ×Ō × A) and the weak convergence topology on M([0, T ] ×Ō × A) is metrizable, in particular, we can define a natural distance on V associated to π.
We recall that the set P([0, T ] ×Ō) endowed with the topology of weak convergence is also metrizable, and hence the product space P([0, T ] ×Ō) × V is metrizable.

B Martingale measures and controlled/stopped martingale problem B.1 Martingale measures
For the sake of clarity we present the definition of martingale measures and some related concepts. This content is taken from [18] and [35]. Throughout the section we fix a filtered probability space (Ω, F, F, P) and a Polish space A with Borel σ-algebra B(A).  (3) There exists a non-decreasing sequence of (A n ) n≥1 ⊂ B(A) such that • For all t ∈ R + , and all n ≥ 1, • For all t ∈ R + , n ≥ 1 and (B k ) k≥1 ⊂ B(A n ) a decreasing sequence such that ∩ k≥1 B k = ∅,  ∧ τ, B) is also a martingale measure. [18]). If M is a martingale measure, then there exists a random σ-finite positive measure ν on R + × A, such that for each B ∈ B(A), (ν([0, t] × B)) t≥0 is the predictable quadratic variation of (M (t, B)) t≥0 . The measure ν is called the intensity of M .

Theorem B.3 (Theorem I-4 in
Let M be a martingale measure with intensity ν and let L 2 ν the set of functions φ : Ω×R + ×A → R measurable with respect to the product of the predictable σ-algebra and B(A), such that Then for any φ ∈ L 2 ν one can construct a stochastic integral of φ with respect to M , which is a function from Ω × R + × B(A) to R. It is denoted by φ · M . We will also denote The construction is analogous to the one of the Itô integral.

a)ψ(s, a)ν(ds, da).
A consequence of this proposition is that This fact allows the use of Burkholder-Davis-Gundy inequality, which can be applied to prove existence of strong solutions to SDEs of the type under standard assumptions.

B.2 Controlled/stopped martingale problem
Recall that the linear operator L is given by Definition B.5. The tuple (Ω, F, F, P, ν, τ, X) is said to be a solution of the controlled/stopped martingale problem if (1) (Ω, F, F, P) is a filtered probability space supporting an F-progressively measurable process ν with values in P(A), an F-stopping time τ and an F-adapted process X.
Suppose that X ·∧τ is continuous, τ is bounded and the coefficients b and σ are bounded. Then, on an extension of the filtered probability space, there exists a continuous martingale measure M with intensity ν t (da)1 t≤τ dt such that Moreover, there exists a Brownian motion W such that M (t, A) = M (t∧τ, A) = W t∧τ . In particular, if σ is uncontrolled, Proof. Using the same proof as in Lemma 3.2 of [31], there exists an F-predictable processν with values in P(A) such thatν t = ν t t-a.e. on [0, T ]. In particular, (Ω, F, F, P,ν, τ, X) is a solution of the controlled/stopped martingale problem. With some abuse of notation we denoteν by ν. For is an F-martingale. DefineX := X ·∧τ and q t (da) = ν t (da)1 t≤τ . Then, for all u ∈ C 2 b (R), is an F-martingale. Moreover, since the processes (ν t ) t∈[0,T ] and (1 t≤τ ) t∈[0,T ] are F-predictable and the map π : R + × P(A) → M(A) given by π(λ, ν) = λν is continuous, we get that the process (q t ) t∈[0,T ] is F-predictable. By Theorem IV-2 in [18], there exists an extension of the filtered probability space, denoted by (Ω ′ , F ′ , F ′ , P ′ ) supporting a martingale measure M with intensity q t (da)dt such that σ(s,X s , a)M (ds, da), t ≥ 0.
Since ν t (A) = 1 for all t ≥ 0, we get that (M (t, A)) t≥0 is a continuous square integrable martingale with quadratic variation (t ∧ τ ) t≥0 . Define M t := M (t, A) for t ≥ 0 and note that since (M t+τ − M τ ) t≥0 is an (F ′ t+τ ) t≥0 martingale, which means that P ′ -a.s., M t = M t∧τ , t ≥ 0. Consider the filtration G given by G t := F ′ t∧τ ⊂ F ′ t . By Theorem 1.7, Chapter V, in [37], on an extension of (Ω ′ , F ′ , G, P ′ ) denoted by (Ω,F ,F,P), there exists anF-Brownian motion W such that W t∧τ = M t∧τ = M t , t ≥ 0. Note that the definition of the stochastic integral We conclude that If σ is uncontrolled, by the construction of the integral with respect to M , one can deduce that In the case where the relaxed control ν is replaced by some strict control α, we can also find a SDE representation with respect to a Brownian motion.
Theorem B.7. Let (Ω, F, F, P, ν, τ, X) be a solution of the controlled/stopped martingale problem. Suppose that X ·∧τ is continuous, ν t = δ αt for some F-progressively measurable process α, τ is bounded and the coefficients b and σ are bounded. Then, on an extension of the filtered probability space, there exists a Brownian motion W such that Proof. Adapting the proof of Theorem 3.3 in [21] to random coefficients, we get the result for the case without stopping time. Using the same techniques as in Theorem B.6, we get the result.

C Link between linear programming and the weak formulation
We have seen in Proposition 2.6 that to any controlled and stopped diffusion we can associate a pair (µ, m) ∈ R. In Theorem C.6 we will prove that any (µ, m) ∈ R can be represented in terms of a controlled and stopped diffusion.
Lemma C.1. Consider a filtered probability space (Ω, F, F, P) supporting an F-Brownian motion W . Let T > 0, ξ an F 0 -measurable random variable supported in O, b a bounded F-progressively measurable process and σ a bounded F-progressively measurable process bounded below by a constant c σ > 0 and above by a constant C σ > 0. Let Y be defined by By Dambis-Dubbins-Schwarz theorem, there exists a Brownian motion W such that We denote by C the event where this result holds true at time τ Y O , which has probability one. Therefore, using that 0 < c σ ≤ σ t ≤ C σ , on the event B c ∩ C, Together, these two results imply that on the event has probability one we conclude the proof.
Let us recall some definitions and results of [28] Section 2. Let E be a complete, separable metric space. We denote by B(E) the set of bounded and measurable functions from E to R. Let L ⊂ B(E) × B(E) be the graph of an operator L (we abuse of notation as it is usual to identify an operator with its graph). Let L S be the linear span of an operator L.
Definition C.2. Let L : D(L) ⊂ B(E) → B(E) an operator and ν 0 ∈ P(E). We say that a measurable P(E)-valued function (we endow P(E) with the Borel σ-algebra generated by the topology of weak convergence) ν on R + is solution of the forward equation for (L, ν 0 ) if for all φ ∈ D(L) and t ∈ R + , is a pre-generator if L is dissipative and there are sequences of functions µ n : E → P(E) and λ n : for each x ∈ E, there exists a solution ν x of the forward equation for (L, δ x ) that is right-continuous (in the weak topology) at zero, then L is a pre-generator.
Now, we will show that any (µ, m) ∈ R has a probabilistic representation in terms of a controlled and stopped diffusion. The first part of the proof is based on the works of Stockbridge and coauthors (see e.g. [29,14,30]) with adaptations to our case. The second part uses the equivalence of the stopped/controlled martingale problem and the diffusions.
Then there exist a filtered probability space (Ω, F, F, P), an F-adapted process X, an F-stopping time τ such that τ ≤ T ∧ τ X O P-a.s., and an F-martingale measure M with intensity ν t,Xt (da)1 t≤τ dt, such that Moreover, if σ is uncontrolled or ν t,x = δ α(t,x) for some measurable function α, then one can replace the martingale measure by a Brownian motion.
Proof. We divide the proof in 4 steps. The first one is the redefinition of the coefficients and measures in order to construct an operator and a measure verifying the stationary equation. The second one contains the verification of the conditions to apply Corollary 1.10 in [30]. In the third step we apply this Corollary to obtain a controlled/stopped martingale problem formulation. Finally, in the fourth step, we go from the controlled/stopped martingale problem to the diffusion representation.
First step: Construction of the operator and the stationary measure. We extend ν t,x onto (R + × R) \ ([0, T ] ×Ō) with the value δ a 0 for an arbitrary a 0 ∈ A. Define the coefficientsb : otherwise. otherwise.
One can find a countable subset of C 1 b (R + ) approximating any function of C 1 b (R + ) under the pointwise convergence of β and β ′ (the same holds for C 2 b (R) with the point-wise convergence of ϕ, ϕ ′ and ϕ ′′ ). Then, the controlled martingale problem associated with L 0 is countably generated. Let us prove that for each (u, v) ∈ U × V , the operator A u,v (βγϕ)(r, s, x) := L 0 (βγϕ)(r, s, x, u, v) is a pre-generator. Suppose first that u = 1, then For z = (r, s, x) ∈ R + × R + × R, define the processes R z t = r, S z t = s + t and X z t = x + v 1 t + √ v 2 W t . For t ≥ 0 and z ∈ R + × R + × R, define the measures ν z Suppose now that u = 0, then We can rewrite the operator as (dr, ds, dx) = δ 0 (dr)δ 0 (ds)m 0 (dx).
By Proposition 10.2 p. 265 in [21], for any initial probability distribution ν on R + × R + × R, there exists a solution to the martingale problem for (A 0,v , ν) with càdlàg paths. This implies existence of a right continuous at zero solution to the forward equation for (A 0,v , δ z ), for any z, which in turn entails by Proposition C.5 that A 0,v is a pre-generator. Finally, the set D(L 0 ) is closed under multiplication and separates points since we can use bump functions.
Third step: Controlled/stopped martingale problem representation. By Corollary 1.10 in [30], there exist a complete probability space (Ω, F, Q) and a stationary R + × R + × R-valued process (R, S, Y ) (which we may assume is defined for all t ∈ R) such that is an (F R,S,Y t+ ) t -martingale for all βγϕ ∈ D(L), where (F R,S,Y t+ ) t is the complete and right continuous augmentation of the natural filtration (F R,S,Y t ) t . Following the same proof as Theorem 3.3 in [14], we arrive to the existence of a complete filtered probability space (Ω, F, F, P), where F satisfies the usual conditions, an F-stopping time τ with values in R + , a processS with values in R + such thatS t 1 t≤τ = t1 t≤τ , an F-progressively measurable process X with values in R such that P • X −1 0 =m 0 . Furthermore, which implies that X 0 ∈ O P-a.s. and On the other hand, since we conclude that τ ≤ T , X τ ∈Ō P-a.s. and Observe also that By the definition ofμ 0 we have Using thatS s 1 s≤τ = s1 s≤τ and taking γ = 1 in (C.1), we get that for all ϕ ∈ C 2 b (R), (ϕ)(s, X s )ds is an F-martingale. Extending b by 0 and σ by 1 for t > T , we obtain that for all ϕ ∈ C 2 b (R), is an F-martingale. Fourth step: SDE representation of the controlled/stopped martingale problem. DefineX t := X t∧τ for all t ∈ R + . Let us show thatX is a continuous process. Settinĝ we get that for all ϕ ∈ C 2 b (R), is an F-martingale. We conclude by Theorem II.2.42 from [26] thatX is a semimartingale with characteristics (B, C, 0) where This means that the compensator of the random measure defined by µX (ω, dt, dx) = s≥0 1 {∆Xs(ω) =0} δ (s,∆Xs(ω)) (dt, dx), is equal to 0 P-a.s. Applying Theorem II.1.8 (i) from [26] with W = 1, we get that µX (·, R + ×R) = 0 a.s., which implies thatX is a continuous process. Using the continuity and (C.2), we can deduce thatX takes values inŌ. Since (Ω, F, F, P, (ν t,Xt ) t≥0 , τ, X) is a solution of the controlled/stopped martingale problem, by Theorem B.6, on an extension of the filtered probability space, there exists a continuous martingale measure M with intensity ν t,Xt (da)1 t≤τ dt such that dX t = A b(t, X t , a)ν t (da)dt + A σ(t, X t , a)M (dt, da), t ≤ τ.
Moreover, there exists a Brownian motion W such that M (t, A) = M (t∧τ, A) = W t∧τ . In particular, if σ is uncontrolled, Let us prove now that τ ≤ τ X O P-a.s. If the first part of Assumption 2 holds, then τX O ≥ T P-a.s. Since X t =X t on {t ≤ τ }, we get that τ ≤ τ X O P-a.s. If we suppose now that the second part of Assumption 2 holds, then since σ is uncontrolled, We define b(s, X s , a)ν s,Xs (da)1 s≤τ ds + t 0 σ(s, X s )dW s .
By Lemma C.1 we get that τ Y O = τ Ȳ O P-a.s. Using that for all t ≥ 0, X t∧τ = Y t∧τ and X t∧τ is O-valued, we get that τ ≤ τ X O P-a.s. The case where ν t,x = δ α(t,x) for some measurable function α, follows by the same arguments and replacing Theorem B.6 by Theorem B.7. D Sufficient condition for the existence of a square integrable density for m t (dx, A) Proposition D.1. Suppose that Assumption 4 (1-6) holds true. Moreover, assume that σ 2 is Lipschitz continuous on [0, T ] ×Ō and m * 0 has a bounded density with respect to the Lebesgue measure. If (µ, m) ∈ R, then m t (dx, A)dt admits an square integrable density with respect to the Lebesgue measure on [0, T ] ×Ō.
Proof. We set η(dt, dx) = m t (dx, A)dt. By Theorem C.6, there exist a filtered probability space (Ω, F, F, P), an F-adapted process X, an F-stopping time τ such that τ ≤ T ∧ τ X O P-a.s., and an F-Brownian motion W , such that X t∧τ = Since τ ≤ τ X O , we get that (λ × P)({(t, ω) : X t (ω) ∈ ∂O, t ≤ τ (ω)}) = 0, which means that η puts 0 mass on ∂O and can thus be treated as a measure on [0, T ] × O. By standard arguments of existence of strong solutions to SDEs, there exists a unique process Y such that Since λ is a bounded process, by Girsanov's Theorem, under Q, . This allows to deduce that η(dt, dx) = η(t, x)dtdx for some non-negative L 1 function η. Moreover we get C (v − g)(t, x)µ ⋆ (dt, dx) = 0.

F Two technical lemmas
Lemma F.1. Let X and Y complete, separable metric spaces, and let ϕ : X × Y → R be bounded and continuous. Then, the map is continuous.

G Some results on set-valued analysis
Let us recall some theory about set-valued analysis, which can be found in Chapter 17 of [1]. For the next definitions, consider a metric space (X, d) and a set valued map ϕ : X → 2 X . The graph of ϕ is defined as the following set: Gr(ϕ) := {(x, y) ∈ X 2 : y ∈ ϕ(x)}.
Definition G.1. The correspondence ϕ is said to be upper hemicontinuous if for any sequence (x n , y n ) n≥1 in the graph of ϕ such that x n → x, the sequence (y n ) n≥1 has a limit point in ϕ(x).
Theorem G.2 (Closed Graph Theorem, Theorem 17.11 in [1]). If X is compact, the following statements are equivalent: (i) ϕ(x) is closed for all x ∈ X and ϕ is upper hemicontinuous.
(ii) The graph of ϕ is closed.
Definition G.3. The correspondence ϕ is said to be lower hemicontinuous if whenever x n → x and y ∈ ϕ(x), there exists a subsequence (x n k ) k≥1 of (x n ) n≥1 and a sequence (y k ) k≥1 , such that y k ∈ ϕ(x n k ) and y k → y.
Definition G.4. We say that ϕ is continuous if it is both upper hemicontinuous and lower hemicontinuous.
Theorem G.5 (Berge's Maximum Theorem, Theorem 17.31 in [1]). Let (X, d) be a metric space. Consider R ⋆ : X → 2 X a continuous correspondence with nonempty compact values and F : Gr(R ⋆ ) → R a continuous function. Define the function Θ : X → 2 X by Θ(x) = arg max F (x, y).
Then Θ is upper hemicontinuous and has nonempty compact values.
Theorem G.6 (Kakutani-Fan-Glicksberg, Corollary 17.55 in [1]). Let K be a nonempty compact convex subset of a locally convex Hausdorff space, and let the correspondence Θ : K → 2 K have closed graph and nonempty convex values. Then the set of fixed points of Θ is compact and nonempty.