LARGE DEVIATIONS AND QUASI-POTENTIAL OF A FLEMING-VIOT PROCESS

The large deviation principle is established for the Fleming-Viot process with neutral mutation when the process starts from a point on the boundary. Since the di(cid:11)usion coe(cid:14)cient is degenerate on the boundary, the boundary behavior of the process is investigated in detail. This leads to the explicit identi(cid:12)cation of the rate function, the quasi-potential, and the structure of the e(cid:11)ective domain of the rate function.


Introduction
Let E = [0, 1], and M 1 (E) be the set of all probability measures on E equipped with the weak topology. C(E) is the set of all continuous functions on E. The set C 2 b (R) contains all functions on the real line R that possess bounded continuous derivatives upto second order. For any θ > 0 and ν 0 in M 1 (E), define Then A is clearly the generator of a Markov jump process on E. Let , ϕ ∈ C(E), µ ∈ M 1 (E)}. For any > 0, consider the following operator on set D: where Q(µ; dx, dy) = µ(dx)δ x (dy) − µ(dx)µ(dy) and δ x stands for the Dirac measure at x ∈ E. The measure-valued process generated by L is called the Fleming-Viot process with neutral mutation. It models the evolution of the distribution of the genotypes in a population under the influence of mutation and replacement sampling. The set E is called the type space or the space of alleles, the operator A describes the generation independent mutation process with mutation rate θ and mutation measure ν 0 , and the last term describes the continuous sampling with sampling rate . When A is replaced by the generator of any other Feller process, the resulting process is still called a Fleming-Viot process. In fact in the original Fleming-Viot process, A is the Laplacian operator. In the sequel, we will use the term Fleming-Viot (henceforth, FV) process solely for the neutral mutation case. Let C([0, +∞), M 1 (E)) be the set of all continuous maps from [0, +∞) to M 1 (E). For any µ(·) in C([0, +∞), M 1 (E)), any fixed n ≥ 2, and 0 = . It is well-known (cf. Ethier and Kurtz [4]) that under the law of the FV process, X(t) = (X 1 (t), · · · , X n−1 (t)) solves the following degenerate stochastic differential equation: where for any 1 ≤ i, j ≤ n − 1, B j (t) are independent Brownian motions, and Because of this partition property, it becomes natural to study the FV process through its finite dimensional analogue. The first objective of this article is to establish the large deviation principle at the path level for the FV process when the sampling rate approaches 0. This problem has been studied in [1] and [2]. The only unresolved case is the lower bound when the mutation measure ν 0 is not absolutely continuous with respect to the initial point µ. In the finite dimensional case, this corresponds to the case when the process starts from a point on the boundary of the simplex {(x 1 , · · · , x n−1 ) : The difficulty in resolving the issue comes from the degeneracy and non-Lipschitz property of σ(x). Our proof of this result reveals that the derivatives of all possible large deviation paths at time zero have to be the same as the derivative of the solution of the limiting dynamic. This is very different from the non-degenerate case. An explicit expression of the rate function is also obtained. All definitions and terminology of large deviations follow those in [3]. Our second objective is to identify the quasi-potential of the process. Consider the following infinite dimensional dynamic: The solution of the dynamic is attracted to ν 0 as t goes to infinity. The FV process can be viewed as a random perturbation of this dynamic. It is thus natural to study the transition from ν 0 to another point ν in M 1 (E) under the perturbation. The quasi-potential V (ν 0 ; ν), if exists, is the minimal energy needed for the transition from ν 0 to ν. It is known that the FV process has a unique reversible measure Π which satisfies a full large deviation principle ( [2]). Roughly speaking for suitable subset A of M 1 (E), we have where I(ν) = θH(ν 0 |ν) and H(ν 0 |ν) is the relative entropy of ν 0 with respect to ν. If we set F (ν) = H(ν 0 |ν), then I(ν) = θ(F (ν) − F (ν 0 )). In physical terms, F (ν) is the free energy functional and θ plays the role of reciprocal temperature. Our second result shows that the quasi-potential equals to I(ν) and the minimal energy is attained by a time-reversed path of (1.2) connecting ν 0 with ν. Large deviation lower bounds are obtained in Section 2, and a detailed description of the effective domain is given in Section 3. Result on quasi-potential is in Section 4.

Finite Dimension
For any fixed n ≥ 1, let In the remaining of this subsection, we assume that x is in L p . Let This contradicts the assumption. Therefore, we may choose ii) For j = i, we have φ j (0) = 0 and hence, for all t in (0, T ], φ j (t) = 0. Therefore, for every t in (0, T ], φ i (t) = 1.
Then lim The conclusion follows easily. If φ i (0) = 1, then φ j (0) = 0 for all j = i and hence,φ j (0) = cp j . Thereforė Proof: Take N large so that δ > 4c where W i are Brownian motions such that λ is a constant determined by Fernique's theorem (cf. Kuo [5]) such that c 1 is finite. Therefore lim sup The conclusion then follows by taking N → ∞.
We are now ready to prove the lower bound.

Theorem 2.5
For δ 0 > 0 and any φ satisfying . Then it is clear that X (t) converges to ψ(t) for all t as goes to zero. For any N ≥ 1, define Choosing N large enough such that ϕ(t) ∈ S • n , the interior of S n , for t ∈ [1/2N, 1/N ], and For any δ < δ 0 , we have where δ is small enough such that |y − ϕ(1/2N )| ≤ δ/2 implies y ∈ S • n . Since X (1/2N ) converges to ψ(1/2N ) as goes to zero, by Lemma 2.4, we get where the next to last inequality follows from Theorem 3.3 in [1]. By direct calculation, we have If x i = 0, then ϕ i (0) = 0 and it is clear that

Infinite Dimension
Let In this subsection, we will assume that ν 0 is not absolutely continuous with respect to µ. Without loss of generality, we also assume that the support of ν 0 is E.

x) g(s, y) Q(µ(s); dx, dy) ds.
Recall that for any µ in M 1 (E), we have Q(µ; dx, dy) = µ(dx)δ x (dy) − µ(dx)µ(dy). Let where the supremum is taken over all f in C ∞ (E) with absolute value bounded by M . Let be the effective domain of S µ (·). Then any µ(·) in H µ is absolutely continuous, and by theorem where A * is the formal adjoint of A defined through the equality A * (µ), f = µ, Af , and for any linear functional ϑ on space C ∞ (E)

Effective Domain
In this subsection, we will analyze the structure of the effective domain. We start with another representation of the rate function. For any µ(·) ∈ C µ ([0, T ], M 1 (E)), let Let H 0 µ be the collection of all absolutely continuous µ(·) in C µ ([0, T ], M 1 (E)) such thatμ(t) − A * (µ(t)) is absolutely continuous with respect to µ(t) as Schwartz distribution with derivative Define Then Similar to Lemma 2.2, we have a t ≤ ct. So The conclusion of the theorem follows from the same arguments as in the proof of Lemma 2.3.

Quasi-Potential
For any ν in M 1 (E), the quasi-potential V (ν 0 ; ν) is defined as It is the least amount of energy needed for a transition from ν 0 to ν under large deviations.