## The Annals of Applied Probability

### Tree-valued Fleming–Viot dynamics with mutation and selection

#### Abstract

The Fleming–Viot measure-valued diffusion is a Markov process describing the evolution of (allelic) types under mutation, selection and random reproduction. We enrich this process by genealogical relations of individuals so that the random type distribution as well as the genealogical distances in the population evolve stochastically. The state space of this tree-valued enrichment of the Fleming–Viot dynamics with mutation and selection (TFVMS) consists of marked ultrametric measure spaces, equipped with the marked Gromov-weak topology and a suitable notion of polynomials as a separating algebra of test functions.

The construction and study of the TFVMS is based on a well-posed martingale problem. For existence, we use approximating finite population models, the tree-valued Moran models, while uniqueness follows from duality to a function-valued process. Path properties of the resulting process carry over from the neutral case due to absolute continuity, given by a new Girsanov-type theorem on marked metric measure spaces.

To study the long-time behavior of the process, we use a duality based on ideas from Dawson and Greven [On the effects of migration in spatial Fleming–Viot models with selection and mutation (2011c) Unpublished manuscript] and prove ergodicity of the TFVMS if the Fleming–Viot measure-valued diffusion is ergodic. As a further application, we consider the case of two allelic types and additive selection. For small selection strength, we give an expansion of the Laplace transform of genealogical distances in equilibrium, which is a first step in showing that distances are shorter in the selective case.

#### Article information

Source
Ann. Appl. Probab., Volume 22, Number 6 (2012), 2560-2615.

Dates
First available in Project Euclid: 23 November 2012

https://projecteuclid.org/euclid.aoap/1353695962

Digital Object Identifier
doi:10.1214/11-AAP831

Mathematical Reviews number (MathSciNet)
MR3024977

Zentralblatt MATH identifier
1316.92048

#### Citation

Depperschmidt, Andrej; Greven, Andreas; Pfaffelhuber, Peter. Tree-valued Fleming–Viot dynamics with mutation and selection. Ann. Appl. Probab. 22 (2012), no. 6, 2560--2615. doi:10.1214/11-AAP831. https://projecteuclid.org/euclid.aoap/1353695962

#### References

• Bakry, D. and Émery, M. (1985). Diffusions hypercontractives. In Séminaire de Probabilités, XIX, 1983/84. Lecture Notes in Math. 1123 177–206. Springer, Berlin.
• Barton, N. H., Etheridge, A. M. and Sturm, A. K. (2004). Coalescence in a random background. Ann. Appl. Probab. 14 754–785.
• Bertoin, J. and Le Gall, J.-F. (2003). Stochastic flows associated to coalescent processes. Probab. Theory Related Fields 126 261–288.
• Bertoin, J. and Le Gall, J.-F. (2005). Stochastic flows associated to coalescent processes. II. Stochastic differential equations. Ann. Inst. Henri Poincaré Probab. Stat. 41 307–333.
• Bertoin, J. and Le Gall, J.-F. (2006). Stochastic flows associated to coalescent processes. III. Limit theorems. Illinois J. Math. 50 147–181 (electronic).
• Dawson, D. A. (1993). Measure-valued Markov processes. In École D’Été de Probabilités de Saint-Flour XXI—1991. (P. L. Hennequin, ed.). Lecture Notes in Math. 1541 1–260. Springer, Berlin.
• Dawson, D. A., Greven, A. and Vaillancourt, J. (1995). Equilibria and quasiequilibria for infinite collections of interacting Fleming–Viot processes. Trans. Amer. Math. Soc. 347 2277–2360.
• Dawson, D. A. and Greven, A. (1999). Hierarchically interacting Fleming–Viot processes with selection and mutation: Multiple space time scale analysis and quasi-equilibria. Electron. J. Probab. 4 81 pp. (electronic).
• Dawson, D. A. and Greven, A. (2011). Duality for spatially interacting Fleming–Viot processes with mutation and selection. Available at arXiv:1104.1099.
• Dawson, D. A. and Greven, A. (2012a). Multiscale analysis: Fisher–Wright diffusions with rare mutations and selection, logistic branching system. In Probability in Complex Physical Systems: In Honour of Erwin Olthausen and Jürgen Gärtner (J.-D. Deuschel, B. Gentz, W. König, M. von Renesse, M. Scheutzow and U. Schmock, eds.). Springer Proceedings in Mathematics 11 373–408. Springer, Berlin.
• Dawson, D. A. and Greven, A. (2012b). On the effects of migration in spatial Fleming–Viot models with selection and mutation. Unpublished manuscript.
• Dawson, D. A. and March, P. (1995). Resolvent estimates for Fleming–Viot operators and uniqueness of solutions to related martingale problems. J. Funct. Anal. 132 417–472.
• Delmas, J.-F., Dhersin, J.-S. and Siri-Jegousse, A. (2010). On the two oldest families for the Wright–Fisher process. Electron. J. Probab. 15 776–800.
• Depperschmidt, A., Greven, A. and Pfaffelhuber, P. (2011). Marked metric measure spaces. Electron. Commun. Probab. 16 174–188.
• Donnelly, P. and Kurtz, T. G. (1996). A countable representation of the Fleming–Viot measure-valued diffusion. Ann. Probab. 24 698–742.
• Donnelly, P. and Kurtz, T. G. (1999). Genealogical processes for Fleming–Viot models with selection and recombination. Ann. Appl. Probab. 9 1091–1148.
• Etheridge, A. (2001). An Introduction to Superprocesses. Amer. Math. Soc., Providence, RI.
• Etheridge, A. M. and Griffiths, R. C. (2009). A coalescent dual process in a Moran model with genic selection. Theoret. Population Biol. 75 320–330. Sam Karlin: Special issue.
• Etheridge, A., Pfaffelhuber, P. and Wakolbinger, A. (2006). An approximate sampling formula under genetic hitchhiking. Ann. Appl. Probab. 16 685–729.
• Ethier, S. N. and Kurtz, T. G. (1986). Markov Processes: Characterization and Convergence. Wiley, New York.
• Ethier, S. N. and Kurtz, T. G. (1993). Fleming–Viot processes in population genetics. SIAM J. Control Optim. 31 345–386.
• Ethier, S. N. and Kurtz, T. G. (1998). Coupling and ergodic theorems for Fleming–Viot processes. Ann. Probab. 26 533–561.
• Ethier, S. N. and Shiga, T. (2000). A Fleming–Viot process with unbounded selection. J. Math. Kyoto Univ. 40 337–361.
• Evans, S. (2000). Kingman’s coalescent as a random metric space. In Stochastic Models: Proceedings of the International Conference on Stochastic Models in Honour of Professor Donald A. Dawson, Ottawa, Canada, June 10-13, 1998 (L. G. Gorostiza and B. G. Ivanoff, eds.). Canadian Math. Soc., Ottawa, ON.
• Evans, S. N. and Lidman, T. (2007). Asymptotic evolution of acyclic random mappings. Electron. J. Probab. 12 1151–1180 (electronic).
• Evans, S. N., Pitman, J. and Winter, A. (2006). Rayleigh processes, real trees, and root growth with re-grafting. Probab. Theory Related Fields 134 81–126.
• Evans, S. N. and Winter, A. (2006). Subtree prune and regraft: A reversible real tree-valued Markov process. Ann. Probab. 34 918–961.
• Fearnhead, P. (2001). Perfect simulation from population genetic models with selection. Theoret. Population Biol. 59 263–279.
• Fearnhead, P. (2002). The common ancestor at a nonneutral locus. J. Appl. Probab. 39 38–54.
• Fleming, W. H. and Viot, M. (1978). Some measure-valued population processes. In Stochastic Analysis (Proc. Internat. Conf., Northwestern Univ., Evanston, Ill., 1978) 97–108. Academic Press, New York.
• Fukushima, M. and Stroock, D. (1986). Reversibility of solutions to martingale problems. In Probability, Statistical Mechanics, and Number Theory. Adv. Math. Suppl. Stud. 9 107–123. Academic Press, Orlando, FL.
• Greven, A., Pfaffelhuber, P. and Winter, A. (2009). Convergence in distribution of random metric measure spaces ($\Lambda$-coalescent measure trees). Probab. Theory Related Fields 145 285–322.
• Greven, A., Pfaffelhuber, P. and Winter, A. (2012). Tree-valued resampling dynamics (Martingale problems and applications). Probab. Theory Related Fields. To appear.
• Hamilton, W. D. (1964a). The genetical evolution of social behaviour. I. J. Theoret. Biol. 7 1–16.
• Hamilton, W. D. (1964b). The genetical evolution of social behaviour. II. J. Theoret. Biol. 7 17–52.
• Itatsu, S. (2002). Ergodic properties of Fleming–Viot processes with selection. Rep. Fac. Sci. Shizuoka Univ. 36 1–14.
• Kaj, I. and Krone, S. M. (2003). The coalescent process in a population with stochastically varying size. J. Appl. Probab. 40 33–48.
• Kallenberg, O. (2002). Foundations of Modern Probability, 2nd ed. Springer, New York.
• Kaplan, N. L., Darden, T. and Hudson, R. R. (1988). The coalescent process in models with selection. Genetics 120 819–829.
• Kaplan, N. L., Hudson, R. R. and Langley, C. H. (1989). The “Hitchhiking effect” revisited. Genetics 123 887–899.
• Kingman, J. F. C. (1978). A simple model for the balance between selection and mutation. J. Appl. Probab. 15 1–12.
• Kingman, J. F. C. (1982). The coalescent. Stochastic Process. Appl. 13 235–248.
• Krone, S. M. and Neuhauser, C. (1997). Ancestral processes with selection. Theoret. Population Biol. 51 210–237.
• Liggett, T. M. (1985). Interacting Particle Systems. Grundlehren der Mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences] 276. Springer, New York.
• Mano, S. (2009). Duality, ancestral and diffusion processes in models with selection. Theoret. Population Biol. 75 164–175.
• Möhle, M. and Sagitov, S. (2001). A classification of coalescent processes for haploid exchangeable population models. Ann. Probab. 29 1547–1562.
• Neuhauser, C. and Krone, S. M. (1997). The genealogy of samples in models with selection. Genetics 154 519–534.
• Pfaffelhuber, P. and Wakolbinger, A. (2006). The process of most recent common ancestors in an evolving coalescent. Stochastic Process. Appl. 116 1836–1859.
• Revuz, D. and Yor, M. (1999). Continuous Martingales and Brownian Motion, 3rd ed. Grundlehren der Mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences] 293. Springer, Berlin.
• Slade, P. F. (2000a). Most recent common ancestor probability distributions in gene genealogies under selection. Theoret. Population Biol. 58 291–305.
• Slade, P. F. (2000b). Simulation of selected genealogies. Theoret. Population Biol. 57 35–49.
• Tajima, F. (1983). Evolutionary relationship of DNA sequences in finite populations. Genetics 105 437–460.
• Taylor, J. E. (2007). The common ancestor process for a Wright–Fisher diffusion. Electron. J. Probab. 12 808–847.
• Uyenoyama, M. K., Feldman, M. W. and Mueller, L. D. (1981). Population genetic theory of kin selection: Multiple alleles at one locus. Proc. Natl. Acad. Sci. USA 78 5036–5040.
• Wakeley, J. and Sargsyan, O. (2009). The conditional ancestral selection graph with strong balancing selection. Theoret. Population Biol. 75 355–364.
• Watterson, G. A. (1975). On the number of segregating sites in genetical models without recombination. Theoret. Population Biol. 7 256–276.
• Zambotti, L. (2001). A reflected stochastic heat equation as symmetric dynamics with respect to the 3-d Bessel bridge. J. Funct. Anal. 180 195–209.
• Zambotti, L. (2002). Integration by parts on Bessel bridges and related stochastic partial differential equations. C. R. Math. Acad. Sci. Paris 334 209–212.
• Zambotti, L. (2003). Integration by parts on $\delta$-Bessel bridges, $\delta>3$ and related SPDEs. Ann. Probab. 31 323–348.