## The Annals of Applied Probability

### Branching Brownian motion and selection in the spatial $\Lambda$-Fleming–Viot process

#### Abstract

We ask the question “when will natural selection on a gene in a spatially structured population cause a detectable trace in the patterns of genetic variation observed in the contemporary population?” We focus on the situation in which “neighbourhood size”, that is the effective local population density, is small. The genealogy relating individuals in a sample from the population is embedded in a spatial version of the ancestral selection graph and through applying a diffusive scaling to this object we show that whereas in dimensions at least three, selection is barely impeded by the spatial structure, in the most relevant dimension, $d=2$, selection must be stronger (by a factor of $\log(1/\mu)$ where $\mu$ is the neutral mutation rate) if we are to have a chance of detecting it. The case $d=1$ was handled in Etheridge, Freeman and Straulino (The Brownian net and selection in the spatial Lambda-Fleming–Viot. Preprint).

The mathematical interest is that although the system of branching and coalescing lineages that forms the ancestral selection graph converges to a branching Brownian motion, this reflects a delicate balance of a branching rate that grows to infinity and the instant annullation of almost all branches through coalescence caused by the strong local competition in the population.

#### Article information

Source
Ann. Appl. Probab., Volume 27, Number 5 (2017), 2605-2645.

Dates
Revised: August 2016
First available in Project Euclid: 3 November 2017

https://projecteuclid.org/euclid.aoap/1509696030

Digital Object Identifier
doi:10.1214/16-AAP1245

Mathematical Reviews number (MathSciNet)
MR3719942

Zentralblatt MATH identifier
1379.92036

Subjects
Primary: 60G99: None of the above, but in this section
Secondary: 92B05: General biology and biomathematics

#### Citation

Etheridge, Alison; Freeman, Nic; Penington, Sarah; Straulino, Daniel. Branching Brownian motion and selection in the spatial $\Lambda$-Fleming–Viot process. Ann. Appl. Probab. 27 (2017), no. 5, 2605--2645. doi:10.1214/16-AAP1245. https://projecteuclid.org/euclid.aoap/1509696030

#### References

• Barton, N. H. (1993). The probability of fixation of a favoured allele in a subdivided population. Genet. Res. 62 149–157.
• Barton, N. H., Etheridge, A. M. and Véber, A. (2010). A new model for evolution in a spatial continuum. Electron. J. Probab. 15 162–216.
• Barton, N. H., Etheridge, A. M., Kelleher, J. and Véber, A. (2013a). Genetic hitchhiking in spatially extended populations. Theor. Popul. Biol. 87 75–89.
• Barton, N. H., Etheridge, A. M., Kelleher, J. and Véber, A. (2013b). Inference in two dimensions: Allele frequencies versus lengths of shared sequence blocks. Theor. Popul. Biol. 87 105–119.
• Billingsley, P. (1995). Probability and Measure, 3rd ed. Wiley, New York.
• Cherry, J. L. (2003). Selection in a subdivided population with local extinction and recolonization. Genetics 164 789–795.
• Durrett, R. and Zähle, I. (2007). On the width of hybrid zones. Stochastic Process. Appl. 117 1751–1763.
• Etheridge, A. M. (2008). Drift, draft and structure: Some mathematical models of evolution. Banach Center Publ. 80 121–144.
• Etheridge, A., Freeman, N. and Straulino, D. (2015). The Brownian net and selection in the spatial Lambda-Fleming-Viot Process. Preprint. Available at arXiv:1506.01158.
• Etheridge, A. M. and Kurtz, T. G. (2014). Genealogical constructions of population models. Preprint. Available at arXiv:1402.6724.
• Etheridge, A. M. and Véber, A. (2012). The spatial $\Lambda$-Fleming-Viot process on a large torus: Genealogies in the presence of recombination. Ann. Appl. Probab. 22 2165–2209.
• Etheridge, A. M., Véber, A. and Yu, F. (2014). Rescaling limits of the spatial Lambda-Fleming-Viot process with selection. Preprint. Available at arXiv:1406.5884.
• Fisher, R. A. (1937). The wave of advance of advantageous genes. Ann. Eugenics 7 355–369.
• Kallenberg, O. (2006). Foundations of Modern Probability. Springer, Berlin.
• Kingman, J. F. C. (1993). Poisson Processes. Oxford Univ. Press, New York.
• Krone, S. M. and Neuhauser, C. (1997). Ancestral processes with selection. Theor. Popul. Biol. 51 210–237.
• Maruyama, T. (1970). On the fixation probability of mutant genes in a subdivided population. Genet. Res. 15 221–225.
• Mueller, C., Mytnik, L. and Quastel, J. (2008). Small noise asymptotics of traveling waves. Markov Process. Related Fields 14 333–342.
• Neuhauser, C. and Krone, S. M. (1997). Genealogies of samples in models with selection. Genetics 145 519–534.
• Véber, A. and Wakolbinger, A. (2015). The spatial Lambda-Fleming–Viot process: An event-based construction and a lookdown representation. Ann. Inst. Henri Poincaré Probab. Stat. 51 570–598.
• Whitlock, M. C. (2003). Fixation probability and time in subdivided populations. Genetics 164 767–779.