## The Annals of Applied Probability

- Ann. Appl. Probab.
- Volume 27, Number 5 (2017), 2605-2645.

### Branching Brownian motion and selection in the spatial $\Lambda$-Fleming–Viot process

Alison Etheridge, Nic Freeman, Sarah Penington, and Daniel Straulino

#### Abstract

We ask the question “when will natural selection on a gene in a spatially structured population cause a detectable trace in the patterns of genetic variation observed in the contemporary population?” We focus on the situation in which “neighbourhood size”, that is the effective local population density, is small. The genealogy relating individuals in a sample from the population is embedded in a spatial version of the ancestral selection graph and through applying a diffusive scaling to this object we show that whereas in dimensions at least three, selection is barely impeded by the spatial structure, in the most relevant dimension, $d=2$, selection must be stronger (by a factor of $\log(1/\mu)$ where $\mu$ is the neutral mutation rate) if we are to have a chance of detecting it. The case $d=1$ was handled in Etheridge, Freeman and Straulino (The Brownian net and selection in the spatial Lambda-Fleming–Viot. Preprint).

The mathematical interest is that although the system of branching and coalescing lineages that forms the ancestral selection graph converges to a branching Brownian motion, this reflects a delicate balance of a branching rate that grows to infinity and the instant annullation of almost all branches through coalescence caused by the strong local competition in the population.

#### Article information

**Source**

Ann. Appl. Probab., Volume 27, Number 5 (2017), 2605-2645.

**Dates**

Received: December 2015

Revised: August 2016

First available in Project Euclid: 3 November 2017

**Permanent link to this document**

https://projecteuclid.org/euclid.aoap/1509696030

**Digital Object Identifier**

doi:10.1214/16-AAP1245

**Mathematical Reviews number (MathSciNet)**

MR3719942

**Zentralblatt MATH identifier**

1379.92036

**Subjects**

Primary: 60G99: None of the above, but in this section

Secondary: 92B05: General biology and biomathematics

**Keywords**

Spatial Lambda–Fleming–Viot process branching coalescing natural selection branching Brownian motion population genetics

#### Citation

Etheridge, Alison; Freeman, Nic; Penington, Sarah; Straulino, Daniel. Branching Brownian motion and selection in the spatial $\Lambda$-Fleming–Viot process. Ann. Appl. Probab. 27 (2017), no. 5, 2605--2645. doi:10.1214/16-AAP1245. https://projecteuclid.org/euclid.aoap/1509696030