## The Annals of Applied Probability

- Ann. Appl. Probab.
- Volume 14, Number 2 (2004), 754-785.

### Coalescence in a random background

N. H. Barton, A. M. Etheridge, and A. K. Sturm

#### Abstract

We consider a single genetic locus which carries two alleles, labelled *P* and *Q*. This locus experiences selection and mutation. It is linked to a second neutral locus with recombination rate *r*. If *r*=0, this reduces to the study of a single selected locus. Assuming a Moran model for the population dynamics, we pass to a diffusion approximation and, assuming that the allele frequencies at the selected locus have reached stationarity, establish the joint generating function for the genealogy of a sample from the population and the frequency of the *P* allele. In essence this is the joint generating function for a coalescent and the random background in which it evolves. We use this to characterize, for the diffusion approximation, the probability of identity in state at the neutral locus of a sample of two individuals (whose type at the selected locus is known) as solutions to a system of ordinary differential equations. The only subtlety is to find the boundary conditions for this system. Finally, numerical examples are presented that illustrate the accuracy and predictions of the diffusion approximation. In particular, a comparison is made between this approach and one in which the frequencies at the selected locus are estimated by their value in the absence of fluctuations and a classical structured coalescent model is used.

#### Article information

**Source**

Ann. Appl. Probab. Volume 14, Number 2 (2004), 754-785.

**Dates**

First available in Project Euclid: 23 April 2004

**Permanent link to this document**

http://projecteuclid.org/euclid.aoap/1082737110

**Digital Object Identifier**

doi:10.1214/105051604000000099

**Mathematical Reviews number (MathSciNet)**

MR2052901

**Zentralblatt MATH identifier**

1060.60100

**Subjects**

Primary: 60J80: Branching processes (Galton-Watson, birth-and-death, etc.) 60J85: Applications of branching processes [See also 92Dxx] 60J70: Applications of Brownian motions and diffusion theory (population genetics, absorption problems, etc.) [See also 92Dxx] 60K35: Interacting random processes; statistical mechanics type models; percolation theory [See also 82B43, 82C43]

**Keywords**

Coalescent selection recombination identity by descent random environment

#### Citation

Barton, N. H.; Etheridge, A. M.; Sturm, A. K. Coalescence in a random background. Ann. Appl. Probab. 14 (2004), no. 2, 754--785. doi:10.1214/105051604000000099. http://projecteuclid.org/euclid.aoap/1082737110.