Combinatorics of ancestral lines for a Wright–Fisher diffusion with selection in a Lévy environment

Abstract

Wright–Fisher diffusions describe the evolution of the type composition of an infinite haploid population with two types (say type 0 and type 1) subject to neutral reproductions, and possibly selection and mutations. In the present paper we study a Wright–Fisher diffusion in a Lévy environment that gives a selective advantage to sometimes one type, sometimes the other. Classical methods using the ancestral selection graph (ASG) fail in the study of this model because of the complexity, resulting from the two-sided selection, of the structure of the information contained in the ASG. We propose a new method that consists in encoding the relevant combinatorics of the ASG into a function. We show that the expectations of the coefficients of this function form a (nonstochastic) semigroup and deduce that they satisfy a linear system of differential equations. As a result we obtain a series representation for the fixation probability $\mathit{h}(\mathit{x})$ (where x is the initial proportion of individuals of type 0 in the population) as an infinite sum of polynomials whose coefficients satisfy explicit linear relations. Our approach then allows to derive Taylor expansions at every order for $\mathit{h}(\mathit{x})$ near $\mathit{x}=0$ and to obtain an explicit recursion formula for the coefficients.