When can one detect overdominant selection in the infinite-alleles model?

Abstract

One of the goals of this paper is to show that the infinite-alleles model with overdominant selection "looks like" the neutral infinite-alleles model when the selection intensity and mutation rate get large together. This rather surprising behavior was noticed by Gillespie (1999) in simulations. To make rigorous and refine Gillespie's observations, we analyze the limiting behavior of the likelihood ratio of the stationary distributions for the model under selection and neutrality, as the mutation rate and selection intensity go to $\infty$ together in a specified manner. In particular, we show that the likelihood ratio tends to 1 as the mutation rate goes to $\infty$, provided the selection intensity is a multiple of the mutation rate raised to a power less than $3/2$. (Gillespie's simulations correspond to the power 1.) This implies that we cannot distinguish between the two models in this setting. Conversely, if the selection intensity grows like a multiple of the mutation rate raised to a power greater than $3/2$, selection can be detected; that is, the likelihood ratio tends to 0 under neutrality and $\infty$ under selection. We also determine the nontrivial limit distributions in the case of the critical exponent $3/2$. We further analyze the limiting behavior when the exponent is less than $3/2$ by determining the rate at which the likelihood ratio converges to 1 and by developing results for the distributions of finite samples.