## The Annals of Applied Probability

### Tree lengths for general $\Lambda$-coalescents and the asymptotic site frequency spectrum around the Bolthausen–Sznitman coalescent

#### Abstract

We study tree lengths in $\Lambda$-coalescents without a dust component from a sample of $n$ individuals. For the total length of all branches and the total length of all external branches, we present laws of large numbers in full generality. The other results treat regularly varying coalescents with exponent 1, which cover the Bolthausen–Sznitman coalescent. The theorems contain laws of large numbers for the total length of all internal branches and of internal branches of order $a$ (i.e., branches carrying $a$ individuals out of the sample). These results immediately transform to sampling formulas in the infinite sites model. In particular, we obtain the asymptotic site frequency spectrum of the Bolthausen–Sznitman coalescent. The proofs rely on a new technique to obtain laws of large numbers for certain functionals of decreasing Markov chains.

#### Article information

Source
Ann. Appl. Probab., Volume 29, Number 5 (2019), 2700-2743.

Dates
Revised: October 2018
First available in Project Euclid: 18 October 2019

https://projecteuclid.org/euclid.aoap/1571385620

Digital Object Identifier
doi:10.1214/19-AAP1462

Mathematical Reviews number (MathSciNet)
MR4019873

#### Citation

Diehl, Christina S.; Kersting, Götz. Tree lengths for general $\Lambda$-coalescents and the asymptotic site frequency spectrum around the Bolthausen–Sznitman coalescent. Ann. Appl. Probab. 29 (2019), no. 5, 2700--2743. doi:10.1214/19-AAP1462. https://projecteuclid.org/euclid.aoap/1571385620

#### References

• [1] Basdevant, A.-L. and Goldschmidt, C. (2008). Asymptotics of the allele frequency spectrum associated with the Bolthausen–Sznitman coalescent. Electron. J. Probab. 13 486–512.
• [2] Berestycki, J., Berestycki, N. and Limic, V. (2014). Asymptotic sampling formulae for $\Lambda$-coalescents. Ann. Inst. Henri Poincaré Probab. Stat. 50 715–731.
• [3] Berestycki, J., Berestycki, N. and Schweinsberg, J. (2007). Beta-coalescents and continuous stable random trees. Ann. Probab. 35 1835–1887.
• [4] Berestycki, N. (2009). Recent Progress in Coalescent Theory. Ensaios Matemáticos [Mathematical Surveys] 16. Sociedade Brasileira de Matemática, Rio de Janeiro.
• [5] Birkner, M., Blath, J. and Eldon, B. (2013). Statistical properties of the site-frequency spectrum associated with $\Lambda$-coalescents. Genetics 195 1037–1053.
• [6] Bolthausen, E. and Sznitman, A.-S. (1998). On Ruelle’s probability cascades and an abstract cavity method. Comm. Math. Phys. 197 247–276.
• [7] Dahmer, I., Kersting, G. and Wakolbinger, A. (2014). The total external branch length of beta-coalescents. Combin. Probab. Comput. 23 1010–1027.
• [8] Delmas, J.-F., Dhersin, J.-S. and Siri-Jegousse, A. (2008). Asymptotic results on the length of coalescent trees. Ann. Appl. Probab. 18 997–1025.
• [9] Dhersin, J.-S. and Möhle, M. (2013). On the external branches of coalescents with multiple collisions. Electron. J. Probab. 18 no. 40, 11.
• [10] Drmota, M., Iksanov, A., Möhle, M. and Rösler, U. (2007). Asymptotic results concerning the total branch length of the Bolthausen–Sznitman coalescent. Stochastic Process. Appl. 117 1404–1421.
• [11] Feller, W. (1971). An Introduction to Probability Theory and Its Applications. Vol. II. Wiley, New York, NY.
• [12] Goldschmidt, C. and Martin, J. B. (2005). Random recursive trees and the Bolthausen–Sznitman coalescent. Electron. J. Probab. 10 718–745.
• [13] Janson, S. and Kersting, G. (2011). On the total external length of the Kingman coalescent. Electron. J. Probab. 16 2203–2218.
• [14] Kersting, G. (2012). The asymptotic distribution of the length of beta-coalescent trees. Ann. Appl. Probab. 22 2086–2107.
• [15] Kersting, G., Pardo, J. C. and Siri-Jégousse, A. (2014). Total internal and external lengths of the Bolthausen–Sznitman coalescent. J. Appl. Probab. 51A 73–86.
• [16] Kersting, G., Schweinsberg, J. and Wakolbinger, A. (2014). The evolving beta coalescent. Electron. J. Probab. 19 no. 64, 27.
• [17] Kimura, M. (1969). The number of heterozygous nucleotide sites maintained in a finite population due to steady flux of mutations. Genetics 61 893–903.
• [18] Kingman, J. F. C. (1982). On the genealogy of large populations. J. Appl. Probab. 19A 27–43.
• [19] Möhle, M. (2010). Asymptotic results for coalescent processes without proper frequencies and applications to the two-parameter Poisson–Dirichlet coalescent. Stochastic Process. Appl. 120 2159–2173.
• [20] Pitman, J. (1999). Coalescents with multiple collisions. Ann. Probab. 27 1870–1902.
• [21] Sagitov, S. (1999). The general coalescent with asynchronous mergers of ancestral lines. J. Appl. Probab. 36 1116–1125.
• [22] Schweinsberg, J. (2000). A necessary and sufficient condition for the $\Lambda$-coalescent to come down from infinity. Electron. Commun. Probab. 5 1–11.
• [23] Spence, J. P., Kamm, J. A. and Song, Y. S. (2016). The site frequency spectrum for general coalescents. Genetics 202 1549–1561.
• [24] Tavaré, S. (2004). Ancestral inference in population genetics. In Lectures on Probability Theory and Statistics. Lecture Notes in Math. 1837 1–188. Springer, Berlin.
• [25] Wakeley, J. (2009). Coalescent Theory: An Introduction. Roberts and Company Publishers, Greenwood Village, CO.
• [26] Watterson, G. A. (1975). On the number of segregating sites in genetical models without recombination. Theor. Popul. Biol. 7 256–276.