## The Annals of Probability

### Beta-coalescents and continuous stable random trees

#### Abstract

Coalescents with multiple collisions, also known as Λ-coalescents, were introduced by Pitman and Sagitov in 1999. These processes describe the evolution of particles that undergo stochastic coagulation in such a way that several blocks can merge at the same time to form a single block. In the case that the measure Λ is the Beta (2−α, α) distribution, they are also known to describe the genealogies of large populations where a single individual can produce a large number of offspring. Here, we use a recent result of Birkner et al. to prove that Beta-coalescents can be embedded in continuous stable random trees, about which much is known due to the recent progress of Duquesne and Le Gall. Our proof is based on a construction of the Donnelly–Kurtz lookdown process using continuous random trees, which is of independent interest. This produces a number of results concerning the small-time behavior of Beta-coalescents. Most notably, we recover an almost sure limit theorem of the present authors for the number of blocks at small times and give the multifractal spectrum corresponding to the emergence of blocks with atypical size. Also, we are able to find exact asymptotics for sampling formulae corresponding to the site frequency spectrum and the allele frequency spectrum associated with mutations in the context of population genetics.

#### Article information

Source
Ann. Probab. Volume 35, Number 5 (2007), 1835-1887.

Dates
First available in Project Euclid: 5 September 2007

Permanent link to this document
https://projecteuclid.org/euclid.aop/1189000930

Digital Object Identifier
doi:10.1214/009117906000001114

Mathematical Reviews number (MathSciNet)
MR2349577

Zentralblatt MATH identifier
1129.60067

#### Citation

Berestycki, Julien; Berestycki, Nathanaël; Schweinsberg, Jason. Beta-coalescents and continuous stable random trees. Ann. Probab. 35 (2007), no. 5, 1835--1887. doi:10.1214/009117906000001114. https://projecteuclid.org/euclid.aop/1189000930

#### References

• Aldous, D. (1991). The continuum random tree. I. Ann. Probab. 19 1--28.
• Aldous, D. (1993). The continuum random tree. III. Ann. Probab. 21 248--289.
• Andrews, G. E., Askey, R. and Roy, R. (1999). Special Functions. Cambridge Univ. Press.
• Arratia, R., Barbour, A. and Tavaré, S. (2003). Logarithmic Combinatorial Structures: A Probabilistic Approach. EMS, Zürich.
• Athreya, K. B. and Ney, P. E. (1972). Branching Processes. Springer, New York--Heidelberg.
• Berestycki, J. (2003). Multifractal spectra of fragmentation processes. J. Statist. Phys. 113 411--430.
• Berestycki, J., Berestycki, N. and Limic, V. (2007). The asymptotic number of blocks in a Lambda coalescent. To appear.
• Berestycki, J., Berestycki, N. and Schweinsberg, J. (2006). Small-time behavior of beta-coalescents. Preprint. Available at http://front.math.ucdavis.edu/math.PR/0601032.
• Bertoin, J. (1999). Subordinators: Examples and applications. Ecole d'été de Probabilités de St-Flour XXVII. Lecture Notes in Math. 1717 1--91. Springer, Berlin.
• Bertoin, J. and Le Gall, J.-F. (2000). The Bolthausen--Sznitman coalescent and the genealogy of continuous-state branching processes. Probab. Theory Related Fields 117 249--266.
• Bertoin, J. and Le Gall, J.-F. (2003). Stochastic flows associated to coalescent processes. Probab. Theory Related Fields 126 261--288.
• Bertoin, J. and Le Gall, J.-F. (2005). Stochastic flows associated to coalescent processes II: Stochastic differential equations. Ann. Inst. H. Poincaré Probab. Statist. 41 307--333.
• Bertoin, J. and Le Gall, J.-F. (2005). Stochastic flows associated to coalescent processes III: Infinite population limits. Illinois J. Math. 50 147--181.
• Birkner, M., Blath, J., Capaldo, M., Etheridge, A., Möhle, M., Schweinsberg, J. and Wakolbinger, A. (2005). Alpha-stable branching and beta-colaescents. Electron. J. Probab. 10 303--325.
• Bolthausen, E. and Sznitman, A.-S. (1998). On Ruelle's probability cascades and an abstract cavity method. Comm. Math. Phys. 197 247--276.
• Donnelly, P. and Kurtz, T. (1999). Particle representations for measure-valued population models. Ann. Probab. 27 166--205.
• Duquesne, T. and Le Gall, J.-F. (2002). Random Trees, Lévy Processes, and Spatial Branching Processes. Astérisque 281.
• Duquesne, T. and Le Gall, J.-F. (2005). Probabilistic and fractal aspects of Lévy trees. Probab. Theory Related Fields 131 553--603.
• Durrett, R. (2004). Probability: Theory and Examples, 3rd ed. Duxbury Press, Belmont, CA.
• Durrett, R. and Schweinsberg, J. (2005). A coalescent model for the effect of advantageous mutations on the genealogyof a population. Stochastic Process. Appl. 115 1628--1657.
• Eldon, B. and Wakeley, J. (2006). Coalescent processes when the distribution of offspring number among individuals is highly skewed. Genetics 172 2621--2633.
• Etheridge, A. (2000). An Introduction to Superprocesses. Amer. Math. Soc., Providence, RI.
• Evans, S. N. (2000). Kingman's coalescent as a random metric space. In Stochastic Models: A Conference in Honour of Professor Donald A. Dawson (L. G. Gorostiza and B. G. Ivanoff, eds.) 105--114. Amer. Math. Soc., Providence, RI.
• Ewens, W. J. (1972). The sampling theory of selectively neutral alleles. Theor. Popul. Biol. 3 87--112.
• Fu, Y. X. and Li, W. H. (1999). Coalescent theory and its applications in population genetics. In Statistics in Genetics (M. E. Halloran and S. Geisser, eds.). Springer, Berlin.
• Gnedin, A., Hansen, B. and Pitman, J. (2007). Notes on the occupancy problem with infinitely many boxes: General asymptotics and power laws. Preprint. Available at http://front.math.ucdavis.edu/math.PR/0701718.
• Grey, D. R. (1974). Asymptotic behaviour of continuous-time continuous-state branching processes. J. Appl. Probab. 11 669--677.
• Griffiths, R. C. and Lessard, S. (2005). Ewens' sampling formula and related formulae: Combinatorial proofs, extension to variable population size, and applications to age of alleles. Theor. Popul. Biol. 68 167--177.
• Hein, J., Schierup, M., Wiuf, C. and Schierup, M. H. (2005). Gene Genealogies, Variation and Evolution: A Primer in Coalescent Theory. Oxford Univ. Press.
• Karlin, S. (1967). Central limit theorems for certain infinite urn schemes. J. Math. Mech. 17 373--401.
• Kimura, M. (1969). The number of heterozygous nucleotide sites maintained in a finite population due to steady flux of mutations. Genetics 61 893--903.
• Kimura, M. and Crow, J. F. (1964). The number of alleles that can be maintained in a finite population. Genetics 49 725--738.
• Kingman, J. F. C. (1982). The coalescent. Stochastic Process. Appl. 13 235--248.
• Kingman, J. F. C. (1982). On the genealogies of large populations. J. Appl. Probab. 19A 27--43.
• Lamperti, J. (1967). The limit of a sequence of branching processes. Z. Wahrsch. Verw. Gebiete 7 271--288.
• Le Gall, J.-F. and Le Jan, Y. (1998). Branching processes in Lévy processes: The exploration process. Ann. Probab. 26 213--252.
• Liu, Q. (2001). Local dimensions of the branching measure on a Galton--Watson tree. Ann. Inst. H. Poincaré Probab. Statist. 37 195--222.
• Lyons, R. and Peres, Y. Probability on trees and networks. Available at http://mypage.iu.edu/~rdlyons/prbtree/prbtree.html.
• Möhle, M. (2006). On sampling distributions for coalescent processes with simultaneous multiple collisions. Bernoulli 12 35--53.
• Möhle, M. and Sagitov, S. (2001). A classification of coalescent processes for haploid exchangeable population models. Ann. Probab. 29 1547--1562.
• Mörters, P. and Shieh, N. R. (2005). Multifractal analysis of branching measure on a Galton--Watson tree. In Proc. Internat. Conference of Chinese Mathematicians, Hong Kong 2004.
• Neveu, J. and Pitman, J. (1989). The branching process in a Brownian excursion. Séminaire de Probabilités XXIII. Lecture Notes in Math. 1372 248--257. Springer, Berlin.
• Pitman, J. (1997). Partition structures derived from Brownian motion and stable subordinators. Bernoulli 3 79--96.
• Pitman, J. (1999). Brownian motion, bridge, excursion and meander characterized by sampling at independent uniform times. Electron. J. Probab. 4 1--33.
• Pitman, J. (1999). Coalescents with multiple collisions. Ann. Probab. 27 1870--1902.
• Pitman, J. (2006). Combinatorial stochastic processes. Ecole d'Eté de Probabilités de Saint-Flour XXXII-2002. Lecture Notes in Math. 1875. Springer, Berlin.
• Revuz, D. and Yor, M. (1999). Continuous Martingales and Brownian Motion. Springer, Berlin.
• Rouault, A. (1978). Lois de Zipf et sources Markoviennes. Ann. Inst. H. Poincaré B 14 169--188.
• Sagitov, S. (1999). The general coalescent with asynchronous mergers of ancestral lines. J. Appl. Probab. 36 1116--1125.
• Schweinsberg, J. (2000). A necessary and sufficient condition for the $\Lambda$-coalescent to come down from infinity. Electron. Comm. Probab. 5 1--11.
• Schweinsberg, J. (2003). Coalescent processes obtained from supercritical Galton--Watson processes. Stochastic Process. Appl. 106 107--139.