## The Annals of Applied Probability

### The coalescent point process of branching trees

#### Abstract

We define a doubly infinite, monotone labeling of Bienaymé–Galton–Watson (BGW) genealogies. The genealogy of the current generation backwards in time is uniquely determined by the coalescent point process $(A_{i};i\ge1)$, where $A_{i}$ is the coalescence time between individuals $i$ and $i+1$. There is a Markov process of point measures $(B_{i};i\ge1)$ keeping track of more ancestral relationships, such that $A_{i}$ is also the first point mass of $B_{i}$.

This process of point measures is also closely related to an inhomogeneous spine decomposition of the lineage of the first surviving particle in generation $h$ in a planar BGW tree conditioned to survive $h$ generations. The decomposition involves a point measure $\rho$ storing the number of subtrees on the right-hand side of the spine. Under appropriate conditions, we prove convergence of this point measure to a point measure on $\mathbb{R}_{+}$ associated with the limiting continuous-state branching (CSB) process. We prove the associated invariance principle for the coalescent point process, after we discretize the limiting CSB population by considering only points with coalescence times greater than $\varepsilon$.

The limiting coalescent point process $(B^{\varepsilon}_{i};i\ge1)$ is the sequence of depths greater than $\varepsilon$ of the excursions of the height process below some fixed level. In the diffusion case, there are no multiple ancestries and (it is known that) the coalescent point process is a Poisson point process with an explicit intensity measure. We prove that in the general case the coalescent process with multiplicities $(B^{\varepsilon}_{i};i\ge1)$ is a Markov chain of point masses and we give an explicit formula for its transition function.

The paper ends with two applications in the discrete case. Our results show that the sequence of $A_{i}$’s are i.i.d. when the offspring distribution is linear fractional. Also, the law of Yaglom’s quasi-stationary population size for subcritical BGW processes is disintegrated with respect to the time to most recent common ancestor of the whole population.

#### Article information

Source
Ann. Appl. Probab., Volume 23, Number 1 (2013), 99-144.

Dates
First available in Project Euclid: 25 January 2013

https://projecteuclid.org/euclid.aoap/1359124383

Digital Object Identifier
doi:10.1214/11-AAP820

Mathematical Reviews number (MathSciNet)
MR3059232

Zentralblatt MATH identifier
1268.60107

#### Citation

Lambert, Amaury; Popovic, Lea. The coalescent point process of branching trees. Ann. Appl. Probab. 23 (2013), no. 1, 99--144. doi:10.1214/11-AAP820. https://projecteuclid.org/euclid.aoap/1359124383

#### References

• [1] Aldous, D. and Popovic, L. (2005). A critical branching process model for biodiversity. Adv. in Appl. Probab. 37 1094–1115.
• [2] Athreya, K. B. and Ney, P. E. (1972). Branching Processes. Die Grundlehren der Mathematischen Wissenschaften, Band 196. Springer, New York.
• [3] Bennies, J. and Kersting, G. (2000). A random walk approach to Galton–Watson trees. J. Theoret. Probab. 13 777–803.
• [4] Bertoin, J. and Le Gall, J.-F. (2000). The Bolthausen–Sznitman coalescent and the genealogy of continuous-state branching processes. Probab. Theory Related Fields 117 249–266.
• [5] Bertoin, J. and Le Gall, J.-F. (2006). Stochastic flows associated to coalescent processes. III. Limit theorems. Illinois J. Math. 50 147–181 (electronic).
• [6] Birkner, M., Blath, J., Capaldo, M., Etheridge, A., Möhle, M., Schweinsberg, J. and Wakolbinger, A. (2005). Alpha-stable branching and beta-coalescents. Electron. J. Probab. 10 303–325 (electronic).
• [7] Chauvin, B., Rouault, A. and Wakolbinger, A. (1991). Growing conditioned trees. Stochastic Process. Appl. 39 117–130.
• [8] Duquesne, T. and Le Gall, J.-F. (2002). Random trees, Lévy processes and spatial branching processes. Astérisque 281 vi+147.
• [9] Evans, S. N. (1993). Two representations of a conditioned superprocess. Proc. Roy. Soc. Edinburgh Sect. A 123 959–971.
• [10] Fleischmann, K. and Siegmund-Schultze, R. (1977). The structure of reduced critical Galton–Watson processes. Math. Nachr. 79 233–241.
• [11] Geiger, J. (1996). Size-biased and conditioned random splitting trees. Stochastic Process. Appl. 65 187–207.
• [12] Geiger, J. (1999). Elementary new proofs of classical limit theorems for Galton–Watson processes. J. Appl. Probab. 36 301–309.
• [13] Gernhard, T. (2008). The conditioned reconstructed process. J. Theoret. Biol. 253 769–778.
• [14] Grimvall, A. (1974). On the convergence of sequences of branching processes. Ann. Probab. 2 1027–1045.
• [15] Kallenberg, O. (1977). Stability of critical cluster fields. Math. Nachr. 77 7–43.
• [16] Kallenberg, O. (2002). Foundations of Modern Probability, 2nd ed. Springer, New York.
• [17] Kingman, J. F. C. (1982). The coalescent. Stochastic Process. Appl. 13 235–248.
• [18] Lambert, A. (2002). The genealogy of continuous-state branching processes with immigration. Probab. Theory Related Fields 122 42–70.
• [19] Lambert, A. (2003). Coalescence times for the branching process. Adv. in Appl. Probab. 35 1071–1089.
• [20] Lambert, A. (2007). Quasi-stationary distributions and the continuous-state branching process conditioned to be never extinct. Electron. J. Probab. 12 420–446.
• [21] Lambert, A. (2008). Population dynamics and random genealogies. Stoch. Models 24 45–163.
• [22] Lambert, A. (2009). The allelic partition for coalescent point processes. Markov Process. Related Fields 15 359–386.
• [23] Lambert, A. (2010). The contour of splitting trees is a Lévy process. Ann. Probab. 38 348–395.
• [24] Le Gall, J.-F. and Le Jan, Y. (1998). Branching processes in Lévy processes: The exploration process. Ann. Probab. 26 213–252.
• [25] Lyons, R., Pemantle, R. and Peres, Y. (1995). Conceptual proofs of $L\log L$ criteria for mean behavior of branching processes. Ann. Probab. 23 1125–1138.
• [26] Popovic, L. (2004). Asymptotic genealogy of a critical branching process. Ann. Appl. Probab. 14 2120–2148.
• [27] Rannala, B. (1997). On the genealogy of a rare allele. Theoret. Popul. Biol. 52 216–223.
• [28] Stadler, T. (2009). On incomplete sampling under birth–death models and connections to the sampling-based coalescent. J. Theoret. Biol. 261 58–66.