Boundary behaviors for a class of continuous-state nonlinear branching processes in critical cases

Using Foster-Lyapunov techniques we establish new conditions on non-extinction, non-explosion, coming down from infinity and staying infinite, respectively, for the general continuous-state nonlinear branching processes introduced in Li et al. (2019). These results can be applied to identify boundary behaviors for the critical cases of the above nonlinear branching processes with power rate functions driven by Brownian motion and (or) stable Poisson random measure, which was left open in Li et al. (2019). In particular, we show that even in the critical cases, a phase transition happens between coming down from infinity and staying infinite.


Introduction
Continuous-state branching processes (CSBPs in short) are nonnegative-valued Markov processes satisfying the additive branching processes. They often arise as time-population scaling limits of discrete-state branching processes, and can also be obtained from spectrally positive Lévy processes via the Lamperti transform. The introduction of CSBP allows the applications of stochastic analysis, Lévy processes and stochastic differential equations (SDE in short) techniques to its study. We refer to Li (2011),  and Kyprianou (2012) and references therein for comprehensive reviews on CSBPs.
Generalized versions of the CSBP have been proposed in recent years to incorporate interactions between individuals and (or) between individuals and the population. A class of CSBPs with nonlinear branching mechanism, obtained by generalized Lamperti transform, is introduced in Li (2019). In , a more general version of the nonlinear CSBP is proposed as the solution to SDE where x > 0, a 0 and a 1 , a 2 ≥ 0 are Borel functions on [0, ∞), W (ds, du) andM (ds, dz, du) denote a Gaussian white noise and an independent compensated Poisson random measure, respectively.
The model of  corresponds to solution (X t ) t≥0 to SDE (1.1) with identical power rate functions a i , i = 0, 1, 2.
These nonlinear CSBPs allow richer behaviors such as coming down from infinity. Some extinction, extinguishing, explosion and coming down from infinity properties are proved in . By analyzing weighted occupation times for spectrally positive Lévy process, asymptotic results on the speeds of coming down from infinity and explosion are obtained in Foucart et al. (2019) and , respectively, for nonlinear CSBP corresponding to solution to SDE (1.1) with identical rate functions a 0 = a 1 = a 2 . Exponential ergodicity for the general continuous-state nonlinear branching processes in  is studied by Li and Wang (2020) using coupling techniques.
A version of SDE (1.1) with a 1 ≡ 0 and power functions a 0 and a 2 is considered earlier in Berestycki et al. (2015) where using the Lamperti transform, a necessary and sufficient condition for extinction is obtained and the pathwise uniqueness of solution is studied. Work on the continuous-state logistic branching process can be found in Lambert (2005), Le et al. (2013) and Le (2014).
Using a martingale approach, the extinction, explosion and coming down from infinity behaviors are further discussed in  and some rather sharp criteria in terms of a 0 , a 1 , a 2 and µ are obtained on characterization of different kinds of boundary behaviors for the nonlinear CSBPs as a Markov process. In Example 2.18 of  where a 0 , a 1 and a 2 are taken to be power functions andM is taken to be an α-stable Poison random measure with index α ∈ (1, 2). The above criteria are further expressed in terms of the coefficients and the powers of functions a i s and the stable index α. But for the critical cases, where the coefficients, the powers and the index α satisfy certain linear equations, the martingale approach fails and the corresponding boundary classification remains an open problem.
The main goal of this paper is to identify the exact boundary behaviors in the above mentioned critical cases for the solution (X) t≥0 to (1.1). For this purpose, we adapt the Foster-Lyapunov approach and select logarithm type test functions to obtain two new conditions under which the nonlinear CSBP never becomes extinct and never explodes, respectively. Similarly, for the boundary at infinity, we also find a Foster-Lyapunov condition with which we can show that an interesting phase transition occurs between coming down from infinity and staying infinite for different choices of coefficients and powers of the power functions and differential values of the stable index α.
The rest of the paper is arranged as follows. We introduce the generalized CSBPs with nonlinear branching in more details and present the main theorem in Section 2. The proofs of preliminary results on Foster-Lyapunov criteria and the main theorem are deferred to Section 3.

Main results
Let U be a Borel set on (0, ∞). Given σ-finite measures µ and ν on (0, ∞) such that (z ∧ z 2 )µ(dz) and (1 ∨ ln(1 + z))ν(dz) are finite measures on U and (0, ∞) \ U , respectively, we consider the following SDE that is a modification of (1.1): We only consider the solution of (2.1) before the minimum of their first times of hitting zero and reaching infinity (that is τ − 0 and τ + ∞ , which will be given in the following), respectively, i.e. The main purpose of this paper is to investigate the extinction, explosion and coming down from infinity behaviors, and the uniqueness of solution to SDE (2.1) is not required.
Throughout this paper we always assume that a 0 , a 1 , a 2 , a 3 are bounded on any bounded interval of [0, ∞) and that process (X t ) t≥0 is defined on filtered probability space (Ω, F , F t , P) which satisfies the usual hypotheses. We use P x to denote the law of a process started at x, and denote by E x the associated expectation. For a, b > 0 we define first passage times with the convention inf ∅ = ∞. Let C 2 ((0, ∞)) denote the space of twice continuously differentiable functions on (0, ∞).
We next introduce several functions. For u > 0 let For ρ, z > 0 and u > 3 let Process a < t} = 0 for all t > 0 and all large a; it comes down from infinity if lim a→∞ lim x→∞ P x {τ − a < t} = 1 for all t > 0. The following main result provides new criteria on non-extinction, non-explosion, coming down from infinity and staying infinite for the solution to SDE (1.1).
for some constant ρ > 0, then the process (X t ) t≥0 does not explode and it comes down from infinity.

Remark 2.2
Since the process (X t ) t≥0 does not explode and comes down from infinity under the assumptions of Theorem 2.1 (iv), then under additional assumptions on functions a i s, (X t ) t≥0 can be extended to a Feller process defined on state space [0, ∞] with ∞ as its entrance boundary; see Foucart et al. (2020).
Until the end of this section we focus on the special case that U = (0, ∞), a 0 , a 1 , a 2 are power functions and µ(dz) is an α-stable measure, that is We further estimate for which we first estimate ∞ 0 K ρ (u, z)µ(dz). Note that for y > 0 and f (y) := y −ρ + ρy − (ρ+ 1), by Taylor's formula we have Then by a change of variable, Observe that ln(1 + z) ≤ C(z ∧ √ z) for all z > 0 and some constant C > 0, which implies that for u > 3, Observe that in this critical case φ(u) = L(ln)(u) = 0, where operator L, to be defined in (3.2), denotes the generator of process X. This inspires us to choose logarithm type test functions for the main proofs.
As the main goal of this paper, applying Theorem 2.1 together with (2.4)-(2.5), we provide an answer to this open problem.

Proofs
Before presenting the proof of Theorem 2.1 we first prove some preliminary Foster-Lyapunov criteria. Suppose that g ∈ C 2 ((0, ∞)) satisfies sup z≥1,u≥v by Taylor's formula. By Itô's formula, Lg(X s )ds.
Lemma 3.1 Given 0 < a < x < b < ∞, for any function g ∈ C 2 ((a, b)) satisfying (3.1) and constant d a,b > 0 satisfying we have Proof. It follows from (3.3) that By Gronwall's lemma, which ends the proof. ✷ Lemma 3.2 Let (X t ) t≥0 be the solution to SDE (2.1).
(iii) If there exists a function g ∈ C 2 ((0, ∞)) strictly positive on [u, ∞) for all large u satisfying (3.1) and lim u→∞ g(u) = 0, and for any large a > 0, there is a constant d(a) > 0 such that Lg(u) ≤ d(a)g(u) for all u > a, then (X t ) t≥0 stays infinite.
Proof. We apply Lemma 3.1 for the proofs.
For part (i), (3.4) holds for all 0 < a < b and with d a,b replaced by d(b). Then using Fatou's lemma we have  > a and with d a,b replaced by d(a). Then using Fatou's lemma again we obtain Since lim u→∞ g(u) = ∞, then P x {τ + ∞ > t ∧ τ − a } = 1 for all t > 0. Letting t → ∞ we get P x {τ + ∞ ≥ τ − a } = 1, which implies the second assertion. For part (iii), given any large a > 0, (3.4) holds for all b > a and with d a,b replaced by d(a) again. We can also get which implies Since lim u→∞ g(u) = 0, then for all t, a > 0, for all t > 0. Then the process stays infinite. ✷ The next lemma provides a condition that associates the probability of coming down from infinity with the probability of non-explosion. Its proof is a modification of Proposition 2.  Then for any t > 0 Consequently, process (X t ) t≥0 comes down from infinity if there is no explosion.
Proof. The proof is a modification of that of Proposition 2.2 in Ren et al. (2019). We present the details for completeness. By (3.3), for all large a < b and then by integration by parts, which implies that Letting t → ∞ in the above inequality and using the dominated convergence we obtain It follows that Observing that lim x→∞ P x {τ − a < t} is increasing in a, the desired inequality then follows from the above inequality. ✷ We are now ready to show the proofs of the main results.
Then we can conclude the proof by the assumptions for this part together with Theorem 2.1(ii) and Lemma 3.3. ✷