## Electronic Journal of Probability

### Nonlinear randomized urn models: a stochastic approximation viewpoint

#### Abstract

This paper extends the link between stochastic approximation ($SA$) theory and randomized urn models developed in [32], and their applications to clinical trials introduced in [2, 3, 4]. We no longer assume that the drawing rule is uniform among the balls of the urn (which contains $d$ colors), but can be reinforced by a function $f$. This is a way to model risk aversion. Firstly, by considering that $f$ is concave or convex and by reformulating the dynamics of the urn composition as an $SA$ algorithm with remainder, we derive the $a.s.$ convergence and the asymptotic normality (Central Limit Theorem, $CLT$) of the normalized procedure by calling upon the so-called $ODE$ and $SDE$ methods. An in depth analysis of the case $d=2$ exhibits two different behaviors: a single equilibrium point when $f$ is concave, and, when $f$ is convex, a transition phase from a single attracting equilibrium to a system with two attracting and one repulsive equilibrium points. The last setting is solved using results on non-convergence toward noisy and noiseless “traps” in order to deduce the $a.s.$ convergence toward one of the attracting points. Secondly, the special case of a Pólya urn (when the addition rule is the $I_{d}$ matrix) is analyzed, still using result from $SA$ theory about “traps”. Finally, these results are used to solve another urn model with a more natural nonlinear drawing rule and we conclude by an example of application to optimal asset allocation in Finance.

#### Article information

Source
Electron. J. Probab., Volume 24 (2019), paper no. 98, 47 pp.

Dates
Accepted: 29 April 2019
First available in Project Euclid: 18 September 2019

https://projecteuclid.org/euclid.ejp/1568793792

Digital Object Identifier
doi:10.1214/19-EJP312

Zentralblatt MATH identifier
07107405

#### Citation

Laruelle, Sophie; Pagès, Gilles. Nonlinear randomized urn models: a stochastic approximation viewpoint. Electron. J. Probab. 24 (2019), paper no. 98, 47 pp. doi:10.1214/19-EJP312. https://projecteuclid.org/euclid.ejp/1568793792

#### References

• [1] K. B. Athreya and S. Karlin. Embedding of urn schemes into continuous time Markov branching processes and related limit theorems. Ann. Math. Statist., 39:1801–1817, 1968.
• [2] Z.-D. Bai and F. Hu. Asymptotic theorems for urn models with nonhomogeneous generating matrices. Stochastic Process. Appl., 80(1):87–101, 1999.
• [3] Z.-D. Bai and F. Hu. Asymptotics in randomized urn models. Ann. Appl. Probab., 15(1B):914–940, 2005.
• [4] Z.-D. Bai, F. Hu, and L. Shen. An adaptive design for multi-arm clinical trials. J. Multivariate Anal., 81(1):1–18, 2002.
• [5] M. Benaïm. Recursive algorithms, urn processes and chaining number of chain recurrent sets. Ergodic Theory Dynam. Systems, 18(1):53–87, 1998.
• [6] M. Benaïm. Dynamics of stochastic approximation algorithms. In Séminaire de Probabilités, XXXIII, volume 1709 of Lecture Notes in Math., pages 1–68. Springer, Berlin, 1999.
• [7] M. Benaïm, I. Benjamini, J. Chen, and Y. Lima. A generalized Pólya’s urn with graph based interactions. Random Structures Algorithms, 46(4):614–634, 2015.
• [8] M. Benaïm and M. W. Hirsch. Asymptotic pseudotrajectories and chain recurrent flows, with applications. J. Dynam. Differential Equations, 8(1):141–176, 1996.
• [9] A. Benveniste, M. Métivier, and P. Priouret. Adaptive algorithms and stochastic approximations, volume 22 of Applications of Mathematics (New York). Springer-Verlag, Berlin, 1990. Translated from the French by Stephen S. Wilson.
• [10] N.H. Bingham, C.M. Goldie, and J.L. Teugels. Regular Variation. Encyclopedia of Mathematics and its Applications. Cambridge University Press, 1989.
• [11] C. Bouton. Approximation gaussienne d’algorithmes stochastiques à dynamique markovienne. Ann. Inst. H. Poincaré Probab. Statist., 24(1):131–155, 1988.
• [12] O. Brandière and M. Duflo. Les algorithmes stochastiques contournent-ils les pièges? Ann. Inst. H. Poincaré Probab. Statist., 32(3):395–427, 1996.
• [13] B. Chauvin, C. Mailler, and N. Pouyanne. Smoothing equations for large Pólya urns. J. Theoret. Probab., 28(3):923–957, 2015.
• [14] B. Chauvin, N. Pouyanne, and R. Sahnoun. Limit distributions for large polya urns. Ann. Appl. Probab., 21(1):1–32, 2011.
• [15] J. Chen and C. Lucas. A generalized Pólya’s urn with graph based interactions: convergence at linearity. Electron. Commun. Probab., 19:no. 67, 13, 2014.
• [16] A. Collevecchio, C. Cotar, and M. LiCalzi. On a preferential attachment and generalized Pòlya’s urn model. Ann. Appl. Probab., 23(3):1219–1253, 2013.
• [17] M. Duflo. Algorithmes stochastiques, volume 23 of Mathématiques & Applications (Berlin) [Mathematics & Applications]. Springer-Verlag, Berlin, 1996.
• [18] M. Duflo. Random iterative models, volume 34 of Applications of Mathematics (New York). Springer-Verlag, Berlin, 1997. Translated from the 1990 French original by Stephen S. Wilson and revised by the author.
• [19] J.-C. Fort and G. Pagès. Convergence of stochastic algorithms: from the Kushner-Clark theorem to the Lyapounov functional method. Adv. in Appl. Probab., 28(4):1072–1094, 1996.
• [20] J.-C. Fort and G. Pagès. Decreasing step stochastic algorithms: a.s. behaviour of weighted empirical measures. Monte Carlo Methods Appl., 8(3):237–270, 2002.
• [21] D. A. Freedman. Bernard Friedman’s urn. Ann. Math. Statist, 36:956–970, 1965.
• [22] P. Hall and C. C. Heyde. Martingale limit theory and its application. Academic Press, Inc. [Harcourt Brace Jovanovich, Publishers], New York-London, 1980. Probability and Mathematical Statistics.
• [23] E. Häusler and H. Luschgy. Stable convergence and stable limit theorems, volume 74 of Probability Theory and Stochastic Modelling. Springer, Cham, 2015.
• [24] B. M. Hill, D. Lane, and W. Sudderth. Exchangeable urn processes. Ann. Probab., 15(4):1586–1592, 1987.
• [25] S. Janson. Functional limit theorems for multitype branching processes and generalized Pólya urns. Stochastic Process. Appl., 110(2):177–245, 2004.
• [26] M. Knape and R. Neininger. Pólya urns via the contraction method. Combin. Probab. Comput., 23(6):1148–1186, 2014.
• [27] H. J. Kushner and D. S. Clark. Stochastic approximation methods for constrained and unconstrained systems, volume 26 of Applied Mathematical Sciences. Springer-Verlag, New York, 1978.
• [28] H. J. Kushner and G. G. Yin. Stochastic approximation and recursive algorithms and applications, volume 35 of Applications of Mathematics (New York). Springer-Verlag, New York, second edition, 2003. Stochastic Modelling and Applied Probability.
• [29] D. Lamberton and G. Pagès. How fast is the bandit? Stoch. Anal. Appl., 26(3):603–623, 2008.
• [30] D. Lamberton and G. Pagès. A penalized bandit algorithm. Electron. J. Probab., 13:no. 13, 341–373, 2008.
• [31] D. Lamberton, G. Pagès, and P. Tarrès. When can the two-armed bandit algorithm be trusted? Ann. Appl. Probab., 14(3):1424–1454, 2004.
• [32] S. Laruelle and G. Pagès. Randomized urn models revisited using stochastic approximation. Ann. Appl. Probab., 23(4):1409–1436, 2013.
• [33] S. Laruelle and G. Pagès. Addendum and corrigendum to “Randomized urn models revisited using stochastic approximation” []. Ann. Appl. Probab., 27(2):1296–1298, 2017.
• [34] N. Lasmar, S. Mailler, and Selmi O. Multiple drawing multi-colour urns by stochastic approximation. J. of Appl. Probab., 55(1):254–281, 2018.
• [35] V. A. Lazarev. Convergence of stochastic approximation procedures in the case of regression equation with several roots. Problemy Peredachi Informatsii, 28(1):75–88, 1992.
• [36] L. Ljung. Analysis of recursive stochastic algorithms. IEEE Trans. Automatic Control, AC-22(4):551–575, 1977.
• [37] R. Pemantle. Nonconvergence to unstable points in urn models and stochastic approximations. Ann. Probab., 18(2):698–712, 1990.
• [38] N. Pouyanne. An algebraic approach to Pólya processes. Ann. Inst. Henri Poincaré, 44(2):293–323, 2008.
• [39] R. van der Hofstad, M. Holmes, A. Kuznetsov, and W. Ruszel. Strongly reinforced Pólya urns with graph-based competition. Ann. Appl. Probab., 26(4):2494–2539, 2016.
• [40] L.-X. Zhang. Central limit theorems of a recursive stochastic algorithm with applications to adaptive design. Ann. Appl. Prob., 26:3630–3658, 2016.
• [41] T. Zhu. Nonlinear Polya urn models and self-organizing processes. ProQuest LLC, Ann Arbor, MI, 2009. Thesis (Ph.D.)–University of Pennsylvania.