Nonlinear randomized urn models: a stochastic approximation viewpoint

Abstract

This paper extends the link between stochastic approximation ($SA$) theory and randomized urn models developed in [32], and their applications to clinical trials introduced in [2, 3, 4]. We no longer assume that the drawing rule is uniform among the balls of the urn (which contains $d$ colors), but can be reinforced by a function $f$. This is a way to model risk aversion. Firstly, by considering that $f$ is concave or convex and by reformulating the dynamics of the urn composition as an $SA$ algorithm with remainder, we derive the $a.s.$ convergence and the asymptotic normality (Central Limit Theorem, $CLT$) of the normalized procedure by calling upon the so-called $ODE$ and $SDE$ methods. An in depth analysis of the case $d=2$ exhibits two different behaviors: a single equilibrium point when $f$ is concave, and, when $f$ is convex, a transition phase from a single attracting equilibrium to a system with two attracting and one repulsive equilibrium points. The last setting is solved using results on non-convergence toward noisy and noiseless “traps” in order to deduce the $a.s.$ convergence toward one of the attracting points. Secondly, the special case of a Pólya urn (when the addition rule is the $I_{d}$ matrix) is analyzed, still using result from $SA$ theory about “traps”. Finally, these results are used to solve another urn model with a more natural nonlinear drawing rule and we conclude by an example of application to optimal asset allocation in Finance.