Stochastic approximation algorithms with constant step size whose average is cooperative
Michel Benaïm and Morris W. Hirsch
Source: Ann. Appl. Probab. Volume 9, Number 1
(1999), 216-241.
Abstract
We consider stochastic approximation algorithms with constant step size whose average ordinary differential equation (ODE) is cooperative and irreducible. We show that, under mild conditions on the noise process, invariant measures and empirical occupations measures of the process weakly converge (as the time goes to infinity and the step size goes to zero) toward measures which are supported by stable equilibria of the ODE. These results are applied to analyzing the long-term behavior of a class of learning processes arising in game theory.
First Page:
Show
Hide
Keywords: Stochastic approximation; ordinary differential equation method; cooperative vector fields; large deviations; weak convergence; theory of learning in games
Full-text: Open access
Links and Identifiers
Permanent link to this document: http://projecteuclid.org/euclid.aoap/1029962603
Mathematical Reviews number (MathSciNet): MR1682576
Digital Object Identifier: doi:10.1214/aoap/1029962603
Zentralblatt MATH identifier: 0983.62046
References
Azencott, R. and Ruget, G. (1977). M´elanges d'´equations differentielles et grands ´ecarts a la loi des grands nombres. Z. Wahrsch. Verw. Gebiete 38 1-54. Bena¨im, M. (1996a). A dy namical sy stem approach to stochastic approximations. Siam J. Control Optim. 34 437-472.
Bena¨im, M. (1998). Recursive algorithms, urn processes and chaining number of chain recurrent sets. Ergodic Theory Dy nam. Sy stems 18 53-87.
Mathematical Reviews (MathSciNet): MR99f:62100
Zentralblatt MATH: 0921.60061
Digital Object Identifier: doi:10.1017/S0143385798097557
Bena¨im, M. and Hirsch, M. W. (1995). Dy namics of Morse Smale urn processes. Ergodic Theory Dy nam. Sy stems 15 1005-1030.
Mathematical Reviews (MathSciNet): MR97f:62138
Zentralblatt MATH: 0846.60054
Bena¨im, M. and Hirsch, M. W. (1996). Asy mptotic pseudo-trajectories and chain-recurrent flows, with applications. J. Dy nam. Differential Equations 8 141-174.
Mathematical Reviews (MathSciNet): MR97d:58165
Zentralblatt MATH: 0878.58053
Digital Object Identifier: doi:10.1007/BF02218617
Bena¨im, M. and Hirsch, M. W. (1997). Information games, Dy namical sy stems and stochastic approximation. Discussion paper for IIASA workshop Oct 17-20.
Benveniste, A. M´etivier, M. and Priouret, P. (1990). Stochastic Approximations and Adaptive Algorithms. Springer, New York.
Chow, S. N. and Hale, J. (1982). Methods of Bifurcation Theory. Springer, New York.
Mathematical Reviews (MathSciNet): MR660633
Zentralblatt MATH: 0487.47039
Conley, C. C. (1978). Isolated Invariant Sets and the Morse Index. Amer. Math. Soc., Providence, RI.
Mathematical Reviews (MathSciNet): MR80c:58009
Zentralblatt MATH: 0397.34056
Duflo, M. (1996). Algorithmes Stochastiques. Springer, Berlin.
Mathematical Reviews (MathSciNet): MR99e:62141
Zentralblatt MATH: 0849.62043
Duflo, M. (1997). Random Iterative Models. Springer, New York.
Mathematical Reviews (MathSciNet): MR98m:62239
Zentralblatt MATH: 0868.62069
Dupuis, P. (1988). Large deviations analysis of some recursive algorithms with state dependent noise. Ann. Probab. 6 1509-1536.
Mathematical Reviews (MathSciNet): MR89i:60062
Zentralblatt MATH: 0661.60045
Digital Object Identifier: doi:10.1214/aop/1176991581
Project Euclid: euclid.aop/1176991581
Dupuis, P. and Ellis R. S. (1997). A Weak Convergence Approach to the Theory of Large Deviations. Wiley, New York.
Mathematical Reviews (MathSciNet): MR99f:60057
Zentralblatt MATH: 0904.60001
Fort, J. C. and Pages, G. (1996). Asy mptotics behaviour of a Markovian constant step size algorithm. Siam J. Control Optim. To appear.
Freidlin, M. I. (1978). The averaging principle and theorems on large deviations. Russian Math. Survey s 33 117-176.
Mathematical Reviews (MathSciNet): MR81k:60059
Freidlin, M. I. and Wentzell, A. D. (1984). Random Perturbation of Dy namical Sy stems. Springer, New York.
Mathematical Reviews (MathSciNet): MR85a:60064
Fudenberg, D. and Levine, D. (1998). Theory of Learning in Games. MIT Press.
Mathematical Reviews (MathSciNet): MR99i:90121
Zentralblatt MATH: 0939.91004
Hirsch, M. W. (1985). Sy stems of differential equations that are competitive or cooperative II: convergence almost every where. SIAM J. Math. Anal. 16 423-439. Hirsch, M. W. (1988a). Sy stems of differential equations which are competitive or cooperative III: competing species. Nonlinearity 1 51-71. Hirsch, M. W. (1988b). Stability and convergence in strongly monotone dy namical sy stems. J. Reine Angew. Math. 383 1-53.
Mathematical Reviews (MathSciNet): MR87a:58137
Zentralblatt MATH: 0658.34023
Digital Object Identifier: doi:10.1137/0516030
Hirsch, M. W. (1996). Chain transitive sets for smooth strongly monotone maps. Preprint, Univ. California, Berkeley.
Jiang, J.-F. (1991). Attractors for strongly monotone flows. J. Math. Anal. Appl. 162 210-222.
Mathematical Reviews (MathSciNet): MR1135272
Zentralblatt MATH: 0754.34055
Digital Object Identifier: doi:10.1016/0022-247X(91)90188-6
Jiang, J.-F. and Yu, S.-X. (1995). Stable cy cles for attractors of strongly monotone discrete-time dy namical sy stems. Comm. Appl. Nonlinear Anal. 2 455-458.
Kifer, Y. (1988). Random Perturbation of Dy namical Sy stems. Birkh¨auser, Boston.
Mathematical Reviews (MathSciNet): MR91e:58159
Kunze, H. and Siegel, D. (1994). A graph theoretic approach to monotonicity with respect to initial conditions. In Comparison Methods and Stability Theory (X. Liu and D. Siegel, eds.) Dekker, New York.
Mathematical Reviews (MathSciNet): MR95g:34065
Kushner, H. J. and Clark, D. S. (1978). Stochastic Approximation Methods for Constrained and Unconstrained Sy stems. Springer, New York.
Mathematical Reviews (MathSciNet): MR80g:62065
Ljung, L. (1977). Analy sis of recursive stochastic algorithms. IEEE Trans. Automat. Control AC22 551-575.
Mathematical Reviews (MathSciNet): MR57:5358
Digital Object Identifier: doi:10.1109/TAC.1977.1101561
Mierczy ´nski, J. (1994). P-arcs in strongly monotone discrete-time dy namical sy stems. Differential Integral Equations 7 1473-1494.
Mathematical Reviews (MathSciNet): MR95c:58121
Smith, H. (1995). Monotone Dy namical Sy stems. Amer. Math. Soc., Providence, RI.
Mathematical Reviews (MathSciNet): MR1319817
Tak´a c, P. (1992). Domains of attraction of generic -limit sets for strongly monotone discretetime semigroups. J. Reine Angew. Math. 423 101-173.
Mathematical Reviews (MathSciNet): MR93c:58127
Zentralblatt MATH: 0729.54022
Tere s c´ak, I. (1994). Dy namics of C1 smooth strongly monotone discrete-time dy namical sy stems. Preprint.
Varadhan, (1984). Large Deviations and Applications. SIAM, Philadelphia.
Mathematical Reviews (MathSciNet): MR86h:60067b
Zentralblatt MATH: 0549.60023
Weibull, J. W. (1995). Evolutionary Game Theory. MIT Press.
Mathematical Reviews (MathSciNet): MR96f:90004
Zentralblatt MATH: 0879.90206
The Annals of Applied Probability
