The Annals of Applied Probability

Stochastic approximation algorithms with constant step size whose average is cooperative

Michel Benaïm and Morris W. Hirsch

Full-text: Open access

Abstract

We consider stochastic approximation algorithms with constant step size whose average ordinary differential equation (ODE) is cooperative and irreducible. We show that, under mild conditions on the noise process, invariant measures and empirical occupations measures of the process weakly converge (as the time goes to infinity and the step size goes to zero) toward measures which are supported by stable equilibria of the ODE. These results are applied to analyzing the long-term behavior of a class of learning processes arising in game theory.

Article information

Source
Ann. Appl. Probab. Volume 9, Number 1 (1999), 216-241.

Dates
First available in Project Euclid: 21 August 2002

Permanent link to this document
http://projecteuclid.org/euclid.aoap/1029962603

Mathematical Reviews number (MathSciNet)
MR1682576

Digital Object Identifier
doi:10.1214/aoap/1029962603

Zentralblatt MATH identifier
0983.62046

Subjects
Primary: 62L20: Stochastic approximation
Secondary: 34C35 34F05: Equations and systems with randomness [See also 34K50, 60H10, 93E03] 93E35: Stochastic learning and adaptive control

Keywords
Stochastic approximation ordinary differential equation method cooperative vector fields large deviations weak convergence theory of learning in games

Citation

Benaïm, Michel; Hirsch, Morris W. Stochastic approximation algorithms with constant step size whose average is cooperative. The Annals of Applied Probability 9 (1999), no. 1, 216--241. doi:10.1214/aoap/1029962603. http://projecteuclid.org/euclid.aoap/1029962603.


Export citation

References

  • Azencott, R. and Ruget, G. (1977). M´elanges d'´equations differentielles et grands ´ecarts a la loi des grands nombres. Z. Wahrsch. Verw. Gebiete 38 1-54. Bena¨im, M. (1996a). A dy namical sy stem approach to stochastic approximations. Siam J. Control Optim. 34 437-472.
  • Bena¨im, M. (1998). Recursive algorithms, urn processes and chaining number of chain recurrent sets. Ergodic Theory Dy nam. Sy stems 18 53-87.
  • Bena¨im, M. and Hirsch, M. W. (1995). Dy namics of Morse Smale urn processes. Ergodic Theory Dy nam. Sy stems 15 1005-1030.
  • Bena¨im, M. and Hirsch, M. W. (1996). Asy mptotic pseudo-trajectories and chain-recurrent flows, with applications. J. Dy nam. Differential Equations 8 141-174.
  • Bena¨im, M. and Hirsch, M. W. (1997). Information games, Dy namical sy stems and stochastic approximation. Discussion paper for IIASA workshop Oct 17-20.
  • Benveniste, A. M´etivier, M. and Priouret, P. (1990). Stochastic Approximations and Adaptive Algorithms. Springer, New York.
  • Chow, S. N. and Hale, J. (1982). Methods of Bifurcation Theory. Springer, New York.
  • Conley, C. C. (1978). Isolated Invariant Sets and the Morse Index. Amer. Math. Soc., Providence, RI.
  • Duflo, M. (1996). Algorithmes Stochastiques. Springer, Berlin.
  • Duflo, M. (1997). Random Iterative Models. Springer, New York.
  • Dupuis, P. (1988). Large deviations analysis of some recursive algorithms with state dependent noise. Ann. Probab. 6 1509-1536.
  • Dupuis, P. and Ellis R. S. (1997). A Weak Convergence Approach to the Theory of Large Deviations. Wiley, New York.
  • Fort, J. C. and Pages, G. (1996). Asy mptotics behaviour of a Markovian constant step size algorithm. Siam J. Control Optim. To appear.
  • Freidlin, M. I. (1978). The averaging principle and theorems on large deviations. Russian Math. Survey s 33 117-176.
  • Freidlin, M. I. and Wentzell, A. D. (1984). Random Perturbation of Dy namical Sy stems. Springer, New York.
  • Fudenberg, D. and Levine, D. (1998). Theory of Learning in Games. MIT Press.
  • Hirsch, M. W. (1985). Sy stems of differential equations that are competitive or cooperative II: convergence almost every where. SIAM J. Math. Anal. 16 423-439. Hirsch, M. W. (1988a). Sy stems of differential equations which are competitive or cooperative III: competing species. Nonlinearity 1 51-71. Hirsch, M. W. (1988b). Stability and convergence in strongly monotone dy namical sy stems. J. Reine Angew. Math. 383 1-53.
  • Hirsch, M. W. (1996). Chain transitive sets for smooth strongly monotone maps. Preprint, Univ. California, Berkeley.
  • Jiang, J.-F. (1991). Attractors for strongly monotone flows. J. Math. Anal. Appl. 162 210-222.
  • Jiang, J.-F. and Yu, S.-X. (1995). Stable cy cles for attractors of strongly monotone discrete-time dy namical sy stems. Comm. Appl. Nonlinear Anal. 2 455-458.
  • Kifer, Y. (1988). Random Perturbation of Dy namical Sy stems. Birkh¨auser, Boston.
  • Kunze, H. and Siegel, D. (1994). A graph theoretic approach to monotonicity with respect to initial conditions. In Comparison Methods and Stability Theory (X. Liu and D. Siegel, eds.) Dekker, New York.
  • Kushner, H. J. and Clark, D. S. (1978). Stochastic Approximation Methods for Constrained and Unconstrained Sy stems. Springer, New York.
  • Ljung, L. (1977). Analy sis of recursive stochastic algorithms. IEEE Trans. Automat. Control AC22 551-575.
  • Mierczy ´nski, J. (1994). P-arcs in strongly monotone discrete-time dy namical sy stems. Differential Integral Equations 7 1473-1494.
  • Smith, H. (1995). Monotone Dy namical Sy stems. Amer. Math. Soc., Providence, RI.
  • Tak´a c, P. (1992). Domains of attraction of generic -limit sets for strongly monotone discretetime semigroups. J. Reine Angew. Math. 423 101-173.
  • Tere s c´ak, I. (1994). Dy namics of C1 smooth strongly monotone discrete-time dy namical sy stems. Preprint.
  • Varadhan, (1984). Large Deviations and Applications. SIAM, Philadelphia.
  • Weibull, J. W. (1995). Evolutionary Game Theory. MIT Press.