The Annals of Applied Probability

On the storage capacity of Hopfield models with correlated patterns

Matthias Löwe

Full-text: Open access

Abstract

We analyze the storage capacity of the Hopfield model with correlated patterns $(\xi_i^{\nu})$. We treat both the case of semantically and spatially correlated patterns (i.e., the patterns are either correlated in $\nu$ but independent in i or vice versa). We show that the standard Hopfield model of neural networks with N neurons can store $N/(\gamma \log N)$ or $\alpha N$ correlated patterns (depending on which notion of storage is used), provided that the correlation comes from a homogeneous Markov chain. This answers the open question whether the standard Hopfield model can store any increasing number of correlated patterns at all in the affirmative. While our bound on the critical value for $\alpha$ decreases with large correlations, the critical $\gamma$ behaves differently for the different types of correlations.

Article information

Source
Ann. Appl. Probab., Volume 8, Number 4 (1998), 1216-1250.

Dates
First available in Project Euclid: 9 August 2002

Permanent link to this document
https://projecteuclid.org/euclid.aoap/1028903378

Digital Object Identifier
doi:10.1214/aoap/1028903378

Mathematical Reviews number (MathSciNet)
MR1661188

Zentralblatt MATH identifier
0941.60090

Subjects
Primary: 82C32: Neural nets [See also 68T05, 91E40, 92B20]
Secondary: 82B44: Disordered systems (random Ising models, random Schrödinger operators, etc.) 60K35: Interacting random processes; statistical mechanics type models; percolation theory [See also 82B43, 82C43]

Keywords
Hopfield model neural networks storage capacity Markov chains large deviations

Citation

Löwe, Matthias. On the storage capacity of Hopfield models with correlated patterns. Ann. Appl. Probab. 8 (1998), no. 4, 1216--1250. doi:10.1214/aoap/1028903378. https://projecteuclid.org/euclid.aoap/1028903378


Export citation

References

  • [1] Amit, D. J. (1987). The properties of models of simple neural networks. Heidelberg Colloquium on Glassy Dy namics. Lecture Notes in Phy s. 275. Springer, Berlin.
  • [2] Amit, D. J., Gutfreund, G. and Sompolinsky, H. (1985). Spin-glass models of neural networks. Phy s. Rev. A 32 1007-1018.
  • [3] Amit, D. J., Gutfreund, G. and Sompolinsky, H. (1987). Statistical mechanics of neural networks near saturation. Ann. physics 173 30-67.
  • [4] Bovier, A. and Gay rard, V. (1992). Rigorous bounds on the storage capacity of the dilute Hopfield model. J. Statist. Phy s. 69 597-627.
  • [5] Bovier, A. and Gay rard, V. (1996). An almost sure large deviation principle for the Hopfield model. Ann. Probab. 24 1444-1475.
  • [6] Bovier, A. and Gay rard, V. (1997). Hopfield models as a generalized mean field model. In Mathematics of Spin Glasses and Neural Networks (A. Bovier and P. Picco, eds.). Birkh¨auser, Boston. To appear.
  • [7] Bovier, A. and Gay rard, V. (1997). The retrieval phase of the Hopfield model. Probab. Theory Related Fields 107 61-98.
  • [8] Bovier, A., Gay rard, V. and Picco, P. (1994). Gibbs states for the Hopfield model in the regime of perfect memory. Probab. Theory Related Fields 100 329-363.
  • [9] Bovier, A., Gay rard, V. and Picco, P. (1995). Large deviation principles for the Hopfield model and the Kac-Hopfield model. Probab. Theory Related Fields 101 511-546.
  • [10] Bovier, A., Gay rard, V. and Picco, P. (1995). Gibbs states for the Hopfield model with extensively many patterns. J. Statist. Phy s. 79 395-414.
  • [11] Drey fus, G., Guy on, I. and Personnaz, L. (1986). Neural network design for efficient infor
  • mation retrieval. Disordered sy stems and biological organization (Les Houches, 1985). NATO Adv. Sci. Inst. Ser. F Comput. Sy stems Sci. 20 227-231.
  • [12] Gentz, B. (1996). A central limit theorem for the overlap parameter in the Hopfield model. Ann. Probab. 24 1809-1841.
  • [13] Georgii, H.-O. (1988). Gibbs measures and phase transition. In Studies in Mathematics 9 (H. V. Bauer, J. Heinz-Kazden and E. Zehnder, eds.). de Gruy ter, Berlin.
  • [14] Hopfield, J. J. (1982). Neural networks and physical sy stems with emergent collective computational abilities. Proc. Nat. Acad. Sci. U.S.A. 79 2554-2558.
  • [15] K ¨uhn, R. and Steffan, H. (1994). Replica sy mmetry breaking in attractor neural network models. Z. Phy s. B 95 249-260.
  • [16] Loukianova, D. (1994). Capacit´e de m´emoire dans le mod ele de Hopfield. C.R. Acad. Sci. Paris 318 157-160.
  • [17] Loukianova, D. (1997). Lower bounds on the restitution error in the Hopfield model. Probab. Theory Related Fields 107 161-176.
  • [18] McEliece, R., Posner, E., Rodemich, E. and Venkatesh, S. (1987). The capacity of the Hopfield associative memory. IEEE Trans. Inform. Theory 33 461-482.
  • [19] Miy ashita, Y. (1988). Neuronal correlate of visual associative long term memory in the primate temporal cortex. Nature 335 817-819.
  • [20] Monasson, R. (1992). Properties of neural networks storing spatially correlated patterns. J. Phy s. A Math. Gen. 335 3701-3720.
  • [21] Newman, C. (1988). Memory capacity in neural networks. Neural Networks 1 223-238.
  • [22] Pastur, L. A. and Figotin, A. L. (1977). Exactly soluble model of a spin-glass. Soviet J. of Low Temperature Phy s. 3 378-383.
  • [23] Petritis, D. (1995). Thermody namic formalism of neural computing. Univ. Rennes I. Preprint.
  • [24] Sandmeier, M. (1997). On the storage capacity of neural networks with temporal association. Ph.D. thesis. Univ. Bielefeld.
  • [25] Talagrand, M. (1995). R´esultats rigoureux pour le mod ele de Hopfield. C.R. Acad. Sci. Paris Ser. I 321 309-312.
  • [26] Talagrand, M. (1996). Rigorous results of the Hopfield model with many patterns. Preprint.
  • [27] Tarkowski, W. and Lewenstein, M. (1993). Storage of sets of correlated data in neural network memories. J. Phy s. A Math. Gen. 26 2453-2469.
  • [28] van Hemmen, L. and K ¨uhn, R. (1991). Collective phenomena in neural networks. In Models of Neural Networks (E. Domany, L. v. Hemmen and R. Schulte, eds.). Springer, Berlin.