The Annals of Applied Probability

On the storage capacity of Hopfield models with correlated patterns

Matthias Löwe

Full-text: Open access

PDF File (212 KB)

Abstract

We analyze the storage capacity of the Hopfield model with correlated patterns $(\xi_i^{\nu})$. We treat both the case of semantically and spatially correlated patterns (i.e., the patterns are either correlated in $\nu$ but independent in i or vice versa). We show that the standard Hopfield model of neural networks with N neurons can store $N/(\gamma \log N)$ or $\alpha N$ correlated patterns (depending on which notion of storage is used), provided that the correlation comes from a homogeneous Markov chain. This answers the open question whether the standard Hopfield model can store any increasing number of correlated patterns at all in the affirmative. While our bound on the critical value for $\alpha$ decreases with large correlations, the critical $\gamma$ behaves differently for the different types of correlations.

Article information

Source
Ann. Appl. Probab., Volume 8, Number 4 (1998), 1216-1250.

Dates
First available in Project Euclid: 9 August 2002

Permanent link to this document
https://projecteuclid.org/euclid.aoap/1028903378

Digital Object Identifier
doi:10.1214/aoap/1028903378

Mathematical Reviews number (MathSciNet)
MR1661188

Zentralblatt MATH identifier
0941.60090

Subjects
Primary: 82C32: Neural nets [See also 68T05, 91E40, 92B20]
Secondary: 82B44: Disordered systems (random Ising models, random Schrödinger operators, etc.) 60K35: Interacting random processes; statistical mechanics type models; percolation theory [See also 82B43, 82C43]

Keywords
Hopfield model neural networks storage capacity Markov chains large deviations

Citation

Löwe, Matthias. On the storage capacity of Hopfield models with correlated patterns. Ann. Appl. Probab. 8 (1998), no. 4, 1216--1250. doi:10.1214/aoap/1028903378. https://projecteuclid.org/euclid.aoap/1028903378


Export citation

References

  • [1] Amit, D. J. (1987). The properties of models of simple neural networks. Heidelberg Colloquium on Glassy Dy namics. Lecture Notes in Phy s. 275. Springer, Berlin.
    Mathematical Reviews (MathSciNet): MR90a:92011
    Digital Object Identifier: doi:10.1007/BFb0057530
  • [2] Amit, D. J., Gutfreund, G. and Sompolinsky, H. (1985). Spin-glass models of neural networks. Phy s. Rev. A 32 1007-1018.
    Mathematical Reviews (MathSciNet): MR86g:92015
    Digital Object Identifier: doi:10.1103/PhysRevA.32.1007
  • [3] Amit, D. J., Gutfreund, G. and Sompolinsky, H. (1987). Statistical mechanics of neural networks near saturation. Ann. physics 173 30-67.
  • [4] Bovier, A. and Gay rard, V. (1992). Rigorous bounds on the storage capacity of the dilute Hopfield model. J. Statist. Phy s. 69 597-627.
    Mathematical Reviews (MathSciNet): MR93k:82049
    Zentralblatt MATH: 0900.82064
    Digital Object Identifier: doi:10.1007/BF01050427
  • [5] Bovier, A. and Gay rard, V. (1996). An almost sure large deviation principle for the Hopfield model. Ann. Probab. 24 1444-1475.
    Mathematical Reviews (MathSciNet): MR98g:60055
    Zentralblatt MATH: 0871.60022
    Digital Object Identifier: doi:10.1214/aop/1065725188
    Project Euclid: euclid.aop/1065725188
  • [6] Bovier, A. and Gay rard, V. (1997). Hopfield models as a generalized mean field model. In Mathematics of Spin Glasses and Neural Networks (A. Bovier and P. Picco, eds.). Birkh¨auser, Boston. To appear.
  • [7] Bovier, A. and Gay rard, V. (1997). The retrieval phase of the Hopfield model. Probab. Theory Related Fields 107 61-98.
    Zentralblatt MATH: 0866.60085
    Mathematical Reviews (MathSciNet): MR1427717
    Digital Object Identifier: doi:10.1007/s004400050077
  • [8] Bovier, A., Gay rard, V. and Picco, P. (1994). Gibbs states for the Hopfield model in the regime of perfect memory. Probab. Theory Related Fields 100 329-363.
    Mathematical Reviews (MathSciNet): MR96c:60115
    Zentralblatt MATH: 0810.60094
    Digital Object Identifier: doi:10.1007/BF01193704
  • [9] Bovier, A., Gay rard, V. and Picco, P. (1995). Large deviation principles for the Hopfield model and the Kac-Hopfield model. Probab. Theory Related Fields 101 511-546.
    Mathematical Reviews (MathSciNet): MR97a:60141
    Zentralblatt MATH: 0826.60090
    Digital Object Identifier: doi:10.1007/BF01202783
  • [10] Bovier, A., Gay rard, V. and Picco, P. (1995). Gibbs states for the Hopfield model with extensively many patterns. J. Statist. Phy s. 79 395-414.
    Mathematical Reviews (MathSciNet): MR96c:82033
    Zentralblatt MATH: 1081.82570
    Digital Object Identifier: doi:10.1007/BF02179395
  • [11] Drey fus, G., Guy on, I. and Personnaz, L. (1986). Neural network design for efficient infor
  • mation retrieval. Disordered sy stems and biological organization (Les Houches, 1985). NATO Adv. Sci. Inst. Ser. F Comput. Sy stems Sci. 20 227-231.
  • [12] Gentz, B. (1996). A central limit theorem for the overlap parameter in the Hopfield model. Ann. Probab. 24 1809-1841.
    Mathematical Reviews (MathSciNet): MR1415230
    Zentralblatt MATH: 0872.60015
    Digital Object Identifier: doi:10.1214/aop/1041903207
    Project Euclid: euclid.aop/1041903207
  • [13] Georgii, H.-O. (1988). Gibbs measures and phase transition. In Studies in Mathematics 9 (H. V. Bauer, J. Heinz-Kazden and E. Zehnder, eds.). de Gruy ter, Berlin.
    Mathematical Reviews (MathSciNet): MR89k:82010
    Zentralblatt MATH: 0657.60122
  • [14] Hopfield, J. J. (1982). Neural networks and physical sy stems with emergent collective computational abilities. Proc. Nat. Acad. Sci. U.S.A. 79 2554-2558.
    Mathematical Reviews (MathSciNet): MR83g:92024
    Digital Object Identifier: doi:10.1073/pnas.79.8.2554
    JSTOR: links.jstor.org
  • [15] K ¨uhn, R. and Steffan, H. (1994). Replica sy mmetry breaking in attractor neural network models. Z. Phy s. B 95 249-260.
  • [16] Loukianova, D. (1994). Capacit´e de m´emoire dans le mod ele de Hopfield. C.R. Acad. Sci. Paris 318 157-160.
  • [17] Loukianova, D. (1997). Lower bounds on the restitution error in the Hopfield model. Probab. Theory Related Fields 107 161-176.
    Mathematical Reviews (MathSciNet): MR98b:60172
    Zentralblatt MATH: 0870.60021
    Digital Object Identifier: doi:10.1007/s004400050081
  • [18] McEliece, R., Posner, E., Rodemich, E. and Venkatesh, S. (1987). The capacity of the Hopfield associative memory. IEEE Trans. Inform. Theory 33 461-482.
    Mathematical Reviews (MathSciNet): MR88h:92019
    Digital Object Identifier: doi:10.1109/TIT.1987.1057328
  • [19] Miy ashita, Y. (1988). Neuronal correlate of visual associative long term memory in the primate temporal cortex. Nature 335 817-819.
  • [20] Monasson, R. (1992). Properties of neural networks storing spatially correlated patterns. J. Phy s. A Math. Gen. 335 3701-3720.
    Zentralblatt MATH: 0791.68140
    Mathematical Reviews (MathSciNet): MR1172070
    Digital Object Identifier: doi:10.1088/0305-4470/25/13/019
  • [21] Newman, C. (1988). Memory capacity in neural networks. Neural Networks 1 223-238.
  • [22] Pastur, L. A. and Figotin, A. L. (1977). Exactly soluble model of a spin-glass. Soviet J. of Low Temperature Phy s. 3 378-383.
  • [23] Petritis, D. (1995). Thermody namic formalism of neural computing. Univ. Rennes I. Preprint.
  • [24] Sandmeier, M. (1997). On the storage capacity of neural networks with temporal association. Ph.D. thesis. Univ. Bielefeld.
  • [25] Talagrand, M. (1995). R´esultats rigoureux pour le mod ele de Hopfield. C.R. Acad. Sci. Paris Ser. I 321 309-312.
  • [26] Talagrand, M. (1996). Rigorous results of the Hopfield model with many patterns. Preprint.
    Mathematical Reviews (MathSciNet): MR1609015
    Zentralblatt MATH: 0897.60041
    Digital Object Identifier: doi:10.1007/s004400050148
  • [27] Tarkowski, W. and Lewenstein, M. (1993). Storage of sets of correlated data in neural network memories. J. Phy s. A Math. Gen. 26 2453-2469.
    Mathematical Reviews (MathSciNet): MR94f:82059
    Zentralblatt MATH: 0788.68118
    Digital Object Identifier: doi:10.1088/0305-4470/26/11/001
  • [28] van Hemmen, L. and K ¨uhn, R. (1991). Collective phenomena in neural networks. In Models of Neural Networks (E. Domany, L. v. Hemmen and R. Schulte, eds.). Springer, Berlin.
    Mathematical Reviews (MathSciNet): MR1107530
    Zentralblatt MATH: 0783.92006

The Institute of Mathematical Statistics

The Annals of Applied Probability

New content alerts

Email RSS ToC RSS Article
  • You have access to this content.
  • You have partial access to this content.
  • You do not have access to this content.
Turn Off MathJax

What is MathJax?

More like this

+ See more