L ARGE A SSOCIATIVE M EMORY P ROBLEM IN N EUROBIOLOGY AND M ACHINE L EARNING
A BSTRACT
Dense Associative Memories or modern Hopﬁeld networks permit storage and reliable retrieval of an exponentially large (in the dimension of feature space) number of memories.
At the same time, their naive implementation is non-biological, since it seemingly requires the existence of many-body synaptic junctions between the neurons.
We show that these models are effective descriptions of a more microscopic (written in terms of biological degrees of freedom) theory that has additional (hidden) neurons and only requires two-body interactions between them.
For this reason our proposed microscopic theory is a valid model of large associative memory with a degree of biological plausibility.
The dynamics of our network and its reduced dimensional equivalent both minimize energy (Lyapunov) functions.
When certain dynamical variables (hidden neurons) are integrated out from our microscopic theory, one can recover many of the models that were previously discussed in the literature, e.g. the model presented in “Hopﬁeld Networks is All You Need” paper.
We also provide an alternative derivation of the energy function and the update rule proposed in the aforementioned paper and clarify the relationships between various models of this class.
I NTRODUCTION
Associative memory is deﬁned in psychology as the ability to remember (link) many sets, called memories, of unrelated items.
Prompted by a large enough subset of items taken from one memory, an animal or computer with an associative memory can retrieve the rest of the items belonging to that memory.
The diverse human cognitive abilities which involve making appropriate responses to stimulus patterns can often be understood as the operation of an associative memory, with the “memories” often being distillations and consolidations of multiple experiences rather than merely corresponding to a single event.
The intuitive idea of associative memory can be described using a “feature space”.
In a mathematical model abstracted from neurobiology, the presence (or absence) of each particular feature i is denoted by the activity (or lack of activity) of a model neuron v i due to being directly driven by a feature signal.
If there are N f possible features, there can be only at most N 2 f distinct connections (synapses) in a neural circuit involving only these neurons.
Typical cortical synapses are not highly reliable, and can store only a few bits of information 1 .
The description of a particular memory requires roughly N f bits of information.
Such a system can therefore store at most ∼ N f unrelated memories.
Artiﬁcial neural network models of associative memory (based on attractor dynamics of feature neurons and understood through an energy function) exhibit this limitation even with precise synapses, with limits of memory storage to less than ∼ 0 .
14 N f memories (Hopﬁeld, 1982).
[Table]
[Figure]
Situations arise in which the number N f is small and the desired number of memories far exceeds ∼ N f , see some examples from biological and AI systems in Section 4.
In these situations the associative memory model of (Hopﬁeld, 1982) would be insufﬁcient, since it would not be able to memorize the required number of patterns.
At the same time, models of associative memory with large storage capacity considered in our paper, can easily solve these problems.
The starting point of this paper is a machine learning approach to associative memory based on an energy function and attractor dynamics in the space of N f variables, called Dense Associative
Memory (Krotov & Hopﬁeld, 2016).
This idea has been shown to dramatically increase the memory storage capacity of the corresponding neural network (Krotov & Hopﬁeld, 2016; Demircigil et al., 2017) and was proposed to be useful for increasing robustness of neural networks to adversarial attacks (Krotov & Hopﬁeld, 2018).
Recently, an extension of this idea to continuous variables, called modern Hopﬁeld network, demonstrated remarkably successful results on the immune repertoire classiﬁcation (Widrich et al., 2020), and provided valuable insights into the properties of attention heads in Transformer architectures (Ramsauer et al., 2020).
Dense Associative Memories or modern Hopﬁeld networks, however, cannot describe biological neural networks in terms of true microscopic degrees of freedom, since they contain many-body interaction terms in equations describing their dynamics and the corresponding energy functions.
To illustrate this point consider two networks: a conventional Hopﬁeld network (Hopﬁeld, 1982) and a Dense Associative Memory with cubic interaction term in the energy function (see Fig. 1).
In the conventional network the dynamics is encoded in the matrix T ij , which represents the strengths of the synaptic connections between feature neurons i and j .
Thus, this network is manifestly describable in terms of only two-body synapses, which is approximately true for many biological synapses.
In contrast, a Dense Associative Memory network with cubic energy function naively requires the synaptic connections to be tensors T ijk with three indices, which are harder, although not impossible, to implement biologically.
Many-body synapses become even more problematic in situations when the interaction term is described by a more complicated function than a simple power (in this case the Taylor expansion of that function would generate a series of terms with increasing powers).
Many-body synapses typically appear in situations when one starts with a microscopic theory described by only two-body synapses and integrates out some of the degrees of freedom (hidden neurons).
The argument described above based on counting the information stored in synapses in conjunction with the fact that modern Hopﬁeld nets and Dense Associative Memories can have a huge storage capacity hints at the same solution.
The reason why these networks have a storage capacity much greater than N f is because they do not describe the dynamics of only N f neurons, but rather involve additional neurons and synapses. Thus, there remains a theoretical question: what does this hidden circuitry look like? Is it possible to introduce a set of hidden neurons with appropriately chosen interaction terms and activation
[Table]
+YXFpeWCyura+sbxc2tujOZ5VDjRhp7nTAHUmiooUAJ16kFphIJjaR/PvIbt2CdMPoKBym0FLvRois4Qy+1i7sxwh3mCpSxA6ohs0a7YUxjSnvtWGXtYiksh2PQvySakhKZotoufsQdwzMFGrlkzjWjMMVWziwKLmFYiDMHKeN9dgNNTzVT4Fr5+JEh3fdKh3aN9aWRjtXvEzlTzg1U4jsVw5777Y3E/7xmht2TVi50miFoPlnUzSRFQ0ep0I6wwFEOPGHcCn8r5T1mGUefXcGHEP1++S+pV8rRYblyeVQ6PZvGsUR2yB45IBE5JqfkglRJjXByTx7JM3kJHoKn4DV4m7TOBNOZbfIDwfsX5M6Z3g==</latexit> ✅ biological functions so that the resulting theory has both large memory storage capacity (signiﬁcantly bigger than N f ), and, at the same time, is manifestly describable in terms on only two-body synapses?
The main contributions of this current paper are the following.
First, we extend the model of (Krotov & Hopﬁeld, 2016) to continuous state variables and continuous time, so that the state of the network is described by a system of non-linear differential equations.
Second, we couple an additional set of
N h “complex neurons” or “memory neurons” or hidden neurons to the N f feature neurons.
When the synaptic couplings and neuron activation functions are appropriately chosen, this dynamical system in N f + N h variables has an energy function describing its dynamics.
The minima (stable points) of this dynamics are at the same locations in the N f - dimensional feature subspace as the minima in the corresponding Dense Associative Memory system.
Importantly, the resulting dynamical system has a mathematical structure of a conventional recurrent neural network, in which the neurons interact only in pairs through a two-body matrix of synaptic connections.
We study three limiting cases of this new theory, which we call models A, B, and C. In one limit (model A) it reduces to Dense Associative Memory model of (Krotov & Hopﬁeld, 2016) or (Demircigil et al., 2017) depending on the choice of the activation function.
In another limit (model B) our model reduces to the network of (Ramsauer et al., 2020).
Finally, we present a third limit (model C) which we call Spherical Memory model.
To the best of our knowledge this model has not been studied in the literature.
However, it has a high degree of symmetry and for this reason might be useful for future explorations of various models of large associative memory and recurrent neural networks in machine learning.
For the purposes of this paper we deﬁned “biological plausiblity” as the absence of many-body synapses.
It is important to note that there other aspects in which our model described by equations (1) below is biologically implausible.
For instance, it assumes that the strengths of two physically different synapses µ → i and i → µ are equal.
This assumption is necessary for the existence of the energy function, which makes it easy to prove the convergence to a ﬁxed point.
It can be relaxed in equations (1), which makes them even more biological, but, at the same time, more difﬁcult to analyse.
M ATHEMATICAL F ORMULATION
In this section, we present a simple mathematical model in continuous time, which, on one hand, permits the storage of a huge number of patterns in the artiﬁcial neural network, and, at the same time, involves only pairwise interactions between the neurons through synaptic junctions.
Thus, this system has the useful associative memory properties of the AI system, while maintaining conventional neural network dynamics and thus a degree of biological plausibility.
The spikes of action potentials in a pre-synaptic cell produce input currents into a postsynaptic neuron.
As a result of a single spike in the pre-synaptic cell the current in the post-synaptic neuron rises instantaneously and then falls off exponentially with a time constant τ .
In the following the currents of the feature neurons are denoted by v i (which are enumerated by the latin indices), and the currents of the complex memory neurons are denoted by h µ ( h stands for hidden neurons, which are enumerated by the greek indices).
A simple cartoon of the network that we discuss is shown in Fig.2.
There are no synaptic connections among the feature neurons or the memory neurons.
A matrix ξ µi denotes the strength of synapses from a feature neuron i to the memory neuron µ .
The synapses are assumed to be symmetric, so that the same value ξ iµ = ξ µi characterizes a different physical synapse from the memory neuron µ to the feature neuron i .
The outputs of the memory neurons and the feature neurons are denoted by f µ and g i , which are non-linear functions of the corresponding currents.
In some situations (model A) these outputs can be interpreted as activation functions for the corresponding neurons, so that f µ = f ( h µ ) and g i =
g ( v i ) with some non-linear functions f ( x ) and g ( x ) .
In other cases (models B and C) these outputs involve contrastive normalization, e.g. a softmax, and can depend on the currents of all the neurons in that layer.
In these cases f µ = f ( { h µ } ) and g i = g ( { v i } ) .
For the most part of this paper one can think about them as ﬁring rates of the corresponding neurons.
In some limiting cases, however, the function g ( v i ) will have both positive and negative signs.
Then it should be interpreted as the input current from a pre-synaptic neuron.
The functions f ( h µ ) and g ( v i ) are the only nonlinearities that appear in our model.
Finally, the time constants for the two groups of neurons are denoted by τ f and τ h .
With these notations our model can be written as
[Equation]
where I i denotes the input current into the feature neurons.
The connectivity of our network has the structure of a bipartite graph, so that the connections exist between two groups of neurons, but not within each of the two groups.
This design of a neural network is inspired by the class of models called Restricted Boltzmann Machines (RBM) (Smolensky, 1986).
There is a body of literature studying thermodynamic properties of these systems and learning rules for the synaptic weights.
In contrast, the goal of our work is to write down a general dynamical system and an energy function so that the network has useful properties of associative memories with a large memory storage capacity, is describable only in terms of manifestly two-body synapses, and is sufﬁciently general so that it can be reduced to various models of this class previously discussed in the literature.
We also note that although we use the notation v i ( v stands for visible neurons), commonly used in the RBM literature, it is more appropriate to think about v i as higher level features.
For example the input to our network can be a latent representation produced by a convolutional neural network or a latent representation of a BERT-like system (Devlin et al., 2018) rather than raw input data.
Additionally, our general formulation makes it possible to use a much broader class of activation functions (e.g. involving contrastive or spherical normalization) than those typically used in the RBM literature.
Also, the relationship between Dense Associative
Memories and RBMs has been previously studied in (Barra et al., 2018; Agliari & De Marzo, 2020).
We also note that a Hopﬁeld network with exponential capacity was studied in (Chaudhuri & Fiete, 2019), but their construction requires speciﬁcally engineered memory vectors and cannot be applied to general arbitrary memory vectors.
Mathematically, equations (1) describe temporal evolution of two groups of neurons.
For each neuron its temporal updates are determined by the inputs from other neurons and its own state (the decay term on the right hand side of the dynamical equations).
For this reason, an energy function for this system is expected to be represented as a sum of three terms: two terms describing the neurons in each speciﬁc group, and the interaction term between the two groups of neurons.
We have chosen the speciﬁc mathematical form of these three terms so that the energy function decreases on the dynamical trajectory.
With these choices the energy function for the network (1) can be written as
[Equation]
Here we introduced two Lagrangian functions L v ( { v i } ) and L h ( { h µ } ) for the feature and the hidden neurons.
They are deﬁned through the following equations, so that derivatives of the Lagrangian functions correspond to the outputs of neurons
[Equation]
With these notations expressions in the square brackets in (2) have a familiar from classical mechanics structure of the Legendre transform between a Lagrangian and an energy function.
By taking time derivative of the energy and using dynamical equations (1) one can show (see Appendix A for details) that the energy monotonically decreases on the dynamical trajectory
[Equation]
The last inequality sign holds provided that the Hessian matrices of the Lagrangian functions are positive semi-deﬁnite.
In addition to decrease of the energy function on the dynamical trajectory it is important to check that for a speciﬁc choice of the activation functions (or Lagrangian functions)
the corresponding energy is bounded from below.
This can be achieved for example by using bounded activation function for the feature neurons g ( v i ) , e.g. hyperbolic tangent or a sigmoid.
Provided that the energy is bounded, the dynamics of the neural network will eventually reach a ﬁxed point, which corresponds to one of the local minima of the energy function 2 .
The proposed energy function has three terms in it: the ﬁrst term depends only on the feature neurons, the second term depends only on the hidden neurons, and the third term is the “interaction” term between the two groups of neurons.
Note, that this third term is manifestly describable by twobody synapses - a function of the activity of the feature neurons is coupled to another function of the activity of the memory neurons, and the strength of this coupling is characterized by the parameters ξ µi .
The absence of many-body interaction terms in the energy function results in the conventional structure (with unconventional activation functions) of the dynamical equations (1).
Each neuron collects outputs of other neurons, weights them with coefﬁcients ξ and generates its own output.
Thus, the network described by equations (1) is biologically plausible according to our deﬁnition (see Introduction).
Lastly, note that the memory patterns ξ µi of our network (1) can be interpreted as the strengths of the synapses connecting feature and memory neurons.
This interpretation is different from the conventional interpretation, in which the strengths of the synapses is determined by matrices
T ij = (cid:80) µ ξ µi ξ µj (see Fig. 1), which are outer products of the memory vectors (or higher order generalizations of the outer products).
E FFECTIVE T HEORY FOR F EATURE N EURONS
In this section we start with the general theory proposed in the previous section and integrate out hidden neurons.
We show that depending on the choice of the activation functions this general theory reduces to some of the models of associative memory previously studied in the literature, such as classical Hopﬁeld networks, Dense Associative Memories, and modern Hopﬁeld networks.
The update rule in the latter case has the same mathematical structure as the dot-product attention (Bahdanau et al., 2014) and is also used in Transformer networks (Vaswani et al., 2017).
3. M ODEL A. D ENSE A SSOCIATIVE M EMORY L IMIT .
Consider the situation when the dynamics of memory neurons h µ is fast.
Mathematically this corresponds to the limit τ h → 0 .
In this case the second equation in (1) equilibrates quickly, and can be solved as
[Equation]
Additionally, assume that the Lagrangian functions for the feature and the memory neurons are additive for individual neurons
[Equation]
where F ( x ) and G ( x ) are some non-linear functions.
In this limit we set G ( x ) =
| x | .
Since, the outputs of the feature neurons are derivatives of the Lagrangian (3), they are given by the sign functions of their currents,which gives a set of binary variables that are denoted by σ i
[Equation]
Since G ( v i ) = | v i | the only term that survives in the ﬁrst square bracket in equation (2) is the one proportional to the input current
I
i .
The ﬁrst term in the second bracket of equation (2) cancels the interaction term because of the steady state condition (5).
Thus, in this limit the energy function (2) reduces to
[Equation]
If there are no input currents
I i =
this is exactly the energy function for Dense Associative
Memory from (Krotov & Hopﬁeld, 2016).
If F ( x ) =
x n is a power function, the network can store N mem ∼ N n − 1 f memories, if F ( x ) =
exp( x ) the network has exponential storage capacity
Demircigil et al.
If power n = 2 this model further reduces to the classical Hopﬁeld network (Hopﬁeld, 1982).
It is important to emphasize that the capacity estimates given above express the maximal number of memories that the associative memory can store given the dimensions of the input, but assuming no limits on the number of hidden neurons.
In all the models considered in this work this capacity is also bounded by the number of those hidden neurons so that N mem ≤ N h .
With this constraint the capacity of model A with power function F ( x ) =
x n should be written as
[Equation]
In many practical applications (see examples in Section 4) the number of hidden neurons can be assumed to be larger than the bound deﬁned by the dimensionality of the input space N f .
It is for this class of problems that Dense Associative Memories or modern Hopﬁeld networks offer a powerful solution to the capacity limitation compared to the standard models of associative memory (Hopﬁeld, 1982; 1984).
Lastly, for the class of additive models (6), which we call models A, the equation for the temporal evolution of the energy function reduces to
[Equation]
Thus, the condition that the Hessians are positive deﬁnite is equivalent to the condition that the activation functions g ( v i ) and f ( h µ ) are monotonically increasing.
Additionally, in Appendix B, we show how standard continuous Hopﬁeld networks (Hopﬁeld, 1984) can be derived as a limiting case of the general theory (1,2).
3. M ODEL B. M ODERN H OPFIELD N ETWORKS L IMIT AND A TTENTION M ECHANISM .
Models B are deﬁned as models having contrastive normalization in the hidden layer.
Speciﬁcally we are interested in
[Equation]
so that L v is still additive, but L h is not.
Using the general deﬁnition of the activation functions (3) one obtains
[Equation]
Similarly to the previous case, consider the limit τ h → 0 , so that equation (5) is satisﬁed.
In this limit the energy function (2) reduces to (currents I i are assumed to be zero)
[Equation]
This is exactly the energy function studied in (Ramsauer et al., 2020) up to additive constants (inverse temperature β was assumed to be equal to one in this derivation).
Notice that we used the notations from (Krotov & Hopﬁeld, 2016), which are different from the notations of (Ramsauer et al., 2020).
In the latter paper the state vector v i is denoted by ξ i and the memory matrix ξ µi is denoted by the matrix X T .
Making substitutions (12) in the ﬁrst equation of (1), using steady state condition (5), and setting input current
I i = 0 results in the following effective equations for the feature neurons, when the memory neurons are integrated out
[Equation]
This is a continuous time counterpart of the update rule of (Ramsauer et al., 2020).
Writing it in ﬁnite differences gives
[Equation]
which for dt = τ f reduces to
[Equation]
This is exactly the update rule derived in (Ramsauer et al., 2020), which, if applied once, is equivalent to the familiar dot-product attention (Bahdanau et al., 2014) and is also used in Transformer networks (Vaswani et al., 2017).
The derivation of this result in (Ramsauer et al., 2020) begins with the energy function for a Dense
Associative Memory model with exponential interactions F ( x ) = exp ( x ) .
Then it is proposed to take a logarithm of that energy (with a minus sign) and add a quadratic term in the state vector v i to ensure that it remains ﬁnite and the energy is bounded from below.
While this is a possible logic, it requires a heuristic step - taking the logarithm, and makes the connection with Dense Associative
Memories less transparent.
In contrast, our derivation follows from the general principles speciﬁed by equations (1,2) for the speciﬁcally chosen Lagrangians.
It is also important to note, that the Hessian matrix for the hidden neurons has a zero mode (zero eigenvalue) for this limit of our model.
3. M ODEL C. S PHERICAL M EMORY .
Models C are deﬁned as having spherical normalization in the feature layer.
We are not aware of a discussion of this class of associative memory models in the literature.
Speciﬁcally,
[Equation]
(17) so that L h is additive, but L v is not.
Using the general deﬁnition of the activation functions (3) one obtains
[Equation] [Equation]
Equations (1) for model C are given by ( I i is assumed to be zero)
[Equation]
Notice, that since the Hessian matrix for the feature neurons has a zero mode proportional to v i in this model,
[Equation]
we can write an arbitrary coefﬁcient α , which can be equal to zero, in front of the decay term for the feature neurons.
Taking the limit τ h → 0 and excluding h µ gives the effective energy
[Equation]
and the corresponding effective dynamical equations
[Equation]
It is also important to notice that the activation function g i that appears in equation (18) implements a canonical computation of divisive normalization widely studied in neuroscience (Carandini &
Heeger, 2012).
Divisive normalization has also been shown to be beneﬁcial in deep CNNs and
RNNs trained on image classiﬁcation and language modelling tasks (Ren et al., 2016).
A F EW E XAMPLES OF L ARGE A SSOCIATIVE M EMORY P ROBLEMS
In this section we provide some examples of problems in AI and biology which may beneﬁt from thinking about them through the lens of associative memory.
Pattern memorization.
Consider a small gray scale image 64 × 64 pixels.
If one treats the intensity of each pixel as an input to a feature neuron the standard associative memory (Hopﬁeld, 1982) would be able to only memorize approximately 0 .
14 · 4096 ≈ 573 distinct patters.
Yet, the number of all possible patterns of this size that one can imagine is far bigger.
For instance, Kuzushiji-Kanji dataset (Clanuwat et al., 2018) includes over 140,000 characters representing 3832 classes with most of the characters recognizable by humans.
A well educated Japanese person can recognize about 3000-5000 character classes, which means that those classes are represented in his/her memory.
In addition, for many characters a person would be able to complete it if only a portion of that character is shown.
Moreover, possible patterns of 64 × 64 pixels are not necessarily Kanji characters, but also include digits, smileys, emojis, etc.
Thus, the overall number of patterns that one might want to memorize is even bigger.
In the problem of immune repertoire classiﬁcation , considered in (Widrich et al., 2020), the number of immune repertoire sequences (number of memories in the modern Hopﬁeld network) is
N (cid:29) 10000 , while the size of the sequence embedding dimension d k = 32 , or N f = 32 using the notations of this current paper.
The ability to solve this problem requires the associative memory used in the aforementioned paper to have a storage capacity much larger than the dimensionality of the feature space.
Cortical-hippocampal system.
The hippocampus has long been hypothesised to be responsible for formation and retrieval of associative memories, see for example (Rolls, 2018; Treves & Rolls, 1994).
Damage to the hippocampus results in deﬁciencies in learning about places and memory recognition visual tasks.
For instance (Parkinson et al., 1988) reports deﬁciencies in object-memory tasks, which require a memory of an object and the place where that object was seen.
One candidate for associative memory network in the hippocampus is the CA3 area, which consists of a large population of pyramidal neurons, approximately 3 · 10 5 in the rat brain (Amaral & Witter, 1989), and 2 . 3 · 10 6 (Seress, 1988) in human, in conjunction with an inhibitory network that keeps the ﬁring rates under control.
There is a substantial recurrent connectivity among the pyramidal neurons (Rolls, 2018), which is necessary for an associative memory network.
There are also several classes of responses of those neurons in behaving animals, one class being place cells (O’Keefe & Dostrovsky, 1971).
In addition to place cells (Ferguson et al., 2011) report existence of cells in the hippocampus that do not respond in experiments designed to drive place cells, but presumably are useful for other tasks.
One possible way of connecting the mathematical model proposed in this paper with the existing anatomical network in the brain is to assume that some of the pyramidal cells in CA correspond to the feature neurons in our model, while the remaining pyramidal cells are the memory neurons.
For example, place cells are believed to emerge as a result of aggregating inputs from the grid cells and environmental features, e.g. landmark objects, environment boundaries, visual and olfactory cues, etc., (Moser et al., 2015).
Thus, it is tempting to think about them as memory neurons (which aggregate information from feature neurons to form a stable memory) in the proposed model.
Another area of the hippocampus potentially related to the mathematical model described in this paper is the area CA1, which, in addition to receiving inputs from CA3, also receives inputs directly from the entorhinal cortex, and projects back to it.
In this interpretation pyramidal cells of the CA would be interpreted as the memory neurons in our mathematical model, while the cells in the layer
III of the entorhinal cortex would be the feature neurons.
The feedback projections from CA1 go primarily to layer V of the entorhinal cortex (Rolls, 2018), but there are also projections to layers
II and III (Witter et al., 2017).
While it is possible to connect the proposed mathematical model of
Dense Associative Memory with existing networks in the hippocampus, it is important to emphasize that the hippocampus is involved in many tasks, for example imagining the future (Hassabis et al., 2007), and not only in retrieving the memories about the past.
For this reason it is difﬁcult at present to separate the network motifs responsible for memory retrievals from the circuitry required for other functions.
Human color vision has three dimensions so that every color sensation can be achieved by mixing three primary lights (Mollon, 2003).
From the neuron’s perspective every color is detected by three kinds of cone photoreceptors ( N f = 3 ) in the retina, so that the degree of excitation of each photoreceptor is described by a continuous number.
Most people know many colors with names for them (e.g. red, orange, yellow, green, blue, indigo, violet, pink, lavender, copper, gold, etc.), descriptions for others.
e.g. ”the color of the sky”.
Experimentally, humans can distinguish about 10 6 different colors (Masaoka et al., 2013), although may not be able to “memorize” all of them.
See also (Meister, 2015) for the discussion of this problem.
Thus, if one thinks about this system as an associative memory for color discrimination, the model of (Hopﬁeld, 1982) and its extensions with O ( N f ) storage capacity would be inadequate since they can only “remember” a few colors.
It is important to emphasize that the memories of the colors are stored in higher areas of the brain, while the color sensation is conveyed to the brain through the cone cells in the retina.
Thus, in this example there are many intermediate neurons and synapses between in feature neurons and memory neurons.
For this reason it is only appropriate to think about this example as a direct associative memory if all these intermediate neurons and synapses are integrated out from this system.
D ISCUSSION AND C ONCLUSIONS
We have proposed a general dynamical system and an energy function that has a large memory storage capacity, and, at the same time, is manifestly describable in terms of two-body synaptic connections.
From the perspective of neuroscience it suggests that Dense Associative Memory models are not just mathematical tools useful in AI, but have a degree of biological plausibility similar to that of the conventional continuous Hopﬁeld networks (Hopﬁeld, 1984).
Compared to the latter, these models have a greater degree of psychological plausibility, since they can store a much larger number of memories, which is necessary to explain memory-based animal behavior.
We want to emphasize that the increase in the memory storage capacity that is achieved by
modern
Hopﬁeld networks is a result of unfolding the effective theory and addition of (hidden) neurons.
By adding these extra neurons we have also added synapses.
Coming back to the information counting argument that we presented in the introduction, the reason why these unfolded models have a larger storage capacity than the conventional Hopﬁeld networks with the same number of input neurons is because they have more synapses, but each of those synapses has the same information capacity as in the conventional case.
From the perspective of AI research our paper provides a conceptually grounded derivation of various associative memory models discussed in the literature, and relationships between them.
We hope that the more general formulation, presented in this work, will assist in the development of new models of this class that could be used as building components of new recurrent neural network architectures.
A CKNOWLEDGEMENTS
We are thankful to J. Brandstetter, S. Hochreiter, M. Kopp, D. Kreil, H.Ramsauer, D. Springer
[Equation]
and F. Tang for useful discussions.
A PPENDIX A
In this appendix we show a step by step derivation of the change of the energy function (2) under dynamics (1).
Time derivative of the energy function can be expressed through time derivatives of the neuron’s activities v i and h µ (the input current
I i is assumed to be time-independent in the calculation below).
Using the deﬁnition of the functions f µ and g i in (3) one can obtain dE
[Equation]
In the last equality sign the right hand sides of dynamical equations (1) are used to replace expressions in the square brackets by the corresponding time derivatives of the neuron’s activities.
This completes the proof that the energy function decreases on the dynamical trajectory described by equations (1) for arbitrary time constants τ f and τ h provided that the Hessians for feature and memory neurons are positive semi-deﬁnite.
A PPENDIX B. T HE LIMIT OF STANDARD CONTINUOUS H OPFIELD NETWORKS .
In this section we explain how the classical formulation of continuous Hopﬁeld networks (Hopﬁeld, 1984) emerges from the general theory (1,2).
Continuous Hopﬁeld networks for neurons with graded response are typically described by the dynamical equations
[Equation]
and the energy function
[Equation]
where, as in Section 3.1, g i =
g ( v i ) , and g − 1 ( z ) is the inverse of the activation function g ( x ) .
According to our classiﬁcation, this model is a special limit of the class of models that we call models A, with the following choice of the Lagrangian functions
[Equation]
, and
[Equation]
[Equation]
that, according to the deﬁnition (3), lead to the activation functions
[Equation]
and
[Equation]
Similarly to Section 3.1, we integrate out the hidden neurons to demonstrate that the system of equations (1) reduces to the equations on the feature neurons (24) with T ij =
N h
[Equation]
The general expression for the energy (2) reduces to the effective energy
[Equation]
While the ﬁrst two terms in equation (25) are the same as those in equation (28), the third terms look superﬁcially different.
In equation (28) it is a Legendre transform of the Lagrangian for the feature neurons, while in (25) the third term is an integral of the inverse activation function.
Nevertheless, these two expressions are in fact equivalent, since the derivatives of a function and its Legendre transform are inverse functions of each other.
The easiest way to see that these two terms are equal explicitly is to differentiate each one with respect to v i .
The results of these differentiations for both expressions are equal to v
i g ( v i ) (cid:48) .
Thus, the two expressions are equal up to an additive constant.
This completes the proof that the classical Hopﬁeld network with continuous states (Hopﬁeld, 1984) is a special limiting case of the general theory (1, 2).