Kernel Attractor Networks: A Unifying Framework for Memory Modeling
Anonymous Author(s)
AffiliationAddress email
Abstract
We consider the problem of training a neural network to store a set of patternswith maximal noise robustness.
A solution, in terms of optimal weights and stateupdate rules, is derived by training each individual neuron to perform either kernelclassification or interpolation with a minimum weight norm.
By applying thismethod to feed-forward and recurrent networks, we derive optimal networks thatinclude, as special cases, many of the hetero- and auto-associative memory modelsthat have been proposed over the past years, such as modern Hopfield networksand Kanerva’s sparse distributed memory.
We generalize Kanerva’s model anddemonstrate a simple way to design a kernel attractor network that can store anexponential number of continuous-valued patterns with a finite basin of attraction.
The framework of kernel attractor networks offers a simple and intuitive wayto understand the storage capacity of previous memory models, and allows fornew biological interpretations in terms of dendritic non-linearities and synapticclustering.
Introduction
Although the classical work on attractor neural networks reached its peak in the late 1980’s, withthe publication of a number of seminal works [e.g., 2, 17, 19, 21], recent years have seen a renewedinterest in the topic, motivated by the popularity of the attention mechanism [58], external memoryaugmented neural networks [20, 59], as well as a new generation of energy-based attractor networksmodels, termed modern Hopfield networks (MHNs), capable of vastly increased memory storage[14, 30].
Recent efforts to understand the theoretical foundation of the attention mechanism have,in fact, shown that it can be linked to Hopfield networks [31, 50], but also to Kanerva’s sparsedistributed memory (SDM) [6, 25], and to the field of kernel machines [56, 61].
The last connectionis particularly intriguing, in light of the many theoretical commonalities between neural networksand kernel methods [7, 8, 23, 41, 60].
In this work, we aim to clarify some of the overlap between these fields, by integrating and formalizinga set of theoretical connections between Hopfield networks, the SDM, kernel machines, and neuronmodels with non-linear dendritic processing.
1.1 Our contribution
• We derive a set of normative models that describe the general mathematical structure of feedforward (i.e., hetero-associative) and recurrent (i.e., auto-associative) memory networks that canperform error-free recall of a given set of patterns with maximal robustness to noise.•
We show that the normative models include, as special cases, the classical and modern Hopfieldnetwork, as well as the SDM.
• We derive a simple attractor network model for storing an exponential number of continuous-valuedpatterns with a finite basin of attraction.
We
discuss
its similarity to attention
Furthermore, we explain how classifiers with non-linear kernels can be interpreted as general formsof neuron models with non-linear dendritic activation functions and synaptic clustering.
1.2 Related work
Our work is related to [6, 30, 31, 50].
While MHNs are extensively analyzed in [30, 31, 50], theapproach is energy-based and makes no statements about the relation between MHNs and othermemory models or kernel methods; a brief comment mentions some similarity to SVMs, but this isnot further explained [50].
The work by [6] focuses on the SDM and its connection to attention.
Itobserves that the classical Hopfield network is a special case of the SDM, but no further generalizationis made, and kernel methods are not mentioned.
In our work, we place MHNs and the SDM in abroader theoretical context by showing that both models are, in fact, special suboptimal cases of afamily of memory networks that can be derived based on a normative kernel-based approach.
Background
In this section, we restate some well-known properties of binary classifiers before moving on to ourmain results.
Consider the following model of hetero-associative memory: a single-layer feed-forwardnetwork consisting of a single output neuron connected to N in inputs with the weights w ∈ R N ϕ .
The output s out ∈ {− 1 , +1 } is given by
[Equation]
(1) where s in is the input vector (also called query), θ the threshold, and ϕ a function that maps the “raw”input to a N ϕ -dimensional feature space, where typically N ϕ ≫ N in .
Suppose that we are given a setof M input-output patterns { ξ µ in , ξ µ out } Mµ =1 , where each entry ξ is randomly drawn from {− 1 , +1 }with sparseness f := P ( ξ =1) .
In order for the neuron to memorize each pattern µ with a maximalattractor basin, it needs to find a set of weights so as to output ξ µ out in response to the input ξ µ in in amanner that maximizes the Euclidean distance between the input and the decision boundary.
Using Gardner’s formalism [17, 19], this problem can be expressed as
[Equation]
where ¯ w > 0 is a constant.
This is equivalent to solving
[Equation]
which can be directly identified as the support vector machine (SVM) problem for separable data[10].
The solution to Eq.
3 can today be found in any textbook on basic machine learning methods,and yields an optimal output that can be formulated in a feature and kernel representation
[Equation]
where we have used the “kernel-trick” K ( x i , x j )
= ϕ
( x i )
⊤ ϕ
( x j ) .
The solution depends on the
Lagrange coefficients α µ ≥ 0 , many of which are typically zero.
Patterns with α µ > 0 are calledsupport vectors .
2.1 Hetero -associative memory as a feed-forward SVM network
We now consider a hetero-associative memory network with an arbitrary number N out of outputneurons, whose combined state we denote s out .
In order for the network as a whole to successfullyrecall memories with maximal noise robustness, we solve Eq.
3 for each neuron independently.
Aseach neuron can have a different classification boundary along with a different set of support vectors, its weights will, in general, be characterized by an independent set of M Lagrange coefficients.
Tosimplify the notation, we represent these coefficients α µi , across neurons i and patterns µ , as entriesin the matrix A ∈ R N out × M ≥, where ( A ) iµ = α µi .
We also combine all thresholds in the vector θ =( θ 1 , . . .
, θ N out ) , and all input and output patterns as columns in the matrices X in = ( ξ 1in , . . .
, ξ M in )and X out = ( ξ 1out , . .
. , ξ M out ) .
The optimal response of the network is compactly summarized in thefollowing property.
Property 1 (Robust hetero-associative memory network) .
The optimal output of a single-layerhetero-associative memory network trained to recall the patterns X out in response to the inputs X in ,with maximal noise robustness, can be written in the form
[Equation]
feature)
[Equation] [Equation]
(kernel) (6) where ⊙ denotes the Hadamard product. Remark. With a linear feature map ϕ ( x ) = x , the optimal output reduces to the forms
[Equation]
= [Equation]
[Equation] [Equation] (7) where W is the optimal weight matrix.  Kernel attractor networks for binary patterns3.1 Auto -associative memory as a recurrent SVM network
We begin our results by first noting that the hetero-associative memory network can be made autoassociative by setting N out
= N in and X out
=
X in .
The network is now effectively recurrent, aseach neuron serves both as an input and output simultaneously (see Fig. 1).
Consider a recurrentnetwork with N neurons, whose state at time point t is denoted s ( t ) ∈ {− 1 , +1 } N , and whosedynamics evolve according to the update rule [Equation]
[Equation]
where w i ∈ R N ϕ is the weight vector to neuron i = 1 , . .
N .
In order to make the patterns{ ξ µ } Mµ =1 attractors of the network dynamics each neuron i is, again, independently trained on everypattern µ to produce the response ξ µi when the rest of the network is initialized in ξ µ .
Moreover, thenoise robustness and, thereby, the size of the attractor basins, can be maximized by maximizing the
Euclidean distance between each neuron’s decision boundary and the patterns [15, 27].
The problemof training the entire network can, in this way, be transformed into the problem of training N separateclassifiers to
[Equation]
[Equation]
The solution can be obtained by slightly modifying Property 1, and is stated below.
Property 2.1 (Robust auto-associative memory) .
A recurrent auto-associative memory networktrained to recall the patterns X with maximal noise robustness has an optimal synchronous updaterule that can be written in the form
[Equation]
feature)
[Equation]
[Equation]
Remark.
With a linear feature map ϕ ( x ) = x , the optimal update is reduced to
[Equation]
[Equation]
where W is the general form of the optimal weight matrix.
The solution described by Property 2.1 does not, in general, prohibit a neuron from having selfconnections.
Applying this constraint yields the following noteworthy result.
[Figure]
Property 2.2 (Robust auto-associative memory without self-connections) .
A recurrent autoassociative memory network without self-connections, that has been trained to recall the patterns
X with maximal noise robustness, has an optimal update rule that can be written in the kernelrepresentation
[Equation]
where K is an inner-product kernel, i.e. K ( x i , x j )
=
k ( x ⊤ i x j ) .
Remark.
When the kernel is linear, i.e. k ( x ⊤ i x j ) =
x ⊤ i x j , the update reduces to
[Equation]
Storage capacity.
An intuition for the storage capacity scaling of the hetero- and auto-associativememory network can be obtained by observing that the network as a whole will be able to successfullyrecall patterns as long as each neuron is able to correctly classify its inputs (or is very unlikely toproduce an error).
The capacity of the network can thereby be derived from the capacity of eachindividual neuron.
It is well-known that a linear binary classifier can learn to correctly discriminatea maximum of M max ≈ 2 D VC random bipolar patterns, where D VC is the Vapnik-Chervonenkis dimension of the model [12, 17, 35, ch. 40].
When the neuron in Eq.
8 has a linear feature mapϕ ( x ) =
x , this results in D VC = N and, thus, M max ≈ 2 N .
Suppose, on the other hand, thatthe kernel is a homogeneous polynomial of degree p , so that K ( x i , x j ) =
( x ⊤ i x j ) p .
In this case,ϕ contains all monomials of degree p composed of the elements in x .
As each element x i isan i.i.d. random variable drawn uniformly from {− 1 , +1 } , each monomial will also be a randomvariable drawn from the same distribution. The ϕ -vector will contain N p /p ! unique monomials (i.e.,independent entries), which yields D VC ∼ O ( N p ) and a capacity that scales as M max ∼ O ( N p ) .
For the exponential kernel, which we can write as K ( x i , x j ) =
exp( x ⊤ i x j ) = (cid:80)
∞
p =0 ( x ⊤ i x j ) p /p !
,the number of independent entries in ϕ is (cid:80) ∞ p =0 N p /p ! =
e N , which gives D VC , M max ∼ O ( e N ) .
Application to special cases.
In the following sections, we will show that many of the modelsof hetero- and auto-associative memory that have been proposed over the past years can be seen asspecific cases of the solutions in Properties 1, 2.1, and 2.2, characterized by specific choices of Aand the functions ϕ and K .
3.2 Kanerva’s sparse distributed memory is a feed-forward SVM network
The sparse distributed memory (SDM), developed by Kanerva [25], is one of the most famousexamples of a hetero-associative memory model.
It has lately received much attention in the contextof generative memory models [62] and attention layers in transformers [6].
The SDM consists of a register of N ϕ memory slots, each associated with an address
z i ∈{− 1 , +1 } N in , i = 1 , . . .
, N ϕ .
All addresses are listed as rows in the matrix Z
= ( z 1 , . . .
, z N ϕ ) ⊤ .
The content of each slot is represented by an N out -dimensional vector, initialized at zero.
Supposethat we wish to store the M patterns X out = ( ξ 1out , . .
. , ξ M out ) in the addresses X in = ( ξ 1in , . . .
, ξ M in ) ,where all entries are random and bipolar.
The basic idea of the SDM is to write the data to, and laterread it from, multiple memory slots at once (hence the distributed storage); this ensures a degree ofnoise-robustness.
In mathematical terms, the read-out of the SDM provided with a query s in , is givenby
[Equation]
where Θ the Heaviside function with bias b = N in − 2 r , and r is a parameter that determines theprecision of the writing and reading process.
Upon comparing Eqs.
15 and 5, the SDM can be directlyidentified as a special case of a suboptimal feed-forward SVM network in the feature representation,with A
= 1 , θ = 0 , and the feature map
[Equation]
When viewed as a kernel method, the function of the SDM is to store the dense addresses X in assparse high-dimensional representations ϕ SDM , to make it easier to later determine the slots closestto a query s in , and retrieve the relevant data.
Capacity.
As the SDM is linear in ϕ SDM , with D VC ≈ N ϕ , it follows from the analysis in Sec. 3.1 that one should expect the capacity to scale as M max ∼ O ( N ϕ ) .
Moreover, one shouldexpect a proportionality constant ∼ 0 .
1 , since the SDM is suboptimal relative the feed-forward SVManalogously to how the classical Hopfield network is suboptimal in relation to the recurrent SVM.
This is consistent with earlier proofs [9, 26].
Kernel of an infinite SDM.
In practice, an SDM with a large number of memory slots N ϕ requirescalculations involving a large address matrix Z .
This can be avoided by applying the kernel-trick inthe limit N ϕ → ∞ .
In this section, we will derive the kernel corresponding to two different variantsof the SDM and demonstrate that both are translation-invariant.
To ensure convergence, we willdefine the SDM-kernel as
[Equation]
As a brief remark, it is interesting to note here that ϕ SDM is equivalent to a single-layer neuralnetwork with N ϕ neurons, weights Z , and bias b .
This means that K SDM is equivalent to the kernelof an infinitely wide neural network [8, 41, 60].
We begin by noticing that
ϕ
SDM ( x ) has a geometrical interpretation [6, 26].
It is a binary vectorthat indicates those memory addresses in Z that differ by at most r bits compared to x .
For any twobipolar vectors z and x , the bit-wise difference can be computed as 12 | z − x | = 14 ∥
z − x ∥ 22 .
Thismeans that ϕ SDM ( x ) indicates all addresses that lie within a sphere centered at x with radius 2 √ r .
Consequently, the inner product ϕ SDM ( x i )
⊤ ϕ SDM ( x j ) is the number of addresses located in theoverlapping volume of two spheres centered at x i and x j .
Although an exact calculation of thisquantity can be found in [6, 25], its connection to the SDM kernel has, to the best of our knowledge,not previously been made.
We modify the previously published expression with a normalizationfactor 1 / 2 N in and state the following property.
Property 3.1 (Kernel of an infinite SDM on the hypercube) .
In the limit N ϕ → ∞ , the read-outof an SDM with N ϕ memory slots, storing M patterns X out ∈ {− 1 , +1 } N out × M in the addresses X in ∈ {− 1 , +1 } N in × M , is given by
[Equation]
where
[Equation]
i [Equation] [Equation] and where r is the bit-wise error threshold and ∆ is the bit-wise difference between x i and x j , givenby ∆ = 12 | x i − x j | = 14 ∥ x i − x j ∥ 22 .
The SDM can also be implemented with continuous addresses, randomly placed on a unit hypersphere of ( N in − 1) dimensions.
The vector ϕ SDM ( x ) now indicates all addresses that lie within ahyperspherical cap centered at x with an angle arccos( b ) between its central axis and the rim.
Theinner product ϕ SDM ( x i )
⊤ ϕ SDM ( x j ) is the number of addresses located in the overlapping area oftwo spherical caps centered at x i and x j .
While a calculation of this quantity, again, can be foundin [6], it has not, to the best of our knowledge, previously been connected to the kernel of an SDM.
We simplify the previously published result and also derive a closed-form approximation, valid forhighly sparse ϕ
SDM (see Appendix A for details).
The results are summarized below.
Property 3.2 (Kernel of an infinite SDM on the unit hypersphere) .
In the limit N ϕ → ∞ , the read-outof an SDM with N ϕ memory slots, storing M patterns X out ∈ {− 1 , +1 } N out × M in the addresses X in ∈ R N in × M , where ∥ ξ µ in ∥ 2 = 1 , ∀ µ , is given by Eq.
18 with the kernel
[Equation]
where α x = 12 arccos( x ⊤ i x j ) , α b = arccos( b ) , and B is the incomplete Beta function (details in
Appendix A.1).
In the highly sparse regime, when 0 . 9 ≲ b < 1 and 1 N ϕ ∥
ϕ
SDM ∥ 2 ≪ 1 , the kernel canbe approximated with
[Equation]
where ∆ = 12 ∥ x i − x j ∥ 2 and ˆ b = sin(arccos( b )) (details in Appendix A.2).
In conclusion, an infinitely large SDM with sparse internal representations ϕ SDM , can be representedas a suboptimal case of a feed-forward SVM network with a translation-invariant kernel.
3.3 The modern Hopfield network is a recurrent SVM network
The Hopfield network [21] is, arguably, the most well-known model of auto-associative memory.
Inits modern form [30], it is a recurrent network of N neurons with the state s ( t ) whose dynamics aregoverned by the energy and state update rule
[Equation]
where F is a smooth function, typically a sigmoid, polynomial, or exponential.
This “generalized”
Hopfield model has a long history [see, e.g., 1, 18, 32] but has received renewed attention in recentyears under the name modern Hopfield network (MHN) or dense associative memory [14, 30].
Bycomparing Eq.
22 with Eq.
13, the state update of the MHN can be identified as a special case of asuboptimal recurrent SVM network in the kernel representation, with k = F ′ , A = 1 , and θ = 0(since f = 0 . 5 ).
With a linear F ′ ( x ) = x , the modern Hopfield reduces to the classical Hopfield,which is identical to the linear SVM network in Eq.
Capacity.
The storage capacity of the MHN has been shown to depend on the shape of F ′ .
In thelinear case, the capacity is famously limited to ∼ 0 .
1 N patterns, depending on the precision of retrieval[2, 37].
If, on the other hand, F ′ is polynomial with degree p , the capacity scales as M max ∼ O ( N p )[30].
Analogously, an exponential F ′ endows the network with a capacity M max ∼ O
( e N ) [14].
From the perspective of the kernel attractor framework, this scaling directly follows from the analysisin Sec.
3.1 with k = F ′ .
Generalization.
Viewing the MHN as a recurrent network of SVMs can also facilitate a moreintuitive understanding of its ability to generalize, when used as a conventional classifier.
In thissetting, one designates a subset of the neurons as input units, and the remaining neurons as outputs.
Given a set of input-output associations, one optimizes the memory patterns ξ µ using, for example,gradient descent.
Such an experiment was performed by Krotov and Hopfield [30] on the MNISTdata set, using a polynomial non-linearity F ( x ) = x p .
Their results showed that the test error first improved as p increased from 2 to 3, but later deteriorated for high degrees, like p = 20 .
While itmay be difficult to explain this behavior within an energy-based framework, it is entirely expectedwhen viewed from the SVM perspective. A kernel of low polynomial degree has too few degrees offreedom to fit the classification boundary in the training set, causing underfitting , while a polynomialof too high degree grants the model too much flexibility, which results in overfitting .
Online learning.
Recently, Tyulmankov et al.
[57] proposed a biologically plausible online learningalgorithm to train the weights in an MHN.
Depending on the implementation, the resulting storagecapacity scales as either ∼ 0 .
16 N or ∼ N .
In a comment on the discrepancy between these resultsand the maximum capacity ∼ 2 N , the authors noted that they are not aware of a learning algorithmthat is able to reach the upper bound.
Here, the perspective of kernel attractor networks can beparticularly helpful, as many of the algorithms that have been developed over the past two decades tooptimize SVMs can be utilized for MHNs as well.
For example, a network formulated in the featurerepresentation can be trained using the batch perceptron rule [11, 29], the passive agressive rules[13], the minnorm rule [4], as well as likelihood maximization applied to logistic regression [24, 40,52].
In the kernel representation, two of the most well-known online algorithms to train linear andnon-linear SVMs are the Adatron [3] and the Kernel-Adatron [16].
The pseudoinverse learning rule.
The coefficients in A are, in general, computed numerically,and cannot be written in closed form.
However, in the special case when Eq. 9 is underdetermined,meaning M < N ϕ , a closed-form (but suboptimal) solution can be obtained using the least-squares
SVM model [53].
The result is a generalized form of the pseudoinverse learning rule [43].
See
Appendix B for details.
Kernel attractor networks for continuous patterns
4.1 Auto -associative memory as a recurrent interpolation
network
So far, we have considered memory models designed to store only bipolar patterns.
We now relaxthis constraint and allow patterns to be continuous-valued.
We first observe that any set of patterns
X ∈ R N × M can be made fixed points of the dynamics by training each neuron i to interpolate ξ µiwhen the rest of the network is initialized in ξ µ , for every pattern µ .
Assuming that the model isequipped with a kernel that allows for each fixed point to also be attracting, we can ensure that alower bounding estimate of the size of the attractor basin is maximized by finding the interpolationwith minimum weight norms (see Appendix C for proof).
These results are summarized below.
Property 4 (Robust auto-associative memory with continuous patterns) .
Suppose that the dynamicsof a recurrent auto-associative memory network evolve according to the synchronous update rule
[Equation]
23) where K = K ( X , X ) =
ϕ ( X )
⊤ ϕ
( X ) is the kernel matrix and K † the Moore-Penrose pseudoinverse,where K † =
K − 1 if ϕ ( X ) is full column rank.
Then, the dynamics of the network is guaranteed tohave the fixed points X .
Moreover, if the points are attracting, Eq.
23 maximizes a lower-boundingestimate of the size of the attractor basins.
4.2 A recurrent SDM for continuous patterns
Memory models for continuous data have generally received less attention than their binary counterparts [22, 28].
Recently, however, Ramsauer et al.
[50] proposed an energy-based model capableof storing an exponential number of continuous-valued patterns (we will refer to this model as thesoftmax network).
While the structure of this model is similar to Eq.
23, it cannot be analyzed withinthe framework of Property 4, as it involves a kernel that is neither symmetric nor positive-definite[61].
Nonetheless, we will in this section demonstrate that it is possible to design an attractor network withconventional kernel methods with similar properties as the softmax network.
We generalize the SDMto continuous patterns and use a translation-invariant kernel with a fixed spatial scale r .
For the sake of simplicity, we choose the exponential power kernel (Exp β )
[Equation]
[Equation]
where β, r > 0 .
These parameters determine the shape of the hyperspherical attractor basin thatsurrounds each pattern.
While r roughly sets the radius of attraction, β represents an inversetemperature which changes the steepness of the boundary of the attractor basin.
Moreover, as long asthe patterns are unique, the kernel matrix is invertible, and we have K † exp β
= K − 1 exp β [39].
We will now analyze the noise robustness and storage capacity of this model.
To make the analysistractable, we will operate in the regime of low temperatures, meaning the limit β → ∞ .
We firstestablish the following property.
Property 5 (The Exp β network at zero temperature) .
Given a set of unique patterns { ξ µ } Mµ =1 withmin µ,ν ̸ = µ ∥ ξ µ − ξ ν ∥ 2 > r , the state update rule for the Exp β network at β → ∞ reduces to
[Equation]
where Θ( · ) is the Heaviside function with Θ(0)
=
e − 1 (see Appendix C.1.1).
Remark.
In geometrical terms, Property 5 states that the boundary of the basin of attraction surrounding each pattern becomes a sharp ( N − 1) -dimensional hypersphere with radius r in the limit β → ∞ .
For lower, finite β , the spherical boundary becomes increasingly fuzzy.
From the perspective of anenergy landscape, each pattern lies in an N -dimensional energy minimum with infinitely steep wallswhen β → ∞ .
As β is lowered, the barriers become progressively smoother.
Property 6.1 (Convergence in one step) .
Given a set of unique patterns { ξ µ } Mµ =1 withmin µ,ν ̸ = µ ∥ ξ µ − ξ ν ∥ 2 > 2 r , the Exp β network at β → ∞ , initialized at s (0) =
ξ µ + ∆ ξ , willconverge to ξ µ in one step if ∥ ∆ ξ ∥ 2 < r .
Property 6.2 (No spurious attractors) .
Given a set of unique patterns { ξ µ } Mµ =1 with min µ,ν ̸ = µ ∥ ξ µ −ξ ν ∥ 2 > 2 r and ∄ µ : ∥ ξ µ ∥ 2 = r/ (1 − e − 1 ) , the only attractors of the dynamics of the Exp β networkat β → ∞ are the set of points [Equation] [Equation]
Remark.
Properties 6.1 and 6.2 can be shown to be true simply by inserting the expression s (0) =
ξ µ + ∆ ξ in Eq.
Assuming no overlaps between the basins of attraction, a quick calculationshows that s (1)
= ξ µ if ∥ ∆
ξ ∥ 2 < r .
If, on the other hand, the network is initialized such that∥ s (0) − ξ µ ∥ 2 > r , ∀ µ , one always obtains s (2) =
ξ 0 , where ξ 0 is either 0 or the pattern closest to 0 .
In other words, the network recalls a pattern only if the state of the network is close enough to it.
Ifthe state is located far from all patterns, the network assumes an “agnostic” state, represented eitherby the origin or the pattern closest to the origin (if the origin happens to be located within a basin ofattraction).
Capacity. We now evaluate how the radius of attraction r determines the maximum level of inputnoise the network can handle and still recover each pattern in at least 50% of trials. We also derive alower bound of the storage capacity, as a function of the noise tolerance.
Property 7 (Robustness to white noise) .
Assume that we are given a set of unique patternsξ 1 , . . .
, ξ M ∼ N ( 0 , I N ) with min µ,ν ̸ = µ ∥ ξ µ − ξ ν ∥ 2 > 2 r , and that the Exp β network is initialized in a distorted pattern s (0)
=
ξ µ + ϵ , where ϵ ∼ N ( 0 , σ 2 I N ) .
Then, at β → ∞ , the minimumradius r min with which ξ µ can be recovered in at least 50% of trials is
[Equation]
Property 8 (Exponential storage capacity) .
At β → ∞ , and for N ≫ 1 , the maximum number ofpatterns that the Exp β network can store and recall without errors is lower-bounded by
[Equation]
[Equation] when each pattern ξ ∈ R N is a random sample from N ( 0 , I N ) and the radius of attraction is largeenough to tolerate white noise fluctuations with a maximum variance σ 2 .
Remark.
Proofs can be found in Appendices C.1.2 and C.1.3.
The derivation of Property 8 relies onbounds that are valid when N is large and when the noise tolerance is moderate, roughly in the rangeσ 2 ≲ 1 / 2 .
While the bounds are fairly tight when σ approaches the higher end of the range, thebounds become loose when σ → 0 .
In this regime, the radius of attraction approaches zero, but thestorage capacity of the Exp β network becomes, in fact, infinite. The reason for this is that the risk ofinterference between patterns becomes highly unlikely when their radius of attraction is very small.
Comparison with the softmax network.
When the patterns in X are randomly placed on ahypersphere instead of being normally distributed, the state update rule in Eq.
25 reduces to the form
X Θ( X ⊤ s ( t ) − θ ) , where θ is a fixed threshold, but the capacity remains exponential (see Appendix
C.2.1). In this case, the basin of attraction surrounding each pattern forms a spherical cap.
ChangingΘ to a softmax , so that the update can be written X softmax( X ⊤ s ( t ) )
[50], changes the shape ofeach basin from having spherical symmetry to being a Voronoi cell with smooth boundaries. Atzero temperature the softmax becomes an arg max , and the basins of attraction parcelate the patternspace into a Voronoi diagram, so that the extent of each basin is maximal in all directions, withoutoverlapping another basin.
The softmax network can, thus, be seen as an interpolation (Eq. 23) with K = softmax and the approximation K ≈ I M .
Discussion
Biological interpretation.
Kernel attractor networks can be mapped to the anatomical propertiesof biological neurons.
Consider an individual neuron in the feature representation of the recurrentnetwork (Eq. 10).
The state of neighboring neurons s is first transformed through ϕ ( s ) and thereafterprojected to the neuron through the weight matrix ( A ⊙ X ) ϕ
( X ) ⊤ .
When the kernel is polynomialof degree p > 1 , so that K ( x i , x j )
=
( x ⊤ i x j + 1) p , the transformation ϕ ( s ) consists of all elementsin s and their cross-terms, up to degree p .
The input to each neuron, in other words, consists of thestates of all other neurons, as well as all possible combinations of their multiplicative interactions.
This neuron model can be viewed as a generalized form of, for example, the multiconnected neuron[42], the clusteron [38], or the sigma-pi unit [51, ch. 2].
These are all perceptrons that includemultiplicative input interactions as a means to model synaptic clustering on non-linear dendrites [48].
In the kernel representation (Eq. 11), each neuron is, again, implicitly comprised of a two-stageprocess, whereby the raw input s is first transformed through the function K ( X , s ) and then projectedthrough the weight matrix ( A ⊙ X ) .
For any inner-product kernel K = k ( x ⊤ i x j ) , the kernelrepresentation can be directly identified as a two-layer neural network, with a hidden layer definedby the weights X and the activation function k .
This interpretation of the recurrent network wasrecently proposed by Krotov and Hopfield [30, 31] and discussed in relation to hippocampal-corticalinteractions involved in memory storage and recall; it is particularly reminiscent of the hippocampalindexing theory [5, 54].
However, the kernel representation can also be viewed as a network in whicheach individual neuron is a generalized form of the two-layered pyramidal cell model [46, 47], whichwas originally proposed as an abstract neuron model augmented with non-linear dendritic processing[36].
It should be noted, however, that the idea of interpreting kernel methods as neural networks hasa longer history, and has been extensively analyzed in the case of, for example, radial basis functions[44, 45].
Summary.
We have shown that conventional kernel methods can be used to derive general expressions for the optimal weights in hetero- and auto-associative memory networks storing binaryor continuous-valued patterns with maximal noise tolerance.
These results give rise to a family ofoptimal state update equations which include, as special cases, well-known memory models, suchas Kanerva’s SDM and MHNs.
We show how the optimal memory models, which we call kernelattractor networks , make it easy to understand the storage capacity of the SDM and MHNs, usingstandard results from the literature on linear classifiers. They also offer new ways to biologicallyinterpret MHNs in terms of non-linear dendritic integration.
In conclusion, our work formalizes the links between kernel methods, attractor networks, and modelsof active dendritic processing.
Our hope is that a proper understanding of the theoretical overlapof contemporary memory models can facilitate the development of qualitatively new and improvedmodeling approaches.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[N/A] Our work is entirely theoreticaland the limitations of our approach is treated in the main text, when discussing, forexample, the validity and assumptions of our derivations.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[N/A]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] This is included in all theoretical results, or in the explanatory text preceding them.
(b) Did you include complete proofs of all theoretical results?
[Yes] These can be found
in the Appendix.
Pointers to the correct sections are given in the main text.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[N/A]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[N/A]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did
you mention the license of the assets?
[N/A](c)
Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’reusing/curating?
[N/A](e)
Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?
[N/A]5.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A](b)
Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A](c)
Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]
Appendices
A Kernel of SDM on the hypersphere
A.1 Derivation of Eq. 20.
We follow the same steps as Bricken and Pehlevan [6], with additional simplifications towards theend.
As stated in the main text, we seek to calculate the overlapping area of two hyperspherical caps.
A formula for this can be found in [33], and is
[Equation]
29) where
[Equation]
where R is the radius, I is the regularized incomplete Beta function, and
[Equation] [Equation]
[Equation]
We insert Eq. 31 in 33 and obtain
[Equation] [Equation]
where we have used tan( α/ 2)
= (1 − cos( α )) / sin( α ) .
Eq.
35 also follows from the symmetry ofthe problem.
This result yields
[Equation]
[Equation]
We rewrite the regularized incomplete Beta function as
[Equation]
where B is the incomplete Beta function.
The area of an ( N in − 1)
-dimensional
hypersphere is
[Equation]
We insert Eq.
37 in
and use the result in Eq. 36.
Using the notation α b = α 1 and α x = α min , andthe identities
[Equation]
[Equation]
the ratio between the overlapping area of the hyperspherical caps and the complete are of thehypersphere can now be calculated as
[Equation]
A.2 Derivation of Eq.
For a large bias b ≳ 0 . 9 , which is equivalent to a small angle α b = arccos( b ) , the hypersphericalcaps surrounding x i and x j will be very small in relation to the whole hypersphere.
In this case, wecan neglect the curvature of the hyperspherical surface and project the area of the hypersphericalcap to the plane that cuts through the rims of the cap.
This projection is a ( N in − 2) -dimensionalhypersphere
(we will refer to it as a mini-sphere).
In three dimensions, for example, the projection ofa hyperspherical cap to the plane would constitute a circle, which is a 1-dimensional sphere.
Theradius of each mini-sphere is
[Equation]
[Equation]
and the distance between the centers of the mini-spheres is
[Equation]
We estimate the overlapping area of the two mini-spheres by calculating their overlapping volume in( N in − 1) dimensions.
The overlapping volume of two identical ( N in − 2) -dimensional hyperspherescan be found in [34] and is given by
[Equation]
We rewrite the regularized incomplete Beta function as
[Equation]
and insert Eq.
44 in 43.
The ratio between the overlapping area of the hyperspherical caps and thecomplete are of the hypersphere can now be estimated as
[Equation]
[Equation]
B Generalized pseudoinverse rule
When the network is linear and underdetermined, meaning M < N , we can make sure that allpatterns are attractors by modeling each neuron as a least-squares SVM [53] instead of a conventional
SVM, so that the weights satisfy
[Equation]
This is a minimum-norm interpolation problem, and yields the solution
[Equation]
where K =
X
⊤ X is the kernel matrix and K † its Moore-Penrose pseudoinverse, and where we haveused the property K † X ⊤ =
( X ⊤ X ) † X ⊤ = X † .
This is the pseudoinverse learning rule [43].
The derivation be extended to MHNs by performing interpolation on the the feature map ϕ ( ξ µ ) .
Assuming that the problem is still underdetermined, so that M < N ϕ , we aim to find the weights
[Equation]
(48) which, analogously to the linear case, produces the optimal state update
[Equation]
(49) where K = K ( Ξ , Ξ )
=
ϕ
( Ξ ) ⊤ ϕ
( Ξ ) .
This can, again, be simplified to
[Equation]
[Equation]
where we can identify the weight matrix W =
Ξ ϕ ( Ξ ) † .
This is the generalized pseudoinverselearning rule .
Note that, if the feature-expanded prototypes { ϕ ( ξ µ ) } Mµ =1 are linearly independent,the kernel matrix is invertible and we have K † =
K − 1 .
C Minimum norm interpolation and attractor basin size
Proof of Property 4.
In this setting, each neuron is modeled as linear regressor whose state isupdated according to
[Equation]
All patterns X are guaranteed to be fixed points of the dynamics by finding the weights that satisfy
[Equation]
[Equation]
In order for each pattern to also be an attractor, the weights must satisfy the added constraint
[Equation]
[Equation]
where we have used the shorthand s
= s (1) , s
= s (0) , and J s is the Jacobian with respect to s 0 .
Eq.
53 states that the spectral norm of the Jacobian of the state update rule must be less than 1when evaluated in each pattern.
This ensures that the fixed points { ξ µ } Mµ =1 are stable attractors and,equivalently, that the state update rule is a contractive map in a region surrounding each pattern [49].
It is assumed that there exists a solution that satisfies both Eqs. 52 and 53.
The basin of attraction ofa fixed point µ can be defined as the set of points S µ edge in the open neighborhood of ξ µ satisfying
[Equation]
Given that the spectral norm of the Jacobian is upper bounded by
[Equation]
55) where ∥·∥ F is the Frobenius norm, we can obtain a lower bound of the extent of the basin of attractionwith the set
[Equation]
Since the Jacobian is given by
[Equation]
) (57) where W = ( w 1 , . . . , w N ) ⊤ , we obtain
[Equation]
58) where the last expression is given by the Cauchy-Schwartz inequality.
Since ∥ J s ( ϕ ( s 0 ))
∥ F is aconstant determined by the kernel in combination with the patterns, Eq.
58 can only be minimized byfinding a set of weights that minimize ∥ W ∥ F .
This is precisely the minimum norm interpolation,given by
[Equation]
[Equation]
The solution is stated in Eq.
This solution maximizes a lower bound of the attractor basin sizesby maximizing the size of the set (cid:98) S µ edge .
□
C.1 Normally distributed patternsC.1.1 Kernel at zero temperature
Proof of Property 5.
Using the notation ∆ =
∥
ξ µ − ξ ν ∥ 2 , we have that
[Equation]
(60) from which it follows that
[Equation]
which is equivalent to Θ( r − ∆) with Θ(0)
=
e − 1 .
We combine this with the assumption that thepatterns are unique and that min µ,ν ̸ = µ ∥ ξ µ − ξ ν ∥ 2 > r and obtain
[Equation]
which can be written compactly as
[Equation]
It directly follows that
[Equation]
and, therefore,
[Equation]
C.1.2 Noise robustness
Proof of Property 7.
With s (0) =
ξ µ + ϵ , we have
[Equation]
[Equation] where ϵ 0 ∼ N ( 0 , I N ) , from which it follows that ∥ ϵ 0 ∥ 22 is a random variable with a χ 2 ( N ) distribution.
According to the central limit theorem, we also have
[Equation]
67) where we have used the fact that each term in ∥ ϵ 0 ∥ 22 is χ 2 (1) -distributed, and has mean 1 and variance2 .
We will therefore make the approximation ∥
ϵ 0 ∥
22 ∼ N ( N, 2 N ) for large N .
This gives usσ
[Equation] [Equation]
N ) . The original pattern ξ µ will only be recovered if r 2 − ∥ ξ µ − s (0) ∥ 22 ≥ 0 , which is satisfied in at least50% of trials if r ≥ σ √ N .
The minimum radius with which this type of recovery still holds is thus
[Equation]
[Equation]
C.1.3 Storage capacity
Proof of Property 8.
In the limit β → ∞ , the boundary of the basin of attraction surrounding eachpattern is sharp.
In this setting, we are guaranteed that each pattern can be recalled without errorsas long as min i,j ̸
= i ∥ ξ i − ξ j ∥ 2 > 2 r .
We will therefore estimate the storage capacity by calculatingthe number of patterns, on average, that can be loaded into the network before at least two attractorbasins overlap and the condition above is violated.
We begin by observing that for two random patterns ξ i , ξ j ∼ N ( 0 , I N ) , we have
[Equation]
70) which, using the central limit theorem as in Eq.
67, can be approximated as N ( N, 2 N ) for large N ,thereby yielding
[Equation]
[Equation] We now assume that the squared Euclidean distance between each pair of patterns ξ i , ξ j in a setof M given patterns { ξ µ } Mµ =1 is an independent sample of a random variable, denoted ∆ 2 , whichis distributed as in Eq. 71.
This is, of course, an approximation which neglects that the pairwisedistances between any set of points are inter-dependent.
Nonetheless, for relatively large N and M ,the approximation accurately describes the empirical distance distribution.
Relying on this assumption, the process of drawing M random patterns becomes equivalent todrawing M ( M − 1) / 2 unique pairwise distances ∆ 2 from the distance distribution.
An upper boundof the probability of drawing a sample ∆ 2 ≤ 4 r 2 can be calculated using the cumulative distributionfunction for the standard normal distribution, given by Φ( x ) = 12 erfc( − x ) , according to
[Equation]
The average number of samples of ∆ 2 one needs to draw before a sample satisfies ∆ 2 ≤ 4 r 2 is givenby P (∆ 2 ≤ 4 r 2 ) − 1 .
This determines the maximum number of patterns that the network, on average,can store, according to
[Equation]
We combine this expression with the approximation M max ( M max − 1) ≈ M 2max (which holds forlarge M max ) and Eq. 72, and obtain
[Equation]
We now parameterize the radius r in terms of the largest tolerable noise amplitude, according to Eq.69.
This gives us
[Equation]
Given that the erfc function can be well approximated using the asymptotic expansion
[Equation]
for large arguments x , we can obtain a tight lower bound of erfc − 1 as long as N is large andσ ≲ 1 / √ 2 with the inverse zeroth order expansion, thereby obtaining
[Equation]
We insert Eq.
77 in [-
-]75 and finally obtain
[Equation]
C.2 Patterns on the hypersphereC.2.1 Storage capacity
Property 9 (Storage capacity: patterns on the hypersphere) .
At β → ∞ , the maximum number ofpatterns that the Exp β network can store and recall without errors is lower-bounded by
[Equation]
when each pattern ξ µ is randomly drawn from S N .
Proof.
We begin by observing that for two random patterns ξ i , ξ j ∈ S N , we have
[Equation]
The probability distribution for the inner product ω = ξ µ ⊤ ξ ν can be found in [55], and is given by
[Equation]
which, for large N , can be approximated as
[Equation]
[Equation]
We use Eq.
82 in
and obtain
[Equation]
[Equation]
The probability of placing a pair of random points on S N with ∆ 2 ≤ 4 r 2 is thus
[Equation]
The average number of samples of ∆ 2 one needs to draw before a sample satisfies ∆ 2 ≤ 4 r 2 is givenby P (∆ 2 ≤ 4 r 2 ) − 1 , and the maximum number of patterns that the network, on average, can store, istherefore
[Equation]
We use the lower bound in Eq.
77 in 85 and finally obtain
[Equation]
C.3 Bipolar patternsC.3.1 Noise robustness
Property 10.1 (Robustness to flipped bits) .
Assume that we are given a set of unique patternsx 1 , . . .
, x M ∈ {− 1 , +1 } N with min µ,ν ̸ = µ ∥ x µ − x ν ∥ 2 > 2 r , and that the Exp β network is initializedin a distorted pattern s (0)
= x µ ⊙ ϵ , where ϵ ∈ {− 1 , +1 } N , with P ( ϵ i = − 1)
= ρ, ∀ i .
Then, atβ → ∞ , the minimum radius r min with which x µ can be recovered in at least 50% of trials is
[Equation]
Proof of Property 10.1.
With s (0) =
ξ µ ⊙ ϵ , we have
[Equation] [Equation]
where ϵ B ∈ { 0 , 1 } N , with each entry being a random variable distributed as ( ϵ B ) i ∼ Bernoulli( ρ ) .
This implies that ∥ ϵ B ∥ 22 ∼ Binomial( N, ρ ) , which can be approximated as N ( ρN, ρ (1 − ρ ) N ) forlarge N .
This gives
[Equation]
Again, the original pattern ξ µ will only be recovered if r 2 − ∥ ξ µ − s (0) ∥ 2 2 ≥ 0 , which is satisfied inat least 50% of trials if r ≥ 2 √ ρN .
The minimum radius with which this type of recovery still holdsis thus
[Equation]
C.3.2 Storage capacity
Property 10.2 (Storage capacity: bipolar patterns) .
At β → ∞ , the maximum number of patternsthat the Exp β network can store and recall without errors is lower-bounded by
[Equation]
when each pattern x ∈ {− 1 , +1 } N is randomly drawn according to P ( x i = 1)
= f, ∀ i , where˜ f = 2 f (1 − f ) and where the radius of attraction is large enough to tolerate a maximum bit-wiseerror probability ρ .
Proof of Property 10.2.
This proof is, again, a slightly modified variant of the proof of Property8.
First, we observe that for two random patterns ξ µ , ξ ν ∈ {− 1 , +1 } N with sparseness f , so that 676 P ( x µ,νi = 1) =
f , it is true that
[Equation]
where ˜ f
= 2 f (1 − f ) denotes the probability that ξ µ and ξ ν differ at any given entry.
For large N ,we can again approximate Binomial( N, ˜ f ) with N ( ˜ fN, ˜ f
(1 − ˜ f ) N ) , which gives us
[Equation]
[Equation]
We use this to compute the upper bound of the probability of drawing a distance ∆ 2 which satisfies∆ 2 ≤ 4 r 2 , as in Eq. 72.
The result is
[Equation]
Following the same derivations used to produce Eqs.
73 and 74, we arrive at
[Equation]
As before, we parameterize the radius r in terms of the maximum tolerable bit-wise error probabilityaccording to Eq. 90 and yield
[Equation]
[Equation]
We finally replace erfc with the lower bound in Eq.
This is valid for large N and whenρ ≲ f (1 − f ) / 2 .
We obtain
M max ≥(cid:32)
[Equation]