Kernel Attractor Networks: A Unifying Framework for Memory Modeling
Anonymous Author(s)
AffiliationAddress
Abstract
We consider the problem of training a neural network to store a set of patternswith maximal noise robustness.
A solution, in terms of optimal weights and stateupdate rules, is derived by training each individual neuron to perform either kernelclassification or interpolation with a minimum weight norm.
By applying thismethod to feed-forward and recurrent networks, we derive optimal networks thatinclude, as special cases, many of the hetero- and auto-associative memory modelsthat have been proposed over the past years, such as modern Hopfield networksand Kanerva’s sparse distributed memory.
We generalize Kanerva’s model anddemonstrate a simple way to design a kernel attractor network that can store anexponential number of continuous-valued patterns with a finite basin of attraction.
The framework of kernel attractor networks offers a simple and intuitive wayto understand the storage capacity of previous memory models, and allows fornew biological interpretations in terms of dendritic non-linearities and synapticclustering.
Introduction
Although the classical work on attractor neural networks reached its peak in the late 1980’s, withthe publication of a number of seminal works [e.g., 2, 18, 20, 23], recent years have seen a renewedinterest in the topic, motivated by the popularity of the attention mechanism [60], external memoryaugmented neural networks [22, 61] as well as a new generation of energy-based attractor networksmodels, termed modern Hopfield networks (MHNs), capable of vastly increased memory storage[15, 32].
Recent efforts to understand the theoretical foundation of the attention mechanism have,in fact, shown that it can be linked to Hopfield networks [33, 52] but also to Kanerva’s sparsedistributed memory (SDM) [7, 27] and to the field of kernel machines [58, 63].
The last connectionis particularly intriguing, in light of the many theoretical commonalities between neural networksand kernel methods [8, 9, 25, 43, 62].
Overall, these results suggest that a unified view can offer newinsights into memory modeling and new tools for leveraging memory in machine learning.
In this work, we aim to clarify some of the overlap between the fields of memory modeling andstatistical learning, by integrating and formalizing a set of theoretical connections between Hopfieldnetworks, the SDM, kernel machines, and neuron models with non-linear dendritic processing.
1.1 Our contribution
• We derive a set of normative kernel-based models that describe the general mathematical structureof feed-forward (i e., hetero-associative) and recurrent (i e., auto-associative) memory networksthat can perform error-free recall of a given set of patterns with maximal robustness to noise.
• We show that the normative models include, as special cases, the classical and modern Hopfield network, as well as the SDM.
• We derive a simple attractor network model for storing an exponential number of continuous-valuedpatterns with a finite basin of attraction.
We
discuss
its similarity to attention.
Furthermore, we explain how classifiers with non-linear kernels can be interpreted as general formsof neuron models with non-linear dendritic activation functions and synaptic clustering.
1.2 Related work
Our work is primarily related to [7, 32, 33, 52].
While MHNs are extensively analyzed in [32,33, 52], the approach is energy-based and makes no statements about the relation between MHNsand other memory models or kernel methods; a brief comment mentions some similarity to SVMs,but this is not further explained [52].
The work by [7] focuses on the SDM and its connection toattention.
It observes that the classical Hopfield network is a special case of the SDM, but no furthergeneralization is made, and kernel methods are not mentioned.
In our work, we place MHNs and the SDM in a broader theoretical context by showing that both models are special suboptimal cases of afamily of memory networks that can be derived based on a normative kernel-based approach.
Background
Consider the following simple model of hetero-associative memory: a single-layer feed-forwardnetwork consisting of a single output neuron connected to N in inputs with the weights w ∈ R N ϕ .
The output s out ∈ {− 1 , +1 } is given by
[Equation]
(1) where s in is the input vector (also called query), θ the threshold, and ϕ a function that maps the“raw” input to a N ϕ -dimensional feature space, where typically N ϕ ≫ N in .
Suppose that we aregiven a set of M input-output patterns { ξ µ in , ξ µ out } Mµ =1 , where each entry ξ is randomly drawn from{− 1 , +1 } with sparseness f := P ( ξ =1) .
In order for the neuron to memorize each pattern µ with amaximal attractor basin, it needs to find a set of weights so as to generate the output ξ µ out in responseto the input ξ µ in in a manner that maximizes the Euclidean distance between the input and the decisionboundary.
Using Gardner’s formalism [18, 20], this problem can be expressed as
[Equation]
where ¯ w > 0 is a constant.
This is equivalent to solving
[Equation]
which can be directly identified as the support vector machine (SVM) problem for separable data[11].
The solution to Eq.
3 can today be found in any textbook on basic machine learning methods,and yields an optimal output that can be formulated in a feature and kernel representation
[Equation]
where we have used the “kernel-trick” K ( x i , x j )
= ϕ
( x i )
⊤ ϕ
( x j ) .
The solution depends on the
Lagrange coefficients α µ ≥ 0 , many of which are typically zero.
Patterns with α µ > 0 are calledsupport vectors .
2.1 Hetero -associative memory as a feed-forward SVM network
We now consider a hetero-associative memory network with an arbitrary number N out of outputneurons, whose combined state we denote s out .
In order for the network as a whole to successfullyrecall memories with maximal noise robustness, we solve Eq.
3 for each neuron independently.
Aseach neuron can have a different classification boundary along with a different set of support vectors,its weights will, in general, be characterized by an independent set of M Lagrange coefficients.
To simplify the notation, we represent these coefficients α µi , across neurons i and patterns µ , as entries inthe matrix A , where ( A ) iµ = α µi .
We also combine all thresholds in the vector θ = ( θ 1 , . .
, θ N out ) ,and all input and output patterns as columns in the matrices X in = ( ξ 1in , . . .
, ξ M in ) and X out =( ξ 1out , . . . , ξ M out ) .
Finally, we assume that all neurons have the same feature map, so that ϕ i
= ϕ , ∀ i(see Fig. 1).
The optimal response of the network is compactly summarized as follows.
Property 1 (Robust hetero-associative memory network) .
The optimal output of a single-layerhetero-associative memory network trained to recall the patterns X out in response to the inputs X in ,with maximal noise robustness, can be written in the form
[Equation]
feature)
[Equation] [Equation]
kernel)
[Equation]
6) where ⊙ denotes the Hadamard product.
Kernel attractor networks for binary patterns3.1 Auto -associative memory as a recurrent SVM network
We begin our results by first noting that the hetero-associative memory network can be made autoassociative by setting N out
= N in and X out
=
X in .
The network is now effectively recurrent, aseach neuron serves both as an input and output simultaneously (see Fig. 1).
Consider a recurrentnetwork with N neurons, whose state at time point t is denoted s ( t ) ∈ {− 1 , +1 } N , and whosedynamics evolve according to the update rule
[Equation]
(7) where w i ∈ R N ϕ is the weight vector to neuron i = 1 , . . .
, N .
In order to make the patterns{ ξ µ } Mµ =1 attractors of the network dynamics, each neuron i is, again, independently trained on everypattern µ to produce the response ξ µi when the rest of the network is initialized in ξ µ .
Moreover, thenoise robustness and, thereby, the size of the attractor basins, can be maximized by maximizing the
Euclidean distance between each neuron’s decision boundary and the patterns [16, 29].
The problemof training the entire network can, in this way, be transformed into the problem of training N separateclassifiers according to
[Equation]
The solution can be obtained by slightly modifying Property 1, and is stated below.
Property 2.1 (Robust auto-associative memory) .
A recurrent auto-associative memory networktrained to recall the patterns X with maximal noise robustness has an optimal synchronous updaterule that can be written in the form
[Equation]
Remark.
With a linear feature map ϕ ( x ) = x , the optimal update is reduced to
[Equation]
where W is the general form of the optimal weight matrix.
The solution described by Property 2.1 does not, in general, prohibit a neuron from having selfconnections.
Applying this constraint yields the following result.
Property 2.2 (Robust auto-associative memory without self-connections) .
A recurrent autoassociative memory network without self-connections, that has been trained to recall the patterns
X with maximal noise robustness, has an optimal update rule that can be written in the kernelrepresentation [Equation] 
[Equation]
where k is an inner-product kernel, i.e. K ( x i , x j )
=
k ( x ⊤ i x j ) .
𝐬 !"
𝑠 #$%,’ =
sgn 𝐰 ’( 𝝓 ’ 𝐬 !"
− 𝜃 ’ 𝐰 !
𝝓 ! 𝐬 !"
𝐬 #$% = sgn 𝐖𝝓 𝐬 !" − 𝜽 𝐖 𝝓 𝐬 !" 𝐬 #$% = sgn
𝐖𝚯 𝐙𝐬 !"
− 𝑏 𝐖 𝐙
A
B
C 𝑠 ’
(*+,) =
sgn 𝐰 ’( 𝝓 ’ 𝐬 (*) − 𝜃 ’ 𝐰 !
𝝓 !
D
E
F 𝑠 .
(*) 𝑠 ’
(*+,) 𝝓 !
𝐖 𝝓 𝐬 (*+,) = sgn
𝐖𝝓 𝐬 (*) − 𝜽
Remark.
When the kernel is linear, i.e. k ( x ⊤ i x j ) =
x ⊤ i x j , the update reduces to
[Equation]
[Equation]
Storage capacity.
An intuition for the storage capacity scaling of the hetero- and auto-associativememory network can be gained by observing that the network as a whole will be able to successfullyrecall patterns as long as each neuron is able to correctly classify its inputs (or is very unlikely toproduce an error).
The capacity of the network can thereby be derived from the capacity of eachindividual neuron.
It is well-known that a linear binary classifier can learn to correctly discriminate amaximum of M max ≈ 2 D VC random patterns, where D VC is the Vapnik-Chervonenkis dimensionof the model [13, 18, 37, ch. 40].
When the neuron in Eq.
7 has a linear feature map ϕ ( x ) =
x ,this results in D VC = N and, thus, the capacity M max ≈ 2 N .
Suppose, on the other hand, thatthe kernel is a homogeneous polynomial of degree p , so that K ( x i , x j ) =
( x ⊤ i x j ) p .
In this case, ϕwill contain all monomials of degree p composed of the entries in x .
As there are O ( N p ) uniquep -degree monomials (see Appendix A), the input dimensionality and M max will be O ( N p ) .
Forthe exponential kernel, which we can write as K ( x i , x j ) =
exp( x ⊤ i x j ) = (cid:80)
∞
p =0 ( x ⊤ i x j ) p /p !
, thedimensionality of ϕ will be (cid:80) Np =0 (cid:0) Np (cid:1)
= 2 N , which gives M max ∼ O ( e N ) .
Application to special cases.
In the following sections, we will show that many of the modelsof hetero- and auto-associative memory that have been proposed over the past years can be seen asspecific cases of the solutions in Properties 1, 2.1, and 2.2, characterized by specific choices of Aand functions ϕ and K .
3.2 Kanerva’s sparse distributed memory is a feed-forward SVM network
The sparse distributed memory (SDM), developed by Kanerva [27], is one of the most famousexamples of a hetero-associative memory model.
It has lately received much attention in the contextof generative memory models [64] and attention layers in transformers [7].
The SDM consists of a register of N ϕ memory slots, each associated with an address
z i ∈{− 1 , +1 } N in , i = 1 , . . .
, N ϕ .
All addresses are listed as rows in the matrix Z
= ( z 1 , . . .
, z N ϕ ) ⊤ .
The content of each slot is represented by an N out -dimensional vector, initialized at zero.
Supposethat we wish to store the M patterns X out = ( ξ 1out , . .
. , ξ M out ) in the addresses X in = ( ξ 1in , . . .
, ξ M in ) ,where all entries are random and bipolar.
The basic idea of the SDM is to write the data to, and laterread it from, multiple memory slots at once (hence the distributed storage); this ensures a degree ofnoise-robustness.
In mathematical terms, the read-out of the SDM provided with a query s in , is givenby
[Equation]
(14) where Θ the Heaviside function with bias b = N in − 2 r , and r is a parameter that determines theprecision of the writing and reading process.
Upon comparing Eqs.
14 and 5, the SDM can be directlyidentified as a special case of a suboptimal feed-forward SVM network in the feature representation,with A
= 1 , θ = 0 , and the feature map ϕ SDM ( x ) =
Θ( Zx − b ) .
When viewed as a kernelmethod, the function of the SDM is to store the dense addresses X in as sparse high-dimensionalrepresentations ϕ SDM , to make it easier to later determine the slots closest to a query s in , and retrievethe relevant data.
Capacity.
As the SDM is linear in ϕ SDM , with D VC ≈ N ϕ , it follows from the analysis in Sec.3.1 that one should expect the capacity to scale as M max ∼ O ( N ϕ ) .
Moreover, one should expecta proportionality constant ∼ 0 .
1 , since the SDM is suboptimal relative to the feed-forward SVMnetwork, analogously to how the classical Hopfield network is suboptimal relative to the recurrent SVM network (see Sec. 3.3).
This is consistent with earlier proofs [10, 28].
Kernel of an infinite SDM.
In practice, an SDM with a large number of memory slots N ϕ requirescalculations involving a large address matrix Z .
This can be avoided by applying the kernel-trick inthe limit N ϕ → ∞ .
In this section, we will derive the kernel corresponding to two different variantsof the SDM and demonstrate that both are translation-invariant.
To ensure convergence, we willdefine the kernel as
[Equation]
It is interesting to note here that ϕ SDM is equivalent to a single-layer neural network with N ϕ neurons,weights Z , and bias b .
This means that K SDM is equivalent to the kernel of an infinitely wide neuralnetwork [9, 43, 62].
We begin by noticing that
ϕ
SDM ( x ) has a geometrical interpretation [7, 28].
It is a binary vectorthat indicates those memory addresses in Z that differ by at most r bits compared to x .
For anytwo bipolar vectors z and x , the bit-wise difference can be computed as 12 | z − x | = 14 ∥
z − x ∥ 22 .
This means that ϕ SDM ( x ) indicates all addresses that lie within a sphere centered at x with radius2 √ r .
Consequently, the inner product ϕ SDM ( x i )
⊤ ϕ SDM ( x j ) is the number of addresses locatedin the overlapping volume of two spheres centered at x i and x j .
Although an exact calculationof this quantity can be found in [7, 27], its connection to the SDM kernel has, to the best of ourknowledge, not previously been made.
We therefore modify the previously published expression witha normalization factor 1 / 2 N in and state the following property.
Property 3.1 (Kernel of an infinite SDM on the hypercube) .
In the limit N ϕ → ∞ , the read-out ofan SDM with N ϕ memory slots, storing M patterns X out in the addresses X in , is given by
[Equation]
16) where
[Equation]
and where r is the bit-wise error threshold and ∆ is the bit-wise difference between x i and x j , givenby ∆ = 12 | x i − x j | = 14 ∥ x i − x j ∥ 22 .
The SDM can also be implemented with continuous addresses, randomly placed on a unit hypersphereof ( N in − 1) dimensions, denoted S N in − 1 .
The vector ϕ SDM ( x ) now indicates all addresses that liewithin a hyperspherical cap centered at x with an angle arccos( b ) between its central axis and the rim.
The inner product ϕ SDM ( x i )
⊤ ϕ SDM ( x j ) is the number of addresses located in the overlapping areaof two spherical caps centered at x i and x j .
While a calculation of this quantity, again, can be foundin [7], it has not previously been connected to the kernel of an SDM.
We simplify the previouslypublished result and also derive a closed-form approximation, valid for highly sparse ϕ SDM (see
Appendix B for details).
The results are summarized below.
Property 3.2 (Kernel of an infinite SDM on the unit hypersphere) .
In the limit N ϕ → ∞ , the read-outof an SDM with N ϕ memory slots, storing M patterns X out in the addresses ξ µ in ∈ S N in − 1 , ∀ µ , isgiven by Eq.
16 with the kernel
[Equation]
where α x = 1 2 arccos( x ⊤ i x j ) , α b = arccos( b ) , and B is the incomplete Beta function (details in
Appendix B.1).
In the highly sparse regime, when 0 . 9 ≲ b < 1 and 1 N ϕ ∥
ϕ
SDM ∥ 0 ≪ 1 , the kernel canbe approximated with
[Equation]
[Equation] where ∆ = 12 ∥ x i − x j ∥ 2 and ˆ b = sin(arccos( b )) (details in Appendix B.2).
In conclusion, an infinitely large SDM with sparse internal representations ϕ SDM , can be representedas a suboptimal case of a feed-forward SVM network with a translation-invariant kernel.
3.3 The modern Hopfield network is a recurrent SVM network
The Hopfield network [23] is, arguably, the most well-known model of auto-associative memory.
Inits modern form [32], it is a recurrent network of N neurons with the state s ( t ) , whose dynamics aregoverned by the energy and state update rule
[Equation]
where F is a smooth function, typically a sigmoid, polynomial, or exponential.
This “generalized”
Hopfield model has a long history [see, e.g., 1, 19, 34] but has received renewed attention in recentyears under the name modern Hopfield network (MHN) or dense associative memory [15, 32].
Bycomparing Eq.
20 with Eq.
12, the state update of the MHN can be identified as a special case of asuboptimal recurrent SVM network in the kernel representation, with k = F ′ , A = 1 , and θ = 0(since f = 0 . 5 ).
With a linear F ′ ( x ) = x , the MHN reduces to the classical Hopfield network, whichis a special case of the linear SVM network in Eq. 13.
Capacity.
The storage capacity of the MHN has been shown to depend on the shape of F ′ .
In thelinear case, the capacity is famously limited to ∼ 0 .
1 N patterns, depending on the precision of retrieval[2, 39].
If, on the other hand, F ′ is polynomial with degree p , the capacity scales as M max ∼ O ( N p )[32].
Analogously, an exponential F ′ endows the network with a capacity M max ∼ O
( e N ) [15].
From the perspective of the kernel attractor framework, this scaling directly follows from the analysisin Sec.
3.1 with k = F ′ .
Generalization.
Viewing the MHN as a recurrent network of SVMs can also facilitate a moreintuitive understanding of its ability to generalize, when used as a conventional classifier.
In thissetting, one designates a subset of the neurons as input units, and the remaining neurons as outputs.
Given a set of input-output associations, one optimizes the memory patterns ξ µ using, for example,gradient descent.
Such an experiment was performed by Krotov and Hopfield [32] on the MNISTdata set, using a polynomial non-linearity F ( x ) = x p .
Their results showed that the test error firstimproved as p increased from 2 to 3, but later deteriorated for high degrees, like p = 20 .
While itmay be difficult to explain this behavior within an energy-based framework, it is entirely expectedwhen viewed from the SVM perspective: a kernel of low polynomial degree has too few degrees offreedom to fit the classification boundary in the training set, causing underfitting , while a polynomialof too high degree grants the model too much flexibility, which results in overfitting .
Online learning.
Recently, Tyulmankov et al.
[59] proposed a biologically plausible online learningalgorithm to train the weights in an MHN.
Depending on the implementation, the resulting storagecapacity scales as either ∼ 0 .
16 N or ∼ N .
In a comment on the discrepancy between these resultsand the maximum capacity ∼ 2 N , the authors noted that they are not aware of a learning algorithmthat is able to reach the upper bound.
Here, the perspective of kernel attractor networks can beparticularly helpful, as many of the algorithms that have been developed over the past two decadesto optimize SVMs can be utilized for MHNs as well.
For example, a network formulated in thefeature representation can be trained with the stochastic batch perceptron rule [12, 31], the passiveaggressive rules [14], the minnorm rule [4], as well as with likelihood maximization applied tologistic regression [26, 42, 54].
In the kernel representation, two of the most well-known onlinealgorithms for training linear and non-linear SVMs are the Adatron [3] and the Kernel-Adatron [17].
A performance comparison between online learning and the modern Hopfield learning rule in an
SVM network can be found in Appendix C.
The pseudoinverse learning rule.
The coefficients in A are, in general, computed numerically,and cannot be written in closed form.
However, in the special case when Eq. 8 is underdetermined,meaning M < N ϕ , a closed-form (but suboptimal) solution can be obtained using the least-squares
SVM method [55].
The result is a generalized form of the pseudoinverse learning rule [45].
See
Appendix D for details.
Kernel attractor networks for continuous patterns
4.1 Auto -associative memory as a recurrent interpolation network
So far, we have considered memory models designed to store only bipolar patterns.
We now relaxthis constraint and allow patterns to be continuous-valued.
We first observe that any set of patterns
X ∈ R N × M can be made fixed points of the dynamics by training each neuron i to interpolate ξ µiwhen the rest of the network is initialized in ξ µ , for every pattern µ .
Assuming that the model isequipped with a kernel that allows for each fixed point to also be attracting, we can ensure that alower bounding estimate of the size of the attractor basin is maximized by finding the interpolationwith minimum weight norms (see Appendix E.1 for proof).
These results are summarized below.
Property 4 (Robust auto-associative memory with continuous patterns) .
Suppose that the dynamicsof a recurrent auto-associative memory network evolve according to the synchronous update rule
[Equation]
(21) where K = K ( X , X ) =
ϕ ( X )
⊤ ϕ
( X ) is the kernel matrix and K † the Moore-Penrose pseudoinverse,where K † =
K − 1 if ϕ ( X ) is full column rank.
Then, the dynamics of the network is guaranteed tohave the fixed points X .
Moreover, if the points are attracting, Eq.
21 maximizes a lower-boundingestimate of the size of the attractor basins.
4.2 A recurrent SDM for continuous patterns
Memory models for continuous data have generally received less attention than their binary counterparts [24, 30].
Recently, however, Ramsauer et al.
[52] proposed an energy-based model capableof storing an exponential number of continuous-valued patterns (we will refer to this model as thesoftmax network).
While the structure of this model is similar to Eq.
21, it cannot be analyzed withinthe framework of Property 4, as it involves a kernel that is neither symmetric nor positive-definite[63].
Nonetheless, we will in this section demonstrate that it is possible to use conventional kernel methodsto design an attractor network with similar properties as the softmax network.
We generalize the SDM to continuous patterns and use a translation-invariant kernel with a fixed spatial scale r .
For thesake of simplicity, we choose the exponential power kernel (Exp β )
[Equation]
where β, r > 0 .
These parameters determine the shape of the hyperspherical attractor basin thatsurrounds each pattern.
While r roughly sets the radius of attraction, β represents an inversetemperature which changes the steepness of the boundary of the attractor basin.
Moreover, as long asthe patterns are unique, the kernel matrix is invertible and we have K † exp β
= K − 1 exp β [41].
We will now analyze the noise robustness and storage capacity of this model.
To make the analysistractable, we will operate in the regime of low temperatures, meaning the limit β → ∞ .
We firstestablish the following three properties.
Property 5.1 (The Exp β network at zero temperature) .
Given a set of unique patterns { ξ µ } Mµ =1 withmin µ,ν ̸ = µ ∥ ξ µ − ξ ν ∥ 2 > r , the state update rule for the Exp β network at β → ∞ reduces to
[Equation]
(23) where Θ( · ) is the Heaviside function with Θ(0)
=
e − 1 (see Appendix E.2.1).
Remark.
In geometrical terms, Property 5.1 states that the boundary of the basin of attractionsurrounding each pattern becomes a sharp ( N − 1) -dimensional hypersphere with radius r in thelimit β → ∞ .
For lower, finite β , the spherical boundary becomes increasingly fuzzy.
From theperspective of an energy landscape, each pattern lies in an N -dimensional energy minimum withinfinitely steep walls when β → ∞ .
As β is lowered, the barriers become progressively smoother.
Property 5.2 (Convergence in one step) .
Given a set of unique patterns { ξ µ } Mµ =1 withmin µ,ν ̸ = µ ∥ ξ µ − ξ ν ∥ 2 > 2 r , the Exp β network at β → ∞ , initialized at s (0) =
ξ µ + ∆ ξ , willconverge to ξ µ in one step if ∥ ∆ ξ ∥ 2 < r .
Property 5.3 (No spurious attractors) .
Given a set of unique patterns { ξ µ } Mµ =1 with min µ,ν ̸ = µ ∥ ξ µ −ξ ν ∥ 2 > 2 r and ∄ µ : ∥ ξ µ ∥ 2 = r/ (1 − e − 1 ) , the only attractors of the dynamics of the Exp β networkat β → ∞ are the points { ξ µ } Mµ =1 , together with 0 if ∄ µ : ∥ ξ µ ∥ 2 ≤ r .
Remark.
Properties 5.2 and 5.3 can be shown to be true simply by inserting the expression s (0) =
ξ µ + ∆ ξ in Eq. 23.
Assuming no overlaps between the basins of attraction, a quick calculationshows that s (1)
= ξ µ if ∥ ∆
ξ ∥ 2 < r .
If, on the other hand, the network is initialized such that∥ s (0) − ξ µ ∥ 2 > r , ∀ µ , one always obtains s (2) =
ξ 0 , where ξ 0 is either 0 or the pattern closest to 0 .
In other words, the network recalls a pattern only if the initialization is close enough to it.
If locatedfar from all patterns, the network assumes an “agnostic” state, represented either by the origin or thepattern closest to the origin (if the origin happens to be located within a basin of attraction).
In the following two properties, we evaluate how the radius of attraction r determines the maximuminput noise tolerance and storage capacity.
Property 6 (Robustness to white noise) .
Assume that we are given a set of unique patternsξ 1 , . . .
, ξ M ∼ N ( 0 , I N ) with min µ,ν ̸ = µ ∥ ξ µ − ξ ν ∥ 2 > 2 r , and that the Exp β network is initialized in a distorted pattern s (0)
=
ξ µ + ϵ , where ϵ ∼ N ( 0 , σ 2 I N ) .
Then, at β → ∞ , the maximumnoise variance σ 2max with which ξ µ can be recovered in at least 50% of trials is
[Equation]
Property 7 (Exponential storage capacity) .
At β → ∞ , and for N ≫ 1 , the average maximumnumber of patterns sampled from N ( 0 , I N ) that the Exp β network can store and recall without errorsis lower-bounded by
[Equation]
where σ 2max is the maximum white noise variance tolerated by the network.
Remark.
Proofs can be found in Appendices E.2.2 and E.2.3.
Note that Property 7 is valid in therange σ 2max ≲ 1 / 2 .
While the bounds are fairly tight at the upper end of the range, they become loosewhen σ 2max → 0 .
In this regime, r → 0 , which allows for infinite storage capacity, as the risk ofinterference between patterns becomes highly unlikely when their radius of attraction is very small.
Comparison to the softmax network.
When the patterns in X are randomly placed on a hypersphere instead of being normally distributed, the state update rule in Eq.
23 reduces to the forms ( t +1) =
X Θ( X ⊤ s ( t ) − θ ) , where θ is a fixed threshold.
The capacity remains exponential (see Appendix E.3.1), but the basin of attraction surrounding each pattern now forms a spherical cap.
We cancompare this to the softmax network at zero temperature, s ( t +1) = lim β →∞ X softmax( β X ⊤ s ( t ) ) = X arg max( X ⊤ s ( t ) ) .
The only difference between the two models is a replacement of Θ witharg max . This changes the shape of each attractor basin from a spherical cap to a Voronoi cell, so thatthe patterns parcellate the surface of the hypersphere into a Voronoi diagram. The extent of each basinis now no longer radially symmetric around the pattern, but instead maximal in all directions, withoutoverlapping another basin (see Appendix E.5).
The softmax network can be seen as an interpolation(Eq. 21) with K = softmax and the approximation K ≈ I M .
Discussion
Biological interpretation.
Kernel attractor networks can be mapped to the anatomical propertiesof biological neurons.
Consider an individual neuron in the feature representation of the recurrentnetwork (Eq. 9).
The state of neighboring neurons s is first transformed through ϕ ( s ) and thereafterprojected to the neuron through the weight matrix ( A ⊙ X ) ϕ
( X ) ⊤ .
When the kernel is polynomial ofdegree p , so that K ( x i , x j )
=
( x ⊤ i x j + 1) p , the transformation ϕ ( s ) consists of all elements in s andtheir cross-terms, up to degree p .
The input to each neuron, in other words, consists of the states of allother neurons, as well as all possible combinations of their multiplicative interactions.
This neuronmodel can be viewed as a generalized form of, for example, the multiconnected neuron [44], theclusteron [40], or the sigma-pi unit [53, p. 73].
These are all perceptrons that include multiplicativeinput interactions as a means to model synaptic clustering on non-linear dendrites [50] (see Fig. 1).
In the kernel representation (Eq. 10), each neuron is, again, implicitly comprised of a two-stageprocess, whereby the raw input s is first transformed through the function K ( X , s ) and then projectedthrough the weight matrix A ⊙ X .
For any inner-product kernel K = k ( x ⊤ i x j ) , this representationcan be directly identified as a two-layer neural network, where the hidden layer is defined by theweights X and the activation function k .
This interpretation of the recurrent network was recentlyproposed in [32, 33] and discussed in relation to hippocampal-cortical interactions involved inmemory storage and recall; it is particularly reminiscent of the hippocampal indexing theory [5, 56].
However, the kernel representation can also be viewed as a network in which each individual neuronis a generalized form of the two-layered pyramidal cell model [48, 49]. This was originally proposedas an abstract neuron model augmented with non-linear dendritic processing [38].
It should be noted,however, that the idea of interpreting kernel methods as neural networks has a longer history, andhas been extensively analyzed in the case of, for example, radial basis functions [46, 47]. For furtherdetails, see Appendix F.
Summary.
We have shown that conventional kernel methods can be used to derive the weightsfor hetero- and auto-associative memory networks storing binary or continuous-valued patterns withmaximal noise tolerance.
The result is a family of optimal memory models, which we call kernelattractor networks , which includes the SDM and MHN as special cases.
This unifying frameworkfacilitates an intuitive understanding of the storage capacity of memory models and offers new waysto biologically interpret these in terms of non-linear dendritic integration.
This work formalizes thelinks between kernel methods, attractor networks, and models of dendritic processing. Future work.
A unifying theoretical framework for memory modeling can be useful for thedevelopment of both improved bio-plausible memory models and for machine learning applications. First, recognizing that there exists online algorithms for training optimally noise-robust classifiers andadapting these to biological constraints can aid in the development of normative synaptic three-factorlearning rules [21]. Second, the theoretical link between neuron models, kernel functions, and storagecapacity enables one to fit kernel attractor networks to neurophysiological data and to analyze thecomputational properties of biophysically informed memory models. Finally, our unifying frameworkreveals that most memory models differ only in the choice of kernel (model complexity) and Lagrangeparameters (model precision).
This categorization simplifies the tailoring of memory models to theirapplication, and allows for the design of models whose properties fundamentally depart from kernelattractor networks, by choosing kernels not associated with a reproducing kernel Hilbert space.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] Our work is entirely theoretical and the limitations of our approach is treated in the main text when discussing, for example, the validity and assumptions of our derivations.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[N/A]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] This isincluded in all theoretical results, or in the explanatory text preceding them.
(b) Did you include complete proofs of all theoretical results?
[Yes] These can be found
in the Appendix.
Pointers to the correct sections are given in the main text.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[N/A]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A](b)
Did you mention the license of the assets?
[N/A](c)
Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’reusing/curating?
[N/A](e)
Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?
[N/A]5.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A](b)
Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A](c)
Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]
Appendices
A Derivation of storage capacity scaling
Suppose that the kernel is a homogeneous polynomial of degree p ≪ N , meaning K ( x i , x j ) =( x ⊤ i x j ) p .
This implies that the associated feature map ϕ ( x ) contains all monomials of degree pcomposed of the entries in x .
Moreover, given that each entry x i is ± 1 , each monomial in ϕ can bewritten as an interaction term of the form x p 1 1 x p 2 2 · · · x p N N , where p i ∈ { 0 , 1 } and (cid:80)
Ni p i ≤ p (i.e., nofactor x
i has an exponent higher than 1 and the sum of exponents is ≤ p ).
The reason for this is that
[Equation]
[Equation]
The number of unique interaction terms of precisely degree p (the highest degree) is (cid:0) Np (cid:1) .
As thebinomial coefficient is known to be bounded according to
[Equation]
we obtain (cid:0) Np (cid:1)
∼ O ( N p ) , for p fixed.
Thus, the effective dimensionality of ϕ scales like O ( N p ) .
For the exponential kernel K ( x i , x j ) =
exp( x ⊤ i x j ) = (cid:80)
∞
p =0 ( x ⊤ i x j ) p /p !
, we first note that themonomials in ϕ now will be interaction terms of all degrees p = 0 , . . .
, N .
No monomial of degreep > N will be possible.
The total number of unique interaction terms will therefore be
[Equation]
[Equation] where the first equality can be found in [6, p. 14]. This gives us an effective dimensionality of ϕ thatscales like O ( e N ) . B Kernel of an infinite SDM on the hypersphereB.1 Derivation of Eq. 18.
We follow the same steps as [7], with additional simplifications towards the end.
As stated in themain text, we seek to calculate the overlapping area of two hyperspherical caps.
A formula for thiscan be found in [35], and can be written as
[Equation]
29) where
[Equation]
where R is the radius, I is the regularized incomplete Beta function, and
[Equation] [Equation] [Equation]
[Equation]
We insert Eq. 31 in 33 and obtain
[Equation] [Equation]
where we have used tan( α/ 2)
= (1 − cos( α )) / sin( α ) .
Eq.
35 also follows from the symmetry ofthe problem.
This result yields
[Equation]
We rewrite the regularized incomplete Beta function as
[Equation]
where B is the incomplete Beta function.
The area of an ( N in − 1)
-dimensional
hypersphere is
[Equation]
[Equation]
We insert Eq.
37 in
and use the result in Eq. 36.
Using the notation α b = α 1 and α x = α min , andthe identities
[Equation]
the ratio between the overlapping area of the hyperspherical caps and the complete are of thehypersphere can now be calculated as
[Equation]
B.2 Derivation of Eq. 19.
For a large bias b ≳ 0 . 9 , which is equivalent to a small angle α b = arccos( b ) , the hyperspherical capssurrounding x i and x j will be very small in relation to the whole hypersphere.
In this case, we canneglect the curvature of the hyperspherical surface and project the area of the hyperspherical cap tothe plane that cuts through the rims of the cap.
This projection is a ( N in − 1) -dimensional
hyperball(we will refer to it as a mini-ball).
In three dimensions, for example, the projection of a spherical capto the plane constitutes a disk, which is a 2-dimensional ball.
The radius of each mini-ball is
[Equation]
and the half-distance between the centers of the mini-balls is
[Equation]
We estimate the overlapping area of the hyperspherical caps by calculating the overlapping volume ofthe mini-balls in ( N in − 1) dimensions.
A formula for the overlapping volume of two hyperballs canbe found in [36] and is
[Equation]
We rewrite the regularized incomplete Beta function as
[Equation]
[Equation]
and insert Eq.
44 in 43.
The ratio between the overlapping area of the hyperspherical caps and thecomplete area of the hypersphere can now be estimated as
[Equation]
A comparison of the exact solution in Eq.
40 and the approximation in Eq.
45 can be seen in Fig.
B.1.
[Figure]
C Online learning in an SVM network
We will in this section compare the noise tolerance of a single neuron in an SVM network whentrained with an online learning rule, and when configured according to the MHN.
First, we choose toequip the neuron with the feature map ϕ pairs ( x ) , which consists of all unique pairs of cross-termsx i x j , i ̸
= j .
This yields a storage capacity scaling of O ( N 2 ) , and we therefore parameterize thestorage load as M/N 2 .
We train the weights of the neuron either with the stochastic batch perceptron[12] or with the one-shot learning rule of the MHN, which is obtained by setting α µ = 1 , ∀ µ , in Eq.4, that is
[Equation]
Finally, we quantify the noise tolerance as the minimum Euclidean distance between the neuron’sdecision boundary and the patterns { ξ µ in } Mµ =1 .
This is equivalent to the minimum classification margin,defined as
[Equation]
We are only interested in the performance regime where all patterns are correctly recalled (i.e.,correctly classified).
This means that we only compare positive margins, since a negative marginindicates that there is one or more patterns that no longer can be correctly recalled.
The results areplotted in Fig.
C.1, and demonstrate that the margin for the MHN quickly drops with increasing load,while the online learning rule achieves a margin close to the theoretical optimum derived by Gardner[20].
Moreover, as the maximum storage capacity M max of each learning rule can be found at theintersection between the margin curve and γ = 0 , the capacity of the online rule can be seen to bemuch higher than that of the MHN, and reaches close to 2 D VC .
D Generalized pseudoinverse rule
When the network is linear and underdetermined, meaning M < N , we can make sure that allpatterns are attractors by modeling each neuron as a least-squares SVM [55] instead of a conventional
SVM, so that the weights satisfy
[Equation]
[Figure]
i f
[Figure]
t i o n m a r g i n
Theoretical limit
Online learning (SBP)
Modern Hopfield rule
Modern Hopfield rule
This is a minimum-norm interpolation problem and yields the solution
[Equation]
where K =
X
⊤ X is the kernel matrix and K † its Moore-Penrose pseudoinverse and where we haveused the property K † X ⊤ =
( X ⊤ X ) † X ⊤ = X † .
This is the pseudoinverse learning rule [45].
The derivation can be extended to MHNs by performing interpolation on the feature map ϕ ( ξ µ ) .
Assuming that the problem is still underdetermined, so that M < N ϕ we aim to find the weights [Equation]
[Equation]
which, analogously to the linear case produces the optimal state update
[Equation]
(51) where K = K ( X , X )
=
ϕ
( X ) ⊤ ϕ
( X ) .
This can, again, be simplified to
[Equation]
52) where we can identify the weight matrix W =
X ϕ ( X ) † .
This is the generalized pseudoinverselearning rule .
Note that, if the feature-expanded patterns { ϕ ( ξ µ ) } M µ =1 are linearly independent, thekernel matrix is invertible and we have K † =
K − 1 .
E The kernel attractor network for continuous patterns
E.1 Minimum norm interpolation and attractor basin size
Proof of Property 4.
In this setting, each neuron is modeled as linear regressor whose state isupdated according to
[Equation]
All patterns X are guaranteed to be fixed points of the dynamics by finding the weights that satisfy
[Equation]
In order for each pattern to also be an attractor, the weights must satisfy the added constraint
[Equation]
where we have used the shorthand s
= s (1) , s
= s (0) , and J s is the Jacobian with respect to s 0 .
Eq.
55 states that the spectral norm of the Jacobian of the state update rule must be less than 1when evaluated in each pattern.
This ensures that the fixed points { ξ µ } Mµ =1 are stable attractors and,equivalently, that the state update rule is a contractive map in a region surrounding each pattern [51].
It is assumed that there exists a solution that satisfies both Eq. 54 and 55.
The basin of attraction of afixed point µ can be defined as the set of points S µ edge in the open neighborhood of ξ µ satisfying [Equation] µ
[Equation]
Given that the spectral norm of the Jacobian is upper bounded by
[Equation]
[Equation] where ∥·∥ F is the Frobenius norm, we can obtain a lower bound of the extent of the basin of attractionwith the set
[Equation]
Since the Jacobian is given by
[Equation]
59) where W = ( w 1 , . . . , w N ) ⊤ , we obtain
[Equation]
60) where the last expression is given by the Cauchy-Schwartz inequality.
Since ∥ J s ( ϕ ( s 0 ))
∥ F dependsonly on the kernel and the patterns, which are both fixed, Eq.
60 can only be minimized by finding aset of weights that minimize ∥ W ∥ F .
This is achieved by the minimum norm interpolation, given by
[Equation]
The solution is stated in Eq.
This solution maximizes a lower bound of the attractor basin sizesby maximizing the size of the set (cid:98) S µ edge .
□
E.2 Normally distributed patternsE.2.1 Kernel at zero temperature
Proof of Property 5.1.
Using the notation ∆ =
∥
ξ µ − ξ ν ∥ 2 , we have that
[Equation]
62) from which it follows that
[Equation]
which is equivalent to Θ( r − ∆) with Θ(0)
=
e − 1 .
We combine this with the assumption that thepatterns are unique and that min µ,ν ̸ = µ ∥ ξ µ − ξ ν ∥ 2 > r and obtain
[Equation]
64) which can be written compactly as
[Equation]
It directly follows that
[Equation]
66) and, therefore,
[Equation]
67) □
E.2.2 Noise robustness
Proof of Property 6.
With s (0) =
ξ µ + ϵ , we have
[Equation]
68) where ϵ 0 ∼ N ( 0 , I N ) , from which it follows that ∥ ϵ 0 ∥ 22 is a random variable with a χ 2 ( N ) distribution.
According to the central limit theorem, we also have
[Equation]
where we have used the fact that each term in ∥ ϵ 0 ∥ 22 is χ 2 (1) -distributed, and has mean 1 and variance2 .
We will therefore make the approximation ∥
ϵ 0 ∥
22 ∼ N ( N, 2 N ) for large N .
This gives us
[Equation] [Equation]
The original pattern ξ µ will only be recovered if r 2 − ∥ ξ µ − s (0) ∥ 22 ≥ 0 , which is satisfied in at least50% of trials if r 2 ≥ σ 2 N .
The maximum variance with which this type of recovery still holds isthus
[Equation]
E.2.3 Storage capacity
Proof of Property 7.
In the limit β → ∞ , the boundary of the basin of attraction surrounding eachpattern is sharp.
In this setting, we are guaranteed that each pattern can be recalled without errorsas long as min i,j ̸
= i ∥ ξ i − ξ j ∥ 2 > 2 r .
We will therefore estimate the storage capacity by calculatingthe number of patterns, on average, that can be loaded into the network before at least two attractorbasins overlap and the condition above is violated (see Fig.
E.1). We begin by observing that for two random patterns ξ i , ξ j ∼ N ( 0 , I N ) , we have
[Equation]
(72) which, using the central limit theorem as in Eq.
69, can be approximated as N ( N, 2 N ) for large N ,thereby yielding
[Equation]
We now assume that the squared Euclidean distance between each pair of patterns ξ i , ξ j in a setof M given patterns { ξ µ } Mµ =1 is an independent sample of a random variable, denoted ∆ 2 , whichis distributed as in Eq.
This is, of course, an approximation which neglects that the pairwisedistances between any set of points are inter-dependent.
Nonetheless, for relatively large N and M ,the approximation accurately describes the empirical distance distribution.
Relying on this assumption, the process of drawing M random patterns becomes equivalent todrawing M ( M − 1) / 2 unique pairwise distances ∆ 2 from the distance distribution.
The probabilityof drawing a sample ∆ 2 ≤ 4 r 2 can be calculated using the cumulative distribution function for thestandard normal distribution, given by Φ( x ) = 12 erfc( − x ) , according to
[Equation]
The average number of samples of ∆ 2 one needs to draw before a sample satisfies ∆ 2 ≤ 4 r 2 is givenby P (∆ 2 ≤ 4 r 2 ) − 1 .
This determines the maximum number of patterns that the network, on average,can store, according to
[Equation]
We combine this expression with the approximation M max ( M max − 1) ≈ M 2max (which holds forlarge M max ) and Eq. 74, and obtain
[Equation]
[Equation] [Figure] We now parameterize the radius r in terms of the largest tolerable noise amplitude, according to Eq.71.
This gives us
[Equation]
Given that the erfc function can be well approximated using the asymptotic expansion
[Equation]
for large arguments x , we can obtain a tight lower bound of erfc − 1 as long as N is large andσ 2max ≲ 1 / 2 with the inverse zeroth order expansion, thereby obtaining
[Equation]
We insert Eq.
79 in {+
+} and finally obtain
[Equation]
E.3 Patterns on the hypersphere
E.3.1 Storage capacity
Property 8 (Storage capacity: patterns on the hypersphere) .
At β → ∞ , the average maximumnumber of patterns that the Exp β network can store and recall without errors is lower-bounded by
[Equation]
when each pattern is randomly drawn from S N − [Equation]
Proof.
We begin by observing that for two random patterns ξ i , ξ j ∈ S N − 1 , we have
[Equation]
The probability distribution for the inner product ω = ξ µ ⊤ ξ ν can be found in [57], and is given by
[Equation]
which, for large N , can be approximated as
[Equation]
We use Eq.
84 in
and obtain
[Equation]
The probability of placing a pair of random points on S N − 1 with ∆ 2 ≤ 4 r 2 is thus
[Equation]
The average number of samples of ∆ 2 one needs to draw before a sample satisfies ∆ 2 ≤ 4 r 2 is givenby P (∆ 2 ≤ 4 r 2 ) − 1 , and the maximum number of patterns that the network, on average, can store, istherefore
[Equation]
We use the lower bound in Eq.
79 in 87 and finally obtain
[Equation]
E.4 Bipolar patterns
E.4.1 Noise robustness
Property 9.1 (Robustness to flipped bits) .
Assume that we are given a set of unique patternsξ 1 , . . .
, ξ M ∈ {− 1 , +1 } N with min µ,ν ̸ = µ ∥ ξ µ − ξ ν ∥ 2 > 2 r , and that the Exp β network is initializedin a distorted pattern s (0) =
ξ µ ⊙ ϵ , where ϵ ∈ {− 1 , +1 } N , with P ( ϵ i = − 1) =
ρ, ∀ i .
Then, atβ → ∞ , the maximum bit-wise error probability ρ max with which ξ µ can be recovered in at least50% of trials is
[Equation]
[Equation] Proof.
With s (0) =
ξ µ ⊙ ϵ , we have
[Equation] [Equation]
where ϵ B ∈ { 0 , 1 } N , with each entry being a random variable distributed as ( ϵ B ) i ∼ Bernoulli( ρ ) .
This implies that ∥ ϵ B ∥ 22 ∼ Binomial( N, ρ ) , which can be approximated as N ( ρN, ρ (1 − ρ ) N ) forlarge N .
This gives
[Equation]
Again, the original pattern ξ µ will only be recovered if r 2 − ∥ ξ µ − s (0) ∥ 22 ≥ 0 , which is satisfied inat least 50% of trials if r 2 ≥ 4 ρN .
The maximum bit-wise error probability with which this type ofrecovery still holds is thus
[Equation]
[Equation]
In Eq.
71, σ roughly quantifies the maximum noise fluctuation around a pattern that is tolerable witha given radius r , while still being able to recover the pattern in a majority of trials.
In the case of Eq.92, ρ instead quantifies the maximum tolerable bit-wise error probability.
E.4.2 Storage capacity
Property 9.2 (Storage capacity: bipolar patterns) .
At β → ∞ , the average maximum number ofbipolar patterns with sparseness f that the Exp β network can store and recall without errors islower-bounded by
[Equation]
where ˜ f
= 2 f (1 − f ) and ρ max is the maximum bit-wise error probability tolerated by the network.
Proof.
This proof is, again, a slightly modified variant of the proof of Property 7.
First, we observethat for two random patterns ξ µ , ξ ν ∈ {− 1 , +1 } N with sparseness f , so that P ( x µ,νi =1) =
f , it istrue that
[Equation] [Equation]
where ˜ f
= 2 f (1 − f ) denotes the probability that ξ µ and ξ ν differ at any given entry.
For large N ,we can again approximate Binomial( N, ˜ f ) with N ( ˜ fN, ˜ f
(1 − ˜ f ) N ) , which gives us
[Equation]
We use this to compute the upper bound of the probability of drawing a distance ∆ 2 which satisfies∆ 2 ≤ 4 r 2 , as in Eq. 74.
The result is
[Equation]
Following the same derivations used to produce Eqs.
75 and 76, we arrive at
[Equation]
As before, we parameterize the radius r in terms of the maximum tolerable bit-wise error probabilityaccording to Eq. 92 and yield
[Equation]
We finally replace erfc with the lower bound in Eq. 79.
This is valid for large N and when ρ ≲ ˜ f/ 4 .
We obtain
[Equation]
E.5 Comparison to the softmax network
To make a illustrative comparison between the softmax and the Exp β network, we plot a set ofrandom patterns on the two-dimensional unit sphere S 2 , together with the boundaries of their attractorbasins at zero temperature (i.e., β → ∞ ). The result is shown in Fig. E.2. While the basins of the
Exp β network form spherical caps around the stored pattern, the basins of the softmax network form
Voronoi cells.
Since the softmax network parcellates the entire pattern space into a Voronoi diagram,its attractor basins are typically larger than those of the Exp β network.
However, it will alwaysconverge to one of the stored patterns, regardless of where we initialize the model (assuming it is notprecisely on a boundary).
In contrast, if the Exp β network is initialized far from all patterns, it willconverge to the origin, which can be interpreted as an agnostic state, indicating that the model cannotassociate the input query with any of its stored patterns.
Finally, as we increase the temperature( β < ∞ ), the boundaries of both networks become progressively fuzzier.
F Comparison to neuron models with active dendrites
To demonstrate how single neurons in kernel attractor networks are generalized models of abstractneurons with active dendrites, we begin by considering a neuron in the feature representation (Eq.5).
We will typically use a Heaviside activation function instead of sgn , and assume patterns andstates in { 0 , 1 } N , where N is the number of inputs.
This, however, does not change the fundamentalproperties of our model, as SVMs can be formulated for binary data with a minor modification.
Assuming a polynomial feature map ϕ of degree p , the feature vector will consist of all possiblemonomials of degree ≤ p composed of the states of its input neurons.
Setting, for example, p = 2gives us
[Equation]
By limiting the feature map to only include a subset of all terms, our model is reduced to the 2-degreesigma-pi unit [53, p. 73], which can be written as
[Equation]
[Equation]
where w i is the weight of cluster i , which consists of a product of all inputs with indices contained inthe set C i .
From a neurophysiological perspective, each product represents the cross-talk between aset of synapses.
By including such multiplicative interactions, synapses can both gate and amplifyeach other, to generate supra-linear input currents.
If we now further constrain this model to includeonly a subset of the cross-terms x i x j , i ̸
= j , and parameterize each weight as w ij = w i w j , our neuron model is reduced to the clusteron [40], which can be written as
[Equation]
w i w j s
[Equation]
i s
[Equation]

 [Equation]

[Equation]
i w
[Equation]
s
[Equation]
i  (cid:88)
j ∈C i w j s in
[Equation]
j   
 (102) where C i now is the set describing all inputs j that input i should be paired with.
We now consider a neuron in the kernel representation (Eq. 6) with an inner-product kernel
K ( x i , x j ) =
k ( x ⊤ i x j ) .
By setting ξ µ out = 1 , ∀ µ , and assuming binary inputs, our model is reduced to
[Equation]
 (103) which is equivalent to the pyramidal cell as a 2-layered neural network, as defined in [48].
Accordingto the original interpretation, this neuron model is comprised of M subunits that are connected toboth the inputs and the output.
Each subunit µ is characterized by the input weights ξ µ in (which masksome of the inputs), the output weight α µ (which determines how strongly µ influences the responseat the soma), and the activation function k .