Kernel Memory Networks: A Unifying Framework for Memory Modeling
Abstract
We consider the problem of training a neural network to store a set of patterns with maximal noise robustness.
A solution, in terms of optimal weights and state update rules, is derived by training each individual neuron to perform either kernel classiﬁcation or interpolation with a minimum weight norm.
By applying this method to feed-forward and recurrent networks, we derive optimal models, termed kernel memory networks, that include, as special cases, many of the hetero- and auto-associative memory models that have been proposed over the past years, such as modern Hopﬁeld networks and Kanerva’s sparse distributed memory.
We modify
Kanerva’s model and demonstrate a simple way to design a kernel memory network that can store an exponential number of continuous-valued patterns with a ﬁnite basin of attraction.
The framework of kernel memory networks offers a simple and intuitive way to understand the storage capacity of previous memory models, and allows for new biological interpretations in terms of dendritic non-linearities and synaptic cross-talk.
Introduction
Although the classical work on attractor neural networks reached its peak in the late 1980’s, with the publication of a number of seminal works [e.g.
2, 20, 22, 26], recent years have seen a renewed interest in the topic, motivated by the popularity of the attention mechanism [65], external memoryaugmented neural networks [24, 66] as well as a new generation of energy-based attractor networks models, termed modern Hopﬁeld networks (MHNs), capable of vastly increased memory storage [17, 35].
Recent efforts to understand the theoretical foundation of the attention mechanism have, in fact, shown that it can be linked to Hopﬁeld networks [36, 57] but also to Kanerva’s sparse distributed memory (SDM) [8, 30] and to the ﬁeld of kernel machines [63, 68].
The last connection is particularly intriguing, in light of the many theoretical commonalities between neural networks and kernel methods [10, 11, 28, 47, 67].
Overall, these results suggest that a uniﬁed view can offer new insights into memory modeling and new tools for leveraging memory in machine learning.
In this work, we aim to clarify some of the overlap between the ﬁelds of memory modeling and statistical learning, by integrating and formalizing a set of theoretical connections between Hopﬁeld networks, the SDM, kernel machines, and neuron models with non-linear dendritic processing.
1.1 Our contribution
• We derive a set of normative kernel-based models that describe the general mathematical structure of feed-forward (i e., hetero-associative) and recurrent (i e., auto-associative) memory networks that can perform error-free recall of a given set of patterns with maximal robustness to noise.
• We show that the normative models include, as special cases, the classical and modern Hopﬁeld network, as well as the SDM.
• We derive a simple attractor network model for storing an exponential number of continuous-valued patterns with a ﬁnite basin of attraction.
We discuss its similarity to attention.
Furthermore, we explain how classiﬁers with non-linear kernels can be interpreted as general forms of neuron models with non-linear dendritic activation functions and synaptic cross-talk.
1.2 Related work
Our work is primarily related to [8, 9, 35, 36, 45, 57].
While MHNs are extensively analyzed in [35, 36, 45, 57], the approach is energy-based and makes no statements about the relation between MHNs and kernel methods; a brief comment in [57] mentions some similarity to SVMs, but this is not further explained.
The work by [8] focuses on the SDM and its connection to attention.
It observes that the classical Hopﬁeld network is a special case of the SDM, but no further generalization is made, and kernel methods are not mentioned.
In our work, we place MHNs and the SDM in a broader theoretical context by showing that both models are special suboptimal cases of a family of memory networks that can be derived with a normative kernel-based approach.
Background
Consider the following simple model of hetero-associative memory: a single-layer feed-forward network consisting of a single output neuron connected to N in inputs with the weights w 2 R N  .
The output s out 2 {± 1 } is given by
[Equation]
where s in is the input vector (also called query), ✓ the threshold, and  a function that maps the “raw” input to a N  -dimensional feature space, where typically N   N in .
Suppose that we are given a set of M input-output patterns { ⇠ µ in , ⇠ µ out } Mµ =1 , in which every entry ⇠ is randomly drawn from {± 1 } with sparseness f := P ( ⇠ =1) .
In order for the neuron to store the patterns in a way that maximizes the amount of noise it can tolerate while still being able to recall all patterns without errors, one needs to ﬁnd the weights that produce the output ⇠ µ out in response to the input ⇠ µ in , 8 µ , and that maximize the smallest Euclidean distance between the inputs and the neuron’s decision boundary.
Using Gardner’s formalism [20, 22], this problem can be expressed as
[Equation]
where ¯ w > 0 is a constant.
This is equivalent to solving
[Equation]
which can be directly identiﬁed as the support vector machine (SVM) problem for separable data [13].
The solution to Eq.
3 can today be found in any textbook on basic machine learning methods, and yields an optimal output rule that can be written in a feature and kernel form
[Equation]
[Equation]
where we, in the latter expression, have used the “kernel-trick” K ( x i , x j )
=
( x i ) >  ( x j ) .
The solution depends on the Lagrange coefﬁcients ↵ µ  0 , many of which are typically zero.
Patterns with ↵ µ > 0 are called support vectors .
3.1 Hetero -associative memory as a feed-forward SVM network
We begin by considering a hetero-associative memory network with an arbitrary number N out output neurons, whose combined state we denote s out .
In order for the network as a whole to be able to tolerate a maximal level of noise and still successfully recall its stored memories, we solve Eq.for each neuron independently.
As each neuron can have a different classiﬁcation boundary along with a different set of support vectors, its weights will, in general, be characterized by an independent set of M Lagrange coefﬁcients.
To simplify the notation, we represent these coefﬁcients ↵ µi , across neurons i and patterns µ , as entries in the matrix A , where ( A ) iµ = ↵ µi .
We also combine all thresholds in the vector ✓ =
( ✓ 1 , . .
. , ✓ N out ) , and all input and output patterns as columns in the matrices X in = ( ⇠ 1in , . . .
, ⇠ M in ) and X out =
( ⇠ 1out , . . .
, ⇠ M out ) .
Finally, we assume that all neurons have the same feature map, so that  i =
, 8 i (see Fig. 1).
All functions are applied column-wise when the argument is a matrix, for example  ( X in )
= (  ( ⇠ 1in ) , . . .
,  ( ⇠ M in )) .
The optimal response of the network can now be compactly summarized as follows.
Property 1 (Robust hetero-associative memory network) .
A single-layer hetero-associative memory network trained to recall the patterns X out in response to the inputs X in with maximal noise robustness, has an optimal output rule that can be written as
[Equation]
where  denotes the Hadamard product.
3.2 Auto -associative memory as a
recurrent SVM network
The hetero-associative network can be made auto-associative by setting N out
= N in and X out
=
X in .
The network is now effectively recurrent, as each neuron can serve both as an input and output simultaneously (see Fig. 1).
Consider a recurrent network with N neurons, whose state at time point t is denoted s ( t ) 2 {± 1 } N , and whose dynamics evolve according to the update rule
[Equation]
where w i 2 R N  is the weight vector to neuron i = 1 , . .
N .
In order to make the patterns { ⇠ µ } Mµ = ﬁxed points of the network dynamics, we train each neuron i independently on every pattern µ to, again, produce the response ⇠ µi when the rest of the network is initialized in ⇠ µ .
Moreover, we maximize the amount of noise that can be tolerated by the network while maintaining error-free recall by maximizing the smallest Euclidean distance between each neuron’s decision boundary and its inputs.
This maximizes the size of the attractor basins [18, 32].
The problem of training the entire network is, in this way, transformed into the problem of training N separate classiﬁers according to
[Equation]
The solution can be obtained by slightly modifying Property 1, and is stated below.
Property 2.1 (Robust auto-associative memory) .
A recurrent auto-associative memory network trained to recall the patterns X with maximal noise robustness has an optimal synchronous update rule that can be written as
[Equation]
Remark.
With a linear feature map  ( x ) = x , the optimal update is reduced to
[Equation]
where ( A  X ) X > can be identiﬁed as the general form of the optimal weight matrix.
The solution described by Property 2.1 does not, in general, prohibit a neuron from having selfconnections.
Applying this constraint yields the following result.
Property 2.2 (Robust auto-associative memory without self-connections) .
A recurrent autoassociative memory network without self-connections, with the inner-product kernel K ( x i , x j ) =
k ( x > i x j ) , that has been trained to recall the patterns X with maximal noise robustness, has an optimal asynchronous update rule that can be written in the kernel form
[Equation]
Storage capacity.
An intuition for the storage capacity scaling of the hetero- and auto-associative memory networks can be gained by observing that the network as a whole will be able to successfully recall patterns as long as each neuron is able to correctly classify its inputs (or is very unlikely to produce an error).
The capacity of the network can thereby be derived from the capacity of each individual neuron.
It is well-known that a linear binary classiﬁer can learn to correctly discriminate a maximum of M max ⇡ 2 D VC random patterns, where D VC is the Vapnik-Chervonenkis dimension of the classiﬁer [15, 20, 40, ch. 40].
For a neuron with N inputs and a linear feature map  ( x ) = x , this results in D VC = N and, thus, the capacity M max ⇡ 2 N .
Suppose, on the other hand, that the kernel is a homogeneous polynomial of degree p , so that K ( x i , x j ) =
( x > i x j ) p .
In this case,  will contain all monomials of degree p composed of the entries in x .
As there are O ( N p ) unique p -degree monomials (see Appendix A.1), the input dimensionality and M max will be O ( N p ) .
For the exponential kernel, which we can write as K ( x i , x j ) =
exp( x > i x j ) =
P 1 p =0 ( x > i x j ) p /p !
, the dimensionality of  will be
[Equation]
which yields M [Equation]
⇠
[Equation]
e N ) .
Special cases.
In the following sections, we will show that many of the models of hetero- and auto-associative memory that have been proposed over the past years are special cases of the solutions in Properties 1, 2.1, and 2.2, characterized by speciﬁc choices of A ,  , and K .
3.3 Kanerva’s sparse distributed memory is a feed-forward SVM network
The sparse distributed memory (SDM), developed by Kanerva [30], is one of the most famous examples of a hetero-associative memory model.
It has lately received much attention in the context of generative memory models [69] and attention layers in transformers [8].
The SDM consists of a register of N  memory slots, each associated with an address z i 2 {± 1 } N in , i = 1 , . . .
, N  .
All addresses are listed as rows in the matrix Z
= ( z 1 , . . .
, z N  ) > .
The content of each slot is represented by an N out -dimensional vector, initialized at zero.
Suppose that we wish to store the M patterns X out = ( ⇠ 1out , . . .
, ⇠ M out ) in the addresses X in = ( ⇠ 1in , . . . , ⇠ M in ) , where all entries are random and bipolar.
The basic idea of the SDM is to write the data to, and later read it from, multiple memory slots at once (hence the distributed storage); this ensures a degree of noise-robustness.
In mathematical terms, the read-out of the SDM provided with a query s in , is given by
[Equation]
where ⇥ the Heaviside function with bias b = N in  2 r , and r is a parameter that determines the precision of the writing and reading process.
Upon comparing Eqs.
13 and 5, the SDM can be directly identiﬁed as a special case of a suboptimal feed-forward SVM network in the feature form, with
A = 1 , ✓ = 0 , and the feature map  SDM ( x ) = ⇥
( Zx  b ) .
When viewed as a kernel method, the function of the SDM is to store the dense addresses X in as sparse high-dimensional representations  SDM , to make it easier to later determine the slots closest to a query s in , and retrieve the relevant data.
Capacity.
As the SDM is linear in  SDM , with D VC ⇡ N  , it follows from the analysis in Sec.
3.2 that one should expect the capacity to scale as M max ⇠ O ( N  ) .
Moreover, one should expect a proportionality constant ⇠ 0 .
1 , since the SDM is suboptimal relative to the feed-forward SVM network, analogously to how the classical Hopﬁeld network is suboptimal relative to the recurrent
SVM network (see Sec. 3.4).
This is consistent with earlier proofs [12, 31].
A ! !" !
#$% = sgn & #$% ' (& !"
− * & ' (! !" − *
C B
B
A
A2 ! !
+ #$%,( = sgn , (& - ( ! !"
− .
#$% = sgn /- ! !"
−# " + ((*+,) = sgn , (& - ( ! (*) − .
(*+,)
= sgn /- !
(*) −+ !"
+ !",( + #$% " =
Kernel of an inﬁnite SDM.
In practice, an SDM with a large number of memory slots N  requires calculations involving a large address matrix Z .
This can be avoided by applying the kernel-trick to
Eq. 13 in the limit N  ! 1 , which allows for the output to be computed with
[Equation]
(14) where we have deﬁned the kernel as
[Equation]
(15) in order to ensure convergence.
In this section, we will derive this kernel for two different variants of the SDM and demonstrate that both are translation-invariant.
It is interesting to note here that  SDM is equivalent to a single-layer neural network with N  neurons, weights Z , and bias b .
This means that K SDM is equivalent to the kernel of an inﬁnitely wide neural network [11, 47, 67].
We begin by noticing that  SDM ( x ) has a geometrical interpretation [8, 31].
It is a binary vector that indicates those memory addresses in Z that differ by at most r bits compared to x .
For any two bipolar vectors z and x , the bit-wise difference can be computed as 12 | z
x | = 14 k z  x k 22 .
This means that  SDM ( x ) indicates all addresses that lie within a sphere centered at x with radius 2 p r .
Consequently, the inner product  SDM ( x i ) >
SDM ( x j ) is the number of addresses located in the overlapping volume of two spheres centered at x i and x j .
Although an exact calculation of this quantity can be found in [8, 30], its connection to the SDM kernel has, to the best of our knowledge, not previously been made.
We therefore modify the previously published expression with a normalization factor 1 / 2 N in and state the following property.
Property 3.1 (Kernel of an inﬁnite SDM on the hypercube) .
In the limit N  ! 1 , the kernel of an
SDM with N  memory slots, whose addresses are randomly drawn from {± 1 } N in , is given by
[Equation]
where r is the bit-wise error threshold and  is the bit-wise difference between x i and x j , given by  =
[Equation] The SDM can also be implemented with continuous addresses, randomly placed on a unit hypersphere of ( N in  1) dimensions, denoted S N in  1 .
The vector  SDM ( x ) now indicates all addresses that lie within a hyperspherical cap centered at x with an angle arccos( b ) between its central axis and the rim.
The inner product  SDM ( x i ) >  SDM ( x j ) is the number of addresses located in the overlapping area of two spherical caps centered at x i and x j .
While a calculation of this quantity, again, can be found in [8], it has not previously been connected to the kernel of an SDM.
We simplify the previously published result and also derive a closed-form approximation, valid for highly sparse  SDM (see
Appendix B for details).
The results are summarized below.
Property 3.2 (Kernel of an inﬁnite SDM on the hypersphere) .
In the limit N  ! 1 , the kernel of an SDM with N  memory slots, whose addresses are randomly drawn from S N in  1 , is given by
[Equation]
where ↵ x = 12 arccos( x > i x j ) , ↵ b = arccos( b ) , and B is the incomplete Beta function.
In the highly sparse regime, when 0 . 9 .
b < 1 and 1 N  k  SDM k 0 ⌧ 1 , the kernel can be approximated with
[Equation]
[Equation]
where  = {+
 k+} x i  x j k 2 and ˆ b = sin(arccos( b )) .
In conclusion, an inﬁnitely large SDM with sparse internal representations  SDM , can be represented as a suboptimal case of a feed-forward SVM network with a translation-invariant kernel.
3.4 The modern Hopﬁeld network is a recurrent SVM network
The Hopﬁeld network [26] is, arguably, the most well-known model of auto-associative memory.
In its modern form [35], it is a recurrent network of N neurons with the state s ( t ) , whose dynamics are governed by the energy and state update rule
[Equation]
where F is a smooth function, typically a sigmoid, polynomial, or exponential.
This “generalized”
Hopﬁeld model has a long history [see, e.g., 1, 21, 25, 37] but has received renewed attention in recent years under the name modern Hopﬁeld network (MHN) or dense associative memory [17, 35].
By comparing Eq.
19 with Eq.
12, the state update of the MHN can be identiﬁed as a special case of a suboptimal recurrent SVM network in the kernel form, with k = F 0 , A = 1 , and ✓ = 0 (since f = 0 . 5 ).
With a linear F 0 ( x ) = x , the MHN reduces to the classical Hopﬁeld network, which is a special case of the recurrent SVM network with the linear kernel k ( x > i x j ) =
x > i x j .
Capacity.
The storage capacity of the MHN has been shown to depend on the shape of F 0 .
In the linear case, the capacity is famously limited to ⇠ 0 .
1 N patterns, depending on the precision of retrieval [2, 42].
If, on the other hand, F 0 is polynomial with degree p , the capacity scales as
M max ⇠ O ( N p ) [35], while an exponential F 0 endows the network with a capacity M max ⇠ O ( e N ) [17].
From the perspective of the kernel memory framework, this scaling directly follows from the analysis in Sec.
3.2 with k = F 0 .
In fact, in the regime of low errors, the kernel memory framework can also be used to derive a more precise capacity scaling for the classical Hopﬁeld network.
We ﬁrst note that any one-shot learning rule that implies A > 0 is equivalent to an SVM network where every stored pattern is a support vector.
Such a heuristic is only likely to be close to the optimal solution and perform well in large networks with very few patterns, as high-dimensional linear SVMs trained on few patterns are highly likely to ﬁnd solutions where all patterns are support vectors; this effect has been termed support vector proliferation [4].
Restricting the network to this regime limits the capacity to M max ⇠ O ( N 2log N ) , consistent with the result in [42] (see Appendix A.2).
Iterative learning rules.
The problem of iteratively training MHNs with biologically plausible online learning rules has recently been studied [64], with a resulting storage capacity ranging from ⇠ 0 .
16 N to ⇠ N , depending on the exact implementation.
The aim, in general, of such studies is to ﬁnd a learning rule capable of producing a capacity close to the theoretical maximum ⇠ 2 N .
For this purpose, the perspective of kernel memory networks can be particularly helpful, as many of the algorithms that have been developed over the past two decades to optimize SVMs can be utilized for MHNs as well.
For example, a network formulated in the feature form can be trained with the stochastic batch perceptron rule [14, 34], the passive aggressive rules [16], the minnorm rule [5], as well as with likelihood maximization applied to logistic regression [29, 46, 59].
In the kernel form, two of the most well-known online algorithms for training linear and non-linear SVMs are the
Adatron [3] and the Kernel-Adatron [19].
A performance comparison between iterative learning and the modern Hopﬁeld learning rule can be found in Appendix C.
Generalization.
Viewing the MHN as a recurrent network of SVMs can also facilitate a more intuitive understanding of its ability to generalize, when used as a conventional classiﬁer.
In this setting, one designates a subset of the neurons as input units, and the remaining neurons as outputs.
Given a set of input-output associations, one optimizes the memory patterns ⇠ µ using, for example, gradient descent.
Such an experiment was performed by Krotov and Hopﬁeld [35] on the MNIST data set, using a polynomial non-linearity F ( x ) = x p .
Results showed that the test error ﬁrst improved as p increased from 2 to 3, but later deteriorated for high degrees, like p = 20 .
While it may be difﬁcult to explain this behavior within an energy-based framework, it is entirely expected when viewed from the SVM perspective: a kernel of low polynomial degree has too few degrees of freedom to ﬁt the classiﬁcation boundary in the training set, causing underﬁtting , while a polynomial of too high degree grants the model too much ﬂexibility, which results in overﬁtting .
The pseudoinverse learning rule.
The coefﬁcients in A are, in general, computed numerically, and cannot be written in closed form.
However, in the special case when Eq. 8 is underdetermined, meaning M < N  , a closed-form (but suboptimal) solution can be obtained using the least-squares
SVM method [60].
The result is a generalized form of the pseudoinverse learning rule [50].
See
Appendix D for details.
Kernel memory networks for continuous patterns 4.1 Auto -associative memory as a recurrent interpolation network
So far, we have considered memory models designed to store only bipolar patterns.
We now relax this constraint and allow patterns to be continuous-valued.
We ﬁrst observe that any set of patterns
X 2 R N ⇥ M can be made ﬁxed points of the dynamics by training each neuron i to interpolate ⇠ µi when the rest of the network is initialized in ⇠ µ , for every pattern µ .
Assuming that the model is equipped with a kernel that allows for each ﬁxed point to also be attracting, we can ensure that a lower bounding estimate of the size of the attractor basin is maximized by ﬁnding the interpolation with minimum weight norm (see Appendix E.1 for proof).
These results are summarized below.
Property 4 (Robust auto-associative memory with continuous patterns) .
Suppose that the dynamics of a recurrent auto-associative memory network evolve according to the synchronous update rule
[Equation]
) where K = K ( X , X ) =
( X ) >  ( X ) is the kernel matrix and K † its Moore-Penrose pseudoinverse, where K † =
K  1 if  ( X ) is full column rank.
Then, the dynamics of the network is guaranteed to have the ﬁxed points X .
Moreover, if the points are attracting, Eq.
20 maximizes a lower bound of the attractor basin sizes.
4.2 A recurrent interpolation network with exponential capacity
Memory models for continuous data [e.g., 27, 33, 48] have generally received less attention than their binary counterparts.
Recently, however, Ramsauer et al.
[57] proposed an energy-based model capable of storing an exponential number of continuous-valued patterns (we will refer to this model as the softmax network).
While the structure of this model is similar to Eq.
20, it cannot be analyzed within the framework of Property 4, as it involves a kernel that is neither symmetric nor positive-deﬁnite [68].
Nonetheless, we will in this section demonstrate that it is possible to use conventional kernel methods to design an attractor network with exponential capacity for continuous patterns.
We utilize the properties of the SDM by using a translation-invariant kernel with a ﬁxed spatial scale r .
For the sake of simplicity, we choose the exponential power kernel (Exp  )
[Equation]
where  , r > 0 .
These parameters determine the shape of the attractor basin that surrounds each pattern.
While r roughly sets the radius of attraction,  represents an inverse temperature which changes the steepness of the boundary of the attractor basin.
Moreover, as long as the patterns are unique, the kernel matrix is invertible and we have K † exp
= K  1 exp  [44].
We will now analyze the noise robustness and storage capacity of this model.
To make the analysis tractable, we will operate in the regime of low temperatures, meaning the limit  !
We ﬁrst establish the following three properties.
Property 5.1 (The Exp  network at zero temperature) .
Given a set of unique patterns { ⇠ µ } M µ =1 with min µ, ⌫ 6 = µ k ⇠ µ  ⇠ ⌫ k 2 > r , the state update rule for the Exp  network at  ! 1 reduces to
[Equation]
(22) where ⇥ ( · ) is the Heaviside function with ⇥ (0) =
e  1 (see Appendix E.2.1).
Remark.
In geometrical terms, Property 5.1 states that the boundary of the basin of attraction surrounding each pattern becomes a sharp ( N  1) -dimensional hypersphere with radius r in the limit  !
For lower, ﬁnite  , the spherical boundary becomes increasingly fuzzy.
From the perspective of an energy landscape, each pattern lies in an N -dimensional energy minimum with inﬁnitely steep walls when  ! 1 .
As  is lowered, the barriers become progressively smoother.
Property 5.2 (Convergence in one step) .
Given a set of unique patterns { ⇠ µ } Mµ =1 with min µ, ⌫ 6 = µ k ⇠ µ  ⇠ ⌫ k 2 > 2 r , the Exp  network at  ! 1 , initialized at s (0) =
⇠ µ +  ⇠ , will converge to ⇠ µ in one step if k  ⇠ k 2 < r .
Property 5.3 (No spurious attractors) .
Given a set of unique patterns { ⇠ µ } Mµ =1 with min µ, ⌫ 6 = µ k ⇠ µ  ⇠ ⌫ k 2 > 2 r and @ µ : k ⇠ µ k 2 = r/ (1  e  1 ) , the only attractors of the dynamics of the Exp  network at  ! 1 are the points { ⇠ µ } Mµ =1 , together with 0 if @ µ : k ⇠ µ k 2  r .
Remark.
Properties 5.2 and 5.3 can be shown to be true simply by inserting the expression s (0) =
⇠ µ +  ⇠ in Eq. 22.
Assuming no overlaps between the basins of attraction, a quick calculation shows that s (1) =
⇠ µ if k  ⇠ k 2 < r .
If, on the other hand, the network is initialized such that k s (0)  ⇠ µ k 2 > r , 8 µ , one always obtains s (2) =
⇠ 0 , where ⇠ 0 is either 0 or the pattern closest to 0 .
In other words, the network recalls a pattern only if the initialization is close enough to it.
If located far from all patterns, the network assumes an “agnostic” state, represented either by the origin or the pattern closest to the origin (if the origin happens to be located within a basin of attraction).
In the following two properties, we evaluate how the radius of attraction r determines the maximum input noise tolerance and storage capacity.
Property 6 (Robustness to white noise) .
Assume that we are given a set of unique patterns ⇠ 1 , . . .
, ⇠ M ⇠ N ( 0 , I N ) with min µ, ⌫ 6 = µ k ⇠ µ  ⇠ ⌫ k 2 > 2 r , and that the Exp  network is initialized in a distorted pattern s (0) =
⇠ µ + ✏ , where ✏ ⇠ N ( 0 ,  2 I N ) .
Then, at  ! 1 , the maximum noise variance  2max with which ⇠ µ can be recovered in at least 50% of trials is
[Equation]
Property 7 (Exponential storage capacity) .
At  ! 1 , and for N  1 , the average maximum number of patterns sampled from N ( 0 , I N ) that the Exp  network can store and recall without errors is lower-bounded according to [Equation]
[Equation]
where  2max is the maximum white noise variance tolerated by the network.
Remark.
Proofs can be found in Appendices E.2.2 and E.2.3.
Note that Property 7 is valid in the range  2max .
While the bounds are fairly tight at the upper end of the range, they become loose when  2max ! 0 .
In this limit, which is equivalent to r ! 0 , the storage capacity tends to inﬁnity, as the risk of interference between patterns vanishes when their radius of attraction becomes inﬁnitesimal.
Comparison to the softmax network.
If patterns are randomly placed on a hypersphere instead of being normally distributed, the state update rule in Eq.
22 reduces to the form s ( t +1) =
X ⇥ ( X > s ( t )  ✓ ) , where ✓ is a ﬁxed threshold.
While the capacity remains exponential (see
Appendix E.3.1), the basin of attraction surrounding each pattern now forms a spherical cap instead of a ball.
We can compare this to the softmax network at zero temperature, given by s ( t +1) =
lim  ! 1 X softmax(  X > s ( t ) ) =
X arg max( X > s ( t ) ) .
This model differs from the Exp  only in a replacement of ⇥ with arg max .
This changes the shape of the attractor basins from spherical caps to Voronoi cells, which parcellate the entire surface of the hypersphere into a Voronoi diagram (see Fig. 2).
The boundary of each basin is now no longer radially symmetric around a pattern, but instead extends as far as possible in all directions.
Consequently, at  ! 1 , the softmax network has larger attractor basins and always converges to one of the stored patterns, regardless of the initialization point (assuming this is not precisely on a boundary).
In contrast, the Exp  network may converge to the origin if initialized far from all patterns.
This can be interpreted as an agnostic response, which indicates that the model cannot associate the input query with any of its stored patterns.
Discussion
Biological interpretation.
Kernel memory networks can be mapped to the anatomical properties of biological neurons.
Consider an individual neuron in the feature form of the recurrent network (Eq.
The state of neighboring neurons s is ﬁrst transformed through  ( s ) and thereafter projected to the neuron through the weight matrix ( A  X )  ( X ) > .
When the kernel is polynomial of degree p , so that K ( x i , x j ) = ( x > i x j + 1) p , the transformation  ( s ) consists of all elements in s and their cross-terms, up to degree p .
The input to each neuron, in other words, consists of the states of all other neurons, as well as all possible combinations of their multiplicative interactions.
This neuron model can be viewed as a generalized form of, for example, the multiconnected neuron [49], the clusteron [43], or the sigma-pi unit [58, p. 73].
These are all perceptrons that include multiplicative input interactions as a means to model synaptic cross-talk and cluster-sensitivity on non-linear dendrites [55] (see Fig. 1).
In the kernel form (Eq. 10), each neuron is, again, implicitly comprised of a two-stage process, whereby the raw input s is ﬁrst transformed through the function K ( X , s ) and then projected through the weight matrix A  X .
For any inner-product kernel K = k ( x > i x j ) , this representation can be directly identiﬁed as a two-layer neural network, where the hidden layer is deﬁned by the weights X and the activation function k .
This interpretation of the recurrent network was recently proposed in [35, 36] and discussed in relation to hippocampal-cortical interactions involved in memory storage and recall; it is particularly reminiscent of the hippocampal indexing theory [6, 61].
However, the kernel form can also be viewed as a network in which each individual neuron is a generalized form of the two-layered pyramidal cell model [53, 54].
This was originally proposed as an abstract neuron model augmented with non-linear dendritic processing [41].
It should be noted, however, that the idea of interpreting kernel methods as neural networks has a longer history, and has been extensively analyzed in the case of, for example, radial basis functions [51, 52].
For further details, see Appendix
F.
Summary.
We have shown that conventional kernel methods can be used to derive the weights for hetero- and auto-associative memory networks storing binary or continuous-valued patterns with maximal noise tolerance.
The result is a family of optimal memory models, which we call kernel memory networks , which includes the SDM and MHN as special cases.
This unifying framework facilitates an intuitive understanding of the storage capacity of memory models and offers new ways to biologically interpret these in terms of non-linear dendritic integration.
This work formalizes the links between kernel methods, attractor networks, and models of dendritic processing.
Future work.
A unifying theoretical framework for memory modeling can be useful for the development of both improved bio-plausible memory models and for machine learning applications.
First, recognizing that there exists algorithms for training optimally noise-robust classiﬁers and adapting these to biological constraints can aid in the development of normative synaptic three-factor learning rules [23].
Second, the theoretical link between neuron models, kernel functions, and storage capacity enables one to ﬁt kernel memory networks to neurophysiological data and to analyze the computational properties of biophysically informed memory models.
Finally, our unifying framework reveals that most memory models differ only in the choice of kernel (model complexity) and Lagrange parameters (model precision).
This categorization simpliﬁes the tailoring of memory models to their application, and allows for the design of models whose properties fundamentally can depart from kernel memory networks, by, for example, choosing kernels not associated with a reproducing kernel
Hilbert space.
Acknowledgments and Disclosure of Funding
This study was supported by funding from the Swiss government’s ETH Board of the Swiss Federal
Institutes of Technology to the Blue Brain Project, a research center of the École Polytechnique
Fédérale de Lausanne (EPFL).
[-As shown recently in [4], support vector proliferation for an SVM trained on M random patterns drawn uniformly from {± 1 } N occurs in the regime N ≳ 
-]
[-40 in 
-]
[-91 in 
-]