Original: • Contraction mapping with a ﬁxed point.
Revised: • Contraction mapping with a ﬁxed point.

Original: We assume that the ﬁrst l patterns are much more probable (and similar to one another) than the other patterns.
Revised: We assume that the ﬁrst l patterns are much more probable (and similar to one another) than the other patterns.

Original: Therefore, we deﬁne:
Revised: Therefore, we deﬁne:

Original: [Equation]
Revised: [Equation]

Original: M is an upper bound on the Euclidean norm of the patterns, which are vectors.
Revised: M is an upper bound on the Euclidean norm of the patterns, which are vectors.

Original: (cid:15) is an upper bound on the probability γ of not choosing one of the ﬁrst l patterns, while 1 − (cid:15) is a lower bound the probability (1 − γ ) of choosing one of the ﬁrst l patterns.
Revised: (cid:15) is an upper bound on the probability γ of not choosing one of the ﬁrst l patterns, while 1 − (cid:15) is a lower bound the probability (1 − γ ) of choosing one of the ﬁrst l patterns.

Original: m
Revised: m

Original: x is the arithmetic mean (the center) of the ﬁrst l patterns.
Revised: x is the arithmetic mean (the center) of the ﬁrst l patterns.

Original: m max is the maximal distance of the patterns to the center m x .
Revised: m max is the maximal distance of the patterns to the center m x .

Original: ˜ p is the probability p normalized for the ﬁrst l patterns.
Revised: ˜ p is the probability p normalized for the ﬁrst l patterns.

Original: The variance of the ﬁrst l patterns is
Revised: The variance of the ﬁrst l patterns is

Original: [Equation]
Revised: [Equation]

Original: We have shown that a ﬁxed point exists.
Revised: We have shown that a ﬁxed point exists.

Original: We want to know how fast the iteration converges to the ﬁxed point.
Revised: We want to know how fast the iteration converges to the ﬁxed point.

Original: Let m ∗ x be the ﬁxed point of the iteration f in the sphere S m .
Revised: Let m ∗ x be the ﬁxed point of the iteration f in the sphere S m .

Original: Using the mean value theorem Lemma A32, we have with J m = (cid:82) 1 0 J
Revised: Using the mean value theorem Lemma A32, we have with J m = (cid:82) 1 0 J

Original: [Equation]
Revised: [Equation]

Original: λ ξ +
Revised: λ ξ +

Original: [Equation]
Revised: [Equation]

Original: ∗
Revised: ∗

Original: [Equation]
Revised: [Equation]

Original: According to Lemma A8 the following bounds on the norm (cid:107) J (cid:107) 2 of the Jacobian of the ﬁxed point iteration hold.
Revised: According to Lemma A8 the following bounds on the norm (cid:107) J (cid:107) 2 of the Jacobian of the ﬁxed point iteration hold.

Original: The γ -bound for (cid:107) J (cid:107) 2 is
Revised: The γ -bound for (cid:107) J (cid:107) 2 is

Original: [Equation]
Revised: [Equation]

Original: while the (cid:15) -bound for (cid:107) J (cid:107) 2 is:
Revised: while the (cid:15) -bound for (cid:107) J (cid:107) 2 is:

Original: [Equation]
Revised: [Equation]

Original: From the last condition we require for a contraction mapping:
Revised: From the last condition we require for a contraction mapping:

Original: [Equation]
Revised: [Equation]

Original: We want to see how large (cid:15) is.
Revised: We want to see how large (cid:15) is.

Original: The separation of center m x from data X = ( x l +1 , . . .
Revised: The separation of center m x from data X = ( x l +1 , . . .

Original: , x N ) is
Revised: , x N ) is

Original: [Equation]
Revised: [Equation]

Original: We need the separation ˜∆ m of ˜ x = λ ξ + (1 − λ )
Revised: We need the separation ˜∆ m of ˜ x = λ ξ + (1 − λ )

Original: m ∗ x from the data.
Revised: m ∗ x from the data.

Original: [Equation]
Revised: [Equation]

Original: We compute a lower bound on ˜∆ m .
Revised: We compute a lower bound on ˜∆ m .

Original: Using the Cauchy-Schwarz inequality
Revised: Using the Cauchy-Schwarz inequality

Original: [Equation]
Revised: [Equation]

Original: we obtain for 1 (cid:54) j (cid:54) N :
Revised: we obtain for 1 (cid:54) j (cid:54) N :

Original: [Equation]
Revised: [Equation]

Original: We have the lower bound
Revised: We have the lower bound

Original: [Equation]
Revised: [Equation]

Original: Since
Revised: Since

Original: [Equation]
Revised: [Equation]

Original: , we have
Revised: , we have

Original: [Equation]
Revised: [Equation]

Original: A.1. P ROPERTIES OF F IXED P OINTS N EAR S TORED P ATTERN
Revised: A.1. P ROPERTIES OF F IXED P OINTS N EAR S TORED P ATTERN

Original: In Subsection A.1.5.3 many stable states that are ﬁxed points near the stored patterns are considered.
Revised: In Subsection A.1.5.3 many stable states that are ﬁxed points near the stored patterns are considered.

Original: We now consider this case.
Revised: We now consider this case.

Original: In the ﬁst subsection we investigate the storage capacity if all patterns are sufﬁciently separated so that metastable states do not appear.
Revised: In the ﬁst subsection we investigate the storage capacity if all patterns are sufﬁciently separated so that metastable states do not appear.

Original: In the next subsection we look into the convergence speed and error when retrieving the stored patterns.
Revised: In the next subsection we look into the updates required and error when retrieving the stored patterns.

Original: For metastable states we can do the same analyses if each metastable state is treated as one state like one pattern.
Revised: For metastable states we can do the same analyses if each metastable state is treated as one state like one pattern.

Original: We see a trade-off that is known from classical Hopﬁeld networks and for modern Hopﬁeld networks.
Revised: We see a trade-off that is known from classical Hopﬁeld networks and for modern Hopﬁeld networks.

Original: Small separation ∆ i of the pattern x
Revised: Small separation ∆ i of the pattern x

Original: i from the other patterns gives high storage capacity.
Revised: i from the other patterns gives high storage capacity.

Original: However the convergence speed is lower and the retrieval error higher.
Revised: However the convergence speed is lower and the retrieval error higher.

Original: In contrast, large separation ∆ i of the pattern x i from the other pattern gives exponentially fast convergence (one update is sufﬁcient) and exponentially lowretrieval error.
Revised: In contrast, large separation ∆ i of the pattern x i from the other pattern allows the retrieval of patterns with one update step and exponentially low error.

Original: A.1.6.1 Exponentially Many Patterns can be Stored.
Revised: A.1.6.1 Exponentially Many Patterns can be Stored.

Original: From Subsection A.1.5.3 need some deﬁnitions.
Revised: From Subsection A.1.5.3 need some deﬁnitions.

Original: We assume to have N patterns, the separation of pattern x i from the other patterns { x 1 , . . .
Revised: We assume to have N patterns, the separation of pattern x i from the other patterns { x 1 , . . .

Original: , x i − 1 , x i +1 , . . .
Revised: , x i − 1 , x i +1 , . . .

Original: , x N } is ∆ i , deﬁned as
Revised: , x N } is ∆ i , deﬁned as

Original: [Equation]
Revised: [Equation]

Original: The pattern is separated from the other data if 0 < ∆ i .
Revised: The pattern is separated from the other data if 0 < ∆ i .

Original: The separation ∆ i can also be expressed as
Revised: The separation ∆ i can also be expressed as

Original: [Equation]
Revised: [Equation]

Original: For (cid:107) x i (cid:107) =
Revised: For (cid:107) x i (cid:107) =

Original: (cid:107) x j (cid:107) we have ∆ i = 1 / 2 min
Revised: (cid:107) x j (cid:107) we have ∆ i = 1 / 2 min

Original: j
Revised: j

Original: [Equation]
Revised: [Equation]

Original: j (cid:54) =
Revised: j (cid:54) =

Original: i (cid:107) x
Revised: i (cid:107) x

Original: i − x j (cid:107) 2 .
Revised: i − x j (cid:107) 2 .

Original: The sphere S i with center x
Revised: The sphere S i with center x

Original: i is deﬁned as
Revised: i is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: The maximal length of a pattern is M = max i
Revised: The maximal length of a pattern is M = max i

Original: [Equation]
Revised: [Equation]

Original: x i
Revised: x i

Original: [Equation]
Revised: [Equation]

Original: We next deﬁne what we mean with storing and retrieving a pattern.
Revised: We next deﬁne what we mean with storing and retrieving a pattern.

Original: Deﬁnition 5 (Pattern Stored and Retrieved) .
Revised: Deﬁnition 5 (Pattern Stored and Retrieved) .

Original: We assume that around every pattern x i a sphere S i is given.
Revised: We assume that around every pattern x i a sphere S i is given.

Original: We say x i is stored if there is a single ﬁxed point x ∗ i ∈ S
Revised: We say x i is stored if there is a single ﬁxed point x ∗ i ∈ S

Original: i to which all points ξ ∈ S i converge, and S i ∩ S j = ∅ for i (cid:54)
Revised: i to which all points ξ ∈ S i converge, and S i ∩ S j = ∅ for i (cid:54)

Original: =
Revised: =

Original: j .
Revised: j .

Original: We say x
Revised: We say x

Original: i is retrieved  if iteration (update rule) Eq.
Revised: i is retrieved for a given (cid:15) if iteration (update rule) Eq.

Original: (91) converged to the single ﬁxed point x ∗ i ∈ S i .
Revised: (91) gives a point ˜ x i that is at least (cid:15) -close to the single ﬁxed point x ∗ i ∈ S i .

Original: The retrieval error is (cid:107) x
Revised: The retrieval error is (cid:107) ˜

Original: i − x∗
Revised: x

Original: i  (cid:107) .
Revised: i − x i (cid:107) .

Original: The sphere S i around pattern x i can be any a sphere and do not have the speciﬁc sphere deﬁned in
Revised: The sphere S i around pattern x i can be any a sphere and do not have the speciﬁc sphere deﬁned in

Original: Def. 3.
Revised: Def. 3.

Original: For a query ξ ∈ S i to converge to a ﬁxed point x ∗ i ∈ S
Revised: For a query ξ ∈ S i to converge to a ﬁxed point x ∗ i ∈ S

Original: i we required for the application of Banach ﬁxed point theorem and for ensuring a contraction mapping the following inequality:
Revised: i we required for the application of Banach ﬁxed point theorem and for ensuring a contraction mapping the following inequality:

Original: [Equation]
Revised: [Equation]

Original: This is the assumption in Lemma A7 to ensure a ﬁxed point in sphere S i .
Revised: This is the assumption in Lemma A7 to ensure a ﬁxed point in sphere S i .

Original: Since replacing ( N − 1) N by N 2 gives
Revised: Since replacing ( N − 1) N by N 2 gives

Original: [Equation]
Revised: [Equation]

Original: the inequality follows from following master inequality
Revised: the inequality follows from following master inequality

Original: [Equation]
Revised: [Equation]

Original: If we assume that S i ∩ S j (cid:54) = ∅ with i (cid:54) =
Revised: If we assume that S i ∩ S j (cid:54) = ∅ with i (cid:54) =

Original: j , then the triangle inequality with a point from the intersection gives
Revised: j , then the triangle inequality with a point from the intersection gives

Original: [Equation]
Revised: [Equation]

Original: Therefore, we have using the Cauchy-Schwarz inequality:
Revised: Therefore, we have using the Cauchy-Schwarz inequality:

Original: [Equation]
Revised: [Equation]

Original: The last inequality is a contraction to Eq.
Revised: The last inequality is a contraction to Eq.

Original: (301) if we assume that
Revised: (301) if we assume that

Original: [Equation]
Revised: [Equation]

Original: With this assumption, the spheres S i and S j do not intersect.
Revised: With this assumption, the spheres S i and S j do not intersect.

Original: Therefore, each x i has its separate ﬁxed point in S i .
Revised: Therefore, each x i has its separate ﬁxed point in S i .

Original: We deﬁne
Revised: We deﬁne

Original: [Equation]
Revised: [Equation]

Original: to obtain the master inequality
Revised: to obtain the master inequality

Original: [Equation]
Revised: [Equation]

Original: • Patterns on a sphere
Revised: • Patterns on a sphere

Original: For simplicity and in accordance with the results of the classical Hopﬁeld network, we assume all patterns being on a sphere with radius M :
Revised: For simplicity and in accordance with the results of the classical Hopﬁeld network, we assume all patterns being on a sphere with radius M :

Original: [Equation]
Revised: [Equation]

Original: Under assumption Eq.
Revised: Under assumption Eq.

Original: (304) we have only to show that the master inequality Eq.
Revised: (304) we have only to show that the master inequality Eq.

Original: (306) is fulﬁlled for each x
Revised: (306) is fulﬁlled for each x

Original: i to have a separate ﬁxed point near each x i .
Revised: i to have a separate ﬁxed point near each x i .

Original: We deﬁned α ij as the angle between x i and x j .
Revised: We deﬁned α ij as the angle between x i and x j .

Original: The minimal angle α min between two data points is
Revised: The minimal angle α min between two data points is

Original: [Equation]
Revised: [Equation]

Original: On the sphere with radius M we have
Revised: On the sphere with radius M we have

Original: [Equation]
Revised: [Equation]

Original: therefore it is sufﬁcient to show the master inequality on the sphere:
Revised: therefore it is sufﬁcient to show the master inequality on the sphere:

Original: [Equation]
Revised: [Equation]

Original: Under assumption Eq.
Revised: Under assumption Eq.

Original: (304) we have only to show that the master inequality Eq.
Revised: (304) we have only to show that the master inequality Eq.

Original: (306) is fulﬁlled for ∆ min .
Revised: (306) is fulﬁlled for ∆ min .

Original: We consider patterns on the sphere, therefore the master inequality Eq.
Revised: We consider patterns on the sphere, therefore the master inequality Eq.

Original: (306) becomes
Revised: (306) becomes

Original: Eq.
Revised: Eq.

Original: First we show results when pattern positions on the sphere are constructed and ∆ min is ensured.
Revised: First we show results when pattern positions on the sphere are constructed and ∆ min is ensured.

Original: Then we move on to random patterns on a sphere, where ∆ min becomes a random variable.
Revised: Then we move on to random patterns on a sphere, where ∆ min becomes a random variable.

Original: • Storage capacity for patterns placed on the sphere.
Revised: • Storage capacity for patterns placed on the sphere.

Original: Next theorem says how many patterns we can stored (ﬁxed point with attraction basin near pattern) if we are allowed to place them on the sphere.
Revised: Next theorem says how many patterns we can stored (ﬁxed point with attraction basin near pattern) if we are allowed to place them on the sphere.

Original: Theorem A3 (Storage Capacity (M=2): Placed Patterns) .
Revised: Theorem A3 (Storage Capacity (M=2): Placed Patterns) .

Original: We assume β = 1 and patterns on the sphere with radius M .
Revised: We assume β = 1 and patterns on the sphere with radius M .

Original: If M = 2 √ d − 1 and the dimension d of the space is d ≥ 4 or if
Revised: If M = 2 √ d − 1 and the dimension d of the space is d ≥ 4 or if

Original: M = 1 . 7 √ d − 1 and the dimension d of the space is d ≥ 50 , then the number of patterns N that can be stored (ﬁxed point with attraction basin near pattern) is at least
Revised: M = 1 . 7 √ d − 1 and the dimension d of the space is d ≥ 50 , then the number of patterns N that can be stored (ﬁxed point with attraction basin near pattern) is at least

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For random patterns on the sphere, we have to show that the master inequality Eq.
Revised: For random patterns on the sphere, we have to show that the master inequality Eq.

Original: (310) holds:
Revised: (310) holds:

Original: [Equation]
Revised: [Equation]

Original: We now place the patterns equidistant on the sphere where the pattern are separated by an angle α min :
Revised: We now place the patterns equidistant on the sphere where the pattern are separated by an angle α min :

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: In a d -dimensional space we can place
Revised: In a d -dimensional space we can place

Original: [Equation]
Revised: [Equation]

Original: points on the sphere.
Revised: points on the sphere.

Original: In a spherical coordinate system a pattern differs from its most closest patterns by an angle α min and there are d − 1 angles.
Revised: In a spherical coordinate system a pattern differs from its most closest patterns by an angle α min and there are d − 1 angles.

Original: Solving for α min gives
Revised: Solving for α min gives

Original: [Equation]
Revised: [Equation]

Original: The number of patterns that can be stored is determined by the largest N that fulﬁls
Revised: The number of patterns that can be stored is determined by the largest N that fulﬁls

Original: [Equation]
Revised: [Equation]

Original: We set N = 2 2( d − 1) and obtain for Eq.
Revised: We set N = 2 2( d − 1) and obtain for Eq.

Original: [Equation]
Revised: [Equation]

Original: This inequality is equivalent to
Revised: This inequality is equivalent to

Original: [Equation]
Revised: [Equation]

Original: The last inequality can be fulﬁlled with M = K √ d − 1 and proper K .
Revised: The last inequality can be fulﬁlled with M = K √ d − 1 and proper K .

Original: For β = 1 , d = 4 and K =the inequality is fulﬁlled.
Revised: For β = 1 , d = 4 and K =the inequality is fulﬁlled.

Original: The left hand side minus the right hand side is 4( d − 1) − 1 / 2 2( d − 1) − 1 − ln(8( d − 1)) − 4( d − 1)
Revised: The left hand side minus the right hand side is 4( d − 1) − 1 / 2 2( d − 1) − 1 − ln(8( d − 1)) − 4( d − 1)

Original: ln 2 .
Revised: ln 2 .

Original: Its derivative with respect to d is strict positive.
Revised: Its derivative with respect to d is strict positive.

Original: Therefore, the inequality holds for d ≥ 4 .
Revised: Therefore, the inequality holds for d ≥ 4 .

Original: For β = 1 , d = 50 and K = 1 .
Revised: For β = 1 , d = 50 and K = 1 .

Original: the inequality is fulﬁlled.
Revised: the inequality is fulﬁlled.

Original: The left hand side minus the right hand side is 2 . 

Revised: The left hand side minus the right hand side is 2 .

Original: ( d − 1) − 1 / 2 2( d − 1) − 1 − ln(5 .
Revised: ( d − 1) − 1 / 2 2( d − 1) − 1 − ln(5 .

Original: 78( d − 1)) − 4( d − 1) ln 2 .
Revised: 78( d − 1)) − 4( d − 1) ln 2 .

Original: Its derivative with respect to d is strict positive.
Revised: Its derivative with respect to d is strict positive.

Original: Therefore, the inequality holds for d ≥ 50 .
Revised: Therefore, the inequality holds for d ≥ 50 .

Original: If we want to store considerably more patterns, then we have to increase the length of the vectors or the dimension of the space where the vectors live.
Revised: If we want to store considerably more patterns, then we have to increase the length of the vectors or the dimension of the space where the vectors live.

Original: The next theorem shows results for the number of patterns N with N = 2 3( d − 1) .
Revised: The next theorem shows results for the number of patterns N with N = 2 3( d − 1) .

Original: Theorem A4 (Storage Capacity (M=5): Placed Patterns) .
Revised: Theorem A4 (Storage Capacity (M=5): Placed Patterns) .

Original: We assume β = 1 and patterns on the sphere with radius M .
Revised: We assume β = 1 and patterns on the sphere with radius M .

Original: If M = 5 √ d − 1 and the dimension d of the space is d ≥ 3 or if
Revised: If M = 5 √ d − 1 and the dimension d of the space is d ≥ 3 or if

Original: M = 4 √ d − 1 and the dimension d of the space is d ≥ 13 , then the number of patterns N that can be stored (ﬁxed point with attraction basin near pattern) is at least
Revised: M = 4 √ d − 1 and the dimension d of the space is d ≥ 13 , then the number of patterns N that can be stored (ﬁxed point with attraction basin near pattern) is at least

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We set N = 2 3( d − 1) and obtain for Eq.
Revised: We set N = 2 3( d − 1) and obtain for Eq.

Original: [Equation]
Revised: [Equation]

Original: This inequality is equivalent to
Revised: This inequality is equivalent to

Original: [Equation]
Revised: [Equation]

Original: The last inequality can be fulﬁlled with M = K √ d − 1 and proper K .
Revised: The last inequality can be fulﬁlled with M = K √ d − 1 and proper K .

Original: For β = 1 , d = 13 and
Revised: For β = 1 , d = 13 and

Original: K
Revised: K

Original: = 4 the inequality is fulﬁlled.
Revised: = 4 the inequality is fulﬁlled.

Original: The left hand side minus the right hand side is 4 .
Revised: The left hand side minus the right hand side is 4 .

Original: 686292( d − 1) − 1 / 2 

Revised: 686292( d − 1) − 1 / 2

Original: ( d − 1) − 1 − ln(32( d − 1))
Revised: ( d − 1) − 1 − ln(32( d − 1))

Original: − 6( d − 1) ln 2 .
Revised: − 6( d − 1) ln 2 .

Original: Its derivative with respect to d is strict positive.
Revised: Its derivative with respect to d is strict positive.

Original: Therefore, the inequality holds for d ≥ 13 .
Revised: Therefore, the inequality holds for d ≥ 13 .

Original: For β = 1 , d = 3 and K = 5 the inequality is fulﬁlled.
Revised: For β = 1 , d = 3 and K = 5 the inequality is fulﬁlled.

Original: The left hand side minus the right hand side is 7 .
Revised: The left hand side minus the right hand side is 7 .

Original: 32233( d − 1) − 1 / 2 

Revised: 32233( d − 1) − 1 / 2

Original: ( d − 1) − 1 − ln(50( d − 1))
Revised: ( d − 1) − 1 − ln(50( d − 1))

Original: − 6( d − 1) ln 2 .
Revised: − 6( d − 1) ln 2 .

Original: Its derivative with respect
Revised: Its derivative with respect

Original: to d is strict positive.
Revised: to d is strict positive.

Original: Therefore, the inequality holds for d ≥ 3 .
Revised: Therefore, the inequality holds for d ≥ 3 .

Original: • Storage capacity for random patterns on the sphere.
Revised: • Storage capacity for random patterns on the sphere.

Original: Next we investigate random points on the sphere.
Revised: Next we investigate random points on the sphere.

Original: Under assumption Eq.
Revised: Under assumption Eq.

Original: (304) we have to show that the master inequality Eq.
Revised: (304) we have to show that the master inequality Eq.

Original: (310) is fulﬁlled for α min , where now α min is now a random variable.
Revised: (310) is fulﬁlled for α min , where now α min is now a random variable.

Original: We use results on the distribution of the minimal angles between random patterns on a sphere according to Cai et al.
Revised: We use results on the distribution of the minimal angles between random patterns on a sphere according to Cai et al.

Original: and Brauchart et al.
Revised: and Brauchart et al.

Original: Theorem 2 in Cai et al.
Revised: Theorem 2 in Cai et al.

Original: (2013) gives the distribution of the minimal angle for random patterns on the unit sphere.
Revised: (2013) gives the distribution of the minimal angle for random patterns on the unit sphere.

Original: Proposition 3.5 in Brauchart et al.
Revised: Proposition 3.5 in Brauchart et al.

Original: (2018) gives a lower bound on the probability of the minimal angle being larger than a given constant.
Revised: (2018) gives a lower bound on the probability of the minimal angle being larger than a given constant.

Original: We require this proposition to derive the probability of pattern having a minimal angle α min .
Revised: We require this proposition to derive the probability of pattern having a minimal angle α min .

Original: Proposition 3.6 in Brauchart et al.
Revised: Proposition 3.6 in Brauchart et al.

Original: (2018) gives the expectation of the minimal angle.
Revised: (2018) gives the expectation of the minimal angle.

Original: We will prove high probability bounds for the expected storage capacity.
Revised: We will prove high probability bounds for the expected storage capacity.

Original: We need the following tail-bound on α min (the minimal angle of random patterns on a sphere):
Revised: We need the following tail-bound on α min (the minimal angle of random patterns on a sphere):

Original: Lemma A13 ((Brauchart et al., 2018)) .
Revised: Lemma A13 ((Brauchart et al., 2018)) .

Original: Let d be the dimension of the pattern space,
Revised: Let d be the dimension of the pattern space,

Original: [Equation]
Revised: [Equation]

Original: and δ > 0 such that κ d − 1 2 δ ( d − 1) (cid:54) 1 .
Revised: and δ > 0 such that κ d − 1 2 δ ( d − 1) (cid:54) 1 .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The statement of the lemma is Eq.
Revised: The statement of the lemma is Eq.

Original: (3-6) from Proposition 3.5 in Brauchart et al.
Revised: (3-6) from Proposition 3.5 in Brauchart et al.

Original: Next we derive upper and lower bounds on the constant κ d since we require them later for proving storage capacity bounds.
Revised: Next we derive upper and lower bounds on the constant κ d since we require them later for proving storage capacity bounds.

Original: Lemma A14.
Revised: Lemma A14.

Original: For κ d deﬁned in Eq.
Revised: For κ d deﬁned in Eq.

Original: (322) we have the following bounds for every d ≥ 1 :
Revised: (322) we have the following bounds for every d ≥ 1 :

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We use for x > 0 the following bound related to Stirling’s approximation formula for the gamma function, c.f.
Revised: We use for x > 0 the following bound related to Stirling’s approximation formula for the gamma function, c.f.

Original: (Olver et al., 2010, (5.6.1)):
Revised: (Olver et al., 2010, (5.6.1)):

Original: [Equation]
Revised: [Equation]

Original: Using Stirling’s formula Eq.
Revised: Using Stirling’s formula Eq.

Original: (325), we upper bound κ d :
Revised: (325), we upper bound κ d :

Original: [Equation]
Revised: [Equation]

Original: For the ﬁrst inequality, we applied Eq.
Revised: For the ﬁrst inequality, we applied Eq.

Original: (325), while for the second we used (1 + 1 d ) d < e for d ≥ 1 .
Revised: (325), while for the second we used (1 + 1 d ) d < e for d ≥ 1 .

Original: Next, we lower bound κ d by again applying Stirling’s formula Eq.
Revised: Next, we lower bound κ d by again applying Stirling’s formula Eq.

Original: [Equation]
Revised: [Equation]

Original: where the last inequality holds because of monotonicity of (1 + 1 d ) d and using the fact that for d =it takes on the value 2.
Revised: where the last inequality holds because of monotonicity of (1 + 1 d ) d and using the fact that for d =it takes on the value 2.

Original: We require a bound on cos to bound the master inequality Eq.
Revised: We require a bound on cos to bound the master inequality Eq.

Original: Lemma A15.
Revised: Lemma A15.

Original: For 0 (cid:54) x (cid:54) π the function cos can be upper bounded by:
Revised: For 0 (cid:54) x (cid:54) π the function cos can be upper bounded by:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We use the inﬁnite product representation of cos , c.f.
Revised: We use the inﬁnite product representation of cos , c.f.

Original: (Olver et al., 2010, (4.22.2)):
Revised: (Olver et al., 2010, (4.22.2)):

Original: [Equation]
Revised: [Equation]

Original: Since it holds that
Revised: Since it holds that

Original: [Equation]
Revised: [Equation]

Original: for | x | (cid:54) π and n ≥ 2 , we can get the following upper bound on Eq.
Revised: for | x | (cid:54) π and n ≥ 2 , we can get the following upper bound on Eq.

Original: [Equation]
Revised: [Equation]

Original: The last but one inequality uses x (cid:54) π , which implies x/π (cid:54) 1 .
Revised: The last but one inequality uses x (cid:54) π , which implies x/π (cid:54) 1 .

Original: Thus Eq.
Revised: Thus Eq.

Original: (328) is proven.
Revised: (328) is proven.

Original: • Exponential storage capacity: the base c as a function of the parameter β , the radius of the sphere M , the probability p , and the dimension d of the space.
Revised: • Exponential storage capacity: the base c as a function of the parameter β , the radius of the sphere M , the probability p , and the dimension d of the space.

Original: We
Revised: We

Original: express the number N of stored patterns by an exponential function with base c > 1 and an exponent linear in d .
Revised: express the number N of stored patterns by an exponential function with base c > 1 and an exponent linear in d .

Original: We derive constraints on he base c as a function of β , the radius of the sphere
Revised: We derive constraints on he base c as a function of β , the radius of the sphere

Original: M , the probability p that all patterns can be stored, and the dimension d of the space.
Revised: M , the probability p that all patterns can be stored, and the dimension d of the space.

Original: With β > 0 ,
Revised: With β > 0 ,

Original: K > 0 , and d ≥ 2 (to ensure a sphere), the following theorem gives our main result.
Revised: K > 0 , and d ≥ 2 (to ensure a sphere), the following theorem gives our main result.

Original: Theorem A5 (Storage Capacity (Main): Random Patterns) .
Revised: Theorem A5 (Storage Capacity (Main): Random Patterns) .

Original: We assume a failure probability 0 < p (cid:54) 1 and randomly chosen patterns on the sphere with radius M :=
Revised: We assume a failure probability 0 < p (cid:54) 1 and randomly chosen patterns on the sphere with radius M :=

Original: K √ d − 1 .
Revised: K √ d − 1 .

Original: We deﬁne
Revised: We deﬁne

Original: [Equation]
Revised: [Equation]

Original: where W 0 is the upper branch of the Lambert W function (Olver et al., 2010, (4.13)) and ensure
Revised: where W 0 is the upper branch of the Lambert W function (Olver et al., 2010, (4.13)) and ensure

Original: [Equation]
Revised: [Equation]

Original: Then with probability 1 − p , the number of random patterns that can be stored is
Revised: Then with probability 1 − p , the number of random patterns that can be stored is

Original: [Equation]
Revised: [Equation]

Original: Examples are c ≥ 3 . 1546 for β = 1 , K = 3 , d = 20 and p = 0 .
Revised: Therefore it is proven for c ≥ 3 . 1546 with β = 1 , K = 3 , d = 20 and p = 0 .

Original: 001 ( a + ln( b ) > 1 . 27 ) and  c ≥ 1 . 3718 for β = 1 , K = 1 , d = 75 , and p = 0 .
Revised: 001 ( a + ln( b ) > 1 . 27 ) and proven for c ≥ 1 . 3718 with β = 1 , K = 1 , d = 75 , and p = 0 .

Original: 001 ( a + ln( b ) < − 0 . 94 ).
Revised: 001 ( a + ln( b ) < − 0 . 94 ).

Original: Proof.
Revised: Proof.

Original: We consider the probability that the master inequality Eq.
Revised: We consider the probability that the master inequality Eq.

Original: (310) is fulﬁlled:
Revised: (310) is fulﬁlled:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Using Eq.
Revised: Using Eq.

Original: (328), we have:
Revised: (328), we have:

Original: [Equation]
Revised: [Equation]

Original: Therefore, with probability 1 − p the storage capacity is largest N that fulﬁlls
Revised: Therefore, with probability 1 − p the storage capacity is largest N that fulﬁlls

Original: [Equation]
Revised: [Equation]

Original: This inequality is equivalent to
Revised: This inequality is equivalent to

Original: Pr
Revised: Pr

Original: [Equation]
Revised: [Equation]

Original: We use Eq.
Revised: We use Eq.

Original: (323) to obtain:
Revised: (323) to obtain:

Original: [Equation]
Revised: [Equation]

Original: For Eq.
Revised: For Eq.

Original: (338) to be fulﬁlled
Revised: (338) to be fulﬁlled

Original: [Equation]
Revised: [Equation]

Original: it is sufﬁcient that
Revised: it is sufﬁcient that

Original: [Equation]
Revised: [Equation]

Original: If we insert the assumption Eq.
Revised: If we insert the assumption Eq.

Original: (333) of the theorem into Eq.
Revised: (333) of the theorem into Eq.

Original: [Equation]
Revised: [Equation]

Original: then we obtain N ≥ 2 .
Revised: then we obtain N ≥ 2 .

Original: We now apply the upper bound κ d − 1 / 2 < κ d − 1 < 1 from Eq.
Revised: We now apply the upper bound κ d − 1 / 2 < κ d − 1 < 1 from Eq.

Original: (324) and the upper bound 2 βN (cid:54) 1 β from
Revised: (324) and the upper bound 2 βN (cid:54) 1 β from

Original: N ≥ 2 to inequality Eq.
Revised: N ≥ 2 to inequality Eq.

Original: In the resulting inequality we insert N = √
Revised: In the resulting inequality we insert N = √

Original: pc
Revised: pc

Original: d − 14 to check whether it is fulﬁlled with this special value of N and obtain:
Revised: d − 14 to check whether it is fulﬁlled with this special value of N and obtain:

Original: [Equation]
Revised: [Equation]

Original: Dividing by p , inserting M = K √ d − 1 , and exponentiation of the left and right side by 2 d − 1 gives:
Revised: Dividing by p , inserting M = K √ d − 1 , and exponentiation of the left and right side by 2 d − 1 gives:

Original: [Equation]
Revised: [Equation]

Original: After some algebraic manipulation, this inequality can be written as
Revised: After some algebraic manipulation, this inequality can be written as

Original: [Equation]
Revised: [Equation]

Original: where we used a :
Revised: where we used a :

Original: [Equation]
Revised: [Equation]

Original: We determine the value ˆ c of c which makes the inequality Eq.
Revised: We determine the value ˆ c of c which makes the inequality Eq.

Original: (343) equal to zero.
Revised: (343) equal to zero.

Original: We solve
Revised: We solve

Original: [Equation]
Revised: [Equation]

Original: for ˆ c :
Revised: for ˆ c :

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where W 0 is the upper branch of the Lambert W function (see Def. A6).
Revised: where W 0 is the upper branch of the Lambert W function (see Def. A6).

Original: Hence, the solution is
Revised: Hence, the solution is

Original: [Equation]
Revised: [Equation]

Original: The solution exist, since the Lambert function W 0 ( x ) (Olver et al., 2010, (4.13)) is deﬁned for − 1 /e <
Revised: The solution exist, since the Lambert function W 0 ( x ) (Olver et al., 2010, (4.13)) is deﬁned for − 1 /e <

Original: x and we have 0 <
Revised: x and we have 0 <

Original: exp( a + ln( b ) .
Revised: exp( a + ln( b ) .

Original: Since ˆ c fulﬁlls inequality Eq.
Revised: Since ˆ c fulﬁlls inequality Eq.

Original: (343) and therefore also Eq.
Revised: (343) and therefore also Eq.

Original: (341), we have a lower bound on the storage capacity N :
Revised: (341), we have a lower bound on the storage capacity N :

Original: [Equation]
Revised: [Equation]

Original: Next we aim at a lower bound on c which does not use the Lambert W function (Olver et al., 2010, (4.13)).
Revised: Next we aim at a lower bound on c which does not use the Lambert W function (Olver et al., 2010, (4.13)).

Original: Therefore, we upper bound W 0 (exp( a + ln( b )) to obtain a lower bound on c , therefore, also a lower bound on the storage capacity N .
Revised: Therefore, we upper bound W 0 (exp( a + ln( b )) to obtain a lower bound on c , therefore, also a lower bound on the storage capacity N .

Original: The lower bound is given in the next corollary.
Revised: The lower bound is given in the next corollary.

Original: Corollary A1.
Revised: Corollary A1.

Original: We assume a failure probability 0 < p (cid:54) 1 and randomly chosen patterns on the sphere with radius M = K √ d − 1 .
Revised: We assume a failure probability 0 < p (cid:54) 1 and randomly chosen patterns on the sphere with radius M = K √ d − 1 .

Original: We deﬁne
Revised: We deﬁne

Original: [Equation]
Revised: [Equation]

Original: Using the omega constant Ω ≈ 0 .
Revised: Using the omega constant Ω ≈ 0 .

Original: 56714329 we set
Revised: 56714329 we set

Original: [Equation]
Revised: [Equation]

Original: and ensure
Revised: and ensure

Original: [Equation]
Revised: [Equation]

Original: Then with probability 1 − p , the number of random patterns that can be stored is
Revised: Then with probability 1 − p , the number of random patterns that can be stored is

Original: [Equation]
Revised: [Equation]

Original: Examples are c ≥ 3 .
Revised: Examples are c ≥ 3 .

Original: 1444 for β = 1 , K = 3 , d = 20 and p = 0 .
Revised: 1444 for β = 1 , K = 3 , d = 20 and p = 0 .

Original: 001 ( a + ln( b ) > 1 . 27 ) and c ≥ 1 .
Revised: 001 ( a + ln( b ) > 1 . 27 ) and c ≥ 1 .

Original: 2585 for β = 1 K = 1 , d = 75 , and p = 0 .
Revised: 2585 for β = 1 K = 1 , d = 75 , and p = 0 .

Original: 001 ( a + ln( b ) < − 0 . 94 ).
Revised: 001 ( a + ln( b ) < − 0 . 94 ).

Original: Proof.
Revised: Proof.

Original: We lower bound the c deﬁned in Theorem A5.
Revised: We lower bound the c deﬁned in Theorem A5.

Original: According to (Hoorfar & Hassani, 2008,
Revised: According to (Hoorfar & Hassani, 2008,

Original: Theorem 2.3) we have for any real u and y > 1 e :
Revised: Theorem 2.3) we have for any real u and y > 1 e :

Original: [Equation]
Revised: [Equation]

Original: To upper bound W 0 ( x ) for x ∈ [0 , 1] , we set
Revised: To upper bound W 0 ( x ) for x ∈ [0 , 1] , we set

Original: [Equation]
Revised: [Equation]

Original: where the Omega constant Ω is
Revised: where the Omega constant Ω is

Original: [Equation]
Revised: [Equation]

Original: See for these equations the special values of the Lambert W function in Lemma A31.
Revised: See for these equations the special values of the Lambert W function in Lemma A31.

Original: We have the upper bound on W 0 :
Revised: We have the upper bound on W 0 :

Original: [Equation]
Revised: [Equation]

Original: At the right hand side of interval [0 , 1] , we have u = 0 and exp( u ) = 1 and get:
Revised: At the right hand side of interval [0 , 1] , we have u = 0 and exp( u ) = 1 and get:

Original: [Equation]
Revised: [Equation]

Original: Therefore, the bound is tight at the right hand side of of interval [0 , 1] , that is for exp( u ) = 1 , i.e. u = 0 .
Revised: Therefore, the bound is tight at the right hand side of of interval [0 , 1] , that is for exp( u ) = 1 , i.e. u = 0 .

Original: We have derived an bound for W 0 (exp( u )) with exp( u ) ∈ [0 , 1] or, equivalently, u ∈ [ −∞ , 0] .
Revised: We have derived an bound for W 0 (exp( u )) with exp( u ) ∈ [0 , 1] or, equivalently, u ∈ [ −∞ , 0] .

Original: We obtain from Hoorfar & Hassani (2008, Corollary 2.6) the following bound on W 0 (exp( u )) for 1 < exp( u ) , or, equivalently 0 < u :
Revised: We obtain from Hoorfar & Hassani (2008, Corollary 2.6) the following bound on W 0 (exp( u )) for 1 < exp( u ) , or, equivalently 0 < u :

Original: [Equation]
Revised: [Equation]

Original: A lower bound on ˆ c is obtained via the upper bounds Eq.
Revised: A lower bound on ˆ c is obtained via the upper bounds Eq.

Original: (356) and Eq.
Revised: (356) and Eq.

Original: (354) on W 0 as W 0 > 0 .
Revised: (354) on W 0 as W 0 > 0 .

Original: We set u = a + ln( b ) and obtain
Revised: We set u = a + ln( b ) and obtain

Original: [Equation]
Revised: [Equation]

Original: We insert this bound into Eq.
Revised: We insert this bound into Eq.

Original: (346), the solution for ˆ c , to obtain the statement of the theorem.
Revised: (346), the solution for ˆ c , to obtain the statement of the theorem.

Original: • Exponential storage capacity: the dimension d of the space as a function of the parameter β , the radius of the sphere M , and the probability p .
Revised: • Exponential storage capacity: the dimension d of the space as a function of the parameter β , the radius of the sphere M , and the probability p .

Original: We express the number N of stored patterns by an exponential function with base c >
Revised: We express the number N of stored patterns by an exponential function with base c >

Original: 1 and an exponent linear in d .
Revised: 1 and an exponent linear in d .

Original: We derive constraints on the dimension d of the space as a function of β , the radius of the sphere M , the probability p that all patterns can be stored, and the base of the exponential storage capacity.
Revised: We derive constraints on the dimension d of the space as a function of β , the radius of the sphere M , the probability p that all patterns can be stored, and the base of the exponential storage capacity.

Original: The following theorem gives this result.
Revised: The following theorem gives this result.

Original: Theorem A6 (Storage Capacity (d computed): Random Patterns) .
Revised: Theorem A6 (Storage Capacity (d computed): Random Patterns) .

Original: We assume a failure probability 0 < p (cid:54) 1 and randomly chosen patterns on the sphere with radius M = K √ d − 1 .
Revised: We assume a failure probability 0 < p (cid:54) 1 and randomly chosen patterns on the sphere with radius M = K √ d − 1 .

Original: We deﬁne
Revised: We deﬁne

Original: [Equation]
Revised: [Equation]

Original: where W is the Lambert W function (Olver et al., 2010, (4.13)).
Revised: where W is the Lambert W function (Olver et al., 2010, (4.13)).

Original: For 0 < a the function W is the upper branch W 0 and for a < 0 we use the lower branch W − 1 .
Revised: For 0 < a the function W is the upper branch W 0 and for a < 0 we use the lower branch W − 1 .

Original: If we ensure that
Revised: If we ensure that

Original: [Equation]
Revised: [Equation]

Original: then with probability 1 − p , the number of random patterns that can be stored is
Revised: then with probability 1 − p , the number of random patterns that can be stored is

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Proof.
Revised: Proof.

Original: We consider the probability that the master inequality Eq.
Revised: We consider the probability that the master inequality Eq.

Original: (310) is fulﬁlled:
Revised: (310) is fulﬁlled:

Original: [Equation]
Revised: [Equation]

Original: Using Eq.
Revised: Using Eq.

Original: (328), we have:
Revised: (328), we have:

Original: [Equation]
Revised: [Equation]

Original: Therefore, with probability 1 − p the storage capacity is largest N that fulﬁlls
Revised: Therefore, with probability 1 − p the storage capacity is largest N that fulﬁlls

Original: [Equation]
Revised: [Equation]

Original: This inequality is equivalent to
Revised: This inequality is equivalent to

Original: [Equation]
Revised: [Equation]

Original: We use Eq.
Revised: We use Eq.

Original: (323) to obtain:
Revised: (323) to obtain:

Original: [Equation]
Revised: [Equation]

Original: For Eq.
Revised: For Eq.

Original: (364) to be fulﬁlled, it is sufﬁcient that
Revised: (364) to be fulﬁlled, it is sufﬁcient that

Original: [Equation]
Revised: [Equation]

Original: If we insert the assumption Eq.
Revised: If we insert the assumption Eq.

Original: (359) of the theorem into Eq.
Revised: (359) of the theorem into Eq.

Original: (360), then we obtain N ≥ 2 .
Revised: (360), then we obtain N ≥ 2 .

Original: We now apply the upper bound κ d − 1 / 2 < κ d − 1 < 1 from Eq.
Revised: We now apply the upper bound κ d − 1 / 2 < κ d − 1 < 1 from Eq.

Original: (324) and the upper bound 2 βN (cid:54) 1 β from
Revised: (324) and the upper bound 2 βN (cid:54) 1 β from

Original: N ≥ 2 to inequality Eq.
Revised: N ≥ 2 to inequality Eq.

Original: In the resulting inequality we insert N = √
Revised: In the resulting inequality we insert N = √

Original: pc
Revised: pc

Original: d − 14 to check whether it is fulﬁlled with this special value of N and obtain:
Revised: d − 14 to check whether it is fulﬁlled with this special value of N and obtain:

Original: [Equation]
Revised: [Equation]

Original: Dividing by p , inserting M = K √ d − 1 , and exponentiation of the left and right side by 2 d − 1 gives:
Revised: Dividing by p , inserting M = K √ d − 1 , and exponentiation of the left and right side by 2 d − 1 gives:

Original: [Equation]
Revised: [Equation]

Original: This inequality Eq.
Revised: This inequality Eq.

Original: (368) can be reformulated as:
Revised: (368) can be reformulated as:

Original: [Equation]
Revised: [Equation]

Original: Using
Revised: Using

Original: [Equation]
Revised: [Equation]

Original: we write inequality Eq.
Revised: we write inequality Eq.

Original: (369) as
Revised: (369) as

Original: [Equation]
Revised: [Equation]

Original: We determine the value ˆ d of d which makes the inequality Eq.
Revised: We determine the value ˆ d of d which makes the inequality Eq.

Original: (371) equal to zero.
Revised: (371) equal to zero.

Original: We solve
Revised: We solve

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: for ˆ d
Revised: for ˆ d

Original: For a (cid:54) = 

Revised: For a (cid:54) =

Original: we have
Revised: we have

Revised: [Equation]

Revised: ⇔

Original: [Equation]
Revised: [Equation]

Original: where W is the Lambert W function (see Def. A6).
Revised: where W is the Lambert W function (see Def. A6).

Original: For a > 0 we have to use the upper branch Wof the Lambert W function and for a < 

Revised: For a > 0 we have to use the upper branch Wof the Lambert W function and for a <

Original: we use the lower branch W − 1 of the Lambert W function (Olver et al., 2010, (4.13)).
Revised: we use the lower branch W − 1 of the Lambert W function (Olver et al., 2010, (4.13)).

Original: We have to ensure that − 1 /e
Revised: We have to ensure that − 1 /e

Original: (cid:54) a exp( − b ) for a solution to exist.
Revised: (cid:54) a exp( − b ) for a solution to exist.

Original: For a = 0 we have ˆ d = 1 + exp( − b ) .
Revised: For a = 0 we have ˆ d = 1 + exp( − b ) .

Original: Hence, the solution is
Revised: Hence, the solution is

Original: [Equation]
Revised: [Equation]

Original: Since ˆ d fulﬁlls inequality Eq.
Revised: Since ˆ d fulﬁlls inequality Eq.

Original: (368) and therefore also Eq.
Revised: (368) and therefore also Eq.

Original: (367), we have a lower bound on the storage capacity N :
Revised: (367), we have a lower bound on the storage capacity N :

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Corollary A2.
Revised: Corollary A2.

Original: We assume a failure probability 0 < p (cid:54) 1 and randomly chosen patterns on the sphere with radius M = K √ d − 1 .
Revised: We assume a failure probability 0 < p (cid:54) 1 and randomly chosen patterns on the sphere with radius M = K √ d − 1 .

Original: We deﬁne
Revised: We deﬁne

Original: [Equation]
Revised: [Equation]

Original: and ensure
Revised: and ensure

Original: [Equation]
Revised: [Equation]

Original: then with probability 1 − p , the number of random patterns that can be stored is
Revised: then with probability 1 − p , the number of random patterns that can be stored is

Original: [Equation]
Revised: [Equation]

Original: Setting β
Revised: Setting β

Original: = 1 , K = 3 , c = 2 and p = 0 .
Revised: = 1 , K = 3 , c = 2 and p = 0 .

Original: 001 yields d < 24 .
Revised: 001 yields d < 24 .

Original: Proof.
Revised: Proof.

Original: For a < 0 the Eq.
Revised: For a < 0 the Eq.

Original: (358) from Theorem (A6) can be written as
Revised: (358) from Theorem (A6) can be written as

Original: [Equation]
Revised: [Equation]

Original: From Alzahrani & Salem (2018, Theorem 3.1) we get the following bound on W − 1 :
Revised: From Alzahrani & Salem (2018, Theorem 3.1) we get the following bound on W − 1 :

Original: [Equation]
Revised: [Equation]

Original: for u > 0 .
Revised: for u > 0 .

Original: We apply Eq.
Revised: We apply Eq.

Original: (380) to Eq.
Revised: (380) to Eq.

Original: (379) with u = − ln
Revised: (379) with u = − ln

Original: − a )
Revised: − a )

Original: +
Revised: +

Original: b − 1 .
Revised: b − 1 .

Original: Since a < 0 we get
Revised: Since a < 0 we get

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: • Storage capacity for the expected minimal separation instead of the probability that all patterns can be stored.
Revised: • Storage capacity for the expected minimal separation instead of the probability that all patterns can be stored.

Original: In contrast to the previous paragraph, we want to argue about the storage capacity for the expected minimal separation.
Revised: In contrast to the previous paragraph, we want to argue about the storage capacity for the expected minimal separation.

Original: Therefore, we will use the following bound on the expectation of α min (minimal angle), which gives also a bound on the expected of ∆ min (minimal separation):
Revised: Therefore, we will use the following bound on the expectation of α min (minimal angle), which gives also a bound on the expected of ∆ min (minimal separation):

Original: Lemma A16 (Proposition 3.6 in Brauchart et al.
Revised: Lemma A16 (Proposition 3.6 in Brauchart et al.

Original: We have the following lower bound on the expectation of α min :
Revised: We have the following lower bound on the expectation of α min :

Original: [Equation]
Revised: [Equation]

Original: The bound is valid for all N ≥ 2 and d ≥ 2 .
Revised: The bound is valid for all N ≥ 2 and d ≥ 2 .

Original: Let us start with some preliminary estimates.
Revised: Let us start with some preliminary estimates.

Original: First of all we need some asymptotics for the constant
Revised: First of all we need some asymptotics for the constant

Original: C d − 1 in Eq.
Revised: C d − 1 in Eq.

Original: Lemma A17.
Revised: Lemma A17.

Original: The following estimate holds for d ≥ 2 :
Revised: The following estimate holds for d ≥ 2 :

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The recursion formula for the Gamma function is (Olver et al., 2010, (5.5.1)):
Revised: The recursion formula for the Gamma function is (Olver et al., 2010, (5.5.1)):

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We use Eq.
Revised: We use Eq.

Original: (324) and the fact that d 1 d ≥ 1 for d ≥ 1 to obtain:
Revised: (324) and the fact that d 1 d ≥ 1 for d ≥ 1 to obtain:

Original: [Equation]
Revised: [Equation]

Original: where in the last step we used the elementary inequality exp( x ) ≥ 1 + x , which follows from the mean value theorem.
Revised: where in the last step we used the elementary inequality exp( x ) ≥ 1 + x , which follows from the mean value theorem.

Original: The next theorem states the number of stored patterns for the expected minimal separation.
Revised: The next theorem states the number of stored patterns for the expected minimal separation.

Original: Theorem A7 (Storage Capacity (expected separation): Random Patterns) .
Revised: Theorem A7 (Storage Capacity (expected separation): Random Patterns) .

Original: We assume patterns on the sphere with radius M = K √ d
Revised: We assume patterns on the sphere with radius M = K √ d

Original: −
Revised: −

Original: that are randomly chosen.
Revised: that are randomly chosen.

Original: Then for all values c ≥ 1 for which
Revised: Then for all values c ≥ 1 for which

Original: [Equation]
Revised: [Equation]

Original: holds, the number of stored patterns for the expected minimal separation is at least
Revised: holds, the number of stored patterns for the expected minimal separation is at least

Original: [Equation]
Revised: [Equation]

Original: The inequality Eq.
Revised: The inequality Eq.

Original: (386) is e.g. fulﬁlled with β = 1 , K = 3 , c = 2 and d ≥ 17 .
Revised: (386) is e.g. fulﬁlled with β = 1 , K = 3 , c = 2 and d ≥ 17 .

Original: Proof.
Revised: Proof.

Original: Instead of considering the probability that the master inequality Eq.
Revised: Instead of considering the probability that the master inequality Eq.

Original: (310) is fulﬁlled we now consider whether this inequality is fulﬁlled for the expected minimal distance.
Revised: (310) is fulﬁlled we now consider whether this inequality is fulﬁlled for the expected minimal distance.

Original: We consider the expectation of the minimal distance ∆ min :
Revised: We consider the expectation of the minimal distance ∆ min :

Original: [Equation]
Revised: [Equation]

Original: For this expectation, the master inequality Eq.
Revised: For this expectation, the master inequality Eq.

Original: (310) becomes
Revised: (310) becomes

Original: [Equation]
Revised: [Equation]

Original: We want to ﬁnd the largest N that fulﬁlls this inequality.
Revised: We want to ﬁnd the largest N that fulﬁlls this inequality.

Original: We apply Eq.
Revised: We apply Eq.

Original: (328) and Jensen’s inequality to deduce the following lower bound:
Revised: (328) and Jensen’s inequality to deduce the following lower bound:

Original: [Equation]
Revised: [Equation]

Original: Now we use Eq.
Revised: Now we use Eq.

Original: (382) and Eq.
Revised: (382) and Eq.

Original: (383) to arrive at
Revised: (383) to arrive at

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: for sufﬁciently large d .
Revised: for sufﬁciently large d .

Original: Thus in order to fulﬁll Eq.
Revised: Thus in order to fulﬁll Eq.

Original: (389), it is enough to ﬁnd values that satisfy
Revised: (389), it is enough to ﬁnd values that satisfy

Original: Eq.
Revised: Eq.

Original: A.1.6.2 Convergence after One Update and Small Retrieval Error. The update rule converges (retrieval error is smaller than some (cid:15) ) after one update for well separated patterns, that is, ∆ i is large.
Revised: A.1.6.

Original: Theorem A8 (Convergence After One Update) .
Revised: Theorem A8 (Pattern Retrieval with One Update) .

Original: With query ξ , after one update the distance of the new point f ( ξ ) to the ﬁxed point x ∗ i is exponentially small in the separation ∆ i .
Revised: With query ξ , after one update the distance of the new point f ( ξ ) to the ﬁxed point x ∗ i is exponentially small in the separation ∆ i .

Original: The precise bounds using the Jacobian J = ∂f ( ξ ) ∂ ξ and its value J m in the mean value theorem are:
Revised: The precise bounds using the Jacobian J = ∂f ( ξ ) ∂ ξ and its value J m in the mean value theorem are:

Original: [Equation]
Revised: [Equation]

Original: For given (cid:15) and sufﬁcient large ∆ i , we have (cid:107) f 
Revised: For given (cid:15) and sufﬁcient large ∆ i , we have (cid:107) f ( ξ ) − x ∗

Original: [Equation] − x ∗ i (cid:107) < (cid:15) .
Revised: i (cid:107) < (cid:15) , that is, retrieval with one update.

Original: Proof.
Revised: Proof.

Original: From Eq.
Revised: From Eq.

Original: (179) we have
Revised: (179) we have

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: After every iteration the mapped point f ( ξ ) is closer to the ﬁxed point x ∗ i than the original point x i :
Revised: After every iteration the mapped point f ( ξ ) is closer to the ﬁxed point x ∗ i than the original point x i :

Original: [Equation]
Revised: [Equation]

Original: For given (cid:15) and sufﬁcient large ∆ i , we have (cid:107) f ( ξ ) − x ∗
Revised: For given (cid:15) and sufﬁcient large ∆ i , we have (cid:107) f ( ξ ) − x ∗

Original: i (cid:107) < (cid:15) , since (cid:107) J m (cid:107) 2 foes exponentially fast to zero with increasing ∆ i .
Revised: i (cid:107) < (cid:15) , since (cid:107) J m (cid:107) 2 foes exponentially fast to zero with increasing ∆ i .

Original: We want to estimate how large ∆ i is.
Revised: We want to estimate how large ∆ i is.

Original: For x i we have:
Revised: For x i we have:

Original: [Equation]
Revised: [Equation]

Original: To estimate how large ∆ i is, assume vectors x ∈ R d and y ∈ R d that have as components standard normally distributed values.
Revised: To estimate how large ∆ i is, assume vectors x ∈ R d and y ∈ R d that have as components standard normally distributed values.

Original: The expected value of the separation of two points with normally distributed components is
Revised: The expected value of the separation of two points with normally distributed components is

Original: [Equation]
Revised: [Equation]

Original: The variance of the separation of two points with normally distributed components is
Revised: The variance of the separation of two points with normally distributed components is

Original: Var

Original: [Equation]
Revised: [Equation]

Original: The expected value for the separation of two random vectors gives:
Revised: The expected value for the separation of two random vectors gives:

Original: [Equation]
Revised: [Equation]

Original: For the exponential storage we set M = 2 √
Revised: For the exponential storage we set M = 2 √

Original: d − 1 .
Revised: d − 1 .

Original: We see the Lipschitz constant (cid:107) J m (cid:107) 2 decreases exponentially with the dimension.
Revised: We see the Lipschitz constant (cid:107) J m (cid:107) 2 decreases exponentially with the dimension.

Original: Therefore, (cid:107) f
Revised: Therefore, (cid:107) f

Original: ( ξ ) − x ∗ i (cid:107) is exponentially small after just one update.
Revised: ( ξ ) − x ∗ i (cid:107) is exponentially small after just one update.

Original: Therefore, the ﬁxed point is well retrieved after one update.
Revised: Therefore, the ﬁxed point is well retrieved after one update.

Original: The retrieval error decreases exponentially with the separation ∆ i .
Revised: The retrieval error decreases exponentially with the separation ∆ i .

Original: Theorem A9 (Exponentially Small Retrieval Error) .
Revised: Theorem A9 (Exponentially Small Retrieval Error) .

Original: The retrieval error (cid:107) x i − x∗ i (cid:107) of pattern x
Revised: The retrieval error (cid:107) f ( ξ ) − x i (cid:107) of pattern x

Original: i is bounded by
Revised: i is bounded by

Original: [Equation]
Revised: [Equation]

Original:  and for (cid:107) xi − x ∗
Revised: M (400) and for (cid:107) x

Original: i [Equation]
Revised: i − x ∗

Original:  (cid:54) 1 2 β M by [Equation]
Revised: i (cid:107) (cid:54) 1 2 β M together with (cid:107) x

Original:  x 
Revised: i − ξ (cid:107) (cid:54) 1 2 β M by (cid:107) x i − x ∗ i (cid:107) (cid:54) 2 e ( N

Original: [Equation]
Revised: [Equation]

Original: Proof. We compute the retrieval error which is just (cid:107) x
Revised: M exp(

Original: i − x∗ i (cid:107) .
Revised: [Equation] Proof. We compute the retrieval error which is just (cid:107) f ( ξ ) − x i (cid:107) .

Original: From Lemma A4 we have 
Revised: From Lemma A4 we have (cid:107) x

Revised: i − f ( ξ

Original: [Equation]
Revised: [Equation]

Revised: (cid:15) M , (402)

Original: From Eq.
Revised: From Eq.

Original: (178) we have
Revised: (178) we have

Original: [Equation]
Revised: [Equation]

Original: We use ξ = x ∗ i  and get
Revised: ∗ [Equation] For (cid:107) x i − x ∗ i (cid:107) (cid:54) 1 2 β M and (cid:107) x i − ξ (cid:107) (cid:54) 1 2 β M Eq. (403) gives (cid:15) (cid:54)

Original: [Equation]
Revised: [Equation]

Original: We obtain
Revised: ( N

Original: [Equation]
Revised: [Equation]

Original: For (cid:107) x i − x ∗ i (cid:107) (cid:54) 1 2 β Minequality Eq. (405) gives
Revised: M

Original: [Equation]
Revised: [Equation]

Original: A.1. L EARNING A SSOCIATIONS
Revised: A.1. L EARNING A SSOCIATIONS

Original: We consider three cases of learning associations, i.e. three cases of how sets are associated.
Revised: We consider three cases of learning associations, i.e. three cases of how sets are associated.

Original: (i) Non of the sets is mapped in an associative space.
Revised: (i) Non of the sets is mapped in an associative space.

Original: The raw state pattern r n is the state (query) pattern ξ n , i.e. ξ n = r n , and the raw stored pattern y s is the stored pattern (key), i.e. x s = y s .
Revised: The raw state pattern r n is the state (query) pattern ξ n , i.e. ξ n = r n , and the raw stored pattern y s is the stored pattern (key), i.e. x s = y s .

Original: (ii)
Revised: (ii)

Original: Either one of the sets is mapped to the space of the other set or an association matrix is learned.
Revised: Either one of the sets is mapped to the space of the other set or an association matrix is learned.

Original: (iia)
Revised: (iia)

Original: The state patterns are equal to the raw patterns, i.e. ξ n = r n , and raw stored patterns are mapped via
Revised: The state patterns are equal to the raw patterns, i.e. ξ n = r n , and raw stored patterns are mapped via

Original: W to the space of the state patterns, i.e. x s = W y s .
Revised: W to the space of the state patterns, i.e. x s = W y s .

Original: (iib)
Revised: (iib)

Original: The stored patterns are equal to the raw patterns, i.e. x s = y s , and raw state patterns are mapped via W to the space of the stored patterns, i.e. ξ n = W T r n .
Revised: The stored patterns are equal to the raw patterns, i.e. x s = y s , and raw state patterns are mapped via W to the space of the stored patterns, i.e. ξ n = W T r n .

Original: (iic)
Revised: (iic)

Original: The matrix W is an association matrix.
Revised: The matrix W is an association matrix.

Original: We will compute the derivative of the new state pattern with respect to W , which is valid for all sub-cases (iib)–(iic).
Revised: We will compute the derivative of the new state pattern with respect to W , which is valid for all sub-cases (iib)–(iic).

Original: (iii) Both set of patterns are mapped in a common associative space.
Revised: (iii) Both set of patterns are mapped in a common associative space.

Original: A raw state pattern r n is mapped by W Q to a state pattern (query) ξ n , that is ξ n = W Q r n .
Revised: A raw state pattern r n is mapped by W Q to a state pattern (query) ξ n , that is ξ n = W Q r n .

Original: A raw stored pattern y s is mapped via W K to stored pattern (key)
Revised: A raw stored pattern y s is mapped via W K to stored pattern (key)

Original: x s , that is x s =
Revised: x s , that is x s =

Original: W K y s .
Revised: W K y s .

Original: We will compute the derivative of the new state pattern with respect to both W Q and W K .
Revised: We will compute the derivative of the new state pattern with respect to both W Q and W K .

Original: A.1.
Revised: of

Original: 1 Association of Raw Patterns – No Mapping in an Associative Space.
Revised: Patterns – No Mapping in an Associative Space.

Original: The sets are associated via their raw patterns, i.e. the raw state pattern r n is the state (query) pattern ξ n , i.e. ξ n = r n , and raw stored pattern y s is the stored pattern (key), i.e. x s = y s .
Revised: The sets are associated via their raw patterns, i.e. the raw state pattern r n is the state (query) pattern ξ n , i.e. ξ n = r n , and raw stored pattern y s is the stored pattern (key), i.e. x s = y s .

Original: There is no mapping in an associative space.
Revised: There is no mapping in an associative space.

Original: The update rule is
Revised: The update rule is

Original: [Equation]
Revised: [Equation]

Original:  where we used
Revised: 405) where we used

Original: [Equation]
Revised: [Equation]

Original: The derivative with respect to ξ is
Revised: The derivative with respect to ξ is

Original: [Equation]
Revised: [Equation]

Original: The derivative with respect to X is
Revised: The derivative with respect to X is

Original: [Equation]
Revised: [Equation]

Original: These derivatives allow to apply the chain rule if a Hopﬁeld layer is integrated into a deep neural network.
Revised: These derivatives allow to apply the chain rule if a Hopﬁeld layer is integrated into a deep neural network.

Original: A.1.7.2 Learning an Association Matrix – Only One Set is Mapped in an Associative Space.
Revised: A.1.7.2 Learning an Association Matrix – Only One Set is Mapped in an Associative Space.

Original: Only one of the sets R or Y is mapped in the space of the patterns of the other set.
Revised: Only one of the sets R or Y is mapped in the space of the patterns of the other set.

Original: Case (a): the state patterns are equal to the raw patterns ξ n = r n and raw stored patterns are mapped via W to the space of the state patterns, i.e. x s = W y s .
Revised: Case (a): the state patterns are equal to the raw patterns ξ n = r n and raw stored patterns are mapped via W to the space of the state patterns, i.e. x s = W y s .

Original: Case (b): the stored patterns are equal to the raw patterns x s = y s and raw state patterns are mapped via W to the space of the stored patterns, i.e. ξ n = W T r n .
Revised: Case (b): the stored patterns are equal to the raw patterns x s = y s and raw state patterns are mapped via W to the space of the stored patterns, i.e. ξ n = W T r n .

Original: Case (c): the matrix W associates the sets R and Y .
Revised: Case (c): the matrix W associates the sets R and Y .

Original: This case also includes that
Revised: This case also includes that

Original: W T = W TK W Q , which is treated in next subsection.
Revised: W T = W TK W Q , which is treated in next subsection.

Original: The next subsection focuses on a low rank approximation of W by deﬁning the dimension d k of associative space and use the matrices W TK and W Q to deﬁne W , or equivalently to map R and Y into the associative space.
Revised: The next subsection focuses on a low rank approximation of W by deﬁning the dimension d k of associative space and use the matrices W TK and W Q to deﬁne W , or equivalently to map R and Y into the associative space.

Original: From a mathematical point of view all these case are equal as they lead to the same update rule.
Revised: From a mathematical point of view all these case are equal as they lead to the same update rule.

Original: Therefore, we consider in the following Case (a) with x s = W y s and ξ n = r n .
Revised: Therefore, we consider in the following Case (a) with x s = W y s and ξ n = r n .

Original: Still, the following formula are valid for all three cases (a)–(c).
Revised: Still, the following formula are valid for all three cases (a)–(c).

Original: The update rule is
Revised: The update rule is

Original: [Equation]
Revised: [Equation]

Original: where we used
Revised: where we used

Original: [Equation]
Revised: [Equation]

Original: We consider the state (query) pattern ξ with result ξ new :
Revised: We consider the state (query) pattern ξ with result ξ new :

Original: [Equation]
Revised: [Equation]

Original: For multiple updates this update rule has to be used.
Revised: For multiple updates this update rule has to be used.

Original: However for a single update, or the last update we consider a simpliﬁed update rule.
Revised: However for a single update, or the last update we consider a simpliﬁed update rule.

Original: Since new state vector ξ new is projected by a weight matrix W V to another vector, we consider the simpliﬁed update rule:
Revised: Since new state vector ξ new is projected by a weight matrix W V to another vector, we consider the simpliﬁed update rule:

Original: [Equation]
Revised: [Equation]

Original: The derivative with respect to W is
Revised: The derivative with respect to W is

Original: [Equation]
Revised: [Equation]

Original: We have the product of the 3-dimensional tensor ∂
Revised: We have the product of the 3-dimensional tensor ∂

Original: ( W T ξ ) ∂ W with the vector a which gives a 2dimensional tensor, i.e. a matrix: 
Revised: ( W T ξ ) ∂ W with the vector a which gives a 2dimensional tensor, i.e. a matrix: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where J is the Jacobian of the update rule deﬁned in Eq.
Revised: where J is the Jacobian of the update rule deﬁned in Eq.

Original: To obtain the derivative of the full update rule Eq.
Revised: To obtain the derivative of the full update rule Eq.

Original: (413) we have to add the term a p T Y T (420) and include the factor W to get
Revised: (411) we have to add the term a p T Y T [Equation] [Equation] and include the factor W to get

Original: [Equation]
Revised: [Equation]

Original: A.1.7.3 Learning Two Association Mappings – Both Sets are Mapped in an Associative Space.
Revised: A.1.7.3 Learning Two Association Mappings – Both Sets are Mapped in an Associative Space.

Original: Both sets R and Y are mapped in an associative space.
Revised: Both sets R and Y are mapped in an associative space.

Original: Every raw state pattern r n is mapped via
Revised: Every raw state pattern r n is mapped via

Original: W Q to a state pattern (query) ξ n = W Q r n .
Revised: W Q to a state pattern (query) ξ n = W Q r n .

Original: Every raw stored pattern y s is mapped via W K to a stored pattern (key) x s = W K y s .
Revised: Every raw stored pattern y s is mapped via W K to a stored pattern (key) x s = W K y s .

Original: In the last subsection we considered a single matrix W .
Revised: In the last subsection we considered a single matrix W .

Original: For
Revised: For

Original: W T = W TK W Q we have the case of the last subsection.
Revised: W T = W TK W Q we have the case of the last subsection.

Original: However in this subsection we are looking for a low rank approximation of W .
Revised: However in this subsection we are looking for a low rank approximation of W .

Original: Toward this end we deﬁne the dimension d k of associative space and use the matrices W TK and W Q to map to the associative space.
Revised: Toward this end we deﬁne the dimension d k of associative space and use the matrices W TK and W Q to map to the associative space.

Original: The update rule is
Revised: The update rule is

Original: [Equation]
Revised: [Equation]

Original: where we used
Revised: where we used

Original: [Equation]
Revised: [Equation]

Original: We consider raw state patterns r n that are mapped to state patterns ξ n = W Q r n with Q T =
Revised: We consider raw state patterns r n that are mapped to state patterns ξ n = W Q r n with Q T =

Original: Ξ = W Q R and raw stored pattern y s that are mapped to stored patterns x s =
Revised: Ξ = W Q R and raw stored pattern y s that are mapped to stored patterns x s =

Original: W K y s with
Revised: W K y s with

Original: K T = X =
Revised: K T = X =

Original: W K Y .
Revised: W K Y .

Original: The update rule is
Revised: The update rule is

Original: [Equation]
Revised: [Equation]

Original: Since new state vector ξ new is projected by a weight matrix W V to another vector, we consider the simpliﬁed update rule:
Revised: Since new state vector ξ new is projected by a weight matrix W V to another vector, we consider the simpliﬁed update rule:

Original: [Equation]
Revised: [Equation]

Original: For the simpliﬁed update rule, the vector ξ new does not live in the associative space but in the space of raw stored pattern y .
Revised: For the simpliﬁed update rule, the vector ξ new does not live in the associative space but in the space of raw stored pattern y .

Original: However W K would map it to the associative space.
Revised: However W K would map it to the associative space.

Original: • Derivative with respect to W Q .
Revised: • Derivative with respect to W Q .

Original: The derivative with respect to W Q is
Revised: The derivative with respect to W Q is

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: We have the product of the 3-dimensional tensor ∂
Revised: We have the product of the 3-dimensional tensor ∂

Original: ( W Q r ) ∂ W Q with the vector a which gives a 2dimensional tensor, i.e. a matrix:
Revised: ( W Q r ) ∂ W Q with the vector a which gives a 2dimensional tensor, i.e. a matrix:

Original: [Equation]
Revised: [Equation]

Original: where J is the Jacobian of the update rule deﬁned in Eq.
Revised: where J is the Jacobian of the update rule deﬁned in Eq.

Original: To obtain the derivative of the full update rule Eq.
Revised: To obtain the derivative of the full update rule Eq.

Original: (424) we have to include the factor W K , then get ∂ a T ξ new
Revised: (422) we have to include the factor W K , then get ∂ a T ξ new

Original: [Equation]
Revised: [Equation]

Original: • Derivative with respect toW K .
Revised: • Derivative with respect to

Original:  The derivative with respect to W K is
Revised: W K The derivative with respect to W K is

Original: [Equation]
Revised: [Equation]

Revised: ξ new

Original: [Equation]
Revised: [Equation]

Original:  We have the product of the 3-dimensional tensor ∂ ( Wr ) ∂ W K with the vector a which gives a 2-dimensional tensor, i.e. a matrix:
Revised: [Equation] We have the product of the 3-dimensional tensor ∂ ( Wr ) ∂ W K with the vector a which gives a 2-dimensional tensor, i.e. a matrix:

Original: [Equation]
Revised: [Equation]

Original: where J is the Jacobian of the update rule deﬁned in Eq.
Revised: where J is the Jacobian of the update rule deﬁned in Eq.

Original: To obtain the derivative of the full update rule Eq.
Revised: To obtain the derivative of the full update rule Eq.

Original: (424) we have to add the term
Revised: (422) we have to add the term

Original: [Equation]
Revised: [Equation]

Original: and to include the factor W K , then get
Revised: and to include the factor W K , then get

Original: [Equation]
Revised: [Equation]

Original: A.1. I NFINITE M ANY P ATTERNS AND F ORGETTING P ATTERNS
Revised: A.1. I NFINITE M ANY P ATTERNS AND F ORGETTING P ATTERNS

Original: In the next subsection we show how the new Hopﬁeld networks can be used for auto-regressive tasks by causal masking.
Revised: In the next subsection we show how the new Hopﬁeld networks can be used for auto-regressive tasks by causal masking.

Original: In the following subsection, we introduce forgetting to the new Hopﬁeld networks by adding a negative value to the softmax which is larger if the pattern was observed more in the past.
Revised: In the following subsection, we introduce forgetting to the new Hopﬁeld networks by adding a negative value to the softmax which is larger if the pattern was observed more in the past.

Original: A.1.8.1 Inﬁnite Many Patterns.
Revised: A.1.8.1 Inﬁnite Many Patterns.

Original: The new Hopﬁeld networks can be used for auto-regressive tasks, that is time series prediction and similar.
Revised: The new Hopﬁeld networks can be used for auto-regressive tasks, that is time series prediction and similar.

Original: Causal masking masks out the future by a large negative value in the softmax.
Revised: Causal masking masks out the future by a large negative value in the softmax.

Original: We assume to have inﬁnite many stored patterns (keys) x 1 , x 2 , . . .
Revised: We assume to have inﬁnite many stored patterns (keys) x 1 , x 2 , . . .

Original: that are represented by the inﬁnite matrix
Revised: that are represented by the inﬁnite matrix

Original: [Equation]
Revised: [Equation]

Original: The pattern index is now a time index, that is, we observe x t at time t .
Revised: The pattern index is now a time index, that is, we observe x t at time t .

Original: The pattern matrix at time t is
Revised: The pattern matrix at time t is

Original: [Equation]
Revised: [Equation]

Original: The query at time t is ξ t .
Revised: The query at time t is ξ t .

Original: For M t = max 1 (cid:54)
Revised: For M t = max 1 (cid:54)

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: , the energy function at time t is E t
Revised: , the energy function at time t is E t

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The update rule is
Revised: The update rule is

Original: [Equation]
Revised: [Equation]

Original: where we used
Revised: where we used

Original: [Equation]
Revised: [Equation]

Original: We can use an inﬁnite pattern matrix with an inﬁnite softmax when using causal masking.
Revised: We can use an inﬁnite pattern matrix with an inﬁnite softmax when using causal masking.

Original: The pattern matrix at time t is
Revised: The pattern matrix at time t is

Original: [Equation]
Revised: [Equation]

Original: with the query ξ t and α → ∞ .
Revised: with the query ξ t and α → ∞ .

Original: The energy function at time t is E t
Revised: The energy function at time t is E t

Original: [Equation]
Revised: [Equation]

Original: For α → ∞ and (cid:107) ξ t (cid:107) >
Revised: For α → ∞ and (cid:107) ξ t (cid:107) >

Original: this becomes
Revised: this becomes

Original: [Equation]
Revised: [Equation]

Original: A.1.8.2 Forgetting Patterns.
Revised: A.1.8.2 Forgetting Patterns.

Original: We introduce forgetting to the new Hopﬁeld networks by adding a negative value in the softmax which increases with patterns that are more in the past.
Revised: We introduce forgetting to the new Hopﬁeld networks by adding a negative value in the softmax which increases with patterns that are more in the past.

Original: We assume to have inﬁnite many patterns x 1 , x 2 , . . .
Revised: We assume to have inﬁnite many patterns x 1 , x 2 , . . .

Original: that are represented by the inﬁnite matrix
Revised: that are represented by the inﬁnite matrix

Original: [Equation]
Revised: [Equation]

Original: The pattern index is now a time index, that is, we observe x t at time t .
Revised: The pattern index is now a time index, that is, we observe x t at time t .

Original: The pattern matrix at time t is
Revised: The pattern matrix at time t is

Original: [Equation]
Revised: [Equation]

Original: The query at time t is ξ t .
Revised: The query at time t is ξ t .

Original: The energy function with forgetting parameter γ at time t is E t
Revised: The energy function with forgetting parameter γ at time t is E t

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: The update rule is
Revised: The update rule is

Original: [Equation]
Revised: [Equation]

Original: where we used
Revised: where we used

Original: [Equation]
Revised: [Equation]

Original: A.1. N UMBER OF S PURIOUS S TATES
Revised: A.1. N UMBER OF S PURIOUS S TATES

Original: The energy E is deﬁned as
Revised: The energy E is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Since the negative exponential function is strict monotonic decreasing, exp( − E) has minima, where
Revised: Since the negative exponential function is strict monotonic decreasing, exp( − E) has minima, where

Original: E has maxima, and has maxima, where as has minima E .
Revised: E has maxima, and has maxima, where as has minima E .

Original: [Equation]
Revised: [Equation]

Original: where C is a positive constant, λ ( x i , β ) =
Revised: where C is a positive constant, λ ( x i , β ) =

Original: exp( 12 β x Ti x i ) and G ( ξ ; x i , β − 1 I ) is the Gaussian with mean x i and covariance matrix β − 1 I .
Revised: exp( 12 β x Ti x i ) and G ( ξ ; x i , β − 1 I ) is the Gaussian with mean x i and covariance matrix β − 1 I .

Original: Since C is a positive constant and x β − 1 = exp( β − 1 ln x ) is strict monotonic for positive x , the minima of E are the maxima of
Revised: Since C is a positive constant and x β − 1 = exp( β − 1 ln x ) is strict monotonic for positive x , the minima of E are the maxima of

Original: [Equation]
Revised: [Equation]

Original: In Carreira-Perpiñán & Williams (2003) it was shown that Eq.
Revised: In Carreira-Perpiñán & Williams (2003) it was shown that Eq.

Original: (459) can have more than N modes, that is, more than N maxima.
Revised: (457) can have more than N modes, that is, more than N maxima.

Original: A. P ROPERTIES OF S OFTMAX , L OG -S UM -E XPONENTIAL , L EGENDRE T RANSFORM , L AMBERT W F UNCTION
Revised: A. P ROPERTIES OF S OFTMAX , L OG -S UM -E XPONENTIAL , L EGENDRE T RANSFORM , L AMBERT W F UNCTION

Original: For β > 0 , the softmax is deﬁned as
Revised: For β > 0 , the softmax is deﬁned as

Original: Deﬁnition A1 (Softmax) .
Revised: Deﬁnition A1 (Softmax) .

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We also need the log-sum-exp function ( lse ), deﬁned as
Revised: We also need the log-sum-exp function ( lse ), deﬁned as

Original: Deﬁnition A2 (Log-Sum-Exp Function) .
Revised: Deﬁnition A2 (Log-Sum-Exp Function) .

Original: [Equation]
Revised: [Equation]

Original: We can formulate the lse in another base:
Revised: We can formulate the lse in another base:

Original: [Equation]
Revised: [Equation]

Original: In particular, the base a = 2 can be used to speed up computations.
Revised: In particular, the base a = 2 can be used to speed up computations.

Original: Next, we give the relation between the softmax and the lse function.
Revised: Next, we give the relation between the softmax and the lse function.

Original: Lemma A18.
Revised: Lemma A18.

Original: The softmax is the gradient of the lse :
Revised: The softmax is the gradient of the lse :

Original: [Equation]
Revised: [Equation]

Original: In the next lemma we report some important properties of the lse function.
Revised: In the next lemma we report some important properties of the lse function.

Original: Lemma A19.
Revised: Lemma A19.

Original: We deﬁne
Revised: We deﬁne

Original: [Equation]
Revised: [Equation]

Original: with L ≥ p T x .
Revised: with L ≥ p T x .

Original: The lse is the maximum of L on the N -dimensional simplex D with D = { z | (cid:80) i z i
Revised: The lse is the maximum of L on the N -dimensional simplex D with D = { z | (cid:80) i z i

Original: = 1 , 0 (cid:54) z i } :
Revised: = 1 , 0 (cid:54) z i } :

Original: [Equation]
Revised: [Equation]

Original: The softmax p = softmax( β x ) is the argument of the maximum of L on the N -dimensional simplex
Revised: The softmax p = softmax( β x ) is the argument of the maximum of L on the N -dimensional simplex

Original: D with D = { z | (cid:80) i
Revised: D with D = { z | (cid:80) i

Original: z i = 1 , 0 (cid:54) z i } :
Revised: z i = 1 , 0 (cid:54) z i } :

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Eq.
Revised: Eq.

Original: (467) is obtained from Equation (8) in Gao & Pavel (2017) and Eq.
Revised: (465) is obtained from Equation (8) in Gao & Pavel (2017) and Eq.

Original: (468) from Equation (11) in Gao & Pavel (2017).
Revised: (466) from Equation (11) in Gao & Pavel (2017).

Original: From a physical point of view, the lse function represents the “free energy” in statistical thermodynamics (Gao & Pavel, 2017).
Revised: From a physical point of view, the lse function represents the “free energy” in statistical thermodynamics (Gao & Pavel, 2017).

Original: Next we consider the Jacobian of the softmax and its properties.
Revised: Next we consider the Jacobian of the softmax and its properties.

Original: Lemma A20.
Revised: Lemma A20.

Original: The Jacobian J s of the softmax p = softmax( β x ) is
Revised: The Jacobian J s of the softmax p = softmax( β x ) is

Original: [Equation]
Revised: [Equation]

Original: which gives the elements
Revised: which gives the elements

Original: [Equation]
Revised: [Equation]

Original: Next we show that J s has eigenvalue 0 .
Revised: Next we show that J s has eigenvalue 0 .

Original: Lemma A21.
Revised: Lemma A21.

Original: The Jacobian J s of the softmax function p = softmax( β x ) has a zero eigenvalue with eigenvector 1 .
Revised: The Jacobian J s of the softmax function p = softmax( β x ) has a zero eigenvalue with eigenvector 1 .

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: Next we show that 0 is the smallest eigenvalue of J s , therefore J s is positive semi-deﬁnite but not (strict) positive deﬁnite.
Revised: Next we show that 0 is the smallest eigenvalue of J s , therefore J s is positive semi-deﬁnite but not (strict) positive deﬁnite.

Original: Lemma A22.
Revised: Lemma A22.

Original: The Jacobian J s of the softmax p = softmax( β ξ ) is symmetric and positive semideﬁnite.
Revised: The Jacobian J s of the softmax p = softmax( β ξ ) is symmetric and positive semideﬁnite.

Original: Proof.
Revised: Proof.

Original: For an arbitrary z , we have
Revised: For an arbitrary z , we have

Original: [Equation]
Revised: [Equation]

Original: The last inequality hold true because the Cauchy-Schwarz inequality says ( a T a )( b T b ) ≥ ( a T b ) 2 , which is the last inequality with a i
Revised: The last inequality hold true because the Cauchy-Schwarz inequality says ( a T a )( b T b ) ≥ ( a T b ) 2 , which is the last inequality with a i

Original: =
Revised: =

Original: z i √ p i and b i = √ p i .
Revised: z i √ p i and b i = √ p i .

Original: Consequently (cid:0) diag( p ) − pp T (cid:1) is positive semi-deﬁnite.
Revised: Consequently (cid:0) diag( p ) − pp T (cid:1) is positive semi-deﬁnite.

Original: Alternatively (cid:80) i p i
Revised: Alternatively (cid:80) i p i

Original: z
Revised: z

Original: 2 i − ( (cid:80) i p i
Revised: 2 i − ( (cid:80) i p i

Original: z i ) 2 can be viewed as the expected second moment minus the mean squared which gives the variance that is larger equal to zero.
Revised: z i ) 2 can be viewed as the expected second moment minus the mean squared which gives the variance that is larger equal to zero.

Original: The Jacobian is 0 < β times a positive semi-deﬁnite matrix, which is a positive semi-deﬁnite matrix.
Revised: The Jacobian is 0 < β times a positive semi-deﬁnite matrix, which is a positive semi-deﬁnite matrix.

Original: Moreover, the softmax is a monotonic map, as described in the next lemma.
Revised: Moreover, the softmax is a monotonic map, as described in the next lemma.

Original: Lemma A23.
Revised: Lemma A23.

Original: The softmax p = softmax( β x ) is monotone, that is,
Revised: The softmax p = softmax( β x ) is monotone, that is,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We use the version of mean value theorem Lemma A32 with the symmetric matrix J ms = (cid:82)
Revised: We use the version of mean value theorem Lemma A32 with the symmetric matrix J ms = (cid:82)

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: since J ms is positive semi-deﬁnite.
Revised: since J ms is positive semi-deﬁnite.

Original: For all λ the Jacobians J s ( λ x + (1 − λ ) x (cid:48) ) are positive semi-deﬁnite according to Lemma A22.
Revised: For all λ the Jacobians J s ( λ x + (1 − λ ) x (cid:48) ) are positive semi-deﬁnite according to Lemma A22.

Original: Since
Revised: Since

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: is an integral over positive values for every x , J ms is positive semi-deﬁnite, too.
Revised: is an integral over positive values for every x , J ms is positive semi-deﬁnite, too.

Original: Next we give upper bounds on the norm of J s .
Revised: Next we give upper bounds on the norm of J s .

Original: Lemma A24.
Revised: Lemma A24.

Original: For a softmax p = softmax( β x ) with m = max i p i (1 − p i ) , the spectral norm of the
Revised: For a softmax p = softmax( β x ) with m = max i p i (1 − p i ) , the spectral norm of the

Original: Jacobian J s of the softmax is bounded:
Revised: Jacobian J s of the softmax is bounded:

Original: [Equation]
Revised: [Equation]

Original: [Equation][Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: In particular everywhere holds
Revised: In particular everywhere holds

Original: [Equation]
Revised: [Equation]

Original: If p max = max i p i ≥ 1 − (cid:15) ≥ 0 .
Revised: If p max = max i p i ≥ 1 − (cid:15) ≥ 0 .

Original: 5 , then for the spectral norm of the Jacobian holds
Revised: 5 , then for the spectral norm of the Jacobian holds

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We consider the maximum absolute column sum norm
Revised: We consider the maximum absolute column sum norm

Original: [Equation]
Revised: [Equation]

Original: and the maximum absolute row sum norm
Revised: and the maximum absolute row sum norm

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: We have for A
Revised: We have for A

Original: =
Revised: =

Original: J s = β (cid:0) diag( p ) − pp T (cid:1) (cid:88)j | a [Equation]   p
Revised: J s = β (cid:0) diag( p ) − pp T (cid:1) (cid:88)

Original: [Equation]

Original: [Equation] p [Equation] + (cid:88)
Revised: j | a ij | = β   p i (1 − p i ) + (cid:88)

Original: j
Revised: j

Original: [Equation]
Revised: [Equation]

Original: j (cid:54) =
Revised: j (cid:54) =

Original: i p i p j 
Revised: i p i p j  

Original: [Equation] 
Revised: [Equation] [Equation]

Revised: p

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: (cid:88)

Original: j p [Equation] [Equation]
Revised: j p j ) (482) = 2 β p i (

Original: [Equation]
Revised: [Equation]

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Original:  (cid:54)
Revised: ) (cid:54)

Original: [Equation]
Revised: [Equation]

Original: (cid:88) i |

Original: a
Revised: m

Original: [Equation]
Revised: [Equation]

Original:  |
Revised: , (cid:88) i | a ij |

Original: [Equation]
Revised: [Equation]

Original:  
Revised:  

Revised: [Equation]

Revised: [Equation]

Revised: p

Original: [Equation]
Revised: [Equation]

Original:  (cid:88)
Revised: + (cid:88)

Revised: i

Original: [Equation]
Revised: [Equation]

Revised: i (cid:54) =

Original:  p 
Revised: j p j p [Equation]  

Original: [Equation]
Revised: [Equation]

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: p 
Revised: p j

Original: [Equation]
Revised: [Equation]

Original: (cid:88) i p 
Revised: (cid:88) i p i

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  p
Revised: = 2 β p

Original: [Equation]
Revised: [Equation]

Original: (cid:54)
Revised: (1 − p

Original: [Equation]
Revised: [Equation]

Original:  m β .
Revised: ) (cid:54) 2 m β .

Original: Therefore, we have (cid:107)
Revised: Therefore, we have (cid:107)

Original: [Equation]
Revised: [Equation]

Original: (486) (cid:107)
Revised: (cid:54)

Original: [Equation]
Revised: [Equation]

Original: (cid:107)
Revised: m

Original: [Equation]
Revised: [Equation]

Original: (cid:54)
Revised: (484) (cid:107)

Original: [Equation]
Revised: [Equation]

Original: m
Revised: (cid:107) ∞ (cid:54)

Original: [Equation]
Revised: [Equation]

Revised: m

Revised: [Equation]

Original: (487) (cid:107)
Revised: , (485) (cid:107)

Original: [Equation]
Revised: [Equation]

Original: The last inequality is a direct consequence of Hölder’s inequality.
Revised: The last inequality is a direct consequence of Hölder’s inequality.

Original: For 0 (cid:54) p i (cid:54) 1 , we have p i (1 − p i ) (cid:54) 0 .
Revised: For 0 (cid:54) p i (cid:54) 1 , we have p i (1 − p i ) (cid:54) 0 .

Original: Therefore, m (cid:54) 0 .
Revised: Therefore, m (cid:54) 0 .

Original: 25 for all values of p i .
Revised: 25 for all values of p i .

Original: If p max ≥ 1 − (cid:15) ≥ 0 .
Revised: If p max ≥ 1 − (cid:15) ≥ 0 .

Original: 5 ( (cid:15) (cid:54) 0 . 5 ), then 1 − p max (cid:54) (cid:15) and for p
Revised: 5 ( (cid:15) (cid:54) 0 . 5 ), then 1 − p max (cid:54) (cid:15) and for p

Original: i (cid:54) =
Revised: i (cid:54) =

Original: p max p i (cid:54) (cid:15) .
Revised: p max p i (cid:54) (cid:15) .

Original: The derivative ∂x (1 − x ) /∂x
Revised: The derivative ∂x (1 − x ) /∂x

Original: = 1 − 2 x > 0 for x < 0 .
Revised: = 1 − 2 x > 0 for x < 0 .

Original: 5 , therefore x (1 − x ) increases with x for x < 0 .
Revised: 5 , therefore x (1 − x ) increases with x for x < 0 .

Original: Using x = 1 − p max and for p
Revised: Using x = 1 − p max and for p

Original: i (cid:54) =
Revised: i (cid:54) =

Original: p max x = p
Revised: p max x = p

Original: i , we obtain p
Revised: i , we obtain p

Original: i (1 − p i ) (cid:54) (cid:15) (1 − (cid:15) ) for all i .
Revised: i (1 − p i ) (cid:54) (cid:15) (1 − (cid:15) ) for all i .

Original: Consequently, we have m (cid:54) (cid:15) (1 − (cid:15) ) .
Revised: Consequently, we have m (cid:54) (cid:15) (1 − (cid:15) ) .

Original: Using the bounds on the norm of the Jacobian, we give some Lipschitz properties of the softmax function.
Revised: Using the bounds on the norm of the Jacobian, we give some Lipschitz properties of the softmax function.

Original: Lemma A25.
Revised: Lemma A25.

Original: The softmax function p = softmax( β x ) is ( β/ 2) -Lipschitz.
Revised: The softmax function p = softmax( β x ) is ( β/ 2) -Lipschitz.

Original: The softmax function p = softmax( β x ) is (2 βm ) -Lipschitz in a convex environment U for which m = max x ∈ U max i p
Revised: The softmax function p = softmax( β x ) is (2 βm ) -Lipschitz in a convex environment U for which m = max x ∈ U max i p

Original: i (1 − p i ) .
Revised: i (1 − p i ) .

Original: For p max = min x ∈ U max i p i = 1 − (cid:15) , the softmax function p = softmax( β x ) is (2 β(cid:15) )
Revised: For p max = min x ∈ U max i p i = 1 − (cid:15) , the softmax function p = softmax( β x ) is (2 β(cid:15) )

Original: -Lipschitz.
Revised: -Lipschitz.

Original: For β < 2 m , the softmax p = softmax( β x ) is contractive in U on which m is deﬁned.
Revised: For β < 2 m , the softmax p = softmax( β x ) is contractive in U on which m is deﬁned.

Original: Proof
Revised: Proof

Original: [Equation]
Revised: [Equation]

Original: According to Lemma A24 for all ˜ x = λ x + (1 − λ ) x (cid:48) )
Revised: According to Lemma A24 for all ˜ x = λ x + (1 − λ ) x (cid:48) )

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where ˜ m = max i ˜ p
Revised: where ˜ m = max i ˜ p

Original: i (1 − ˜ p i ) .
Revised: i (1 − ˜ p i ) .

Original: Since x ∈ U and x (cid:48) ∈ U we have ˜ x ∈ U , since U is convex.
Revised: Since x ∈ U and x (cid:48) ∈ U we have ˜ x ∈ U , since U is convex.

Original: For m = max x ∈ U max i p
Revised: For m = max x ∈ U max i p

Original: i (1 − p i )
Revised: i (1 − p i )

Original: we have ˜ m
Revised: we have ˜ m

Original: (cid:54) m for all ˜ m .
Revised: (cid:54) m for all ˜ m .

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: which also holds for the mean:
Revised: which also holds for the mean:

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: From Lemma A24 we know m (cid:54) 1 / 4 globally.
Revised: From Lemma A24 we know m (cid:54) 1 / 4 globally.

Original: For p max = min x ∈ U max i p i = 1 − (cid:15) we have according to Lemma A24: m (cid:54) (cid:15) .
Revised: For p max = min x ∈ U max i p i = 1 − (cid:15) we have according to Lemma A24: m (cid:54) (cid:15) .

Original: For completeness we present a result about cocoercivity of the softmax:
Revised: For completeness we present a result about cocoercivity of the softmax:

Original: Lemma A26.
Revised: Lemma A26.

Original: For m = max x ∈ U max i p
Revised: For m = max x ∈ U max i p

Original: i (1 − p i ) , softmax function p = softmax( β x ) is 1 / (2 mβ ) - cocoercive in U , that is,
Revised: i (1 − p i ) , softmax function p = softmax( β x ) is 1 / (2 mβ ) - cocoercive in U , that is,

Original: [Equation]
Revised: [Equation]

Original: In particular the softmax function p = softmax( β x ) is (2 /β )
Revised: In particular the softmax function p = softmax( β x ) is (2 /β )

Original: -cocoercive everywhere.
Revised: -cocoercive everywhere.

Original: With p max = min x ∈ U max i p i = 1 − (cid:15) , the softmax function p = softmax( β x ) is 1 / (2 β(cid:15) )
Revised: With p max = min x ∈ U max i p i = 1 − (cid:15) , the softmax function p = softmax( β x ) is 1 / (2 β(cid:15) )

Original: -cocoercive in U .
Revised: -cocoercive in U .

Original: Proof.
Revised: Proof.

Original: We apply the Baillon-Haddad theorem (e.g. Theorem 1 in Gao & Pavel (2017)) together with
Revised: We apply the Baillon-Haddad theorem (e.g. Theorem 1 in Gao & Pavel (2017)) together with

Original: Lemma A25.
Revised: Lemma A25.

Original: Finally, we introduce the Legendre transform and use it to describe further properties of the lse .
Revised: Finally, we introduce the Legendre transform and use it to describe further properties of the lse .

Original: We start with the deﬁnition of the convex conjugate.
Revised: We start with the deﬁnition of the convex conjugate.

Original: Deﬁnition A3 (Convex Conjugate) .
Revised: Deﬁnition A3 (Convex Conjugate) .

Original: The Convex Conjugate (Legendre-Fenchel transform) of a function f from a Hilbert Space X to [ −∞ , ∞ ] is f ∗ which is deﬁned as
Revised: The Convex Conjugate (Legendre-Fenchel transform) of a function f from a Hilbert Space X to [ −∞ , ∞ ] is f ∗ which is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: See page 219 Def.
Revised: See page 219 Def.

Original: 13.1 in Bauschke & Combettes (2017) and page 134 in Garling (2017).
Revised: 13.1 in Bauschke & Combettes (2017) and page 134 in Garling (2017).

Original: Next we deﬁne the Legendre transform, which is a more restrictive version of the convex conjugate.
Revised: Next we deﬁne the Legendre transform, which is a more restrictive version of the convex conjugate.

Original: Deﬁnition A4 (Legendre Transform) .
Revised: Deﬁnition A4 (Legendre Transform) .

Original: The Legendre transform of a convex function f from a convex set X ⊂ R n to R ( f : X → R ) is f ∗ , which is deﬁned as
Revised: The Legendre transform of a convex function f from a convex set X ⊂ R n to R ( f : X → R ) is f ∗ , which is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: See page 91 in Boyd & Vandenberghe (2009).
Revised: See page 91 in Boyd & Vandenberghe (2009).

Original: Deﬁnition A

Revised: Deﬁnition A

Original: (Epi-Sum) .
Revised: (Epi-Sum) .

Original: Let f and g be two functions from X to ( −∞ , ∞ ] , then the inﬁmal convolution (or epi-sum) of f and g is [Equation] (cid:3) g [Equation]
Revised: Let f and g be two functions from X to ( −∞ , ∞ ] , then the inﬁmal convolution (or epi-sum) of f and g is f (cid:3) g : X → [ −∞ , ∞ ] , x (cid:55)→ inf y ∈ X ( f ( y )

Original: X →

Original: [Equation]

Original: , ∞ ] , x (cid:55)→ inf

Original: [Equation]

Original:  y ))[Equation]
Revised: + g ( x − y ))

Original: See Def.
Revised: See Def.

Original: 12.1 in Bauschke & Combettes (2017).
Revised: 12.1 in Bauschke & Combettes (2017).

Original: Lemma A27.
Revised: Lemma A27.

Original: Let f and g be functions from X to ( −∞ , ∞ ] .
Revised: Let f and g be functions from X to ( −∞ , ∞ ] .

Original: Then the following hold:
Revised: Then the following hold:

Original: Convex Conjugate of norm squared
Revised: Convex Conjugate of norm squared

Original: [Equation]
Revised: [Equation]

Original: Convex Conjugate of a function multiplied by scalar 0 < α ∈ R
Revised: Convex Conjugate of a function multiplied by scalar 0 < α ∈ R

Original: [Equation]
Revised: [Equation]

Original: Convex Conjugate of the sum of a function and a scalar
Revised: Convex Conjugate of the sum of a function and a scalar

Original: [Equation]
Revised: [Equation]

Original: Convex Conjugate of afﬁne transformation of the arguments.
Revised: Convex Conjugate of afﬁne transformation of the arguments.

Original: Let A be a non-singular matrix and b a vector
Revised: Let A be a non-singular matrix and b a vector

Original: [Equation]
Revised: [Equation]

Original: Convex Conjugate of epi-sums
Revised: Convex Conjugate of epi-sums

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Since h ( t ) :
Revised: Since h ( t ) :

Original: = t 2 2 is a non-negative convex function and h ( t ) = 

Revised: = t 2 2 is a non-negative convex function and h ( t ) =

Original: ⇐⇒ t = 

Revised: ⇐⇒ t =

Original: we have because of Proposition 11.3.3 in Garling (2017)
Revised: we have because of Proposition 11.3.3 in Garling (2017)

Original: that h ( (cid:107) x
Revised: that h ( (cid:107) x

Original: (cid:107) ) ∗
Revised: (cid:107) ) ∗

Original: =
Revised: =

Original: h ∗ ( (cid:107) x ∗ (cid:107) ) .
Revised: h ∗ ( (cid:107) x ∗ (cid:107) ) .

Original: Additionally, by example (a) on page 137 we get for 1 < p < ∞ and 1 p + 1 q = 1 that (cid:16) | t | p p (cid:17) ∗ = | t ∗ | q q .
Revised: Additionally, by example (a) on page 137 we get for 1 < p < ∞ and 1 p + 1 q = 1 that (cid:16) | t | p p (cid:17) ∗ = | t ∗ | q q .

Original: Putting all together we get the desired result.
Revised: Putting all together we get the desired result.

Original: The same result can also be deduced from page 222 Example 13.6 in Bauschke & Combettes (2017).
Revised: The same result can also be deduced from page 222 Example 13.6 in Bauschke & Combettes (2017).

Original: 2. Follows immediately from the deﬁnition since
Revised: 2. Follows immediately from the deﬁnition since

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: From Proposition 13.24 (i) in Bauschke & Combettes (2017) and Proposition 11.4.2 in Garling (2017) we get
Revised: From Proposition 13.24 (i) in Bauschke & Combettes (2017) and Proposition 11.4.2 in Garling (2017) we get

Original: [Equation]
Revised: [Equation]

Original: Lemma A28.
Revised: Lemma A28.

Original: The Legendre transform of the lse is the negative entropy function, restricted to the probability simplex and vice versa.
Revised: The Legendre transform of the lse is the negative entropy function, restricted to the probability simplex and vice versa.

Original: For the log-sum exponential f ( x ) =
Revised: For the log-sum exponential f ( x ) =

Original: ln (cid:32) n (cid:88) i = exp( x i ) (cid:33) [Equation] [Equation] the Legendre transform is the negative entropy function, restricted to the probability simplex:
Revised: ln (cid:32) n (cid:88) i = exp( x i ) (cid:33) , (502) the Legendre transform is the negative entropy function, restricted to the probability simplex:

Original: [Equation]
Revised: [Equation]

Original: for 0 (cid:54) x ∗ i and (cid:80) ni
Revised: for 0 (cid:54) x ∗ i and (cid:80) ni

Original: =1 x ∗ i = [Equation] ∞ otherwise .[Equation]
Revised: =1 x ∗ i =∞ otherwise .

Original: [Equation]
Revised: [Equation]

Original: For the negative entropy function, restricted to the probability simplex:
Revised: For the negative entropy function, restricted to the probability simplex:

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: x i
Revised: x i

Original: [Equation]
Revised: [Equation]

Original: for 0 (cid:54) x i and (cid:80) ni
Revised: for 0 (cid:54) x i and (cid:80) ni

Original: [Equation]
Revised: [Equation]

Original: 1 x i
Revised: 1 x i

Original: [Equation]
Revised: [Equation]

Original: ∞ otherwise .
Revised: ∞ otherwise .

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: the Legendre transform is the log-sum exponential
Revised: the Legendre transform is the log-sum exponential

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: See page 93 Example 3.25 in Boyd & Vandenberghe (2009) and (Gao & Pavel, 2017).
Revised: See page 93 Example 3.25 in Boyd & Vandenberghe (2009) and (Gao & Pavel, 2017).

Original: If f is a regular convex function (lower semi-continuous convex function), then f ∗∗ = f
Revised: If f is a regular convex function (lower semi-continuous convex function), then f ∗∗ = f

Original: according to page 135 Exercise 11.2.3 in Garling (2017).
Revised: according to page 135 Exercise 11.2.3 in Garling (2017).

Original: If f is lower semi-continuous and convex, then f ∗∗ = f according to Theorem 13.37 (Fenchel-Moreau) in Bauschke & Combettes (2017).
Revised: If f is lower semi-continuous and convex, then f ∗∗ = f according to Theorem 13.37 (Fenchel-Moreau) in Bauschke & Combettes (2017).

Original: The log-sum-exponential is continuous and convex.
Revised: The log-sum-exponential is continuous and convex.

Original: Lemma A29.
Revised: Lemma A29.

Original: Let XX T be non-singular and X a Hilbert space.
Revised: Let XX T be non-singular and X a Hilbert space.

Original: We deﬁne
Revised: We deﬁne

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: The Legendre transform of lse( β, X T ξ ) with ξ ∈ X is
Revised: The Legendre transform of lse( β, X T ξ ) with ξ ∈ X is

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: with ξ ∗ ∈ X ∗ and v ∈ X v .
Revised: with ξ ∗ ∈ X ∗ and v ∈ X v .

Original: The domain of (cid:0) lse( β, X T ξ ) (cid:1) ∗ is X ∗ .
Revised: The domain of (cid:0) lse( β, X T ξ ) (cid:1) ∗ is X ∗ .

Original: Furthermore we have
Revised: Furthermore we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We use the deﬁnition of the Legendre transform:
Revised: We use the deﬁnition of the Legendre transform:

Original: [Equation]
Revised: [Equation]

Original: where we used v ∗
Revised: where we used v ∗

Original: =
Revised: =

Original: X T (cid:0) XX T (cid:1) − 1 ξ ∗ .
Revised: X T (cid:0) XX T (cid:1) − 1 ξ ∗ .

Original: According to page 93 Example 3.25 in Boyd & Vandenberghe (2009), the equations for the maximum max v ∈ X v v T v ∗ − lse( β, v ) are solvable if and only if 0 < v ∗ =
Revised: According to page 93 Example 3.25 in Boyd & Vandenberghe (2009), the equations for the maximum max v ∈ X v v T v ∗ − lse( β, v ) are solvable if and only if 0 < v ∗ =

Original: X T (cid:0) XX T (cid:1) − 1 ξ ∗ and 1 T v ∗ = 1 T X T (cid:0) XX T (cid:1) − 1 ξ ∗ = 1 .
Revised: X T (cid:0) XX T (cid:1) − 1 ξ ∗ and 1 T v ∗ = 1 T X T (cid:0) XX T (cid:1) − 1 ξ ∗ = 1 .

Original: Therefore, we assumed ξ ∗ ∈ X ∗ .
Revised: Therefore, we assumed ξ ∗ ∈ X ∗ .

Original: The domain of (cid:0)
Revised: The domain of (cid:0)

Original: Using
Revised: Using

Original: [Equation]
Revised: [Equation]

Original: the Hessian of lse( β, X T ξ )
Revised: the Hessian of lse( β, X T ξ )

Original: [Equation]
Revised: [Equation]

Original: is positive semi-deﬁnite since diag( p ) − pp T is positive semi-deﬁnite according to Lemma A22.
Revised: is positive semi-deﬁnite since diag( p ) − pp T is positive semi-deﬁnite according to Lemma A22.

Original: Therefore, lse( β, X T ξ ) is convex and continuous.
Revised: Therefore, lse( β, X T ξ ) is convex and continuous.

Original: If f is a regular convex function (lower semi-continuous convex function), then f ∗∗ = f
Revised: If f is a regular convex function (lower semi-continuous convex function), then f ∗∗ = f

Original: according to page 135 Exercise 11.2.3 in Garling (2017).
Revised: according to page 135 Exercise 11.2.3 in Garling (2017).

Original: If f is lower semi-continuous and convex, then f ∗∗ = f according to Theorem 13.37 (Fenchel-Moreau) in Bauschke & Combettes (2017).
Revised: If f is lower semi-continuous and convex, then f ∗∗ = f according to Theorem 13.37 (Fenchel-Moreau) in Bauschke & Combettes (2017).

Original: Consequently we have
Revised: Consequently we have

Original: [Equation]
Revised: [Equation]

Original: We introduce the Lambert W function and some of its properties, since it is needed to derive bounds on the storage capacity of our new Hopﬁeld networks.
Revised: We introduce the Lambert W function and some of its properties, since it is needed to derive bounds on the storage capacity of our new Hopﬁeld networks.

Original: Deﬁnition A6 (Lambert Function) .
Revised: Deﬁnition A6 (Lambert Function) .

Original: The Lambert W function (Olver et al., 2010, (4.13)) is the inverse function of
Revised: The Lambert W function (Olver et al., 2010, (4.13)) is the inverse function of

Original: [Equation]
Revised: [Equation]

Original: The Lambert W function has an upper branch W 0 for − 1 (cid:54) y and a lower branch W − 1 for y (cid:54) − 1 .
Revised: The Lambert W function has an upper branch W 0 for − 1 (cid:54) y and a lower branch W − 1 for y (cid:54) − 1 .

Original: We use W if a formula holds for both branches.
Revised: We use W if a formula holds for both branches.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original:  We present some identities for the Lambert W function (Olver et al., 2010, (4.13)):
Revised: [Equation] We present some identities for the Lambert W function (Olver et al., 2010, (4.13)):

Original: Lemma A30.
Revised: Lemma A30.

Original: Identities for the Lambert W function are
Revised: Identities for the Lambert W function are

Original: [Equation]
Revised: [Equation]

Original: W ( x ) =
Revised: W ( x ) =

Original: W ( x ) x , (521) e nW ( x ) =
Revised: W ( x ) x , (519) e nW ( x ) =

Original: (cid:18) x
Revised: (cid:18) x

Original: W ( x ) (cid:19) n , (522)
Revised: W ( x ) (cid:19) n , (520)

Original: W 0 ( x ln x ) =
Revised: W 0 ( x ln x ) =

Original: ln x for x ≥ 1 e , (523)
Revised: ln x for x ≥ 1 e , (521)

Original: W − 1 ( x ln x ) =
Revised: W − 1 ( x ln x ) =

Original: ln x for x (cid:54) 1 e , (524)
Revised: ln x for x (cid:54) 1 e , (522)

Original: W ( x ) =
Revised: W ( x ) =

Original: ln x
Revised: ln x

Original: W ( x ) for x ≥ − 1 e , (525)
Revised: W ( x ) for x ≥ − 1 e , (523)

Original: W (cid:32) n x n
Revised: W (cid:32) n x n

Original: W ( x ) n −(cid:33)
Revised: W ( x ) n −(cid:33)

Original: = n W ( x ) for n, x > 0 , (526)
Revised: = n W ( x ) for n, x > 0 , (524)

Original: W ( x ) + W ( y ) =
Revised: W ( x ) + W ( y ) =

Original: W (cid:18) x y (cid:18)
Revised: W (cid:18) x y (cid:18)

Original: W ( x ) +
Revised: W ( x ) +

Original: W ( y ) (cid:19)(cid:19) for x, y > 0 , (527)
Revised: W ( y ) (cid:19)(cid:19) for x, y > 0 , (525)

Original: W(cid:18)
Revised: W(cid:18)

Original: − ln x x (cid:19) =
Revised: − ln x x (cid:19) =

Original: − ln x for 0 < x (cid:54) e , (528)
Revised: − ln x for 0 < x (cid:54) e , (526)

Original: W −(cid:18)
Revised: W −(cid:18)

Original: − ln x x (cid:19) =
Revised: − ln x x (cid:19) =

Original: − ln x for x > e , (529) e − W ( − ln x ) =
Revised: − ln x for x > e , (527) e − W ( − ln x ) =

Original: W ( − ln x ) − ln x for x (cid:54) = 1 .
Revised: W ( − ln x ) − ln x for x (cid:54) = 1 .

Original: We also present some special values for the Lambert W function (Olver et al., 2010, (4.13)):
Revised: We also present some special values for the Lambert W function (Olver et al., 2010, (4.13)):

Original: Lemma A31.
Revised: Lemma A31.

Original: [Equation]
Revised: [Equation]

Original: where the Omega constant Ω is
Revised: where the Omega constant Ω is

Original: [Equation]
Revised: [Equation]

Original: We need in some proofs a version of the mean value theorem as given in the next lemma.
Revised: We need in some proofs a version of the mean value theorem as given in the next lemma.

Original: Lemma A32 (Mean Value Theorem) .
Revised: Lemma A32 (Mean Value Theorem) .

Original: Let U ⊂ R n be open, f : U → R m continuously differentiable, and x ∈ U as well as h ∈ R n vectors such that the line segment x + t h for 0 (cid:54) t (cid:54) 1 is in U .
Revised: Let U ⊂ R n be open, f : U → R m continuously differentiable, and x ∈ U as well as h ∈ R n vectors such that the line segment x + t h for 0 (cid:54) t (cid:54) 1 is in U .

Original: Then the following holds:
Revised: Then the following holds:

Original: [Equation]
Revised: [Equation]

Original: where J is the Jacobian of f and the integral of the matrix is component-wise.
Revised: where J is the Jacobian of f and the integral of the matrix is component-wise.

Original: Proof.
Revised: Proof.

Original: Let f 1 , . . .
Revised: Let f 1 , . . .

Original: f m denote the components of f and deﬁne g i : [0 , 1]
Revised: f m denote the components of f and deﬁne g i : [0 , 1]

Original: → R by
Revised: → R by

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: then we obtain
Revised: then we obtain

Original: [Equation]
Revised: [Equation]

Original: The statement follows since the Jacobian J has as entries ∂f i ∂x
Revised: The statement follows since the Jacobian J has as entries ∂f i ∂x

Original: j
Revised: j

Original: . A. M ODERN H OPFIELD N ETWORKS : B INARY S TATES (K ROTOV AND H OPFIELD ) A.3. M ODERN H OPFIELD N ETWORKS : I NTRODUCTION A.3.
Revised: . A. M ODERN H OPFIELD N ETWORKS : B INARY S TATES (K ROTOV AND H OPFIELD ) A.3. M ODERN H OPFIELD N ETWORKS : I NTRODUCTION A.3.1.1 Additional Memory

Original: .1 Additional

Original: Memory and Attention for Neural Networks.
Revised: and Attention for Neural Networks.

Original: Modern Hopﬁeld networks may serve as additional memory for neural networks.
Revised: Modern Hopﬁeld networks may serve as additional memory for neural networks.

Original: Different approaches have been suggested to equip neural networks with an additional memory beyond recurrent connections.
Revised: Different approaches have been suggested to equip neural networks with an additional memory beyond recurrent connections.

Original: The neural
Revised: The neural

Original: Turing machine (NTM) is a neural network equipped with an external memory and an attention process (Graves et al., 2014).
Revised: Turing machine (NTM) is a neural network equipped with an external memory and an attention process (Graves et al., 2014).

Original: The NTM can write to the memory and can read from it.
Revised: The NTM can write to the memory and can read from it.

Original: A memory network (Weston et al., 2014) consists of a memory together with the components: (1) input feature map (converts the incoming input to the internal feature representation) (2) generalization (updates old memories given the new input), (3) output feature map (produces a new output), (4) response
Revised: A memory network (Weston et al., 2014) consists of a memory together with the components: (1) input feature map (converts the incoming input to the internal feature representation) (2) generalization (updates old memories given the new input), (3) output feature map (produces a new output), (4) response

Original: (converts the output into the response format).
Revised: (converts the output into the response format).

Original: Memory networks are generalized to an end-to-end trained model, where the arg max memory call is replaced by a differentiable softmax (Sukhbaatar et al., 2015a;b).
Revised: Memory networks are generalized to an end-to-end trained model, where the arg max memory call is replaced by a differentiable softmax (Sukhbaatar et al., 2015a;b).

Original: Linear Memory Network use a linear autoencoder for sequences as a memory (Carta et al., 2020).
Revised: Linear Memory Network use a linear autoencoder for sequences as a memory (Carta et al., 2020).

Original: To enhance RNNs with additional associative memory like Hopﬁeld networks have been proposed (Ba et al., 2016a;b).
Revised: To enhance RNNs with additional associative memory like Hopﬁeld networks have been proposed (Ba et al., 2016a;b).

Original: The associative memory stores hidden states of the RNN, retrieves stored states if they are similar to actual ones, and has a forgetting parameter.
Revised: The associative memory stores hidden states of the RNN, retrieves stored states if they are similar to actual ones, and has a forgetting parameter.

Original: The forgetting and storing parameters of the RNN associative memory have been generalized to learned matrices (Zhang & Zhou, 2017).
Revised: The forgetting and storing parameters of the RNN associative memory have been generalized to learned matrices (Zhang & Zhou, 2017).

Original: LSTMs with associative memory via Holographic Reduced Representations have been proposed (Danihelka et al., 2016).
Revised: LSTMs with associative memory via Holographic Reduced Representations have been proposed (Danihelka et al., 2016).

Original: Recently most approaches to new memories are based on attention.
Revised: Recently most approaches to new memories are based on attention.

Original: The neural Turing machine (NTM) is equipped with an external memory and an attention process (Graves et al., 2014).
Revised: The neural Turing machine (NTM) is equipped with an external memory and an attention process (Graves et al., 2014).

Original: End to end memory networks (EMN) make the attention scheme of memory networks (Weston et al., 2014) differentiable by replacing arg max through a softmax (Sukhbaatar et al., 2015a;b).
Revised: End to end memory networks (EMN) make the attention scheme of memory networks (Weston et al., 2014) differentiable by replacing arg max through a softmax (Sukhbaatar et al., 2015a;b).

Original: EMN with dot products became very popular and implement a key-value attention (Daniluk et al., 2017) for self-attention.
Revised: EMN with dot products became very popular and implement a key-value attention (Daniluk et al., 2017) for self-attention.

Original: An enhancement of EMN is the transformer (Vaswani et al., 2017a;b) and its extensions (Dehghani et al., 2018).
Revised: An enhancement of EMN is the transformer (Vaswani et al., 2017a;b) and its extensions (Dehghani et al., 2018).

Original: The transformer had great impact on the natural language processing (NLP) community as new records in NLP benchmarks have been achieved (Vaswani et al., 2017a;b).
Revised: The transformer had great impact on the natural language processing (NLP) community as new records in NLP benchmarks have been achieved (Vaswani et al., 2017a;b).

Original: MEMO uses the transformer attention mechanism for reasoning over longer distances (Banino et al., 2020).
Revised: MEMO uses the transformer attention mechanism for reasoning over longer distances (Banino et al., 2020).

Original: Current state-of-the-art for language processing is a transformer architecture called “the Bidirectional
Revised: Current state-of-the-art for language processing is a transformer architecture called “the Bidirectional

Original: Encoder Representations from Transformers” (BERT) (Devlin et al., 2018; 2019).
Revised: Encoder Representations from Transformers” (BERT) (Devlin et al., 2018; 2019).

Original: A.3.1.2 Modern Hopﬁeld networks: Overview. The
Revised: A.3.1.2 Modern Hopﬁeld networks: Overview. The

Original: storage capacity of classical binary Hopﬁeld networks (Hopﬁeld, 1982) has been shown to be very limited.
Revised: storage capacity of classical binary Hopﬁeld networks (Hopﬁeld, 1982) has been shown to be very limited.

Original: In a d -dimensional space, the standard Hopﬁeld model can store d uncorrelated patterns without errors but only Cd/ ln( d ) random patterns with C < 1 / 2 for a ﬁxed stable pattern or C < 1 / 4 if all patterns are stable (McEliece et al., 1987).
Revised: In a d -dimensional space, the standard Hopﬁeld model can store d uncorrelated patterns without errors but only Cd/ ln( d ) random patterns with C < 1 / 2 for a ﬁxed stable pattern or C < 1 / 4 if all patterns are stable (McEliece et al

Original: Thesame bound holds for nonlinear learning rules (Mazza, 1997).
Revised: The

Original: Usingtricks-of-trade and allowing small retrieval errors, the storage capacity is about 0 .
Revised: Using and allowing small retrieval errors, the storage capacity is about 0 .

Original: 138 d (Crisanti et al., 1986; Hertz et al., 1991; Torres et al., 2002).
Revised: 138 d (Crisanti et al., 1986; Hertz et al., 1991; Torres et al., 2002).

Original: If the learning rule is not related to the Hebb rule then up to d patterns can be stored (Abu-Mostafa & StJacques, 1985).
Revised: If the learning rule is not related to the Hebb rule then up to d patterns can be stored (Abu-Mostafa & StJacques, 1985).

Original: Using Hopﬁeld networks with non-zero diagonal matrices, the storage can be increased to Cd ln( d ) (Folli et al., 2017).
Revised: Using Hopﬁeld networks with non-zero diagonal matrices, the storage can be increased to Cd ln( d ) (Folli et al., 2017).

Original: In contrast to the storage capacity, the number of energy minima (spurious states, stable states) of Hopﬁeld networks is exponentially in d (Tanaka & Edwards, 1980; Bruck & Roychowdhury, 1990; Wainrib & Touboul, 2013).
Revised: In contrast to the storage capacity, the number of energy minima (spurious states, stable states) of Hopﬁeld networks is exponentially in d (Tanaka & Edwards, 1980; Bruck & Roychowdhury, 1990; Wainrib & Touboul, 2013).

Original: Recent advances in the ﬁeld of binary Hopﬁeld networks (Hopﬁeld, 1982) led to new properties of Hopﬁeld networks.
Revised: Recent advances in the ﬁeld of binary Hopﬁeld networks (Hopﬁeld, 1982) led to new properties of Hopﬁeld networks.

Original: The stability of spurious states or metastable states was sensibly reduced by a Hamiltonian treatment for the new relativistic Hopﬁeld model (Barra et al., 2018).
Revised: The stability of spurious states or metastable states was sensibly reduced by a Hamiltonian treatment for the new relativistic Hopﬁeld model (Barra et al., 2018).

Original: Recently the storage capacity of Hopﬁeld networks could be increased by new energy functions.
Revised: Recently the storage capacity of Hopﬁeld networks could be increased by new energy functions.

Original: Interaction functions of the form F ( x ) = x n lead to storage capacity of α n d n − 1 , where α n depends on the allowed error probability (Krotov & Hopﬁeld, 2016; 2018; Demircigil et al., 2017) (see (Krotov &
Revised: Interaction functions of the form F ( x ) = x n lead to storage capacity of α n d n − 1 , where α n depends on the allowed error probability (Krotov & Hopﬁeld, 2016; 2018; Demircigil et al., 2017) (see (Krotov &

Original: Hopﬁeld, 2018) for the non-binary case).
Revised: Hopﬁeld, 2018) for the non-binary case).

Original: Interaction functions of the form F ( x ) = x n lead to storage capacity of α n d n − 1 c n ln d for c n > 2(2 n − 3)!!
Revised: Interaction functions of the form F ( x ) = x n lead to storage capacity of α n d n − 1 c n ln d for c n > 2(2 n − 3)!!

Original: (Demircigil et al., 2017).
Revised: (Demircigil et al., 2017).

Original: Interaction functions of the form F ( x ) = exp( x ) lead to exponential storage capacity of 2 d/ 2 where all stored patterns are ﬁxed points but the radius of attraction vanishes (Demircigil et al., 2017).
Revised: Interaction functions of the form F ( x ) = exp( x ) lead to exponential storage capacity of 2 d/ 2 where all stored patterns are ﬁxed points but the radius of attraction vanishes (Demircigil et al., 2017).

Original: It has been shown that the network converges even after one update (Demircigil et al., 2017).
Revised: It has been shown that the network converges with high probability after one update (Demircigil et al., 2017).

Original: A.3. E NERGY AND U PDATE R ULE FOR B INARY M ODERN H OPFIELD N ETWORKS
Revised: A.3. E NERGY AND U PDATE R ULE FOR B INARY M ODERN H OPFIELD N ETWORKS

Original: We follow (Demircigil et al., 2017) where the goal is to store a set of input data x 1 , . . .
Revised: We follow (Demircigil et al., 2017) where the goal is to store a set of input data x 1 , . . .

Original: , x N that are represented by the matrix
Revised: , x N that are represented by the matrix

Original: [Equation]
Revised: [Equation]

Original: The x i is pattern with binary components x ij ∈ {− 1 , +1 } for all i and j .
Revised: The x i is pattern with binary components x ij ∈ {− 1 , +1 } for all i and j .

Original: ξ is the actual state of the units of the Hopﬁeld model.
Revised: ξ is the actual state of the units of the Hopﬁeld model.

Original: Krotov and Hopﬁeld (Krotov & Hopﬁeld, 2016) deﬁned the energy function E with the interaction function F that evaluates the dot product between patterns x i and the actual
Revised: Krotov and Hopﬁeld (Krotov & Hopﬁeld, 2016) deﬁned the energy function E with the interaction function F that evaluates the dot product between patterns x i and the actual

Original: [Equation]
Revised: [Equation]

Original: ξ :
Revised: ξ :

Original: [Equation]
Revised: [Equation]

Original: with F ( a ) = a n , where n = 2 gives the energy function of the classical Hopﬁeld network.
Revised: with F ( a ) = a n , where n = 2 gives the energy function of the classical Hopﬁeld network.

Original: This allows to store α n d n − 1 patterns (Krotov & Hopﬁeld, 2016).
Revised: This allows to store α n d n − 1 patterns (Krotov & Hopﬁeld, 2016).

Original: Krotov and Hopﬁeld (Krotov & Hopﬁeld, 2016) suggested for minimizing this energy an asynchronous updating dynamics T =
Revised: Krotov and Hopﬁeld (Krotov & Hopﬁeld, 2016) suggested for minimizing this energy an asynchronous updating dynamics T =

Original: ( T j ) for component ξ j :
Revised: ( T j ) for component ξ j :

Original: [Equation]
Revised: [Equation]

Original: While Krotov and Hopﬁeld used F ( a ) = a n , Demircigil et al.
Revised: While Krotov and Hopﬁeld used F ( a ) = a n , Demircigil et al.

Original: (Demircigil et al., 2017) went a step further and analyzed the model with the energy function F ( a ) =
Revised: (Demircigil et al., 2017) went a step further and analyzed the model with the energy function F ( a ) =

Original: exp( a ) , which leads to an exponential storage capacity of N = 2 d/ 2 .
Revised: exp( a ) , which leads to an exponential storage capacity of N = 2 d/ 2 .

Original: Furthermore with a single update the ﬁnal pattern is recovered with high probability.
Revised: Furthermore with a single update the ﬁnal pattern is recovered with high probability.

Original: These statements are given in next theorem.
Revised: These statements are given in next theorem.

Original: Theorem A10 (Storage Capacity for Binary Modern Hopﬁeld Nets (Demircigil et al. 2017)) .
Revised: Theorem A10 (Storage Capacity for Binary Modern Hopﬁeld Nets (Demircigil et al. 2017)) .

Original: Consider the generalized Hopﬁeld model with the dynamics described in Eq.
Revised: Consider the generalized Hopﬁeld model with the dynamics described in Eq.

Original: (546) and interaction function F given by F ( x ) =
Revised: (544) and interaction function F given by F ( x ) =

Original: e x .
Revised: e x .

Original: For a ﬁxed 0 < α < ln(2) / 2 let N = exp ( αd ) + 1 and let x 1 , . . .
Revised: For a ﬁxed 0 < α < ln(2) / 2 let N = exp ( αd ) + 1 and let x 1 , . . .

Original: , x N be N patterns chosen uniformly at random from {− 1 , +1 } d .
Revised: , x N be N patterns chosen uniformly at random from {− 1 , +1 } d .

Original: Moreover ﬁx (cid:37) ∈ [0 , 1 / 2) .
Revised: Moreover ﬁx (cid:37) ∈ [0 , 1 / 2) .

Original: For any i and any (cid:101) x i taken uniformly at random from the Hamming sphere with radius (cid:37)d centered in x i , S ( x i , (cid:37)d ) , where (cid:37)d is assumed to be an integer, it holds that
Revised: For any i and any (cid:101) x i taken uniformly at random from the Hamming sphere with radius (cid:37)d centered in x i , S ( x i , (cid:37)d ) , where (cid:37)d is assumed to be an integer, it holds that

Original: [Equation]
Revised: [Equation]

Original: if α is chosen in dependence of (cid:37) such that
Revised: if α is chosen in dependence of (cid:37) such that

Original: [Equation]
Revised: [Equation]

Original: with
Revised: with

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof can be found in Demircigil et al.
Revised: The proof can be found in Demircigil et al.

Original: The number of patterns N = exp ( αd ) + 1 is exponential in the number d of components.
Revised: The number of patterns N = exp ( αd ) + 1 is exponential in the number d of components.

Original: The result
Revised: The result

Original: [Equation]
Revised: [Equation]

Original: means that one update for each component is sufﬁcient to recover the pattern with high probability.
Revised: means that one update for each component is sufﬁcient to recover the pattern with high probability.

Original: The constraint α < I (1 − 2 (cid:37) )on α gives the trade-off between the radius of attraction (cid:37)d and the number N = exp ( αd ) + 1 of pattern that can be stored.
Revised: The constraint α < I (1 − 2 (cid:37) )on α gives the trade-off between the radius of attraction (cid:37)d and the number N = exp ( αd ) + 1 of pattern that can be stored.

Original: Theorem A10 in particular implies that
Revised: Theorem A10 in particular implies that

Original: [Equation]
Revised: [Equation]

Original: as d → ∞ , i.e. with a probability converging to 1 , all the patterns are ﬁxed points of the dynamics.
Revised: as d → ∞ , i.e. with a probability converging to 1 , all the patterns are ﬁxed points of the dynamics.

Original: In this case we can have α → I (1)2 = ln(2) / 2 .
Revised: In this case we can have α → I (1)2 = ln(2) / 2 .

Original: Krotov and Hopﬁeld deﬁne the update dynamics T j ( ξ ) in Eq.
Revised: Krotov and Hopﬁeld deﬁne the update dynamics T j ( ξ ) in Eq.

Original: (546) via energy differences of the energy in Eq.
Revised: (544) via energy differences of the energy in Eq.

Original: First we express the energy in Eq.
Revised: First we express the energy in Eq.

Original: (545) with F ( a ) =
Revised: (543) with F ( a ) =

Original: exp( a ) (Demircigil et al., 2017) by the lse function.
Revised: exp( a ) (Demircigil et al., 2017) by the lse function.

Original: Then we use the mean value theorem to express the update dynamics T j ( ξ ) in Eq.
Revised: Then we use the mean value theorem to express the update dynamics T j ( ξ ) in Eq.

Original: (546) by the softmax function.
Revised: (544) by the softmax function.

Original: For simplicity, we set β = 1 in the following.
Revised: For simplicity, we set β = 1 in the following.

Original: There exists a v ∈ [ − 1 , 1] with
Revised: There exists a v ∈ [ − 1 , 1] with

Original: [Equation]
Revised: [Equation]

Original: where e j is the Cartesian unit vector with a one at position j and zeros elsewhere, [ . ]
Revised: where e j is the Cartesian unit vector with a one at position j and zeros elsewhere, [ . ]

Original: j is the projection to the j -th component, and
Revised: j is the projection to the j -th component, and

Original: [Equation]
Revised: [Equation]

Original: A. H OPFIELD U PDATE R ULE IS A TTENTION OF T HE T RANSFORMER
Revised: A. H OPFIELD U PDATE R ULE IS A TTENTION OF T HE T RANSFORMER

Original: The Hopﬁeld network update rule is the attention mechanism used inthe transformer and BERT  (see Fig.
Revised: The Hopﬁeld network update rule is the attention mechanism used in transformer and BERT models (see Fig.

Original: A.2).
Revised: A.2).

Original: To see this, we assume  patterns
Revised: To see this, we assume N stored (key) patterns y i and S state (query) patterns

Original: y i that are mapped to the Hopﬁeld space of dimension d k .
Revised: r i that are mapped to the Hopﬁeld space of dimension d k .

Original: We set x i =
Revised: We set x i =

Original: W TK
Revised: W TK

Original: y i , ξ i =
Revised: y i , ξ i =

Original: W TQ y i , and multiply the result of our update rule with W V .
Revised: W TQ r i , and multiply the result of our update rule with W V .

Original: The matrix Y =
Revised: The matrices Y =

Original: ( y 1 , . . .
Revised: ( y 1 , . . .

Revised: , y N ) T and R =

Revised: ( r 1 , . .

Original: , y N ) T combines the y i  as row vectors.
Revised: , r S ) T combine the y i and r i as row vectors.

Original: We deﬁne the matrices X T =
Revised: We deﬁne the matrices X T =

Original: K = Y W K ,  Q = Y W Q , and V =
Revised: K = Y W K , Ξ T = Q = RW Q , and V =

Original: Y W K W V =
Revised: Y W K W V =

Original: X T W V , where W K ∈ R d y × d k , W Q ∈ R d y × d k , W V ∈ 
Revised: X T W V , where W K ∈ R d y × d k , W Q ∈ R d r × d k , W V ∈ R

Original: R d k × d v .
Revised: d k × d v .

Original: For combining all queries in matrix Q , β = 1 / √ d k, and softmax ∈ R N  changed to a row vector, we obtain for the update rule Eq.
Revised: If β = 1 / √ d k and softmax ∈ R N is changed to a row vector, we obtain for the update rule Eq.

Original: (27) multiplied by W V :
Revised: (3) multiplied by W V :

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Revised: The left part of Eq.

Revised: (547) is the transformer attention.

Revised: Besides the attention mechanism, Hopﬁeld networks allow for other functionalities in deep network architectures, which we introduce via speciﬁc layers in the next section.

Revised: The right part of Eq.

Revised: (547) serves as starting point for these speciﬁc layers.

Revised: of binary modern Hopﬁeld networks for allowing continuous states keeping fast and

Original: This formula is the transformer attention.
Revised: We deﬁned for the new energy also a new update rule that minimizes the energy. The new update rule is the attention mechanism of the transformer. Formulae are modiﬁed to express softmax as row vector as for transformers. " = "-sign means "keeps the properties".

Original: A. E XPERIMENTS A.5. E XPERIMENT 1: A TTENTION IN T RANSFORMERS DESCRIBED BY H OPFIELD DYNAMICS
Revised: A. E XPERIMENTS A.5. E XPERIMENT 1: A TTENTION IN T RANSFORMERS DESCRIBED BY H OPFIELD DYNAMICS

Revised: A.5.1.1 Analysis of operating modes of the heads of a pre-trained BERT model.

Revised: We analyzed pre-trained BERT models from Hugging Face Inc.

Original: A.5.1.1 Experimental Setup.
Revised: (Wolf et al., 2019) according to these operating classes. In Fig. A.3 in the appendix the distribution of the pre-trained bert-base-cased model is depicted (for other models see appendix Section A.5.1.4). Operating classes (II) (large metastable states) and (IV) (small metastable states) are often observed in the middle layers.

Revised: Operating class (I) (averaging over a very large number of patterns) is abundant in lower layers.

Revised: Similar observations have been reported in other studies (Toneva & Wehbe, 2019a;b; Tay et al., 2020).

Revised: Operating class (III) (medium metastable states) is predominant in the last layers.

Revised: A.5.1.2 Experimental Setup.

Original: Transformer architectures are known for their high computational demands.
Revised: Transformer architectures are known for their high computational demands.

Original: To investigate the learning dynamics of such a model and at the same time keeping training time manageable, we adopted the BERT-small setting from ELECTRA (Clark et al., 2020).
Revised: To investigate the learning dynamics of such a model and at the same time keeping training time manageable, we adopted the BERT-small setting from ELECTRA (Clark et al., 2020).

Original: It has 12 layers, 4 heads and a reduced hidden size, the sequence length is shortened from 512 totokens and the batch size is reduced from 256 to 128 .
Revised: It has 12 layers, 4 heads and a reduced hidden size, the sequence length is shortened from 512 totokens and the batch size is reduced from 256 to 128 .

Original: Additionally, the hidden dimension is reduced from 768 to 256 and the embedding dimension is reduced from 768 to 

Revised: Additionally, the hidden dimension is reduced from 768 to 256 and the embedding dimension is reduced from 768 to

Original: (Clark et al., 2020).
Revised: (Clark et al., 2020).

Original: The training of such a BERT-small model for 1 . 45 million update steps takes roughly four days on a single NVIDIA V100 GPU. 
Revised: The training of such a BERT-small model for 1 . 45 million update steps takes roughly four days on a single NVIDIA V100 GPU. L a y e r Head Head

Revised: Head

Revised: Head

Original: As the code base we use the transformers repository from Hugging Face, Inc (Wolf et al., 2019).
Revised: Head Head Head Head

Revised: Head

Revised: Head

Revised: Head

Revised: Head

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: L a y e r

Revised: [Figure]

Revised: 300 400 k patterns

Revised: L a y e r

Revised: [Figure]

Revised: patterns

Revised: [Figure]

Revised: patterns

Revised: [Figure]

Revised: patterns

Revised: [Figure]

Revised: patterns

Revised: [Figure]

Revised: patterns

Revised: [Figure]

Original: {+patterns 0 
+}
Revised: {+patterns 0 
+}

Revised: [Figure]

Original: {+k patterns 0 
+}
Revised: {+k patterns 0 
+}

Revised: [Figure]

Original: {+k patterns 0 
+}
Revised: {+k patterns 0 
+}

Revised: [Figure]

Original: {+k patterns 0 
+}
Revised: {+k patterns 0 
+}

Revised: [Figure]

Revised: As the code base we use the transformers repository from Hugging Face, Inc (Wolf et al., 2019).

Original: We aim to reproduce the dataset of Devlin et al.
Revised: We aim to reproduce the dataset of Devlin et al.

Original: (2019) as close as possible, which consists of the English
Revised: (2019) as close as possible, which consists of the English

Original: Wikipedia dataset and the Toronto BookCorpus dataset (Zhu et al., 2015).
Revised: Wikipedia dataset and the Toronto BookCorpus dataset (Zhu et al., 2015).

Original: Due to recent copyright claims the later is not publicly available anymore.
Revised: Due to recent copyright claims the later is not publicly available anymore.

Original: Therefore, the pre-training experiments use an uncased snapshot of the original BookCorpus dataset.
Revised: Therefore, the pre-training experiments use an uncased snapshot of the original BookCorpus dataset.

Original: A 2 Hopﬁeld Operating Classes of Transformer and BERT Models.
Revised: A.5.1.3 Hopﬁeld Operating Classes of Transformer and BERT Models.

Original: To better understand how operation modes in attention heads develop, we tracked the distribution of counts k
Revised: To better understand how operation modes in attention heads develop, we tracked the distribution of counts k

Original: (see main paper) over time in a BERT-small model.
Revised: (see main paper) over time in a BERT-small model.

Original: At the end of training we visualized the count distribution, grouped into four classes (see Figure A.3).
Revised: At the end of training we visualized the count distribution, grouped into four classes (see Figure A.4).

Original: The thresholds for the classes were chosen according to the thresholds of Figure 2 in the main paper.
Revised: The thresholds for the classes were chosen according to the thresholds of Figure 2 in the main paper.

Original: However, they are divided by a factor of 4 to adapt to the shorter sequence length of 128 compared to 512 .
Revised: However, they are divided by a factor of 4 to adapt to the shorter sequence length of 128 compared to 512 .

Original: From this plot it is clear, that the attention in heads of Class IV commit very early to the operating class of small metastable states.
Revised: From this plot it is clear, that the attention in heads of Class IV commit very early to the operating class of small metastable states.

Original: A.5.1.3 Learning Dynamics of Transformer and BERT Models.
Revised: A.5.1.4 Learning Dynamics of Transformer and BERT Models.

Original: To observe this behavior in the early phase of training, we created a ridge plot of the distributions of counts k for the ﬁrst 20 ,steps (see Figure A.4 (a)).
Revised: To observe this behavior in the early phase of training, we created a ridge plot of the distributions of counts k for the ﬁrst 20 ,steps (see Figure A.5 (a)).

Original: This plot shows that the attention in heads of middle layers often change the operation mode to Class IV around 9 , 000 to 10 , 000 steps.
Revised: This plot shows that the attention in heads of middle layers often change the operation mode to Class IV around 9 , 000 to 10 , 000 steps.

Original: At the same time the second big drop in the loss occurs.
Revised: At the same time the second big drop in the loss occurs.

Original: The question arises whether this is functionally important or whether it is an artefact which could be even harmful.
Revised: The question arises whether this is functionally important or whether it is an artefact which could be even harmful.

Original: To check if the attention mechanism is still able to learn after the change in the operation mode we analyzed the gradient ﬂow through the softmax function.
Revised: To check if the attention mechanism is still able to learn after the change in the operation mode we analyzed the gradient ﬂow through the softmax function.

Original: For every token we calculate the Frobenius norm of the Jacobian of the softmax over multiple samples.
Revised: For every token we calculate the Frobenius norm of the Jacobian of the softmax over multiple samples.

Original: Then, for every head we plot the distribution of the norm (see Figure A.4(b)).
Revised: Then, for every head we plot the distribution of the norm (see Figure A.5(b)).

Original: The gradients with respect to the weights are determined by the Jacobian J deﬁned in Eq.
Revised: The gradients with respect to the weights are determined by the Jacobian J deﬁned in Eq.

Original: (58) as can be seen in Eq.
Revised: (58) as can be seen in Eq.

Original: Eq.
Revised: Eq.

Original: (430), and Eq.
Revised: (428), and Eq.

Original: We can see that the attention in heads of Class IV remain almost unchanged during the rest of the training.
Revised: We can see that the attention in heads of Class IV remain almost unchanged during the rest of the training.

Original: A.5.1.4 Attention Heads Replaced by Gaussian Averaging Layers.
Revised: A.5.1.5 Attention Heads Replaced by Gaussian Averaging Layers.

Original: The self-attention mechanism proposed in Vaswani et al.
Revised: The self-attention mechanism proposed in Vaswani et al.

Original: (2017a) utilizes the softmax function to compute the coefﬁcients of a convex combination over the embedded tokens, where the softmax is conditioned on the input.
Revised: (2017a) utilizes the softmax function to compute the coefﬁcients of a convex combination over the embedded tokens, where the softmax is conditioned on the input.

Original: However, our analysis showed that especially in lower layers many heads perform averaging over a very large number of patterns.
Revised: However, our analysis showed that especially in lower layers many heads perform averaging over a very large number of patterns.

Original: This suggests that at this level neither the dependency on the input nor a ﬁne grained attention to individual positions is necessary.
Revised: This suggests that at this level neither the dependency on the input nor a ﬁne grained attention to individual positions is necessary.

Original: As an alternative to the original mechanism we propose Gaussian averaging heads which are computationally more efﬁcient.
Revised: As an alternative to the original mechanism we propose Gaussian averaging heads which are computationally more efﬁcient.

Original: Here, the softmax function is replaced by a discrete Gaussian kernel, where the location µ and the scale σ are learned.
Revised: Here, the softmax function is replaced by a discrete Gaussian kernel, where the location µ and the scale σ are learned.

Original: In detail, for a sequence length of N tokens we are given a vector of location parameters µ = ( µ 1 , . . . , µ N ) T and a vector of corresponding scale parameters σ = ( σ 1 , . . .
Revised: In detail, for a sequence length of N tokens we are given a vector of location parameters µ = ( µ 1 , . . . , µ N ) T and a vector of corresponding scale parameters σ = ( σ 1 , . . .

Original: , σ N ) T .
Revised: , σ N ) T .

Original: We subdivide the interval [ − 1 , 1] into N equidistant supporting points { s j } Nj =1 , where
Revised: We subdivide the interval [ − 1 , 1] into N equidistant supporting points { s j } Nj =1 , where

Original: [Equation]
Revised: [Equation]

Original: The attention [ A ] i,j from the i -th token to the j -th position is calculated as
Revised: The attention [ A ] i,j from the i -th token to the j -th position is calculated as

Original: [Equation]
Revised: [Equation]

Original: where z i normalizes the i -th row of the attention matrix
Revised: where z i normalizes the i -th row of the attention matrix

Original: A to sum up to one:
Revised: A to sum up to one:

Original: [Equation]
Revised: [Equation]

Original: For initialization we uniformly sample a location vector µ ∈ [ − 1 , 1] N and a scale vector σ ∈ [0 .
Revised: For initialization we uniformly sample a location vector µ ∈ [ − 1 , 1] N and a scale vector σ ∈ [0 .

Original: N per head.
Revised: N per head.

Original: A simple way to consider the individual position of each token at initialization is to use the supporting points µ i =
Revised: A simple way to consider the individual position of each token at initialization is to use the supporting points µ i =

Original: s i (see Figure A.5).
Revised: s i (see Figure A.6).

Original: In practice no difference to the random initialization was observed.
Revised: In practice no difference to the random initialization was observed.

Original: • Number of parameters.
Revised: • Number of parameters.

Original: Gaussian averaging heads can reduce the number of parameters signiﬁcantly.
Revised: Gaussian averaging heads can reduce the number of parameters signiﬁcantly.

Original: For an input size of N tokens, there are 2 · N parameters per head.
Revised: For an input size of N tokens, there are 2 · N parameters per head.

Original: In contrast, a standard self-attention head with word embedding dimension d y and projection dimension d k has two weight matrices W Q ,
Revised: In contrast, a standard self-attention head with word embedding dimension d y and projection dimension d k has two weight matrices L a y e r Head Head

Revised: Head

Revised: Head

Original: W K ∈ R d k × d y , which together amount to 2 · d k · d y parameters.
Revised: L a y e r L a y e r L a y e r L a y e r

Revised: L a y e r

Original: As a 
Revised: L a y e r L a y e r L a y e r L a y e r

Revised: L a y e r       100 k patterns

Revised: L a y e r    100 k patterns   100 k patterns   100 k patterns

Revised: [Figure]

Revised: W Q , W K ∈ R

Revised: d k × d y , which together amount to 2 · d k · d y parameters.

Original:  concrete example, the
Revised: As a concrete example, the

Original: BERT-base model from Devlin et al.
Revised: BERT-base model from Devlin et al.

Original: (2019) has an embedding dimension d y = 768 , a projection dimension d k = 64 and a sequence length of N = 512 .
Revised: (2019) has an embedding dimension d y = 768 , a projection dimension d k = 64 and a sequence length of N = 512 .

Original: Compared to the Gaussian head, in this case (2 · 768 · 64) / (2 · 512) =
Revised: Compared to the Gaussian head, in this case (2 · 768 · 64) / (2 · 512) =

Original: 5 times more parameters are trained for the attention mechanism itself.
Revised: 5 times more parameters are trained for the attention mechanism itself.

Original: Only for very long sequences (and given that the word embedding dimension stays the same) the dependence on N may become a disadvantage.
Revised: Only for very long sequences (and given that the word embedding dimension stays the same) the dependence on N may become a disadvantage.

Original: But of course, due to the independence from the input the Gaussian averaging head is less expressive in comparison to the original attention mechanism.
Revised: But of course, due to the independence from the input the Gaussian averaging head is less expressive in comparison to the original attention mechanism.

Original: A recently proposed input independent replacement for self-attention is the so called
Revised: A recently proposed input independent replacement for self-attention is the so called

Original: Random Synthesizer (Tay et al., 2020).
Revised: Random Synthesizer (Tay et al., 2020).

Original: Here the softmax -attention is directly parametrized with an
Revised: Here the softmax -attention is directly parametrized with an

Original: N × N matrix.
Revised: N × N matrix.

Original: This amounts to 0 .
Revised: This amounts to 0 .

Original: 5 · N more parameters than Gaussian averaging.L a y e r Head Head
Revised: 5 · N more parameters than Gaussian averaging.

Original: Head

Original: Head

Original: L a y e r

Original: L a y e r

Original: L a y e r

Original: L a y e r

Original: L a y e r

Original: L a y e r

Original: L a y e r

Original: L a y e r

Original: L a y e r

Original: L a y e r       100 k patterns

Original: L a y e r    100 k patterns   100 k patterns   100 k patterns

Original: [Figure]
Revised: [Figure]

Original: 

Original:       WRNHQ   

Original:    

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: A.5.2.1 Immune Repertoire Classiﬁcation.
Revised: A.5.2.1 Immune Repertoire Classiﬁcation.

Original: An architecture called DeepRC, is based on our modern Hopﬁeld networks, for immune repertoire classiﬁcation and compared to other machine learning approaches.
Revised: An architecture called DeepRC, is based on our modern Hopﬁeld networks, for immune repertoire classiﬁcation and compared to other machine learning approaches.

Original: For DeepRC, we consider immune repertoires as input objects, which are represented as bags of instances.
Revised: For DeepRC, we consider immune repertoires as input objects, which are represented as bags of instances.

Original: In a bag, each instance is an immune receptor sequence and each bag can contain a large number of sequences.
Revised: In a bag, each instance is an immune receptor sequence and each bag can contain a large number of sequences.

Original: At its core, DeepRC consists of a modern Hopﬁeld network that extracts information from each repertoire.
Revised: At its core, DeepRC consists of a modern Hopﬁeld network that extracts information from each repertoire.

Original: The stored patterns (keys) are representations of the immune amino acid sequences (instances) that are obtained by an 1D convolutional network with position encoding.
Revised: The stored patterns (keys) are representations of the immune amino acid sequences (instances) that are obtained by an 1D convolutional network with position encoding.

Original: Each state pattern (query) is static and learned via backpropagation.
Revised: Each state pattern (query) is static and learned via backpropagation.

Original: For details see Widrich et al.
Revised: For details see Widrich et al.

Original: (2020a;b).
Revised: (2020a;b).

Original: Our new Hopﬁeld network has been integrated into a deep learning architecture for immune repertoire classiﬁcation, a massive multiple instance learning task (Widrich et al., 2020a;b).
Revised: Our new Hopﬁeld network has been integrated into a deep learning architecture for immune repertoire classiﬁcation, a massive multiple instance learning task (Widrich et al., 2020a;b).

Original: Theorem 3 states that modern Hopﬁeld networks possess an exponential storage capacity which enables to tackle massive multiple instance learning (MIL) problems (Dietterich et al., 1997).
Revised: Theorem 3 states that modern Hopﬁeld networks possess an exponential storage capacity which enables to tackle massive multiple instance learning (MIL) problems (Dietterich et al., 1997).

Original: Immune repertoire classiﬁcation (Emerson et al., 2017) typically requires to extract few patterns from a large set of sequences, the repertoire, that are indicative for the respective immune status.
Revised: Immune repertoire classiﬁcation (Emerson et al., 2017) typically requires to extract few patterns from a large set of sequences, the repertoire, that are indicative for the respective immune status.

Original: Most MIL methods fail due the large number of instances.
Revised: Most MIL methods fail due the large number of instances.

Original: Data is obtained by experimentally observed immune receptors as well as simulated sequences sequence motifs
Revised: Data is obtained by experimentally observed immune receptors as well as simulated sequences sequence motifs

Original: (Akbar et al., 2019; Weber et al., 2020) with low yet varying degrees of frequency are implanted.
Revised: (Akbar et al., 2019; Weber et al., 2020) with low yet varying degrees of frequency are implanted.

Original: Four different categories of datasets are constructed: (a) Simulated immunosequencing data with implanted motifs, (b) immunosequencing data generated by long short-term memory (LSTM) with implanted motifs, (c) real-world immunosequencing data with implanted motifs, and (d) real-world immunosequencing data with known immune status (Emerson et al., 2017).
Revised: Four different categories of datasets are constructed: (a) Simulated immunosequencing data with implanted motifs, (b) immunosequencing data generated by long short-term memory (LSTM) with implanted motifs, (c) real-world immunosequencing data with implanted motifs, and (d) real-world immunosequencing data with known immune status (Emerson et al., 2017).

Original: Categories (a), (b), and (d) contain approx.
Revised: Categories (a), (b), and (d) contain approx.

Original: 300,000 instances per immune repertoire.
Revised: 300,000 instances per immune repertoire.

Original: With over 30 billion sequences in total, this represents one of the largest multiple instance learning experiments ever conducted (Carbonneau et al., 2018).
Revised: With over 30 billion sequences in total, this represents one of the largest multiple instance learning experiments ever conducted (Carbonneau et al., 2018).

Original: Despite the massive number of instances as well as the low frequency of sequences indicative of the respective immune status, deep learning architectures with modern
Revised: Despite the massive number of instances as well as the low frequency of sequences indicative of the respective immune status, deep learning architectures with modern

Original: Hopﬁeld networks outperform all competing methods with respect to average area under the ROC curve in all four categories, (a), (b), (c) and (d) (for details see Widrich et al.
Revised: Hopﬁeld networks outperform all competing methods with respect to average area under the ROC curve in all four categories, (a), (b), (c) and (d) (for details see Widrich et al.

Original: (2020a)).
Revised: (2020a)).

Original: We evaluate and compare the performance of DeepRC to a set of machine learning methods that serve as baseline, were suggested, or can readily be adapted to immune repertoire classiﬁcation.
Revised: We evaluate and compare the performance of DeepRC to a set of machine learning methods that serve as baseline, were suggested, or can readily be adapted to immune repertoire classiﬁcation.

Original: The methods comprise (i) known motif, which counts how often the known implanted motifs occur, (ii)
Revised: The methods comprise (i) known motif, which counts how often the known implanted motifs occur, (ii)

Original: Support Vector Machine (SVM) approach that uses a ﬁxed mapping from a bag of sequences to the corresponding k -mer counts and used the MinMax and Jaccard kernel, (iii)
Revised: Support Vector Machine (SVM) approach that uses a ﬁxed mapping from a bag of sequences to the corresponding k -mer counts and used the MinMax and Jaccard kernel, (iii)

Original: k -Nearest Neighbor (KNN) with k -mer representation, transforming MinMax and Jaccard kernel to distances, (iv) logistic regression on the k -mer representation, (v) burden test that ﬁrst identiﬁes sequences or k -mers and then computes a burden score per individual, and (vi) logistic multiple instance learning (lMIL).
Revised: k -Nearest Neighbor (KNN) with k -mer representation, transforming MinMax and Jaccard kernel to distances, (iv) logistic regression on the k -mer representation, (v) burden test that ﬁrst identiﬁes sequences or k -mers and then computes a burden score per individual, and (vi) logistic multiple instance learning (lMIL).

Original: On the real-world dataset DeepRC achieved an AUC of 0 .
Revised: On the real-world dataset DeepRC achieved an AUC of 0 .

Original: 832 ± 0 . 002 , followed by the SVM with
Revised: 832 ± 0 . 022 , followed by the SVM with

Original: MinMax kernel (AUC 0 .
Revised: MinMax kernel (AUC 0 .

Original: 825 ± 0 . 022 ) and the burden test with an AUC of 0 .
Revised: 825 ± 0 . 022 ) and the burden test with an AUC of 0 .

Original: 699 ± 0 .
Revised: 699 ± 0 .

Original: Overall on all datasets, DeepRC outperformed all competing methods with respect to average AUC (see Widrich et al.
Revised: Overall on all datasets, DeepRC outperformed all competing methods with respect to average AUC (see Widrich et al.

Original: (2020a;b)).
Revised: (2020a;b)).

Original: Table A.1 reports the average performance in the simulated immunosequencing datasets (last column) and the performance on datasets of the remaining three categories.
Revised: Table A.1 reports the average performance in the simulated immunosequencing datasets (last column) and the performance on datasets of the remaining three categories.

Original: DeepRC outperforms all competing methods with respect to average AUC.
Revised: DeepRC outperforms all competing methods with respect to average AUC.

Original: Across categories, the runner-up methods are either the SVM for MIL problems with MinMax kernel or the burden test.
Revised: Across categories, the runner-up methods are either the SVM for MIL problems with MinMax kernel or the burden test.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure] [Table] implanted signals and varying difﬁculties. The error reported is the standard deviation of the AUC values across the 18 datasets.
Revised: A

Original: A.5.2.2 Multiple Instance Learning Benchmark Datasets.
Revised: 2 Multiple Instance Learning Benchmark Datasets.

Original: Classical benchmarking datasets comprisemultiple versions of the Musk dataset (Dietterich et al., 1997), UCSB breast cancer classiﬁcation (Kandemir et al., 2014), or web mining (Zhou et al., 2005).
Revised: Classical benchmarking datasets comprise UCSB breast cancer classiﬁcation (Kandemir et al., 2014), and the Elephant, Fox, Tiger datasets (Andrews et al., 2003).

Original: Elphant and tiger are MIL datasets for image annotation (Andrews et al., 2003).
Revised: Elephant, Fox and Tiger are MIL datasets for image annotation which comprise color images from the

Original: Color images from the Corel dataset  have been preprocessed and segmented.
Revised: Corel dataset that have been preprocessed and segmented.

Original: An image consists of a set of segments (or blobs), each characterized by color, texture and shape descriptors.
Revised: An image consists of a set of segments (or blobs), each characterized by color, texture and shape descriptors.

Original: The datasets have 100 positive and 100 negative example images.
Revised: The datasets have 100 positive and 100 negative example images.

Original: The latter have been randomly drawn from a pool of photos of other animals.
Revised: The latter have been randomly drawn from a pool of photos of other animals.

Original: Elephant has 1391 instances and 230 features.
Revised: Elephant has 1391 instances and 230 features.

Revised: Fox has 1320 instances and 230 features.

Original: Tiger has 1220 instances and 230 features.
Revised: Tiger has 1220 instances and 230 features.

Revised: Furthermore, we use the UCSB breast cancer classiﬁcation (Kandemir et al., 2014) dataset, which consists of 2,002 instances across 58 input objects.

Revised: An instance represents a patch of a histopathological image of cancerous or normal tissue.

Revised: The layer

Revised: HopfieldPooling is used, which allows for computing a per-input-object representation by

Original: [Table]
Revised: [Table]

Revised: extracting an average of instances that are indicative for one of the two classes.

Revised: The input to the

Revised: HopfieldPooling layer is a set of embedded instances Y and a trainable but ﬁxed state (query) pattern Q used for averaging of class-indicative instances.

Revised: This averaging enables a compression of variable-sized bags to a ﬁxed-sized representation to discriminate the bags.

Revised: We performed a manual hyperparameter search on a validation set.

Revised: In detail, we used the following architecture to perform the given task on the Elephant, Fox, Tiger and UCSCB breast cancer datasets: (I) we apply fully connected linear embedding layers with ReLU activation.

Original: [Table]
Revised: (II) The output of this embedding serves as the input to our HopfieldPooling layer where the above described pooling operation is performed. (III) Thereafter we use ’ReLU - Linear blocks’ as the ﬁnal linear output layers that perform the classiﬁcation.

Revised: Among other hyperparameters, different hidden layer widths (for the fully connected pre- and post-

Revised: HopfieldPooling layers), learning rates and batch sizes were tried.

Original: A
Revised: Additionally our focus resided on the hyperparameters of the HopfieldPooling layer. Among those were the number of heads, the head dimension and the scaling factor β . All models were trained for 160 epochs using the AdamW optimizer (Loshchilov & Hutter, 2017) with exponential learning rate decay (see Table A.2), and validated by 10-fold nested cross validation repeated ﬁve times with different splits on the data sets. The reported ROC AUC scores are the average of these repetitions.

Revised: As overﬁtting imposed quite a problem, bag dropout was applied as the regularization technique of choice.

Revised: A.5. E XPERIMENT 3: C LASSIFICATION ON S MALL UCI B ENCHMARK D ATASETS

Original: 3.1 Motivation.
Revised: A.5.3.1 Motivation.

Original: Datasets with a small number of samples, like the UCI benchmark datasets, are particularly difﬁcult for neural networks to generalize on.
Revised: Datasets with a small number of samples, like the UCI benchmark datasets, are particularly difﬁcult for neural networks to generalize on.

Original: In contrast to their performance on larger datasets, they are consistently outperformed by methods like e.g. Random Forests (RF) and Support Vector Machines (SVMs).
Revised: In contrast to their performance on larger datasets, they are consistently outperformed by methods like e.g. gradient boosting, random forests (RF) and support vector machines (SVMs).

Original: Finding samples or even learning prototypes that are highly indicative for the class of a sample (query) suggest the use of Hopﬁeld networks.
Revised: Finding samples or even learning prototypes that are highly indicative for the class of a sample (query) suggest the use of Hopﬁeld networks.

Revised: We applied a modern Hopﬁeld network via the layer Hopfield .

Revised: The input vector is mapped to R using a self-normalizing net (SNN) and W K is learned, where the dimension of W K (the number of stored ﬁxed pattern) is a hyperparameter.

Revised: The output Z of Hopfield enters the output layer.

Revised: A.5.3.2 Methods compared.

Revised: Modern Hopﬁeld networks via the layer Hopﬁeld are compared to 17 groups of methods (Fernández-Delgado et al., 2014; Klambauer et al., 2017a):

Original: A.5.3.2 Experimental design and implementation details.
Revised: Support Vector Machines 2. Random Forest 3. Multivariate adaptive regression splines (MARS) 4. Boosting 5.

Revised: Rule-based Methods 6.

Revised: Logistic and Multinomial Regression (LMR) 7.

Revised: Discriminant Analysis (DA) 8.

Revised: Bagging 9.

Revised: Nearest Neighbor 10.

Revised: Decision Trees 11.

Revised: Other Ensembles 12.

Revised: Neural Networks (standard NN, BatchNorm, WeighNorm, MSRAinit, LayerNorm, ResNet, Self-Normalizing Nets) 13.

Revised: Bayesian Methods 14.

Revised: Other Methods 15.

Revised: Generalized linear models (GLM) 16.

Revised: Partial Least Squares and Principal Component Regression (PLSR) 17.

Revised: Stacking (Wolpert)

Revised: A.5.3.3 Experimental design and implementation details.

Original: As speciﬁed in the main paper, we consider 75 datasets of the UC Irvine Machine Learning Repository , which contain less than 1 ,samples per dataset, following the dataset separation into large and small dataset in Klambauer et al.
Revised: As speciﬁed in the main paper, we consider 75 datasets of the UC Irvine Machine Learning Repository , which contain less than 1 ,samples per dataset, following the dataset separation into large and small dataset in Klambauer et al.

Original: (2017a).
Revised: (2017a).

Original: On each dataset, we performed a grid-search to determine the best hyperparameter setting and model per dataset.
Revised: On each dataset, we performed a grid-search to determine the best hyperparameter setting and model per dataset.

Original: The hyperparameter search-space of the grid-search is listed in table A.3.
Revised: The hyperparameter search-space of the grid-search is listed in Table A.3.

Original: All models were trained for 100 epochs with a mini-batch size of 4 samples using the cross entropy loss and the PyTorch SGD module for stochastic gradient descent without momentum and without weight decay or dropout.
Revised: All models were trained for 100 epochs with a mini-batch size of 4 samples using the cross entropy loss and the PyTorch SGD module for stochastic gradient descent without momentum and without weight decay or dropout.

Original: After each epoch, the model accuracy was computed on a separated validation set.
Revised: After each epoch, the model accuracy was computed on a separated validation set.

Original: Using early stopping, the model with the best validation set accuracy averaged over 16 consecutive epochs was selected as ﬁnal model.
Revised: Using early stopping, the model with the best validation set accuracy averaged over 16 consecutive epochs was selected as ﬁnal model.

Original: This ﬁnal model was then evaluated against a separated test set to determine the accuracy, as reported in Tables 2 and Table uci_detailed_results.csv in the supplemental materials.
Revised: This ﬁnal model was then evaluated against a separated test set to determine the accuracy, as reported in Tables 2 and Table uci_detailed_results.csv in the supplemental materials.

Original: As network architecture, we use { 0 , 1 , 7 } fully connected embedding layers with SeLU Klambauer et al.
Revised: As network architecture, we use { 0 , 1 , 7 } fully connected embedding layers with SELU Klambauer et al.

Original: (2017a) activation functions and { 128 , 1024 } hidden units per embedding layer.
Revised: (2017a) activation functions and { 128 , 1024 } hidden units per embedding layer.

Original: These embedding layers are followed bya Hopfield Hopﬁeld layer, with a learned mapping of the input features to stored patterns, state patterns, and value projections.
Revised: These embedding layers are followed by the layer Hopfield .

Original: The number of hidden units is also used as number of dimensions for the Hopﬁeld association space with a number of { 1 , 32 } heads.
Revised: The number of hidden units is also used as number of dimensions for the Hopﬁeld association space with a number of { 1 , 32 } heads.

Revised: The layer

Original: Hopfield is followed by a mappingof its output values to the  number of classes.
Revised: Hopfield is followed by a mapping to the output vector, which has as dimension the number of classes.

Original: Finally, the softmax function is applied to obtain the predicted probability for a class. 
Revised: Finally, the softmax function is applied to obtain the predicted probability for a class. [Table]

Original: A.5.3.3 Results.
Revised: A.5.3.4 Results.

Original: We compared the performance of 25 methods based on their method rank.
Revised: We compared the performance of 25 methods based on their method rank.

Original: For this we computed the rank per method per dataset based on the accuracy on the test set, which was then averaged over all 75 datasets for each method to obtain the method rank.
Revised: For this we computed the rank per method per dataset based on the accuracy on the test set, which was then averaged over all 75 datasets for each method to obtain the method rank.

Original: For the baseline methods we used the scores summarized by (Klambauer et al., 2017a).
Revised: For the baseline methods we used the scores summarized by (Klambauer et al., 2017a).

Original: As shown in Table 2, the modern Hopﬁeld network outperforms all compared methods, where the difference is signiﬁcant with a p -value of 0 .
Revised: As shown in Table 2, the modern Hopﬁeld network outperforms all compared methods, where the difference is signiﬁcant with a p -value of 0 .

Original: (Wilcoxon signed rank test).
Revised: (Wilcoxon signed rank test).

Original: The accuracy values of all method groups for all datasets are listed in Table uci_detailed_results.
Revised: The accuracy values of all method groups for all datasets are listed in Table uci_detailed_results.csv in the supplemental materials.

Original: [Table]
Revised: [Table]

Original: A.5. E XPERIMENT 4: D RUG D ESIGN B ENCHMARK D ATASETS
Revised: A.5. E XPERIMENT 4: D RUG D ESIGN B ENCHMARK D ATASETS

Original: A. 1 Experimental design and implementation details.
Revised: A.5.4.1 Experimental design and implementation details.

Original: We test Hopﬁeld layers on 4 classiﬁcation datasets from MoleculeNet (Wu et al., 2017), which are challenging for deep learning methods.
Revised: We test Hopﬁeld layers on 4 classiﬁcation datasets from MoleculeNet (Wu et al., 2017), which are challenging for deep learning methods.

Original: The ﬁrst dataset is HIV, which was introduced by the Drug Therapeutics Program (DTP)
Revised: The ﬁrst dataset is HIV, which was introduced by the Drug Therapeutics Program (DTP)

Original: AIDS Antiviral Screen.
Revised: AIDS Antiviral Screen.

Original: The second dataset is BACE, which has IC50 measurements for binding afﬁnities of inhibitors (molecules) to the human β -secretase 1 (BACE-1).
Revised: The second dataset is BACE, which has IC50 measurements for binding afﬁnities of inhibitors (molecules) to the human β -secretase 1 (BACE-1).

Original: The third dataset is BBBP (blood-brain barrier permeability), which stems from modeling and predicting the blood-brain barrier permeability (Martins et al., 2012).
Revised: The third dataset is BBBP (blood-brain barrier permeability), which stems from modeling and predicting the blood-brain barrier permeability (Martins et al., 2012).

Original: The fourth dataset is SIDER (Side Effect Resource)
Revised: The fourth dataset is SIDER (Side Effect Resource)

Original: Kuhn et al.
Revised: Kuhn et al.

Original: and contains 1427 approved drugs.
Revised: and contains 1427 approved drugs.

Original: These datasets represent four areas of modeling tasks in drug discovery, concretely to develop accurate models for predicting a) new anti-virals (HIV), b) new protein inhibitors (BACE), c) metabolic effects (BBBP), and d) side effects of a chemical compound (SIDER).
Revised: These datasets represent four areas of modeling tasks in drug discovery, concretely to develop accurate models for predicting a) new anti-virals (HIV), b) new protein inhibitors (BACE), c) metabolic effects (BBBP), and d) side effects of a chemical compound (SIDER).

Original: We implemented a Hopﬁeld layer, in which we used the training-input as stored-pattern or key, the training-label as pattern-projection  or value and the input as state-pattern  or query.
Revised: We implemented a Hopﬁeld layer HopfieldLayer , in which we used the training-input as storedpattern Y or key, the training-label as pattern-projection Y W V or value and the input as state-pattern R or query.

Revised: As described in section A.6 by concatenation of input z i and target t

Revised: i the matrices W K and W V can be designed such that inside the softmax the input z

Revised: i is used and outside the softmax the target t i .

Original: All hyperparameters were selected on separate validation sets and we selected the model with the highest validation AUC on ﬁve different random splits. 
Revised: All hyperparameters were selected on separate validation sets and we selected the model with the highest validation AUC on ﬁve different random splits. A.5.4.2 Results.

Original: [Table]
Revised: We compared the Hopﬁeld layer Hopfieldlayer to Support Vector Machines (SVMs) (Cortes & Vapnik, 1995; Schölkopf & Smola, 2002), Extreme Gradient Boosting (XGBoost) (Chen & Guestrin, 2016), Random Forest (RF) (Breiman, 2001), Deep Neural Networks (DNNs) (LeCun et al., 2015; Schmidhuber, 2015), and to graph neural networks (GNN) like Graph Convolutional Networks (GCNs) (Kipf & Welling, 2016), Graph Attention Networks (GATs) (Veli˘ckovi´c et al., 2018), Message Passing Neural Networks (MPNNs) (Gilmer et al., 2017), and Attentive FP (Xiong et al., 2020). Our architecture with HopfieldLayer has reached state-of-theart for predicting side effects on SIDER 0 .

Original: 672 ± 0 .
Revised: 672 ± 0 .

Original: A
Revised: 019 as well as for predicting β -secretase BACE 0 . 902 ± 0 . See Table A.5 for all results, where the results of other methods are taken from Jiang et al.

Original: Results
Revised: [Table]

Original: . A. P Y T ORCH I MPLEMENTATION OF A H OPFIELD L AYER A.6. I NTRODUCTION
Revised: A. P Y T ORCH I MPLEMENTATION OF A H OPFIELD L AYER A.6. I NTRODUCTION

Original: In this section, we describe the implementation of a Hopﬁeld layer in PyTorch (Paszke et al., 2017; 2019) and, additionally, provide a brief usage manual.
Revised: In this section, we describe the implementation of a Hopﬁeld layer in PyTorch (Paszke et al. and, additionally, provide a brief usage manual.

Original: Possible applications for a Hopﬁeld layer in a deep network architecture comprise:
Revised: Possible applications for a Hopﬁeld layer in a deep network architecture comprise:

Original: • multiple instance learning (MIL)
Revised: • multiple instance learning (MIL)

Original: (Dietterich et al., 1997), • processing of and learning with point sets (Qi et al., 2017a;b; Xu et al., 2018), • set-based and permutation invariant learning (Guttenberg et al., 2016; Ravanbakhsh et al., 2016;
Revised: (Dietterich et al., 1997), • processing of and learning with point sets (Qi et al., 2017a;b; Xu et al., 2018), • set-based and permutation invariant learning (Guttenberg et al., 2016; Ravanbakhsh et al., 2016;

Original: Zaheer et al., 2017; Korshunova et al., 2018; Ilse et al., 2018; Zhai et al., 2020), • attention-based learning (Vaswani et al., 2017a), • associative learning, • natural language processing, • sequence analysis and time series prediction, and • storing and retrieving reference or experienced data, e.g. to store training data and retrieve it by the model or to store experiences for reinforcement learning.
Revised: Zaheer et al., 2017; Korshunova et al., 2018; Ilse et al., 2018; Zhai et al., 2020), • attention-based learning (Vaswani et al., 2017a), • associative learning, • natural language processing, • sequence analysis and time series prediction, and • storing and retrieving reference or experienced data, e.g. to store training data and retrieve it by the model or to store experiences for reinforcement learning.

Original: The Hopﬁeld layer in a deep neural network architecture can implement:
Revised: The Hopﬁeld layer in a deep neural network architecture can implement:

Original: • a memory (storage) with associative retrieval (Danihelka et al., 2016; Ba et al., 2016a), • conditional pooling and averaging operations (Wang et al., 2018; Ilse et al., 2020), • combining data by associations (Agrawal et al., 1993), 
Revised: • a memory (storage) with associative retrieval (Danihelka et al., 2016; Ba et al., 2016a), • conditional pooling and averaging operations (Wang et al., 2018; Ilse et al., 2020), • combining data by associations (Agrawal et al., 1993), • associative credit assignment (e

Original: • associative credit assignment (e.g. Rescorla-Wagner model or value estimation) (Sutton & Barto, 2018), and • attention mechanisms (Vaswani et al., 2017a; Bahdanau et al., 2014).
Revised: g. Rescorla-Wagner model or value estimation) (Sutton & Barto, 2018), and • attention mechanisms (Vaswani et al., 2017a; Bahdanau et al., 2014).

Original: In particular a Hopﬁeld layer can substitute attention layers in architectures of transformer and BERT models.
Revised: In particular, a Hopﬁeld layer can substitute attention layers in architectures of transformer and BERT models.

Original: The Hopﬁeld layer is designed to be used as plug-in replacement for existing layers like
Revised: The Hopﬁeld layer is designed to be used as plug-in replacement for existing layers like

Original: • pooling layers (max-pooling or average pooling), • permutation equivariant layers (Guttenberg et al., 2016; Ravanbakhsh et al., 2016), • GRU & LSTM layers, and • attention layers.
Revised: • pooling layers (max-pooling or average pooling), • permutation equivariant layers (Guttenberg et al., 2016; Ravanbakhsh et al., 2016), • GRU & LSTM layers, and • attention layers.

Original: In contrast to classical Hopﬁeld networks, the Hopﬁeld layer is based on the modern Hopﬁeld networks with continuous states that have increased storage capacity, as discussed in the main paper.
Revised: In contrast to classical Hopﬁeld networks, the Hopﬁeld layer is based on the modern Hopﬁeld networks with continuous states that have increased storage capacity, as discussed in the main paper.

Original: Like classical Hopﬁeld networks, the dynamics of the single heads of a Hopﬁeld layer follow a energy minimization dynamics.
Revised: Like classical Hopﬁeld networks, the dynamics of the single heads of a Hopﬁeld layer follow a energy minimization dynamics.

Original: The energy minimization empowers our Hopﬁeld layer with several advantages over other architectural designs like memory cells, associative memory, or attention mechanisms.
Revised: The energy minimization empowers our Hopﬁeld layer with several advantages over other architectural designs like memory cells, associative memory, or attention mechanisms.

Original: For example, the Hopﬁeld layer has more functionality than a transformer self-attention layer (Vaswani et al., 2017a) as described in Sec.
Revised: For example, the Hopﬁeld layer has more functionality than a transformer self-attention layer (Vaswani et al., 2017a) as described in Sec.

Original: A.6.2.
Revised: A.6.2.

Original: Possible use cases are given in Sec.
Revised: Possible use cases are given in Sec.

Original: A.6.3.
Revised: A.6.3.

Original: Source code will be provided under github .
Revised: Source code will be provided under github .

Original: A.6. F UNCTIONALITY
Revised: A.6. F UNCTIONALITY

Original: Non-standard functionalities that are added by a Hopﬁeld layer are
Revised: Non-standard functionalities that are added by a Hopﬁeld layer are

Original: • Association of two sets , • Multiple Updates for precise ﬁxed points, • Variable Beta that determines the kind of ﬁxed points, • Dimension of the associative space for controlling the storage capacity, • Static Patterns for ﬁxed pattern search, and • Pattern Normalization to control the ﬁxed point dynamics by norm of the patterns and shift of the patterns.
Revised: • Association of two sets , • Multiple Updates for precise ﬁxed points, • Variable Beta that determines the kind of ﬁxed points, • Dimension of the associative space for controlling the storage capacity, • Static Patterns for ﬁxed pattern search, and • Pattern Normalization to control the ﬁxed point dynamics by norm of the patterns and shift of the patterns.

Original: A functional sketch of our Hopﬁeld layer is shown in Fig.
Revised: A functional sketch of our Hopﬁeld layer is shown in Fig.

Original: A.6.
Revised: A.7.

Original: • Association of two sets.
Revised: Association of two sets.

Original: The Hopﬁeld layer makes it possible to associate two sets of vectors.
Revised: The Hopﬁeld layer makes it possible to associate two sets of vectors.

Original: This general functionality allows
Revised: This general functionality allows

Original: • for transformer-like self-attention, • for decoder-encoder attention, • for time series prediction (maybe with positional encoding), • for sequence analysis, • for multiple instance learning, • for learning with point sets, • for combining data sources by associations, • for constructing a memory, • for averaging and pooling operations, and • for many more.
Revised: • for transformer-like self-attention, • for decoder-encoder attention, • for time series prediction (maybe with positional encoding), • for sequence analysis, • for multiple instance learning, • for learning with point sets, • for combining data sources by associations, • for constructing a memory, • for averaging and pooling operations, and • for many more.

Original: The ﬁrst set of vectors consists of S raw state patterns R =
Revised: The ﬁrst set of vectors consists of S raw state patterns R =

Original: ( r 1 , . .
Revised: ( r 1 , . .

Original: , r S ) T with r s ∈ R d r and the second set of vectors consists of N raw stored patterns Y
Revised: , r S ) T with r s ∈ R d r and the second set of vectors consists of N raw stored patterns Y

Original: =
Revised: =

Original: ( y 1 , . . .
Revised: ( y 1 , . . .

Original: , y N ) T with y i ∈
Revised: , y N ) T with y i ∈

Original: R d y .
Revised: R d y .

Original: Both the S raw state patterns and N raw stored patterns are mapped to an associative space in R d k via the matrices W Q ∈ R d r × d k and W K ∈ R d y × d k , respectively.
Revised: Both the S raw state patterns and N raw stored patterns are mapped to an associative space in R d k via the matrices W Q ∈ R d r × d k and W K ∈ R d y × d k , respectively.

Original: We deﬁne a matrix Q ( Ξ T ) of state patterns ξ n = W Q r n in an associative space R d k and a matrix K ( X T ) of stored patterns x i =
Revised: We deﬁne a matrix Q ( Ξ T ) of state patterns ξ n = W Q r n in an associative space R d k and a matrix K ( X T ) of stored patterns x i =

Original: W K y s in the associative space R d
Revised: W K y s in the associative space R d

Original: k :
Revised: k :

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: In the main paper, Eq.
Revised: In the main paper, Eq.

Original: (3) deﬁnes the novel update rule:
Revised: (3) deﬁnes the novel update rule:

Original: [Equation]
Revised: [Equation]

Original: For multiple patterns, Eq.
Revised: For multiple patterns, Eq.

Original: (3) becomes:
Revised: (3) becomes:

Original: [Equation]
Revised: [Equation]

Original: where Ξ = ( ξ 1 , . . . , ξ N ) is the matrix of N state (query) patterns, X is the matrix of stored (key) patterns, and Ξ new is the matrix of new state patterns, which are averages over stored patterns.
Revised: where Ξ = ( ξ 1 , . . . , ξ N ) is the matrix of N state (query) patterns, X is the matrix of stored (key) patterns, and Ξ new is the matrix of new state patterns, which are averages over stored patterns.

Original: A new state pattern can also be very similar to a single stored pattern, in which case we call the stored pattern to be retrieved.
Revised: A new state pattern can also be very similar to a single stored pattern, in which case we call the stored pattern to be retrieved.

Original: These matrices allow to rewrite Eq.
Revised: These matrices allow to rewrite Eq.

Original: (553) as:
Revised: (551) as:

Original: [Equation]
Revised: [Equation]

Original: For β = 1 / √ d k and changing in Eq.
Revised: For β = 1 / √ d k and changing in Eq.

Original:  softmax ∈ R N to a row vector (and evaluating a row vector), we obtain:
Revised: (552) softmax ∈ R N to a row vector (and evaluating a row vector), we obtain:

Original: [Equation]
Revised: [Equation]

Original: where Q new is again the matrix of new state patterns.
Revised: where Q new is again the matrix of new state patterns.

Original: The new state patterns Ξ new are projected via W V to the result patterns Z =
Revised: The new state patterns Ξ new are projected via W V to the result patterns Z =

Original: Ξ new W V , where W V ∈ R
Revised: Ξ new W V , where W V ∈ R

Original: d
Revised: d

Original: k × d v .
Revised: k × d v .

Original: With the pattern projection
Revised: With the pattern projection

Original: V = KW V , we obtain the update rule Eq.
Revised: V = KW V , we obtain the update rule Eq.

Original: (10) from the main paper:
Revised: (10) from the main paper:

Original: [Equation]
Revised: [Equation]

Original: • Multiple Updates.
Revised: • Multiple Updates.

Original: The update Eq.
Revised: The update Eq.

Original: (554) can be iteratively applied to the initial state ξ of every Hopﬁeld layer head.
Revised: (552) can be iteratively applied to the initial state ξ of every Hopﬁeld layer head.

Original: After the last update, the new states Ξ new are projected via W V to the result patterns Z =
Revised: After the last update, the new states Ξ new are projected via W V to the result patterns Z =

Original: Ξ new W V .
Revised: Ξ new W V .

Original: Therefore, the Hopﬁeld layer allows multiple update steps in the forward pass without changing the number of parameters.
Revised: Therefore, the Hopﬁeld layer allows multiple update steps in the forward pass without changing the number of parameters.

Original: The number of update steps can be given for every Hopﬁeld head individually.
Revised: The number of update steps can be given for every Hopﬁeld head individually.

Original: Furthermore, it is possible to set a threshold for the number of updates of every Hopﬁeld head based on (cid:107) ξ − ξ new (cid:107) 2 .
Revised: Furthermore, it is possible to set a threshold for the number of updates of every Hopﬁeld head based on (cid:107) ξ − ξ new (cid:107) 2 .

Original: In the general case of multiple initial states Ξ , the maximum over the individual norms is taken.
Revised: In the general case of multiple initial states Ξ , the maximum over the individual norms is taken.

Original: • Variable β .
Revised: • Variable β .

Original: In the main paper, we have identiﬁed β as a crucial parameter for the ﬁxed point dynamics of the Hopﬁeld network, which governs the operating mode of the attention heads.
Revised: In the main paper, we have identiﬁed β as a crucial parameter for the ﬁxed point dynamics of the Hopﬁeld network, which governs the operating mode of the attention heads.

Original: In appendix, e.g. in Lemma A7 or in Eq.
Revised: In appendix, e g. in Lemma A7 or in Eq.

Original: (101) and Eq.
Revised: (101) and Eq.

Original: (102), we showed that the characteristics of the ﬁxed points of the new modern Hopﬁeld network are determined by: β , M (maximal pattern norm), m max (spread of the similar patterns), and (cid:107) m x (cid:107) (center of the similar patterns).
Revised: (102), we showed that the characteristics of the ﬁxed points of the new modern Hopﬁeld network are determined by: β , M (maximal pattern norm), m max (spread of the similar patterns), and (cid:107) m x (cid:107) (center of the similar patterns).

Original: Low values of β induce global averaging and higher values of β metastable states.
Revised: Low values of β induce global averaging and higher values of β metastable states.

Original: In the transformer attention, the β parameter is set to β = 1 / √ d k as in Eq. 
Revised: In the transformer attention, the β parameter is set to β = 1 / √ d k as in Eq. The Hopﬁeld

Original: The Hopﬁeld layer, however, allows to freely choose β > 0 , since the ﬁxed point dynamics does not only depend on the dimension of the associative space d k .
Revised: layer , however, allows to freely choose β > 0 , since the ﬁxed point dynamics does not only depend on the dimension of the associative space d k .

Original: Additionally, β heavily inﬂuences the gradient ﬂow tothe matrices W Q and W K .
Revised: Additionally, β heavily inﬂuences the gradient ﬂow to

Revised: the matrices

Revised: W

Revised: Q and

Revised: W

Revised: K .

Original: Thus, ﬁnding the right β for the respective application can becrucial.
Revised: Thus, ﬁnding the right β for the respective application can be

Revised: crucial.

Original: • Variable dimension of the associative space.
Revised: • Variable dimension of the associative space.

Original: Theorem A5 says that the storage capacity of the modern Hopﬁeld network grows exponentially with the dimension of the associative space.
Revised: Theorem A5 says that the storage capacity of the modern Hopﬁeld network grows exponentially with the dimension of the associative space.

Original: However higher dimension of the associative space also means less averaging and smaller metastable states.
Revised: However higher dimension of the associative space also means less averaging and smaller metastable states.

Original: The dimension of the associative space trades off storage capacity against the size of metastable states, e.g. over how many pattern is averaged.
Revised: The dimension of the associative space trades off storage capacity against the size of metastable states, e.g. over how many pattern is averaged.

Original: In Eq.
Revised: In Eq.

Original: (551) and in Eq.
Revised: (549) and in Eq.

Original: (550), we assumed N raw state patterns R =
Revised: (548), we assumed N raw state patterns R =

Original: ( r 1 , . .
Revised: ( r 1 , . .

Original: , r N ) T and S raw stored patterns Y =
Revised: , r N ) T and S raw stored patterns Y =

Original: ( y 1 , . . .
Revised: ( y 1 , . . .

Original: , y S ) T that are mapped to a d k -dimensional associative space via the matrices W Q 
Revised: , y S ) T that are mapped to a d k -dimensional associative space via the matrices W Q ∈ R d r × d k and W K ∈ R d

Original: ∈ R d r × d k and W K ∈ R d y × d k , respectively.
Revised: y × d k , respectively.

Original: In the associative space R d k , we obtain the state patterns Q =
Revised: In the associative space R d k , we obtain the state patterns Q =

Original: Ξ T = RW Q and the stored patterns K = X T =
Revised: Ξ T = RW Q and the stored patterns K = X T =

Original: Y W K .
Revised: Y W K .

Original: The Hopﬁeld view relates the dimension d k to the number of input patterns N that have to be processed.
Revised: The Hopﬁeld view relates the dimension d k to the number of input patterns N that have to be processed.

Original: The storage capacity depends exponentially on the dimension
Revised: The storage capacity depends exponentially on the dimension

Original: d k (the dimension of the associative space) and the size to metastable states is governed by this dimension, too.
Revised: d k (the dimension of the associative space) and the size to metastable states is governed by this dimension, too.

Original: Consequently, d k should be chosen with respect to the number N of patterns one wants to store and the desired size of metastable states, which is the number of patterns one wants to average over.
Revised: Consequently, d k should be chosen with respect to the number N of patterns one wants to store and the desired size of metastable states, which is the number of patterns one wants to average over.

Original: For example, if the input consists of many low dimensional input patterns, it makes sense to project the patterns into a higher dimensional space to allow a proper ﬁxed point dynamics.
Revised: For example, if the input consists of many low dimensional input patterns, it makes sense to project the patterns into a higher dimensional space to allow a proper ﬁxed point dynamics.

Original: Intuitively, this coincides with the construction of a richer feature space for the patterns.
Revised: Intuitively, this coincides with the construction of a richer feature space for the patterns.

Original: • Static Patterns.
Revised: • Static Patterns.

Original: In Eq.
Revised: In Eq.

Original: (551) and Eq.
Revised: (549) and Eq.

Original: (550), the N raw state patterns R =
Revised: (548), the N raw state patterns R =

Original: ( r 1 , . .
Revised: ( r 1 , . .

Original: , r N ) T and S raw stored patterns Y =
Revised: , r N ) T and S raw stored patterns Y =

Original: ( y 1 , . . .
Revised: ( y 1 , . . .

Original: , y S ) T are mapped to an associative space via the matrices W Q ∈ R d r × d k and W K ∈ R d y × d k , which gives the state patterns Q = Ξ T = RW Q and the stored patterns K = X T =
Revised: , y S ) T are mapped to an associative space via the matrices W Q ∈ R d r × d k and W K ∈ R d y × d k , which gives the state patterns Q = Ξ T = RW Q and the stored patterns K = X T =

Original: Y W K .
Revised: Y W K .

Original: We allow for static state and static stored patterns.
Revised: We allow for static state and static stored patterns.

Original: Static pattern means that the pattern does not depend on the network input, i.e. it is determined by the bias weights and remains constant across different network inputs.
Revised: Static pattern means that the pattern does not depend on the network input, i.e. it is determined by the bias weights and remains constant across different network inputs.

Original: Static state patterns allow to determine whether particular ﬁxed patterns are among the stored patterns and vice versa.
Revised: Static state patterns allow to determine whether particular ﬁxed patterns are among the stored patterns and vice versa.

Original: The static pattern functionality is typically needed if particular patterns must be identiﬁed in the data, e.g. as described for immune repertoire classiﬁcation in the main paper, where a ﬁxed d k -dimensional state vector ξ is used.
Revised: The static pattern functionality is typically needed if particular patterns must be identiﬁed in the data, e.g. as described for immune repertoire classiﬁcation in the main paper, where a ﬁxed d k -dimensional state vector ξ is used.

Original: • Pattern Normalization.
Revised: • Pattern Normalization.

Original: In the appendix, e.g. in Lemma A7 or in Eq.
Revised: In the appendix, e.g. in Lemma A7 or in Eq.

Original: (101) and Eq.
Revised: (101) and Eq.

Original: (102), we showed that the characteristics of the ﬁxed points of the new modern Hopﬁeld network are determined by: β , M (maximal pattern norm), m max (spread of the similar patterns), and (cid:107) m x (cid:107) (center of the similar patterns).
Revised: (102), we showed that the characteristics of the ﬁxed points of the new modern Hopﬁeld network are determined by: β , M (maximal pattern norm), m max (spread of the similar patterns), and (cid:107) m x (cid:107) (center of the similar patterns).

Original: We already discussed the parameter β while the spread of the similar patterns m max is given by the data.
Revised: We already discussed the parameter β while the spread of the similar patterns m max is given by the data.

Original: The remaining variables M and m x that both control the ﬁxed point dynamics are adjusted pattern normalization.
Revised: The remaining variables M and m x that both control the ﬁxed point dynamics are adjusted pattern normalization.

Original: M is the maximal pattern norm and m x the center of the similar patterns.
Revised: M is the maximal pattern norm and m x the center of the similar patterns.

Original: Theorem A5 says that larger M allows for more patterns to be stored.
Revised: Theorem A5 says that larger M allows for more patterns to be stored.

Original: However, the size of metastable states will decrease with increasing M .
Revised: However, the size of metastable states will decrease with increasing M .

Original: The vector m x says how well the (similar) patterns are centered.
Revised: The vector m x says how well the (similar) patterns are centered.

Original: If the norm (cid:107) m x (cid:107) is large, then this leads to smaller metastable states.
Revised: If the norm (cid:107) m x (cid:107) is large, then this leads to smaller metastable states.

Original: The two parameters M and m x are controlled by pattern normalization and determine the size and convergence properties of metastable states.
Revised: The two parameters M and m x are controlled by pattern normalization and determine the size and convergence properties of metastable states.

Original: These two parameters are important for creating large gradients if heads start with global averaging which has small gradient.
Revised: These two parameters are important for creating large gradients if heads start with global averaging which has small gradient.

Original: These two parameters can shift a head towards small metastable states which have largest gradient as shown in Fig.
Revised: These two parameters can shift a head towards small metastable states which have largest gradient as shown in Fig.

Original: A.4(b).
Revised: A.5(b).

Original: We allow for three different pattern normalizations:
Revised: We allow for three different pattern normalizations:

Original: • pattern normalization of the input patterns, • pattern normalization after mapping into the associative space, • no pattern normalization.
Revised: • pattern normalization of the input patterns, • pattern normalization after mapping into the associative space, • no pattern normalization.

Original: The default setting is a pattern normalization of the input patterns.MatMul Q K
Revised: The default setting is a pattern normalization of the input patterns.

Original: Scale

Original: Softmax

Original: Mask

Original: [Figure]

Original: MatMul

Original: MatMul

Original: K

Original: [Figure]

Original: [Figure]

Original: A.6. U SAGE
Revised: A.6. U SAGE

Original: As outlined in Sec.
Revised: As outlined in Sec.

Original: A.6.1, there are a variety of possible use cases for the Hopﬁeld layer, e.g. to build memory networks or transformer models.
Revised: A.6.1, there are a variety of possible use cases for the Hopﬁeld layer, e.g. to build memory networks or transformer models.

Original: The goal of the implementation is therefore to provide an easy to use Hopﬁeld module that can be used in a wide range of applications, be it as part of a larger architecture or as a standalone module.
Revised: The goal of the implementation is therefore to provide an easy to use Hopﬁeld module that can be used in a wide range of applications, be it as part of a larger architecture or as a standalone module.

Original: Consequently, the focus of the Hopﬁeld layer interface is set on its core parameters: the association of two sets, the scaling parameter β , the maximum number of updates, the dimension of the associative space, the possible usage of static patterns, and the pattern normalization.
Revised: Consequently, the focus of the Hopﬁeld layer interface is set on its core parameters: the association of two sets, the scaling parameter β , the maximum number of updates, the dimension of the associative space, the possible usage of static patterns, and the pattern normalization.