Original: For β = 1 / √ d k and changing in Eq.
Revised: For β = 1 / √ d k and changing in Eq.

Original: (552) softmax ∈ R N to a row vector (and evaluating a row vector), we obtain:
Revised: (553) softmax ∈ R N to a row vector (and evaluating a row vector), we obtain:

Original: [Equation]
Revised: [Equation]

Original: where Q new is again the matrix of new state patterns.
Revised: where Q new is again the matrix of new state patterns.

Original: The new state patterns Ξ new are projected via W V to the result patterns Z =
Revised: The new state patterns Ξ new are projected via W V to the result patterns Z =

Original: Ξ new W V , where W V ∈ R
Revised: Ξ new W V , where W V ∈ R

Original: d
Revised: d

Original: k × d v .
Revised: k × d v .

Original: With the pattern projection
Revised: With the pattern projection

Original: V = KW V , we obtain the update rule Eq.
Revised: V = KW V , we obtain the update rule Eq.

Original: (10) from the main paper:
Revised: (10) from the main paper:

Original: [Equation]
Revised: [Equation]

Original: • Multiple Updates.
Revised: • Multiple Updates.

Original: The update Eq.
Revised: The update Eq.

Original: (552) can be iteratively applied to the initial state ξ of every Hopﬁeld layer head.
Revised: (553) can be iteratively applied to the initial state ξ of every Hopﬁeld layer head.

Original: After the last update, the new states Ξ new are projected via W V to the result patterns Z =
Revised: After the last update, the new states Ξ new are projected via W V to the result patterns Z =

Original: Ξ new W V .
Revised: Ξ new W V .

Original: Therefore, the Hopﬁeld layer allows multiple update steps in the forward pass without changing the number of parameters.
Revised: Therefore, the Hopﬁeld layer allows multiple update steps in the forward pass without changing the number of parameters.

Original: The number of update steps can be given for every Hopﬁeld head individually.
Revised: The number of update steps can be given for every Hopﬁeld head individually.

Original: Furthermore, it is possible to set a threshold for the number of updates of every Hopﬁeld head based on (cid:107) ξ − ξ new (cid:107) 2 .
Revised: Furthermore, it is possible to set a threshold for the number of updates of every Hopﬁeld head based on (cid:107) ξ − ξ new (cid:107) 2 .

Original: In the general case of multiple initial states Ξ , the maximum over the individual norms is taken.
Revised: In the general case of multiple initial states Ξ , the maximum over the individual norms is taken.

Original: • Variable β .
Revised: • Variable β .

Original: In the main paper, we have identiﬁed β as a crucial parameter for the ﬁxed point dynamics of the Hopﬁeld network, which governs the operating mode of the attention heads.
Revised: In the main paper, we have identiﬁed β as a crucial parameter for the ﬁxed point dynamics of the Hopﬁeld network, which governs the operating mode of the attention heads.

Original: In appendix, e g. in Lemma A7 or in Eq.
Revised: In appendix, e.g. in Lemma A7 or in Eq.

Original: (101) and Eq.
Revised: (102) and Eq.

Original: (102), we showed that the characteristics of the ﬁxed points of the new modern Hopﬁeld network are determined by: β , M (maximal pattern norm), m max (spread of the similar patterns), and (cid:107) m x (cid:107) (center of the similar patterns).
Revised: , we showed that the characteristics of the ﬁxed points of the new modern Hopﬁeld network are determined by: β , M (maximal pattern norm), m max (spread of the similar patterns) and (cid:107) m x (cid:107) (center of the similar patterns).

Original: Low values of β induce global averaging and higher values of β metastable states.
Revised: Low values of β induce global averaging and higher values of β metastable states.

Original: In the transformer attention, the β parameter is set to β = 1 / √ d k as in Eq.
Revised: In the transformer attention, the β parameter is set to β = 1 / √ d k as in Eq.

Original: The Hopﬁeld

Original: layer , however, allows to freely choose β > 0 , since the ﬁxed point dynamics does not only depend on the dimension of the associative space d k .
Revised: The Hopﬁeld layer, however, allows to freely choose β > 0 , since the ﬁxed point dynamics does not only depend on the dimension of the associative space d k .

Original: Additionally, β heavily inﬂuences the gradient ﬂow to 
Revised: Additionally, β heavily inﬂuences the gradient ﬂow to the matrices W Q and W K .

Original: the matrices

Original: W

Original: Q and

Original: W

Original: K .

Original: Thus, ﬁnding the right β for the respective application can be 
Revised: Thus, ﬁnding the right β for the respective application can be crucial.

Original: crucial.

Original: • Variable dimension of the associative space.
Revised: • Variable dimension of the associative space.

Original: Theorem A5 says that the storage capacity of the modern Hopﬁeld network grows exponentially with the dimension of the associative space.
Revised: Theorem A5 says that the storage capacity of the modern Hopﬁeld network grows exponentially with the dimension of the associative space.

Original: However higher dimension of the associative space also means less averaging and smaller metastable states.
Revised: However higher dimension of the associative space also means less averaging and smaller metastable states.

Original: The dimension of the associative space trades off storage capacity against the size of metastable states, e.g. over how many pattern is averaged.
Revised: The dimension of the associative space trades off storage capacity against the size of metastable states, e.g. over how many pattern is averaged.

Original: In Eq.
Revised: In Eq.

Original: (549) and in Eq.
Revised: (550) and in Eq.

Original: (548), we assumed N raw state patterns R =
Revised: (549), we assumed N raw state patterns R =

Original: ( r 1 , . .
Revised: ( r 1 , . .

Original: , r N ) T and S raw stored patterns Y =
Revised: , r N ) T and S raw stored patterns Y =

Original: ( y 1 , . . .
Revised: ( y 1 , . . .

Original: , y S ) T that are mapped to a d k -dimensional associative space via the matrices W Q ∈ R d r × d k and W K ∈ R d
Revised: , y S ) T that are mapped to a d k -dimensional associative space via the matrices W Q ∈ R d r × d k and W K ∈ R d

Original: y × d k , respectively.
Revised: y × d k , respectively.

Original: In the associative space R d k , we obtain the state patterns Q =
Revised: In the associative space R d k , we obtain the state patterns Q =

Original: Ξ T = RW Q and the stored patterns K = X T =
Revised: Ξ T = RW Q and the stored patterns K = X T =

Original: Y W K .
Revised: Y W K .

Original: The Hopﬁeld view relates the dimension d k to the number of input patterns N that have to be processed.
Revised: The Hopﬁeld view relates the dimension d k to the number of input patterns N that have to be processed.

Original: The storage capacity depends exponentially on the dimension
Revised: The storage capacity depends exponentially on the dimension

Original: d k (the dimension of the associative space) and the size to metastable states is governed by this dimension, too.
Revised: d k (the dimension of the associative space) and the size to metastable states is governed by this dimension, too.

Original: Consequently, d k should be chosen with respect to the number N of patterns one wants to store and the desired size of metastable states, which is the number of patterns one wants to average over.
Revised: Consequently, d k should be chosen with respect to the number N of patterns one wants to store and the desired size of metastable states, which is the number of patterns one wants to average over.

Original: For example, if the input consists of many low dimensional input patterns, it makes sense to project the patterns into a higher dimensional space to allow a proper ﬁxed point dynamics.
Revised: For example, if the input consists of many low dimensional input patterns, it makes sense to project the patterns into a higher dimensional space to allow a proper ﬁxed point dynamics.

Original: Intuitively, this coincides with the construction of a richer feature space for the patterns.
Revised: Intuitively, this coincides with the construction of a richer feature space for the patterns.

Original: • Static Patterns.
Revised: • Static Patterns.

Original: In Eq.
Revised: In Eq.

Original: (549) and Eq.
Revised: (550) and Eq.

Original: (548), the N raw state patterns R =
Revised: (549), the N raw state patterns R =

Original: ( r 1 , . .
Revised: ( r 1 , . .

Original: , r N ) T and S raw stored patterns Y =
Revised: , r N ) T and S raw stored patterns Y =

Original: ( y 1 , . . .
Revised: ( y 1 , . . .

Original: , y S ) T are mapped to an associative space via the matrices W Q ∈ R d r × d k and W K ∈ R d y × d k , which gives the state patterns Q = Ξ T = RW Q and the stored patterns K = X T =
Revised: , y S ) T are mapped to an associative space via the matrices W Q ∈ R d r × d k and W K ∈ R d y × d k , which gives the state patterns Q = Ξ T = RW Q and the stored patterns K = X T =

Original: Y W K .
Revised: Y W K .

Original: We allow for static state and static stored patterns.
Revised: We allow for static state and static stored patterns.

Original: Static pattern means that the pattern does not depend on the network input, i.e. it is determined by the bias weights and remains constant across different network inputs.
Revised: Static pattern means that the pattern does not depend on the network input, i.e. it is determined by the bias weights and remains constant across different network inputs.

Original: Static state patterns allow to determine whether particular ﬁxed patterns are among the stored patterns and vice versa.
Revised: Static state patterns allow to determine whether particular ﬁxed patterns are among the stored patterns and vice versa.

Original: The static pattern functionality is typically needed if particular patterns must be identiﬁed in the data, e.g. as described for immune repertoire classiﬁcation in the main paper, where a ﬁxed d k -dimensional state vector ξ is used.
Revised: The static pattern functionality is typically needed if particular patterns must be identiﬁed in the data, e.g. as described for immune repertoire classiﬁcation in the main paper, where a ﬁxed d k -dimensional state vector ξ is used

Original: • Pattern Normalization.
Revised: Pattern Normalization.

Original: In the appendix, e.g. in Lemma A7 or in Eq.
Revised: In the appendix, e.g. in Lemma A7 or in Eq.

Original: (101) and Eq.
Revised: (102) and Eq.

Original: (102), we showed that the characteristics of the ﬁxed points of the new modern Hopﬁeld network are determined by: β , M (maximal pattern norm), m max (spread of the similar patterns), and (cid:107) m x (cid:107) (center of the similar patterns).
Revised: we showed that the characteristics of the ﬁxed points of the new modern Hopﬁeld network are determined by: β , M (maximal pattern norm), m max (spread of the similar patterns), and (cid:107) m x (cid:107) (center of the similar patterns).

Original: We already discussed the parameter β while the spread of the similar patterns m max is given by the data.
Revised: We already discussed the parameter β while the spread of the similar patterns m max is given by the data.

Original: The remaining variables M and m x that both control the ﬁxed point dynamics are adjusted pattern normalization.
Revised: The remaining variables M and m x that both control the ﬁxed point dynamics are adjusted pattern normalization.

Original: M is the maximal pattern norm and m x the center of the similar patterns.
Revised: M is the maximal pattern norm and m x the center of the similar patterns.

Original: Theorem A5 says that larger M allows for more patterns to be stored.
Revised: Theorem A5 says that larger M allows for more patterns to be stored.

Original: However, the size of metastable states will decrease with increasing M .
Revised: However, the size of metastable states will decrease with increasing M .

Original: The vector m x says how well the (similar) patterns are centered.
Revised: The vector m x says how well the (similar) patterns are centered.

Original: If the norm (cid:107) m x (cid:107) is large, then this leads to smaller metastable states.
Revised: If the norm (cid:107) m x (cid:107) is large, then this leads to smaller metastable states.

Original: The two parameters M and m x are controlled by pattern normalization and determine the size and convergence properties of metastable states.
Revised: The two parameters M and m x are controlled by pattern normalization and determine the size and convergence properties of metastable states.

Original: These two parameters are important for creating large gradients if heads start with global averaging which has small gradient.
Revised: These two parameters are important for creating large gradients if heads start with global averaging which has small gradient.

Original: These two parameters can shift a head towards small metastable states which have largest gradient as shown in Fig.
Revised: These two parameters can shift a head towards small metastable states which have largest gradient as shown in Fig.

Original: A.5(b).
Revised: A.5(b).

Original: We allow for three different pattern normalizations:
Revised: We allow for three different pattern normalizations:

Original: • pattern normalization of the input patterns, • pattern normalization after mapping into the associative space, • no pattern normalization.
Revised: • pattern normalization of the input patterns, • pattern normalization after mapping into the associative space, • no pattern normalization.

Original: The default setting is a pattern normalization of the input patterns.
Revised: The default setting is a pattern normalization of the input patterns.

Original: A.6. U SAGE
Revised: A.6. U SAGE

Original: As outlined in Sec.
Revised: As outlined in Sec.

Original: A.6.1, there are a variety of possible use cases for the Hopﬁeld layer, e.g. to build memory networks or transformer models.
Revised: A.6.1, there are a variety of possible use cases for the Hopﬁeld layer, e.g. to build memory networks or transformer models.

Original: The goal of the implementation is therefore to provide an easy to use Hopﬁeld module that can be used in a wide range of applications, be it as part of a larger architecture or as a standalone module.
Revised: The goal of the implementation is therefore to provide an easy to use Hopﬁeld module that can be used in a wide range of applications, be it as part of a larger architecture or as a standalone module.

Original: Consequently, the focus of the Hopﬁeld layer interface is set on its core parameters: the association of two sets, the scaling parameter β , the maximum number of updates, the dimension of the associative space, the possible usage of static patterns, and the pattern normalization.
Revised: Consequently, the focus of the Hopﬁeld layer interface is set on its core parameters: the association of two sets, the scaling parameter β , the maximum number of updates, the dimension of the associative space, the possible usage of static patterns, and the pattern normalization.

Original: The integration into the PyTorch framework is built such that with all the above functionalities disabled, the “HopﬁeldEncoderLayer” and the “HopﬁeldDecoderLayer”, both extensions of the Hopﬁeld module, can be used as a one-to-one plug-in replacement for the TransformerEncoderLayer and the TransformerDecoderLayer , respectively, of the PyTorch transformer module.
Revised: The integration into the PyTorch framework is built such that with all the above functionalities disabled, the “HopﬁeldEncoderLayer” and the “HopﬁeldDecoderLayer”, both extensions of the Hopﬁeld module, can be used as a one-to-one plug-in replacement for the TransformerEncoderLayer and the TransformerDecoderLayer , respectively, of the PyTorch transformer module.

Original: The Hopﬁeld layer can be used to implement or to substitute different layers:
Revised: The Hopﬁeld layer can be used to implement or to substitute different layers:

Original: • Pooling layers: We consider the Hopﬁeld layer as a pooling layer if only one static state (query) pattern exists.
Revised: • Pooling layers: We consider the Hopﬁeld layer as a pooling layer if only one static state (query) pattern exists.

Original: Then, it is de facto a pooling over the sequence, which results from the softmax values applied on the stored patterns.
Revised: Then, it is de facto a pooling over the sequence, which results from the softmax values applied on the stored patterns.

Original: Therefore, our Hopﬁeld layer can act as a pooling layer.
Revised: Therefore, our Hopﬁeld layer can act as a pooling layer.

Original: • Permutation equivariant layers: Our Hopﬁeld layer can be used as a plug-in replacement for permutation equivariant layers.
Revised: • Permutation equivariant layers: Our Hopﬁeld layer can be used as a plug-in replacement for permutation equivariant layers.

Original: Since the Hopﬁeld layer is an associative memory it assumes no dependency between the input patterns.
Revised: Since the Hopﬁeld layer is an associative memory it assumes no dependency between the input patterns.

Original: • GRU & LSTM layers: Our Hopﬁeld layer can be used as a plug-in replacement for GRU & LSTM layers.
Revised: • GRU & LSTM layers: Our Hopﬁeld layer can be used as a plug-in replacement for GRU & LSTM layers.

Original: Optionally, for substituting GRU & LSTM layers, positional encoding might be considered.
Revised: Optionally, for substituting GRU & LSTM layers, positional encoding might be considered.

Original: • Attention layers: Our Hopﬁeld layer can act as an attention layer, where state (query) and stored (key) patterns are different, and need to be associated.
Revised: • Attention layers: Our Hopﬁeld layer can act as an attention layer, where state (query) and stored (key) patterns are different, and need to be associated.

Original: • Finally, the extensions of the Hopﬁeld layer are able to operate as a self-attention layer (HopﬁeldEncoderLayer) and as cross-attention layer (HopﬁeldDecoderLayer), as described in (Vaswani et al., 2017a).
Revised: • Finally, the extensions of the Hopﬁeld layer are able to operate as a self-attention layer (HopﬁeldEncoderLayer) and as cross-attention layer (HopﬁeldDecoderLayer), as described in (Vaswani et al., 2017a).

Original: As such, it can be used as building block of transformer-based or general architectures.
Revised: As such, it can be used as building block of transformer-based or general architectures

Original: MatMul
Revised: MatMul

Original: Q K Scale
Revised: [Figure]

Original: Softmax
Revised: Softmax

Original: Mask
Revised: Mask

Original: [Figure]
Revised: [Figure]

Original: MatMul
Revised: MatMul

Revised: [Figure]

Original: MatMul
Revised: multipleupdates

Original: [Figure] [Figure]
Revised: [Figure] [Figure]