Original: In particular, a Hopﬁeld layer can substitute attention layers in architectures of transformer and BERT models.
Revised: In particular, a Hopﬁeld layer can substitute attention layers in architectures of transformer and BERT models.

Original: The Hopﬁeld layer is designed to be used as plug-in replacement for existing layers like
Revised: The Hopﬁeld layer is designed to be used as plug-in replacement for existing layers like

Original: • pooling layers (max-pooling or average pooling), • permutation equivariant layers (Guttenberg et al., 2016; Ravanbakhsh et al., 2016), • GRU & LSTM layers, and • attention layers.
Revised: • pooling layers (max-pooling or average pooling), • permutation equivariant layers (Guttenberg et al., 2016; Ravanbakhsh et al., 2016), • GRU & LSTM layers, and • attention layers.

Original: In contrast to classical Hopﬁeld networks, the Hopﬁeld layer is based on the modern Hopﬁeld networks with continuous states that have increased storage capacity, as discussed in the main paper.
Revised: In contrast to classical Hopﬁeld networks, the Hopﬁeld layer is based on the modern Hopﬁeld networks with continuous states that have increased storage capacity, as discussed in the main paper.

Original: Like classical Hopﬁeld networks, the dynamics of the single heads of a Hopﬁeld layer follow a energy minimization dynamics.
Revised: Like classical Hopﬁeld networks, the dynamics of the single heads of a Hopﬁeld layer follow a energy minimization dynamics.

Original: The energy minimization empowers our Hopﬁeld layer with several advantages over other architectural designs like memory cells, associative memory, or attention mechanisms.
Revised: The energy minimization empowers our Hopﬁeld layer with several advantages over other architectural designs like memory cells, associative memory, or attention mechanisms.

Original: For example, the Hopﬁeld layer has more functionality than a transformer self-attention layer (Vaswani et al., 2017a) as described in Sec.
Revised: For example, the Hopﬁeld layer has more functionality than a transformer self-attention layer (Vaswani et al., 2017a) as described in Sec.

Original: A.6.2.
Revised: A.6.2.

Original: Possible use cases are given in Sec.
Revised: Possible use cases are given in Sec.

Original: A.6.3.
Revised: A.6.3.

Original: Source code will be provided under github .
Revised: Source code will be provided under github .

Original: A.6. F UNCTIONALITY
Revised: A.6. F UNCTIONALITY

Original: Non-standard functionalities that are added by a Hopﬁeld layer are
Revised: Non-standard functionalities that are added by a Hopﬁeld layer are

Original: • Association of two sets , • Multiple Updates for precise ﬁxed points, • Variable Beta that determines the kind of ﬁxed points, • Dimension of the associative space for controlling the storage capacity, • Static Patterns for ﬁxed pattern search, and • Pattern Normalization to control the ﬁxed point dynamics by norm of the patterns and shift of the patterns.
Revised: • Association of two sets , • Multiple Updates for precise ﬁxed points, • Variable Beta that determines the kind of ﬁxed points, • Dimension of the associative space for controlling the storage capacity, • Static Patterns for ﬁxed pattern search, and • Pattern Normalization to control the ﬁxed point dynamics by norm of the patterns and shift of the patterns.

Original: A functional sketch of our Hopﬁeld layer is shown in Fig.
Revised: A functional sketch of our Hopﬁeld layer is shown in Fig.

Original: A.7.
Revised: A.7.

Original: • Association of two sets.
Revised: • Association of two sets.

Original: The Hopﬁeld layer makes it possible to associate two sets of vectors.
Revised: The Hopﬁeld layer makes it possible to associate two sets of vectors.

Original: This general functionality allows • for transformer-like self-attention, • for decoder-encoder attention, • for time series prediction (maybe with positional encoding), • for sequence analysis, • for multiple instance learning, • for learning with point sets, • for combining data sources by associations, • for constructing a memory, • for averaging and pooling operations, and • for many more.
Revised: This general functionality allows • for transformer-like self-attention, • for decoder-encoder attention, • for time series prediction (maybe with positional encoding), • for sequence analysis, • for multiple instance learning, • for learning with point sets, • for combining data sources by associations, • for constructing a memory, • for averaging and pooling operations, and • for many more.

Original: The ﬁrst set of vectors consists of S raw state patterns R =
Revised: The ﬁrst set of vectors consists of S raw state patterns R =

Original: ( r 1 , . .
Revised: ( r 1 , . .

Original: , r S ) T with r s ∈ R d r and the second set of vectors consists of N raw stored patterns Y
Revised: , r S ) T with r s ∈ R d r and the second set of vectors consists of N raw stored patterns Y

Original: =
Revised: =

Original: ( y 1 , . . .
Revised: ( y 1 , . . .

Original: , y N ) T with y i ∈
Revised: , y N ) T with y i ∈

Original: R d y .
Revised: R d y .

Original: Both the S raw state patterns and N raw stored patterns are mapped to an associative space in R d k via the matrices W Q ∈ R d r × d k and W K ∈ R d y × d k , respectively.
Revised: Both the S raw state patterns and N raw stored patterns are mapped to an associative space in R d k via the matrices W Q ∈ R d r × d k and W K ∈ R d y × d k , respectively.

Original: We deﬁne a matrix Q ( Ξ T ) of state patterns ξ n = W Q r n in an associative space R d k and a matrix K ( X T ) of stored patterns x i =
Revised: We deﬁne a matrix Q ( Ξ T ) of state patterns ξ n = W Q r n in an associative space R d k and a matrix K ( X T ) of stored patterns x i =

Original: W K y s in the associative space R d
Revised: W K y s in the associative space R d

Original: k :
Revised: k :

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: In the main paper, Eq.
Revised: In the main paper, Eq.

Original: (3) deﬁnes the novel update rule:
Revised: (3) deﬁnes the novel update rule:

Original: [Equation]
Revised: [Equation]

Original: For multiple patterns, Eq.
Revised: For multiple patterns, Eq.

Original: (3) becomes:
Revised: (3) becomes:

Original: [Equation]
Revised: [Equation]

Original: where Ξ = ( ξ 1 , . . . , ξ N ) is the matrix of N state (query) patterns, X is the matrix of stored (key) patterns, and Ξ new is the matrix of new state patterns, which are averages over stored patterns.
Revised: where Ξ = ( ξ 1 , . . . , ξ N ) is the matrix of N state (query) patterns, X is the matrix of stored (key) patterns, and Ξ new is the matrix of new state patterns, which are averages over stored patterns.

Original: A new state pattern can also be very similar to a single stored pattern, in which case we call the stored pattern to be retrieved.
Revised: A new state pattern can also be very similar to a single stored pattern, in which case we call the stored pattern to be retrieved.

Original: These matrices allow to rewrite Eq.
Revised: These matrices allow to rewrite Eq.

Original: (552) as:
Revised: (552) as:

Original: [Equation]
Revised: [Equation]

Original: For β = 1 / √ d k and changing in Eq.
Revised: For β = 1 / √ d k and changing in Eq.

Original: (553) softmax ∈ R N to a row vector (and evaluating a row vector), we obtain:
Revised: (553) softmax ∈ R N to a row vector (and evaluating a row vector), we obtain:

Original: [Equation]
Revised: [Equation]

Original: where Q new is again the matrix of new state patterns.
Revised: where Q new is again the matrix of new state patterns.

Original: The new state patterns Ξ new are projected via W V to the result patterns Z =
Revised: The new state patterns Ξ new are projected via W V to the result patterns Z =

Original: Ξ new W V , where W V ∈ R
Revised: Ξ new W V , where W V ∈ R

Original: d
Revised: d

Original: k × d v .
Revised: k × d v .

Original: With the pattern projection
Revised: With the pattern projection

Original: V = KW V , we obtain the update rule Eq.
Revised: V = KW V , we obtain the update rule Eq.

Original: (10) from the main paper:
Revised: (10) from the main paper:

Original: [Equation]
Revised: [Equation]

Original: • Multiple Updates.
Revised: • Multiple Updates.

Original: The update Eq.
Revised: The update Eq.

Original: (553) can be iteratively applied to the initial state ξ of every Hopﬁeld layer head.
Revised: (553) can be iteratively applied to the initial state ξ of every Hopﬁeld layer head.

Original: After the last update, the new states Ξ new are projected via W V to the result patterns Z =
Revised: After the last update, the new states Ξ new are projected via W V to the result patterns Z =

Original: Ξ new W V .
Revised: Ξ new W V .

Original: Therefore, the Hopﬁeld layer allows multiple update steps in the forward pass without changing the number of parameters.
Revised: Therefore, the Hopﬁeld layer allows multiple update steps in the forward pass without changing the number of parameters.

Original: The number of update steps can be given for every Hopﬁeld head individually.
Revised: The number of update steps can be given for every Hopﬁeld head individually.

Original: Furthermore, it is possible to set a threshold for the number of updates of every Hopﬁeld head based on (cid:107) ξ − ξ new (cid:107) 2 .
Revised: Furthermore, it is possible to set a threshold for the number of updates of every Hopﬁeld head based on (cid:107) ξ − ξ new (cid:107) 2 .

Original: In the general case of multiple initial states Ξ , the maximum over the individual norms is taken.
Revised: In the general case of multiple initial states Ξ , the maximum over the individual norms is taken.

Original: • Variable β .
Revised: • Variable β .

Original: In the main paper, we have identiﬁed β as a crucial parameter for the ﬁxed point dynamics of the Hopﬁeld network, which governs the operating mode of the attention heads.
Revised: In the main paper, we have identiﬁed β as a crucial parameter for the ﬁxed point dynamics of the Hopﬁeld network, which governs the operating mode of the attention heads.

Original: In appendix, e.g. in Lemma A7 or in Eq.
Revised: In appendix, e.g. in Lemma A7 or in Eq.

Original: (102) and Eq.
Revised: (102) and Eq.

Original: , we
Revised: , we

Original: showed that
Revised: showed that

Original: the characteristics of the ﬁxed points of the new modern Hopﬁeld network are determined by: β , M (maximal pattern norm), m max (spread of the similar patterns)
Revised: the characteristics of the ﬁxed points of the new modern Hopﬁeld network are determined by: β , M (maximal pattern norm), m max (spread of the similar patterns)

Original: and (cid:107) m x (cid:107) (center of the similar patterns).
Revised: and (cid:107) m x (cid:107) (center of the similar patterns).

Original: Low values of β induce global averaging and higher values of β metastable states.
Revised: Low values of β induce global averaging and higher values of β metastable states.

Original: In the transformer attention, the β parameter is set to β = 1 / √ d k as in Eq.
Revised: In the transformer attention, the β parameter is set to β = 1 / √ d k as in Eq.

Original: The Hopﬁeld layer, however, allows to freely choose β > 0 , since the ﬁxed point dynamics does not only depend on the dimension of the associative space d k .
Revised: The Hopﬁeld layer, however, allows to freely choose β > 0 , since the ﬁxed point dynamics does not only depend on the dimension of the associative space d k .

Original: Additionally, β heavily inﬂuences the gradient ﬂow to the matrices W Q and W K .
Revised: Additionally, β heavily inﬂuences the gradient ﬂow to the matrices W Q and W K .

Original: Thus, ﬁnding the right β for the respective application can be crucial.
Revised: Thus, ﬁnding the right β for the respective application can be crucial.

Original: • Variable dimension of the associative space.
Revised: • Variable dimension of the associative space.

Original: Theorem A5 says that the storage capacity of the modern Hopﬁeld network grows exponentially with the dimension of the associative space.
Revised: Theorem A5 says that the storage capacity of the modern Hopﬁeld network grows exponentially with the dimension of the associative space.

Original: However higher dimension of the associative space also means less averaging and smaller metastable states.
Revised: However higher dimension of the associative space also means less averaging and smaller metastable states.

Original: The dimension of the associative space trades off storage capacity against the size of metastable states, e.g. over how many pattern is averaged.
Revised: The dimension of the associative space trades off storage capacity against the size of metastable states, e.g. over how many pattern is averaged.

Original: In Eq.
Revised: In Eq.

Original: (550) and in Eq.
Revised: (550) and in Eq.

Original: (549), we assumed N raw state patterns R =
Revised: (549), we assumed N raw state patterns R =

Original: ( r 1 , . .
Revised: ( r 1 , . .

Original: , r N ) T and S raw stored patterns Y =
Revised: , r N ) T and S raw stored patterns Y =

Original: ( y 1 , . . .
Revised: ( y 1 , . . .

Original: , y S ) T that are mapped to a d k -dimensional associative space via the matrices W Q ∈ R d r × d k and W K ∈ R d
Revised: , y S ) T that are mapped to a d k -dimensional associative space via the matrices W Q ∈ R d r × d k and W K ∈ R d

Original: y × d k , respectively.
Revised: y × d k , respectively.

Original: In the associative space R d k , we obtain the state patterns Q =
Revised: In the associative space R d k , we obtain the state patterns Q =

Original: Ξ T = RW Q and the stored patterns
Revised: Ξ T = RW Q and the stored patterns

Original: K = X T =
Revised: K = X T =

Original: Y W K .
Revised: Y W K .

Original: The Hopﬁeld view relates the dimension d k to the number of input patterns
Revised: The Hopﬁeld view relates the dimension d k to the number of input patterns

Original: N that have to be processed.
Revised: N that have to be processed.

Original: The storage capacity depends exponentially on the dimension
Revised: The storage capacity depends exponentially on the dimension

Original: d k (the dimension of the associative space) and the size to metastable states is governed by this dimension, too.
Revised: d k (the dimension of the associative space) and the size to metastable states is governed by this dimension, too.

Original: Consequently, d k should be chosen with
Revised: Consequently, d k should be chosen with

Original: respect to the number N of patterns one wants to store and the desired size of metastable states, which is the number of patterns one wants to average over.
Revised: respect to the number N of patterns one wants to store and the desired size of metastable states, which is the number of patterns one wants to average over.

Original: For example, if the input consists of many low dimensional input patterns, it makes sense to project the patterns into a higher dimensional space to allow a proper ﬁxed point dynamics.
Revised: For example, if the input consists of many low dimensional input patterns, it makes sense to project the patterns into a higher dimensional space to allow a proper ﬁxed point dynamics.

Original: Intuitively, this coincides with the construction of a richer feature space for the patterns.
Revised: Intuitively, this coincides with the construction of a richer feature space for the patterns.

Original: • Static Patterns.
Revised: • Static Patterns.

Original: In Eq.
Revised: In Eq.

Original: (550) and Eq.
Revised: (550) and Eq.

Original: (549), the N raw state patterns R =
Revised: (549), the N raw state patterns R =

Original: ( r 1 , . .
Revised: ( r 1 , . .

Original: , r N ) T and S raw stored patterns Y =
Revised: , r N ) T and S raw stored patterns Y =

Original: ( y 1 , . . .
Revised: ( y 1 , . . .

Original: , y S ) T are mapped to an associative space via the matrices W Q ∈ R d r × d k and W K ∈ R d y × d k , which gives the state patterns Q = Ξ T = RW Q and the stored patterns K = X T =
Revised: , y S ) T are mapped to an associative space via the matrices W Q ∈ R d r × d k and W K ∈ R d y × d k , which gives the state patterns Q = Ξ T = RW Q and the stored patterns K = X T =

Original: Y W K .
Revised: Y W K .

Original: We allow for static state and static stored patterns.
Revised: We allow for static state and static stored patterns.

Original: Static pattern means that the pattern does not depend on the network input, i.e. it is determined by the bias weights and remains constant across different network inputs.
Revised: Static pattern means that the pattern does not depend on the network input, i.e. it is determined by the bias weights and remains constant across different network inputs.

Original: Static state patterns allow to determine whether
Revised: Static state patterns allow to determine whether

Original: particular ﬁxed patterns are among the stored patterns and vice versa.
Revised: particular ﬁxed patterns are among the stored patterns and vice versa.

Original: The static pattern functionality is typically needed if particular patterns must be identiﬁed in the data, e.g. as described for immune repertoire classiﬁcation in the main paper, where a ﬁxed d k -dimensional state vector ξ is used
Revised: The static pattern functionality is typically needed if particular patterns must be identiﬁed in the data, e.g. as described for immune repertoire classiﬁcation in the main paper, where a ﬁxed d k -dimensional state vector ξ is used

Original: Pattern
Revised: Pattern

Original: Normalization.
Revised: Normalization.

Original: In the appendix, e.g. in Lemma A7 or in Eq.
Revised: In the appendix, e.g. in Lemma A7 or in Eq.

Original: (102) and Eq.
Revised: (102) and Eq.

Original: we showed that the characteristics of the ﬁxed points of the new modern Hopﬁeld network are determined by: β , M (maximal pattern norm), m max (spread of the similar patterns), and (cid:107) m x (cid:107) (center of the similar patterns).
Revised: we showed that the characteristics of the ﬁxed points of the new modern Hopﬁeld network are determined by: β , M (maximal pattern norm), m max (spread of the similar patterns), and (cid:107) m x (cid:107) (center of the similar patterns).

Original: We already discussed the parameter β while the spread of the similar patterns m max is given by the data.
Revised: We already discussed the parameter β while the spread of the similar patterns m max is given by the data.

Original: The remaining variables M and m x that both control the ﬁxed point dynamics are adjusted pattern normalization.
Revised: The remaining variables M and m x that both control the ﬁxed point dynamics are adjusted pattern normalization.

Original: M is the maximal pattern norm and m x the center of the similar patterns.
Revised: M is the maximal pattern norm and m x the center of the similar patterns.

Original: Theorem A5 says that larger M allows for more patterns to be stored.
Revised: Theorem A5 says that larger M allows for more patterns to be stored.

Original: However, the size of metastable states will decrease with increasing M .
Revised: However, the size of metastable states will decrease with increasing M .

Original: The vector m x says how well the (similar) patterns are centered.
Revised: The vector m x says how well the (similar) patterns are centered.

Original: If the norm (cid:107) m x (cid:107) is large, then this leads to smaller metastable states.
Revised: If the norm (cid:107) m x (cid:107) is large, then this leads to smaller metastable states.

Original: The two parameters M and m x are controlled by pattern normalization and determine the size and convergence properties of metastable states.
Revised: The two parameters M and m x are controlled by pattern normalization and determine the size and convergence properties of metastable states.

Original: These two parameters are important for creating large gradients if heads start with global averaging which has small gradient.
Revised: These two parameters are important for creating large gradients if heads start with global averaging which has small gradient.

Original: These two parameters can shift a head towards small metastable states which have largest gradient as shown in Fig.
Revised: These two parameters can shift a head towards small metastable states which have largest gradient as shown in Fig.

Original: A.5(b).
Revised: A.5(b).

Original: We allow for three different pattern normalizations:
Revised: We allow for three different pattern normalizations:

Original: • pattern normalization of the input patterns, • pattern normalization after mapping into the associative space, • no pattern normalization.
Revised: • pattern normalization of the input patterns, • pattern normalization after mapping into the associative space, • no pattern normalization.

Original: The default setting is a pattern normalization of the input patterns.
Revised: The default setting is a pattern normalization of the input patterns.

Original: A.6. U SAGE
Revised: A.6. U SAGE

Original: As outlined in Sec.
Revised: As outlined in Sec.

Original: A.6.1, there are a variety of possible use cases for the Hopﬁeld layer, e.g. to build memory networks or transformer models.
Revised: A.6.1, there are a variety of possible use cases for the Hopﬁeld layer, e.g. to build memory networks or transformer models.

Original: The goal of the implementation is therefore to provide an easy to use Hopﬁeld module that can be used in a wide range of applications, be it as part of a larger architecture or as a standalone module.
Revised: The goal of the implementation is therefore to provide an easy to use Hopﬁeld module that can be used in a wide range of applications, be it as part of a larger architecture or as a standalone module.

Original: Consequently, the focus of the Hopﬁeld layer interface is set on its core parameters: the association of two sets, the scaling parameter β , the maximum number of updates, the dimension of the associative space, the possible usage of static patterns, and the pattern normalization.
Revised: Consequently, the focus of the Hopﬁeld layer interface is set on its core parameters: the association of two sets, the scaling parameter β , the maximum number of updates, the dimension of the associative space, the possible usage of static patterns, and the pattern normalization.

Original: The integration into the PyTorch framework is built such that with all the above functionalities disabled, the “HopﬁeldEncoderLayer” and the “HopﬁeldDecoderLayer”, both extensions of the Hopﬁeld module, can be used as a one-to-one plug-in replacement for the TransformerEncoderLayer and the TransformerDecoderLayer , respectively, of the PyTorch transformer module.
Revised: The integration into the PyTorch framework is built such that with all the above functionalities disabled, the “HopﬁeldEncoderLayer” and the “HopﬁeldDecoderLayer”, both extensions of the Hopﬁeld module, can be used as a one-to-one plug-in replacement for the TransformerEncoderLayer and the TransformerDecoderLayer , respectively, of the PyTorch transformer module.

Original: The Hopﬁeld layer can be used to implement or to substitute different layers:
Revised: The Hopﬁeld layer can be used to implement or to substitute different layers:

Original: • Pooling layers: We consider the Hopﬁeld layer as a pooling layer if only one static state (query) pattern exists.
Revised: • Pooling layers: We consider the Hopﬁeld layer as a pooling layer if only one static state (query) pattern exists.

Original: Then, it is de facto a pooling over the sequence, which results from the softmax values applied on the stored patterns.
Revised: Then, it is de facto a pooling over the sequence, which results from the softmax values applied on the stored patterns.

Original: Therefore, our Hopﬁeld layer can act as a pooling layer.
Revised: Therefore, our Hopﬁeld layer can act as a pooling layer.

Original: • Permutation equivariant layers: Our Hopﬁeld layer can be used as a plug-in replacement for permutation equivariant layers.
Revised: • Permutation equivariant layers: Our Hopﬁeld layer can be used as a plug-in replacement for permutation equivariant layers.

Original: Since the Hopﬁeld layer is an associative memory it assumes no dependency between the input patterns.
Revised: Since the Hopﬁeld layer is an associative memory it assumes no dependency between the input patterns.

Original: • GRU & LSTM layers: Our Hopﬁeld layer can be used as a plug-in replacement for GRU & LSTM layers.
Revised: • GRU & LSTM layers: Our Hopﬁeld layer can be used as a plug-in replacement for GRU & LSTM layers.

Original: Optionally, for substituting GRU & LSTM layers, positional encoding might be considered.
Revised: Optionally, for substituting GRU & LSTM layers, positional encoding might be considered.

Original: • Attention layers: Our Hopﬁeld layer can act as an attention layer, where state (query) and stored (key) patterns are different, and need to be associated.
Revised: • Attention layers: Our Hopﬁeld layer can act as an attention layer, where state (query) and stored (key) patterns are different, and need to be associated.

Original: • Finally, the extensions of the Hopﬁeld layer are able to operate as a self-attention layer (HopﬁeldEncoderLayer) and as cross-attention layer (HopﬁeldDecoderLayer), as described in (Vaswani et al., 2017a).
Revised: • Finally, the extensions of the Hopﬁeld layer are able to operate as a self-attention layer (HopﬁeldEncoderLayer) and as cross-attention layer (HopﬁeldDecoderLayer), as described in (Vaswani et al., 2017a).

Original: As such, it can be used as building block of transformer-based or general architectures
Revised: As such, it can be used as building block of transformer-based or general architectures

Original: MatMul
Revised: MatMul

Original: [Figure]
Revised: [Figure]

Original: Softmax
Revised: Softmax

Original: Mask
Revised: Mask

Original: [Figure]
Revised: [Figure]

Original: MatMul
Revised: MatMul

Original: [Figure]
Revised: [Figure]

Original: multipleupdates
Revised: multipleupdates

Original: [Figure] [Figure]
Revised: [Figure] [Figure]