Original: The integration into the PyTorch framework is built such that with all the above functionalities disabled, the “HopﬁeldEncoderLayer” and the “HopﬁeldDecoderLayer”, both extensions of the Hopﬁeld module, can be used as a one-to-one plug-in replacement for the TransformerEncoderLayer and the TransformerDecoderLayer , respectively, of the PyTorch transformer module.
Revised: The integration into the PyTorch framework is built such that with all the above functionalities disabled, the “HopﬁeldEncoderLayer” and the “HopﬁeldDecoderLayer”, both extensions of the Hopﬁeld module, can be used as a one-to-one plug-in replacement for the TransformerEncoderLayer and the TransformerDecoderLayer , respectively, of the PyTorch transformer module.

Original: The Hopﬁeld layer can be used to implement or to substitute different layers:
Revised: The Hopﬁeld layer can be used to implement or to substitute different layers:

Original: • Pooling layers: We consider the Hopﬁeld layer as a pooling layer if only one static state (query) pattern exists.
Revised: • Pooling layers: We consider the Hopﬁeld layer as a pooling layer if only one static state (query) pattern exists.

Original: Then, it is de facto a pooling over the sequence, which results from the softmax values applied on the stored patterns.
Revised: Then, it is de facto a pooling over the sequence, which results from the softmax values applied on the stored patterns.

Original: Therefore, our Hopﬁeld layer can act as a pooling layer.
Revised: Therefore, our Hopﬁeld layer can act as a pooling layer.

Original: • Permutation equivariant layers: Our Hopﬁeld layer can be used as a plug-in replacement for permutation equivariant layers.
Revised: • Permutation equivariant layers: Our Hopﬁeld layer can be used as a plug-in replacement for permutation equivariant layers.

Original: Since the Hopﬁeld layer is an associative memory it assumes no dependency between the input patterns.
Revised: Since the Hopﬁeld layer is an associative memory it assumes no dependency between the input patterns.

Original: • GRU & LSTM layers: Our Hopﬁeld layer can be used as a plug-in replacement for GRU & LSTM layers.
Revised: • GRU & LSTM layers: Our Hopﬁeld layer can be used as a plug-in replacement for GRU & LSTM layers.

Original: Optionally, for substituting GRU & LSTM layers, positional encoding might be considered.
Revised: Optionally, for substituting GRU & LSTM layers, positional encoding might be considered.

Original: • Attention layers: Our Hopﬁeld layer can act as an attention layer, where state (query) and stored (key) patterns are different, and need to be associated.
Revised: • Attention layers: Our Hopﬁeld layer can act as an attention layer, where state (query) and stored (key) patterns are different, and need to be associated.

Original: • Finally, the extensions of the Hopﬁeld layer are able to operate as a self-attention layer (HopﬁeldEncoderLayer) and as cross-attention layer (HopﬁeldDecoderLayer), as described in (Vaswani et al., 2017a).
Revised: • Finally, the extensions of the Hopﬁeld layer are able to operate as a self-attention layer (HopﬁeldEncoderLayer) and as cross-attention layer (HopﬁeldDecoderLayer), as described in (Vaswani et al., 2017a).

Original: As such, it can be used as building block of transformer-based or general architectures. 
Revised: As such, it can be used as building block of transformer-based or general architectures. MatMul Q K

Revised: Scale

Revised: Softmax

Revised: Mask

Revised: [Figure]

Revised: MatMul

Revised: MatMul

Revised: [Figure]

Revised: [Figure]