Original: Hopﬁeldnetworks are energy-based, binary associative memories, which popularized artiﬁcial neural networks[Figure] [Figure] in the 1980s (Hopﬁeld, 1982, 1984).
Revised: Hopﬁeld networks are energy-based, binary associative memories, which popularized artiﬁcial neural networks in the 1980s (Hopﬁeld, 1982, 1984).

Original: Associative memory networks have been designed to store andretrieve samples.
Revised: Associative memory networks have been designed to store and retrieve samples.

Original: Their storage capacity can be considerably increased by polynomial terms in theenergy function (Chen et al., 1986; Psaltis & Cheol, 1986; Baldi & Venkatesh, 1987; Gardner, 1987;
Revised: Their storage capacity can be considerably increased by polynomial terms in the energy function (Chen et al., 1986; Psaltis & Cheol, 1986; Baldi & Venkatesh, 1987; Gardner, 1987;

Original: Abbott & Arian, 1987; Horn & Usher, 1988; Caputo & Niemann, 2002; Krotov & Hopﬁeld, 2016).
Revised: Abbott & Arian, 1987; Horn & Usher, 1988; Caputo & Niemann, 2002; Krotov & Hopﬁeld, 2016).

Original: In contrast to these binary memory networks, we use continuous associative memory networks withvery high storage capacity.
Revised: In contrast to these binary memory networks, we use continuous associative memory networks with very high storage capacity.

Original: These modern Hopﬁeld networks for deep learning architectures have anenergy function with continuous states and can retrieve samples with only one update (Ramsauer et al., 
Revised: These modern Hopﬁeld networks for deep learning architectures have an energy function with continuous states and can retrieve samples with only one update (Ramsauer et al., 2021).

Original: Modern Hopﬁeld Networks have been successfully applied to immune repertoire classiﬁcation(Widrich et al., 2020) and chemical reaction prediction (Seidl et al., 2021).
Revised: Modern Hopﬁeld Networks have been successfully applied to immune repertoire classiﬁcation (Widrich et al., 2020) and chemical reaction prediction (Seidl et al., 2021).

Original: We assume a set of patterns { u 1 , . . .
Revised: We assume a set of patterns { u 1 , . . .

Original: , u N } ⊂ R d that are stacked as columns to the matrix U =( u 1 , . . .
Revised: , u N } ⊂ R d that are stacked as columns to the matrix U = ( u 1 , . . .

Original: , u N ) and a state pattern (query) ξ ∈ R d that represents the current state.
Revised: , u N ) and a state pattern (query) ξ ∈ R d that represents the current state.

Original: The largest normof a stored pattern is M = max i (cid:107) u i (cid:107) .
Revised: The largest norm of a stored pattern is M = max i (cid:107) u i (cid:107) .

Original: Continuous modern Hopﬁeld networks with state ξ have theenergy
Revised: Continuous modern Hopﬁeld networks with state ξ have the energy

Original: [Equation]
Revised: [Equation]

Original: For energy E and state ξ , the update rule
Revised: For energy E and state ξ , the update rule

Original: [Equation]
Revised: [Equation]

Original: has been proven to converge globally to stationary points of the energy E , which are almost alwayslocal minima (Ramsauer et al., 2021).
Revised: has been proven to converge globally to stationary points of the energy E , which are almost always local minima (Ramsauer et al., 2021).

Original: The update rule Eq.
Revised: The update rule Eq.

Original: (A179) is also the formula of the wellknown transformer attention mechanism (Ramsauer et al., 2021), therefore Hopﬁeld retrieval andtransformer attention coincide.
Revised: (A179) is also the formula of the wellknown transformer attention mechanism (Ramsauer et al., 2021), therefore Hopﬁeld retrieval and transformer attention coincide.

Original: The separation ∆ i of a pattern u i is deﬁned as its minimal dot product difference to any of theother patterns:
Revised: The separation ∆ i of a pattern u i is deﬁned as its minimal dot product difference to any of the other patterns:

Original: ∆ i = min j,j (cid:54) = i (cid:0)
Revised: ∆ i = min j,j (cid:54) = i (cid:0)

Original: u Ti u i − u Ti u j (cid:1) .
Revised: u Ti u i − u Ti u j (cid:1) .

Original: A pattern is well-separated from the data if∆ i ≥ 2 βN + 1 β log (cid:0) 2( N − 1) NβM 2 (cid:1) .
Revised: A pattern is well-separated from the data if ∆ i ≥ 2 βN + 1 β log (cid:0) 2( N − 1) NβM 2 (cid:1) .

Original: If the patterns u i are well separated, the iterate Eq.
Revised: If the patterns u i are well separated, the iterate Eq.

Original: (A179)converges to a ﬁxed point close to a stored pattern.
Revised: (A179) converges to a ﬁxed point close to a stored pattern.

Original: If some patterns are similar to one another and,therefore, not well separated, the update rule Eq.
Revised: If some patterns are similar to one another and, therefore, not well separated, the update rule Eq.

Original: (A179) converges to a ﬁxed point close to the meanof the similar patterns.
Revised: (A179) converges to a ﬁxed point close to the mean of the similar patterns.

Original: This ﬁxed point is a metastable state of the energy function and averages oversimilar patterns.
Revised: This ﬁxed point is a metastable state of the energy function and averages over similar patterns.

Original: The next theorem states that the update rule Eq.
Revised: The next theorem states that the update rule Eq.

Original: (A179) typically converges after one update if thepatterns are well separated.
Revised: (A179) typically converges after one update if the patterns are well separated.

Original: Furthermore, it states that the retrieval error is exponentially small in theseparation ∆ i .
Revised: Furthermore, it states that the retrieval error is exponentially small in the separation ∆ i .

Original: Theorem A4 (Modern Hopﬁeld Networks: Retrieval with One Update) .
Revised: Theorem A4 (Modern Hopﬁeld Networks: Retrieval with One Update) .

Original: With query ξ , after oneupdate the distance of the new point f ( ξ ) to the ﬁxed point u ∗ i is exponentially small in the separation∆ i .
Revised: With query ξ , after one update the distance of the new point f ( ξ ) to the ﬁxed point u ∗ i is exponentially small in the separation ∆ i .

Original: The precise bounds using the Jacobian J = ∂f ( ξ ) ∂ ξ and its value J m in the mean value theoremare:
Revised: The precise bounds using the Jacobian J = ∂f ( ξ ) ∂ ξ and its value J m in the mean value theorem are:

Original: [Equation]
Revised: [Equation]

Original: For given (cid:15) and sufﬁcient large ∆ i , we have (cid:107) f ( ξ ) − u ∗
Revised: For given (cid:15) and sufﬁcient large ∆ i , we have (cid:107) f ( ξ ) − u ∗

Original: i (cid:107) < (cid:15) , that is, retrieval with one update.
Revised: i (cid:107) < (cid:15) , that is, retrieval with one update.

Original: The retrieval error (cid:107) f ( ξ ) − u i (cid:107) of pattern u i is bounded by
Revised: The retrieval error (cid:107) f ( ξ ) − u i (cid:107) of pattern u i is bounded by

Original: [Equation]
Revised: [Equation]

Original: (A182) The main requirement of modern Hopﬁeld networks to be suited for contrastive learning is that theycan store and retrieve enough embeddings if the batch size is large.
Revised: The main requirement of modern Hopﬁeld networks to be suited for contrastive learning is that they can store and retrieve enough embeddings if the batch size is large.

Original: We want to store a potentiallylarge set of embeddings.
Revised: We want to store a potentially large set of embeddings.

Original: We ﬁrst deﬁne what we mean by storing and retrieving patterns from amodern Hopﬁeld network.
Revised: We ﬁrst deﬁne what we mean by storing and retrieving patterns from a modern Hopﬁeld network.

Original: Deﬁnition A1 (Pattern Stored and Retrieved) .
Revised: Deﬁnition A1 (Pattern Stored and Retrieved) .

Original: We assume that around every pattern u i a sphere
Revised: We assume that around every pattern u i a sphere

Original: S i is given.
Revised: S i is given.

Original: We say u i is stored if there is a single ﬁxed point u ∗ i ∈ S
Revised: We say u i is stored if there is a single ﬁxed point u ∗ i ∈ S

Original: i to which all points ξ ∈ S iconverge, and S i ∩ S j = ∅ for i (cid:54) =
Revised: i to which all points ξ ∈ S i converge, and S i ∩ S j = ∅ for i (cid:54) =

Original: j .
Revised: j .

Original: We say u i is retrieved for a given (cid:15) if iteration (update rule)
Revised: We say u i is retrieved for a given (cid:15) if iteration (update rule)

Original: Eq.
Revised: Eq.

Original: (A179) gives a point ˜
Revised: (A179) gives a point ˜

Original: x i that is at least (cid:15) -close to the single ﬁxed point u ∗ i ∈ S i .
Revised: x i that is at least (cid:15) -close to the single ﬁxed point u ∗ i ∈ S i .

Original: The retrievalerror is (cid:107) ˜
Revised: The retrieval error is (cid:107) ˜

Original: x
Revised: x

Original: i − u
Revised: i − u

Original: i (cid:107) .
Revised: i (cid:107) .

Original: As with classical Hopﬁeld networks, we consider patterns on the sphere, i.e. patterns with a ﬁxednorm.
Revised: As with classical Hopﬁeld networks, we consider patterns on the sphere, i.e. patterns with a ﬁxed norm.

Original: For randomly chosen patterns, the number of patterns that can be stored is exponential in thedimension d of the space of the patterns ( u i ∈ R d ).
Revised: For randomly chosen patterns, the number of patterns that can be stored is exponential in the dimension d of the space of the patterns ( u i ∈ R d ).

Original: Theorem A5 (Modern Hopﬁeld Networks: Exponential Storage Capacity) .
Revised: Theorem A5 (Modern Hopﬁeld Networks: Exponential Storage Capacity) .

Original: We assume a failureprobability 0 < p (cid:54) 1 and randomly chosen patterns on the sphere with radius M :=
Revised: We assume a failure probability 0 < p (cid:54) 1 and randomly chosen patterns on the sphere with radius M :=

Original: K √ d − 1 .
Revised: K √ d − 1 .

Original: Wedeﬁne a := 2 d − 1 (1 + ln(2 βK 2 p ( d − 1))) , b := 2 K 2 β 5 , and c :=
Revised: We deﬁne a := 2 d − 1 (1 + ln(2 βK 2 p ( d − 1))) , b := 2 K 2 β 5 , and c :=

Original: b W 0 (exp( a +ln( b )) , where W 0 is theupper branch of the Lambert W function (Olver et al., 2010, (4.13) ) , and ensure c ≥ (cid:16) 2 √ p (cid:17) 4 d − 1 .
Revised: b W 0 (exp( a +ln( b )) , where W 0 is the upper branch of the Lambert W function (Olver et al., 2010, (4.13) ) , and ensure c ≥ (cid:16) 2 √ p (cid:17) 4 d − 1 .

Original: Then with probability 1 − p , the number of random patterns that can be stored is
Revised: Then with probability 1 − p , the number of random patterns that can be stored is

Original: [Equation]
Revised: [Equation]

Original: Therefore it is proven for c ≥ 3 .
Revised: Therefore it is proven for c ≥ 3 .

Original: 1546 with β = 1 , K = 3 , d = 20 and p = 0 .
Revised: 1546 with β = 1 , K = 3 , d = 20 and p = 0 .

Original: 001 ( a + ln( b ) > 1 . 27 )and proven for c ≥ 1 . 3718 with β = 1 , K = 1 , d = 75 , and p = 0 .
Revised: 001 ( a + ln( b ) > 1 . 27 ) and proven for c ≥ 1 . 3718 with β = 1 , K = 1 , d = 75 , and p = 0 .

Original: 001 ( a + ln( b ) < − 0 . 94 ).
Revised: 001 ( a + ln( b ) < − 0 . 94 ).

Original: For a proof see (Ramsauer et al., 2021).
Revised: For a proof see (Ramsauer et al., 2021).

Original: This theorem justiﬁes to use continuous modern Hopﬁeld networks for using retrieved embeddingsinstead of the original embeddings for large batch sizes.
Revised: This theorem justiﬁes to use continuous modern Hopﬁeld networks for using retrieved embeddings instead of the original embeddings for large batch sizes.

Original: Even for hundreds of thousands of embeddings, the continuous modern Hopﬁeld network is able to retrieve the embeddings if the dimension ofthe embeddings is large enough.
Revised: Even for hundreds of thousands of embeddings, the continuous modern Hopﬁeld network is able to retrieve the embeddings if the dimension of the embeddings is large enough.

Original: A.4 Further Related Work
Revised: A.4 Further Related Work

Original: With the advent of large corpora of unlabeled data in vision and language, self-supervised learningvia contrastive learning has become highly successful.
Revised: With the advent of large corpora of unlabeled data in vision and language, self-supervised learning via contrastive learning has become highly successful.

Original: Some contrastive learning objectives, such asthose of BYOL (Grill et al., 2020) and SimSiam (Chen & He, 2021), do not require negative samples.
Revised: Some contrastive learning objectives, such as those of BYOL (Grill et al., 2020) and SimSiam (Chen & He, 2021), do not require negative samples.

Original: However, the most popular objective for contrastive learning is InfoNCE (van den Oord et al., 2018),in which for an anchor sample, a positive sample is contrasted with negative samples.
Revised: However, the most popular objective for contrastive learning is InfoNCE (van den Oord et al., 2018), in which for an anchor sample, a positive sample is contrasted with negative samples.

Original: The idea to use objectives with negative samples is well known in deep learning (Gutmann & Hyvärinen, 2010; Chen et al., 2017; Mikolov et al., 2013).
Revised: The idea to use objectives with negative samples is well known in deep learning (Gutmann & Hyvärinen, 2010; Chen et al., 2017; Mikolov et al., 2013).

Original: For contrastive learning, the most successfulobjective is InfoNCE, which has been introduced as Contrastive Predictive Coding (CPC) (van den
Revised: For contrastive learning, the most successful objective is InfoNCE, which has been introduced as Contrastive Predictive Coding (CPC) (van den

Original: Oord et al., 2018).
Revised: Oord et al., 2018).

Original: InfoNCE has been applied to transfer learning (Hénaff et al., 2019), to naturallanguage response suggestion (Henderson et al., 2017), to learning sentence representations fromunlabelled data (Logeswaran & Lee, 2018), and to unsupervised feature learning by maximizingdistinctions between instances (Wu et al., 2018).
Revised: InfoNCE has been applied to transfer learning (Hénaff et al., 2019), to natural language response suggestion (Henderson et al., 2017), to learning sentence representations from unlabelled data (Logeswaran & Lee, 2018), and to unsupervised feature learning by maximizing distinctions between instances (Wu et al., 2018).

Original: InfoNCE has been used for learning visual representations in Pretext-Invariant Representation Learning (PIRL) (Misra & vanDerMaaten, 2020), in
Revised: InfoNCE has been used for learning visual representations in Pretext-Invariant Representation Learning (PIRL) (Misra & vanDerMaaten, 2020), in

Original: Momentum Contrast (MoCo) (He et al., 2020), and in SimCLR (Chen et al., 2020).
Revised: Momentum Contrast (MoCo) (He et al., 2020), and in SimCLR (Chen et al., 2020).

Original: SimCLR becamewell known as is was highly effective for transfer learning.
Revised: SimCLR became well known as is was highly effective for transfer learning.

Original: Zero-shot transfer learning (Lampertet al., 2009) is one of the most ambitious goals in vision, since it would improve various real-worlddownstream applications.
Revised: Zero-shot transfer learning (Lampert et al., 2009) is one of the most ambitious goals in vision, since it would improve various real-world downstream applications.

Original: Current models in natural language processing and vision perform verywell on standard benchmarks, but they fail at new data, new applications, deployments in the wild,and stress tests (D’Amour et al., 2020; Recht et al., 2019; Taori et al., 2020; Lapuschkin et al., 2019;
Revised: Current models in natural language processing and vision perform very well on standard benchmarks, but they fail at new data, new applications, deployments in the wild, and stress tests (D’Amour et al., 2020; Recht et al., 2019; Taori et al., 2020; Lapuschkin et al., 2019;

Original: Geirhos et al., 2020).
Revised: Geirhos et al., 2020).

Original: A model with high zero-shot transfer learning performance will not fail on suchdata, therefore will be trusted by practitioners.
Revised: A model with high zero-shot transfer learning performance will not fail on such data, therefore will be trusted by practitioners.

Original: Multiple works have proposed improvements to InfoNCE.
Revised: Multiple works have proposed improvements to InfoNCE.

Original: Joint Contrastive Learning (JCL) studiesthe effect of sampling multiple positives for each anchor.
Revised: Joint Contrastive Learning (JCL) studies the effect of sampling multiple positives for each anchor.

Original: (Cai et al., 2020).
Revised: (Cai et al., 2020).

Original: Sampling negativesaround each positive leads to higher bias but lower variance than InfoNCE (Wu et al., 2021).
Revised: Sampling negatives around each positive leads to higher bias but lower variance than InfoNCE (Wu et al., 2021).

Original: InfoNCEhas been generalized to C-InfoNCE and WeaC-InfoNCE, which are conditional contrastive learningapproaches to remove undesirable information in self-supervised representations (Tsai et al., 2021).
Revised: InfoNCE has been generalized to C-InfoNCE and WeaC-InfoNCE, which are conditional contrastive learning approaches to remove undesirable information in self-supervised representations (Tsai et al., 2021).

Original: ProtoNCE is a generalized version of the InfoNCE, which pushes representations to be closer totheir assigned prototypes (Li et al., 2021).
Revised: ProtoNCE is a generalized version of the InfoNCE, which pushes representations to be closer to their assigned prototypes (Li et al., 2021).

Original: ProtoNCE combines contrastive learning with clustering.
Revised: ProtoNCE combines contrastive learning with clustering.

Original: SimCSE employs InfoNCE for contrastive learning to learn sentence embeddings (Gao et al., 2021).
Revised: SimCSE employs InfoNCE for contrastive learning to learn sentence embeddings (Gao et al., 2021).

Original: InfoNCE has been extended to video representation learning (Han et al., 2020).
Revised: InfoNCE has been extended to video representation learning (Han et al., 2020).

Original: CLOOB uses InfoLOOB, which is an upper bound on the mutual information.
Revised: CLOOB uses InfoLOOB, which is an upper bound on the mutual information.

Original: An alternative upperbound on the mutual information would be Contrastive Log-ratio Upper Bound (CLUB), whichwas used for minimizing the mutual information (Cheng et al., 2020).
Revised: An alternative upper bound on the mutual information would be Contrastive Log-ratio Upper Bound (CLUB), which was used for minimizing the mutual information (Cheng et al., 2020).

Original: So far CLUB was only usedfor minimizing the mutual information, except for the analysis in (Wang & Liu, 2021).
Revised: So far CLUB was only used for minimizing the mutual information, except for the analysis in (Wang & Liu, 2021).

Original: In ourexperiments, maximizing CLUB failed as conﬁrmed in (Wang & Liu, 2021).
Revised: In our experiments, maximizing CLUB failed as conﬁrmed in (Wang & Liu, 2021).

Original: The reason is that theembedding distribution is not uniform as required for successful contrastive learning (Wang & Isola,2020; Wang & Liu, 2021).
Revised: The reason is that the embedding distribution is not uniform as required for successful contrastive learning (Wang & Isola, 2020; Wang & Liu, 2021).

Original: Many follow up works have been based on the CLIP model.
Revised: Many follow up works have been based on the CLIP model.

Original: The CLIP model is used in Vision-andLanguage tasks (Shen et al., 2021).
Revised: The CLIP model is used in Vision-andLanguage tasks (Shen et al., 2021).

Original: The CLIP model guided generative models via an additionaltraining objective (Bau et al., 2021;
Revised: The CLIP model guided generative models via an additional training objective (Bau et al., 2021;

Original: Galatolo et al., 2021; Frans et al., 2021) and improved clustering oflatent representations (Pakhomov et al., 2021).
Revised: Galatolo et al., 2021; Frans et al., 2021) and improved clustering of latent representations (Pakhomov et al., 2021).

Original: It is used in studies of out of distribution performance(Devillers et al., 2021; Milbich et al., 2021; Miller et al., 2021), of ﬁne-tuning robustness (Wortsmanet al., 2021), of zero-shot prompts (Zhou et al., 2021) and of adversarial attacks to uncurated datasets(Carlini & Terzis, 2021).
Revised: It is used in studies of out of distribution performance (Devillers et al., 2021; Milbich et al., 2021; Miller et al., 2021), of ﬁne-tuning robustness (Wortsman et al., 2021), of zero-shot prompts (Zhou et al., 2021) and of adversarial attacks to uncurated datasets (Carlini & Terzis, 2021).

Original: It stirred discussions about more holistic evaluation schemes in computervision (Agarwal et al., 2021).
Revised: It stirred discussions about more holistic evaluation schemes in computer vision (Agarwal et al., 2021).

Original: Multiple methods utilize the CLIP model in a straightforward way toperform text-to-video retrieval (Fang et al., 2021; Luo et al., 2021; Narasimhan et al., 2021).
Revised: Multiple methods utilize the CLIP model in a straightforward way to perform text-to-video retrieval (Fang et al., 2021; Luo et al., 2021; Narasimhan et al., 2021).

Original: [Figure]
Revised: [Figure]

Original: HorseRiding
Revised: HorseRiding

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: DodgeCaliberWagon
Revised: DodgeCaliberWagon

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: horizontalstripenoentry
Revised: horizontalstripenoentry

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: mosque
Revised: mosque

Original: [Figure]
Revised: [Figure]

Original: cottontailrabbit
Revised: cottontailrabbit

Original: [Figure]
Revised: [Figure]

Original: correctrank:
Revised: correctrank:

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: ChryslerAspenSUV correctrank:
Revised: ChryslerAspenSUV correctrank:

Original: [Figure]
Revised: [Figure]

Original: redandwhitecirclered truckandblackcarnopassing correctrank:
Revised: redandwhitecirclered truckandblackcarnopassing correctrank:

Original: [Figure]
Revised: [Figure]

Original: magnolia correctrank:
Revised: magnolia correctrank:

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Greece
Revised: Greece

Original: [Figure]
Revised: [Figure]

Original: GreatBlueHeron
Revised: GreatBlueHeron

Original: [Figure]
Revised: [Figure]

Original: threshingmachine
Revised: threshingmachine

Original: [Figure]
Revised: [Figure]

Original: longhornbeetle
Revised: longhornbeetle

Original: [Figure]
Revised: [Figure]