Original: To elaborate on the memory and time consumption of CLOOB, Table A9 compares the experimentsdone in Section 5.1.

Original: The memory consumption of CLOOB is the same as CLIP since only embeddings from the mini-batch are used both in the objective and in the Hopﬁeld memories.

Original: The timeconsumption is approximately 5% higher, which is because of the retrieval of the modern Hopﬁeldnetworks.

Original: The added complexity of modern Hopﬁeld networks is O ( N ) per sample, where N denotesthe batch size.

Original: [Table]

Original: A.3 Review of Modern Hopﬁeld Networks
Revised: A.3 Review of Modern Hopﬁeld Networks

Original: We brieﬂy review continuous modern Hopﬁeld networks that are used for deep learning architectures.
Revised: We brieﬂy review continuous modern Hopﬁeld networks that are used for deep learning architectures.

Original: They are continuous and differentiable, therefore they a work with gradient descent in deep architectures.
Revised: They are continuous and differentiable, therefore they a work with gradient descent in deep architectures.

Original: They retrieve with one update only, therefore they can be activated like other deep learninglayers.
Revised: They retrieve with one update only, therefore they can be activated like other deep learninglayers.

Original: They have exponential storage capacity, therefore they can tackle large problems.
Revised: They have exponential storage capacity, therefore they can tackle large problems.

Original: Hopﬁeldnetworks are energy-based, binary associative memories, which popularized artiﬁcial neural networks
Revised: Hopﬁeld

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: in the 1980s (Hopﬁeld, 1982, 1984).
Revised: networks are energy-based, binary associative memories, which popularized artiﬁcial neural networksin the 1980s (Hopﬁeld, 1982, 1984).

Original: Associative memory networks have been designed to store andretrieve samples.
Revised: Associative memory networks have been designed to store andretrieve samples.

Original: Their storage capacity can be considerably increased by polynomial terms in theenergy function (Chen et al., 1986; Psaltis & Cheol, 1986; Baldi & Venkatesh, 1987; Gardner, 1987;
Revised: Their storage capacity can be considerably increased by polynomial terms in theenergy function (Chen et al., 1986; Psaltis & Cheol, 1986; Baldi & Venkatesh, 1987; Gardner, 1987;

Original: Abbott & Arian, 1987; Horn & Usher, 1988; Caputo & Niemann, 2002; Krotov & Hopﬁeld, 2016).
Revised: Abbott & Arian, 1987; Horn & Usher, 1988; Caputo & Niemann, 2002; Krotov & Hopﬁeld, 2016).

Original: In contrast to these binary memory networks, we use continuous associative memory networks withvery high storage capacity.
Revised: In contrast to these binary memory networks, we use continuous associative memory networks withvery high storage capacity.

Original: These modern Hopﬁeld networks for deep learning architectures have anenergy function with continuous states and can retrieve samples with only one update (Ramsauer et al.,
Revised: These modern Hopﬁeld networks for deep learning architectures have anenergy function with continuous states and can retrieve samples with only one update (Ramsauer et al.,

Original: Modern Hopﬁeld Networks have been successfully applied to immune repertoire classiﬁcation(Widrich et al., 2020) and chemical reaction prediction (Seidl et al., 2021).
Revised: Modern Hopﬁeld Networks have been successfully applied to immune repertoire classiﬁcation(Widrich et al., 2020) and chemical reaction prediction (Seidl et al., 2021).

Original: We assume a set of patterns { u 1 , . . .
Revised: We assume a set of patterns { u 1 , . . .

Original: , u N } ⊂ R d that are stacked as columns to the matrix U =( u 1 , . . .
Revised: , u N } ⊂ R d that are stacked as columns to the matrix U =( u 1 , . . .

Original: , u N ) and a state pattern (query) ξ ∈ R d that represents the current state.
Revised: , u N ) and a state pattern (query) ξ ∈ R d that represents the current state.

Original: The largest normof a stored pattern is M = max i (cid:107) u i (cid:107) .
Revised: The largest normof a stored pattern is M = max i (cid:107) u i (cid:107) .

Original: Continuous modern Hopﬁeld networks with state ξ have theenergy
Revised: Continuous modern Hopﬁeld networks with state ξ have theenergy

Original: [Equation]
Revised: [Equation]

Original: For energy E and state ξ , the update rule
Revised: For energy E and state ξ , the update rule

Original: [Equation]
Revised: [Equation]

Original: has been proven to converge globally to stationary points of the energy E , which are almost alwayslocal minima (Ramsauer et al., 2021).
Revised: has been proven to converge globally to stationary points of the energy E , which are almost alwayslocal minima (Ramsauer et al., 2021).

Original: The update rule Eq.
Revised: The update rule Eq.

Original: (A179) is also the formula of the wellknown transformer attention mechanism (Ramsauer et al., 2021), therefore Hopﬁeld retrieval andtransformer attention coincide.
Revised: (A179) is also the formula of the wellknown transformer attention mechanism (Ramsauer et al., 2021), therefore Hopﬁeld retrieval andtransformer attention coincide.

Original: The separation ∆ i of a pattern u i is deﬁned as its minimal dot product difference to any of theother patterns:
Revised: The separation ∆ i of a pattern u i is deﬁned as its minimal dot product difference to any of theother patterns:

Original: ∆ i = min j,j (cid:54) = i (cid:0)
Revised: ∆ i = min j,j (cid:54) = i (cid:0)

Original: u Ti u i − u Ti u j (cid:1) .
Revised: u Ti u i − u Ti u j (cid:1) .

Original: A pattern is well-separated from the data
Revised: A pattern is well-separated from the data

Original: if∆ i ≥ 2 βN + 1 β log (cid:0) 2( N − 1)
Revised: if∆ i ≥ 2 βN + 1 β log (cid:0) 2( N − 1)

Original: NβM 2 (cid:1) .
Revised: NβM 2 (cid:1) .

Original: If the patterns u i are well separated, the iterate Eq.
Revised: If the patterns u i are well separated, the iterate Eq.

Original: (A179)converges to a ﬁxed point close to a stored pattern.
Revised: (A179)converges to a ﬁxed point close to a stored pattern.

Original: If some patterns are similar to one another and,therefore, not well separated, the update rule Eq.
Revised: If some patterns are similar to one another and,therefore, not well separated, the update rule Eq.

Original: (A179) converges to a ﬁxed point close to the meanof the similar patterns.
Revised: (A179) converges to a ﬁxed point close to the meanof the similar patterns.

Original: This ﬁxed point is a metastable state of the energy function and averages oversimilar patterns.
Revised: This ﬁxed point is a metastable state of the energy function and averages oversimilar patterns.

Original: The next theorem states that the update rule Eq.
Revised: The next theorem states that the update rule Eq.

Original: (A179) typically converges after one update if thepatterns are well separated.
Revised: (A179) typically converges after one update if thepatterns are well separated.

Original: Furthermore, it states that the retrieval error is exponentially small in theseparation ∆ i .
Revised: Furthermore, it states that the retrieval error is exponentially small in theseparation ∆ i .

Original: Theorem A4 (Modern Hopﬁeld Networks: Retrieval with One Update) .
Revised: Theorem A4 (Modern Hopﬁeld Networks: Retrieval with One Update) .

Original: With query ξ , after oneupdate the distance of the new point f ( ξ ) to the ﬁxed point u ∗ i is exponentially small in the separation∆ i .
Revised: With query ξ , after oneupdate the distance of the new point f ( ξ ) to the ﬁxed point u ∗ i is exponentially small in the separation∆ i .

Original: The precise bounds using the Jacobian J = ∂f ( ξ ) ∂ ξ and its value J m in the mean value theoremare:
Revised: The precise bounds using the Jacobian J = ∂f ( ξ ) ∂ ξ and its value J m in the mean value theoremare:

Original: [Equation]
Revised: [Equation]

Original: For given (cid:15) and sufﬁcient large ∆ i , we have (cid:107) f ( ξ ) − u ∗
Revised: For given (cid:15) and sufﬁcient large ∆ i , we have (cid:107) f ( ξ ) − u ∗

Original: i (cid:107) < (cid:15) , that is, retrieval with one update.
Revised: i (cid:107) < (cid:15) , that is, retrieval with one update.

Original: The retrieval error (cid:107) f ( ξ ) − u i (cid:107) of pattern u i is bounded by
Revised: The retrieval error (cid:107) f ( ξ ) − u i (cid:107) of pattern u i is bounded by

Original: [Equation]
Revised: [Equation]

Original: (A182)
Revised: A182)

Revised: For proof see ( et al., 2021).

Original: The main requirement of modern Hopﬁeld networks to be suited for contrastive learning is that theycan store and retrieve enough embeddings if the batch size is large.
Revised: The main requirement of modern Hopﬁeld networks to be suited for contrastive learning is that theycan store and retrieve enough embeddings if the batch size is large.

Original: We want to store a potentiallylarge set of embeddings.
Revised: We want to store a potentiallylarge set of embeddings.

Original: We ﬁrst deﬁne what we mean by storing and retrieving patterns from amodern Hopﬁeld network.
Revised: We ﬁrst deﬁne what we mean by storing and retrieving patterns from amodern Hopﬁeld network.

Original: Deﬁnition A1 (Pattern Stored and Retrieved) .
Revised: Deﬁnition A1 (Pattern Stored and Retrieved) .

Original: We assume that around every pattern u i a sphere
Revised: We assume that around every pattern u i a sphere

Original: S i is given.
Revised: S i is given.

Original: We say u i is stored if there is a single ﬁxed point u ∗ i ∈ S
Revised: We say u i is stored if there is a single ﬁxed point u ∗ i ∈ S

Original: i to which all points ξ ∈ S iconverge, and S i ∩ S j = ∅ for i (cid:54) =
Revised: i to which all points ξ ∈ S iconverge, and S i ∩ S j = ∅ for i (cid:54) =

Original: j .
Revised: j .

Original: We say u i is retrieved for a given (cid:15) if iteration (update rule)
Revised: We say u i is retrieved for a given (cid:15) if iteration (update rule)

Original: Eq.
Revised: Eq.

Original: (A179) gives a point ˜
Revised: (A179) gives a point ˜

Original: x i that is at least (cid:15) -close to the single ﬁxed point u ∗ i ∈ S i .
Revised: x i that is at least (cid:15) -close to the single ﬁxed point u ∗ i ∈ S i .

Original: The retrievalerror is (cid:107) ˜
Revised: The retrievalerror is (cid:107) ˜

Original: x
Revised: x

Original: i − u
Revised: i − u

Original: i (cid:107) .
Revised: i (cid:107) .

Original: As with classical Hopﬁeld networks, we consider patterns on the sphere, i.e. patterns with a ﬁxednorm.
Revised: As with classical Hopﬁeld networks, we consider patterns on the sphere, i.e. patterns with a ﬁxednorm.

Original: For randomly chosen patterns, the number of patterns that can be stored is exponential in thedimension d of the space of the patterns ( u i ∈ R d ).
Revised: For randomly chosen patterns, the number of patterns that can be stored is exponential in thedimension d of the space of the patterns ( u i ∈ R d ).

Original: Theorem A5 (Modern Hopﬁeld Networks: Exponential Storage Capacity) .
Revised: Theorem A5 (Modern Hopﬁeld Networks: Exponential Storage Capacity) .

Original: We assume a failureprobability 0 < p (cid:54) 1 and randomly chosen patterns on the sphere with radius M :=
Revised: We assume a failureprobability 0 < p (cid:54) 1 and randomly chosen patterns on the sphere with radius M :=

Original: K √ d − 1 .
Revised: K √ d − 1 .

Original: Wedeﬁne a := 2 d − 1 (1 + ln(2 βK 2 p ( d − 1))) , b := 2 K 2 β 5 , and c :=
Revised: Wedeﬁne a := 2 d − 1 (1 + ln(2 βK 2 p ( d − 1))) , b := 2 K 2 β 5 , and c :=

Original: b W 0 (exp( a +ln( b )) , where W 0 is theupper branch of the Lambert W function (Olver et al., 2010, (4.13) ) , and ensure c ≥ (cid:16) 2 √ p (cid:17) 4 d − 1 .
Revised: b W 0 (exp( a +ln( b )) , where W 0 is theupper branch of the Lambert W function (Olver et al., 2010, (4.13) ) , and ensure c ≥ (cid:16) 2 √ p (cid:17) 4 d − 1 .

Original: Then with probability 1 − p , the number of random patterns that can be stored is
Revised: Then with probability 1 − p , the number of random patterns that can be stored is

Original: [Equation]
Revised: [Equation]

Original: Therefore it is proven for c ≥ 3 .
Revised: Therefore it is proven for c ≥ 3 .

Original: 1546 with β = 1 , K = 3 , d = 20 and p = 0 .
Revised: 1546 with β = 1 , K = 3 , d = 20 and p = 0 .

Original: 001 ( a + ln( b ) > 1 . 27 )and proven for c ≥ 1 . 3718 with β = 1 , K = 1 , d = 75 , and p = 0 .
Revised: 001 ( a + ln( b ) > 1 . 27 )and proven for c ≥ 1 . 3718 with β = 1 , K = 1 , d = 75 , and p = 0 .

Original: 001 ( a + ln( b ) < − 0 . 94 ).
Revised: 001 ( a + ln( b ) < − 0 . 94 ).

Original: For a proof see (Ramsauer et al., 2021).
Revised: For a proof see (Ramsauer et al., 2021).

Original: This theorem justiﬁes to use continuous modern Hopﬁeld networks for using retrieved embeddingsinstead of the original embeddings for large batch sizes.
Revised: This theorem justiﬁes to use continuous modern Hopﬁeld networks for using retrieved embeddingsinstead of the original embeddings for large batch sizes.

Original: Even for hundreds of thousands of embeddings, the continuous modern Hopﬁeld network is able to retrieve the embeddings if the dimension ofthe embeddings is large enough.
Revised: Even for hundreds of thousands of embeddings, the continuous modern Hopﬁeld network is able to retrieve the embeddings if the dimension ofthe embeddings is large enough.

Original: A.4 Further Related Work
Revised: A.4 Further Related Work

Original: With the advent of large corpora of unlabeled data in vision and language, self-supervised learningvia contrastive learning has become highly successful.
Revised: With the advent of large corpora of unlabeled data in vision and language, self-supervised learningvia contrastive learning has become highly successful.

Original: Some contrastive learning objectives, such asthose of BYOL (Grill et al., 2020) and SimSiam (Chen & He, 2021), do not require negative samples.
Revised: Some contrastive learning objectives, such asthose of BYOL (Grill et al., 2020) and SimSiam (Chen & He, 2021), do not require negative samples.

Original: However, the most popular objective for contrastive learning is InfoNCE (van den Oord et al., 2018),in which for an anchor sample, a positive sample is contrasted with negative samples.
Revised: However, the most popular objective for contrastive learning is InfoNCE (van den Oord et al., 2018),in which for an anchor sample, a positive sample is contrasted with negative samples.

Original: The idea to use objectives with negative samples is well known in deep learning (Gutmann & Hyvärinen, 2010; Chen et al., 2017; Mikolov et al., 2013).
Revised: The idea to use objectives with negative samples is well known in deep learning (Gutmann & Hyvärinen, 2010; Chen et al., 2017; Mikolov et al., 2013).

Original: For contrastive learning, the most successfulobjective is InfoNCE, which has been introduced as Contrastive Predictive Coding (CPC) (van den
Revised: For contrastive learning, the most successfulobjective is InfoNCE, which has been introduced as Contrastive Predictive Coding (CPC) (van den

Original: Oord et al., 2018).
Revised: Oord et al., 2018).

Original: InfoNCE has been applied to transfer learning (Hénaff et al., 2019), to naturallanguage response suggestion (Henderson et al., 2017), to learning sentence representations fromunlabelled data (Logeswaran & Lee, 2018), and to unsupervised feature learning by maximizingdistinctions between instances (Wu et al., 2018).
Revised: InfoNCE has been applied to transfer learning (Hénaff et al., 2019), to naturallanguage response suggestion (Henderson et al., 2017), to learning sentence representations fromunlabelled data (Logeswaran & Lee, 2018), and to unsupervised feature learning by maximizingdistinctions between instances (Wu et al., 2018).

Original: InfoNCE has been used for learning visual representations in Pretext-Invariant Representation Learning (PIRL) (Misra & vanDerMaaten, 2020), in
Revised: InfoNCE has been used for learning visual representations in Pretext-Invariant Representation Learning (PIRL) (Misra & vanDerMaaten, 2020), in

Original: Momentum Contrast (MoCo) (He et al., 2020), and in SimCLR (Chen et al., 2020).
Revised: Momentum Contrast (MoCo) (He et al., 2020), and in SimCLR (Chen et al., 2020).

Original: SimCLR becamewell known as is was highly effective for transfer learning.
Revised: SimCLR becamewell known as is was highly effective for transfer learning.

Original: Zero-shot transfer learning (Lampertet al., 2009) is one of the most ambitious goals in vision, since it would improve various real-worlddownstream applications.
Revised: Zero-shot transfer learning (Lampertet al., 2009) is one of the most ambitious goals in vision, since it would improve various real-worlddownstream applications.

Original: Current models in natural language processing and vision perform verywell on standard benchmarks, but they fail at new data, new applications, deployments in the wild,and stress tests (D’Amour et al., 2020; Recht et al., 2019; Taori et al., 2020; Lapuschkin et al., 2019;
Revised: Current models in natural language processing and vision perform verywell on standard benchmarks, but they fail at new data, new applications, deployments in the wild,and stress tests (D’Amour et al., 2020; Recht et al., 2019; Taori et al., 2020; Lapuschkin et al., 2019;

Original: Geirhos et al., 2020).
Revised: Geirhos et al., 2020).

Original: A model with high zero-shot transfer learning performance will not fail on suchdata, therefore will be trusted by practitioners.
Revised: A model with high zero-shot transfer learning performance will not fail on suchdata, therefore will be trusted by practitioners.

Original: Multiple works have proposed improvements to InfoNCE.
Revised: Multiple works have proposed improvements to InfoNCE.

Original: Joint Contrastive Learning (JCL) studiesthe effect of sampling multiple positives for each anchor.
Revised: Joint Contrastive Learning (JCL) studiesthe effect of sampling multiple positives for each anchor.

Original: (Cai et al., 2020).
Revised: (Cai et al., 2020).

Original: Sampling negativesaround each positive leads to higher bias but lower variance than InfoNCE (Wu et al., 2021).
Revised: Sampling negativesaround each positive leads to higher bias but lower variance than InfoNCE (Wu et al., 2021).

Original: InfoNCEhas been generalized to C-InfoNCE and WeaC-InfoNCE, which are conditional contrastive learningapproaches to remove undesirable information in self-supervised representations (Tsai et al., 2021).
Revised: InfoNCEhas been generalized to C-InfoNCE and WeaC-InfoNCE, which are conditional contrastive learningapproaches to remove undesirable information in self-supervised representations (Tsai et al., 2021).

Original: ProtoNCE is a generalized version of the InfoNCE, which pushes representations to be closer totheir assigned prototypes (Li et al., 2021).
Revised: ProtoNCE is a generalized version of the InfoNCE, which pushes representations to be closer totheir assigned prototypes (Li et al., 2021).

Original: ProtoNCE combines contrastive learning with clustering.
Revised: ProtoNCE combines contrastive learning with clustering.

Original: SimCSE employs InfoNCE for contrastive learning to learn sentence embeddings (Gao et al., 2021).
Revised: SimCSE employs InfoNCE for contrastive learning to learn sentence embeddings (Gao et al., 2021).

Original: InfoNCE has been extended to video representation learning (Han et al., 2020).
Revised: InfoNCE has been extended to video representation learning (Han et al., 2020).

Original: CLOOB uses InfoLOOB, which is an upper bound on the mutual information.
Revised: CLOOB uses InfoLOOB, which is an upper bound on the mutual information.

Original: An alternative upperbound on the mutual information would be Contrastive Log-ratio Upper Bound (CLUB), whichwas used for minimizing the mutual information (Cheng et al., 2020).
Revised: An alternative upperbound on the mutual information would be Contrastive Log-ratio Upper Bound (CLUB), whichwas used for minimizing the mutual information (Cheng et al., 2020).

Original: So far CLUB was only usedfor minimizing the mutual information, except for the analysis in (Wang & Liu, 2021).
Revised: So far CLUB was only usedfor minimizing the mutual information, except for the analysis in (Wang & Liu, 2021).

Original: In ourexperiments, maximizing CLUB failed as conﬁrmed in (Wang & Liu, 2021).
Revised: In ourexperiments, maximizing CLUB failed as conﬁrmed in (Wang & Liu, 2021).

Original: The reason is that theembedding distribution is not uniform as required for successful contrastive learning (Wang & Isola,2020; Wang & Liu, 2021).
Revised: The reason is that theembedding distribution is not uniform as required for successful contrastive learning (Wang & Isola,2020; Wang & Liu, 2021).

Original: Many follow up works have been based on the CLIP model.
Revised: Many follow up works have been based on the CLIP model.

Original: The CLIP model is used in Vision-andLanguage tasks (Shen et al., 2021).
Revised: The CLIP model is used in Vision-andLanguage tasks (Shen et al., 2021).

Original: The CLIP model guided generative models via an additionaltraining objective (Bau et al., 2021;
Revised: The CLIP model guided generative models via an additionaltraining objective (Bau et al., 2021;

Original: Galatolo et al., 2021; Frans et al., 2021) and improved clustering oflatent representations (Pakhomov et al., 2021).
Revised: Galatolo et al., 2021; Frans et al., 2021) and improved clustering oflatent representations (Pakhomov et al., 2021).

Original: It is used in studies of out of distribution performance(Devillers et al., 2021; Milbich et al., 2021; Miller et al., 2021), of ﬁne-tuning robustness (Wortsmanet al., 2021), of zero-shot prompts (Zhou et al., 2021) and of adversarial attacks to uncurated datasets(Carlini & Terzis, 2021).
Revised: It is used in studies of out of distribution performance(Devillers et al., 2021; Milbich et al., 2021; Miller et al., 2021), of ﬁne-tuning robustness (Wortsmanet al., 2021), of zero-shot prompts (Zhou et al., 2021) and of adversarial attacks to uncurated datasets(Carlini & Terzis, 2021).

Original: It stirred discussions about more holistic evaluation schemes in computervision (Agarwal et al., 2021).
Revised: It stirred discussions about more holistic evaluation schemes in computervision (Agarwal et al., 2021).

Original: Multiple methods utilize the CLIP model in a straightforward way toperform text-to-video retrieval (Fang et al., 2021; Luo et al., 2021; Narasimhan et al., 2021).
Revised: Multiple methods utilize the CLIP model in a straightforward way toperform text-to-video retrieval (Fang et al., 2021; Luo et al., 2021; Narasimhan et al., 2021).

Original: [Figure]
Revised: [Figure]

Original: HorseRiding
Revised: HorseRiding

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: DodgeCaliberWagon
Revised: DodgeCaliberWagon

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: horizontalstripenoentry
Revised: horizontalstripenoentry

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: mosque
Revised: mosque

Original: [Figure]
Revised: [Figure]

Original: cottontailrabbit
Revised: cottontailrabbit

Original: [Figure]
Revised: [Figure]

Original: correctrank:
Revised: correctrank:

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: ChryslerAspenSUV correctrank:
Revised: ChryslerAspenSUV correctrank:

Original: [Figure]
Revised: [Figure]

Original: redandwhitecirclered truckandblackcarnopassing correctrank:
Revised: redandwhitecirclered truckandblackcarnopassing correctrank:

Original: [Figure]
Revised: [Figure]

Original: magnolia correctrank:
Revised: magnolia correctrank:

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Greece
Revised: Greece

Original: [Figure]
Revised: [Figure]

Original: GreatBlueHeron
Revised: GreatBlueHeron

Original: [Figure]
Revised: [Figure]

Original: threshingmachine
Revised: threshingmachine

Original: [Figure]
Revised: [Figure]

Original: longhornbeetle
Revised: longhornbeetle

Original: [Figure]
Revised: [Figure]