Original: CLOOB: Modern Hopﬁeld Networks with InfoLOOB Outperform CLIP
Revised: CLOOB: Modern Hopﬁeld Networks with InfoLOOB Outperform CLIP

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: CLIP yielded impressive results on zero-shot transfer learning tasks and is considered as a foundation model like BERT or GPT3.
Revised: CLIP yielded impressive results on zero-shot transfer learning tasks and is considered as a foundation model like BERT or GPT3.

Original: CLIP vision models thathave a rich representation are pre-trained using the InfoNCE objective and naturallanguage supervision before they are ﬁne-tuned on particular tasks.
Revised: CLIP vision models thathave a rich representation are pre-trained using the InfoNCE objective and naturallanguage supervision before they are ﬁne-tuned on particular tasks.

Original: Though CLIPexcels at zero-shot transfer learning, it suffers from an explaining away problem,that is, it focuses on one or few features, while neglecting other relevant features.
Revised: Though CLIPexcels at zero-shot transfer learning, it suffers from an explaining away problem,that is, it focuses on one or few features, while neglecting other relevant features.

Original: This problem is caused by insufﬁciently extracting the covariance structure in theoriginal multi-modal data.
Revised: This problem is caused by insufﬁciently extracting the covariance structure in theoriginal multi-modal data.

Original: We suggest to use modern Hopﬁeld networks to tacklethe problem of explaining away.
Revised: We suggest to use modern Hopﬁeld networks to tacklethe problem of explaining away.

Original: Their retrieved embeddings have an enrichedcovariance structure derived from co-occurrences of features in the stored embeddings.
Revised: Their retrieved embeddings have an enrichedcovariance structure derived from co-occurrences of features in the stored embeddings.

Original: However, modern Hopﬁeld networks increase the saturation effect of the
Revised: However, modern Hopﬁeld networks increase the saturation effect of the

Original: InfoNCE objective which hampers learning.
Revised: InfoNCE objective which hampers learning.

Original: We propose to use the InfoLOOBobjective to mitigate this saturation effect.
Revised: We propose to use the InfoLOOBobjective to mitigate this saturation effect.

Original: We introduce the novel “Contrastive
Revised: We introduce the novel “Contrastive

Original: Leave One Out Boost” (CLOOB), which uses modern Hopﬁeld networks for covariance enrichment together with the InfoLOOB objective.
Revised: Leave One Out Boost” (CLOOB), which uses modern Hopﬁeld networks for covariance enrichment together with the InfoLOOB objective.

Original: In experiments wecompare CLOOB to CLIP after pre-training on the Conceptual Captions and the
Revised: In experiments wecompare CLOOB to CLIP after pre-training on the Conceptual Captions and the

Original: YFCC dataset with respect to their zero-shot transfer learning performance on otherdatasets.
Revised: YFCC dataset with respect to their zero-shot transfer learning performance on otherdatasets.

Original: CLOOB consistently outperforms CLIP at zero-shot transfer learningacross all considered architectures and datasets.
Revised: CLOOB consistently outperforms CLIP at zero-shot transfer learningacross all considered architectures and datasets.

Original:  Introduction
Revised:  Introduction

Original: Contrastive Language-Image Pre-training (CLIP) showed spectacular performances at zero-shottransfer learning (Radford et al., 2021).
Revised: Contrastive Language-Image Pre-training (CLIP) showed spectacular performances at zero-shottransfer learning (Radford et al., 2021).

Original: CLIP learns expressive image embeddings directly fromraw text, thereby leverages a much richer source of supervision than just labels.
Revised: CLIP learns expressive image embeddings directly fromraw text, thereby leverages a much richer source of supervision than just labels.

Original: The CLIP modelis considered as an important foundation model (Bommasani et al., 2021), therefore a plethora offollow-up work has been published (see Appendix Section A.4).
Revised: The CLIP modelis considered as an important foundation model (Bommasani et al., 2021), therefore a plethora offollow-up work has been published (see Appendix Section A.4).

Original: CLIP as a contrastive learningmethod has two simultaneous goals, namely (i) increasing the similarity of matched language-imagepairs and (ii) decreasing the similarity of unmatched language-image pairs.
Revised: CLIP as a contrastive learningmethod has two simultaneous goals, namely (i) increasing the similarity of matched language-imagepairs and (ii) decreasing the similarity of unmatched language-image pairs.

Original: Though CLIP yieldedstriking zero-shot transfer learning results, it still suffers from “explaining away”.
Revised: Though CLIP yieldedstriking zero-shot transfer learning results, it still suffers from “explaining away”.

Original: Explaining away isknown in reasoning as the concept that the conﬁrmation of one cause of an observed event dismissesalternative causes (Pearl, 1988; Wellman & Henrion, 1993).
Revised: Explaining away isknown in reasoning as the concept that the conﬁrmation of one cause of an observed event dismissesalternative causes (Pearl, 1988; Wellman & Henrion, 1993).

Original: CLIP’s explaining away problem isits focus on one or few features while neglecting other relevant features.
Revised: CLIP’s explaining away problem isits focus on one or few features while neglecting other relevant features.

Original: This problem is causedby insufﬁciently extracting feature co-occurrences and covariance structures in the original multimodal data.
Revised: This problem is causedby insufﬁciently extracting feature co-occurrences and covariance structures in the original multimodal data.

Original: Humans extract co-occurrences and covariances by associating current perceptions withmemories (Bonner & Epstein, 2021; Potter, 2012).
Revised: Humans extract co-occurrences and covariances by associating current perceptions withmemories (Bonner & Epstein, 2021; Potter, 2012).

Original: In analogy to these human cognitive processes,we suggest to use modern Hopﬁeld networks to amplify co-occurrences and covariance structures ofthe original data.
Revised: In analogy to these human cognitive processes,we suggest to use modern Hopﬁeld networks to amplify co-occurrences and covariance structures ofthe original data.

Original: Hopﬁeld networks are energy-based, binary associative memories, which popularized artiﬁcial neuralnetworks in the 1980s (Amari, 1972; Hopﬁeld, 1982, 1984).
Revised: Hopﬁeld networks are energy-based, binary associative memories, which popularized artiﬁcial neuralnetworks in the 1980s (Amari, 1972; Hopﬁeld, 1982, 1984).

Original: Associative memory networks havebeen designed to store and retrieve samples.
Revised: Associative memory networks havebeen designed to store and retrieve samples.

Original: Their storage capacity can be considerably increased bypolynomial terms in the energy function (Chen et al., 1986; Psaltis & Cheol, 1986; Baldi & Venkatesh,1987; Gardner, 1987; Abbott & Arian, 1987; Horn & Usher, 1988; Caputo & Niemann, 2002; Krotov& Hopﬁeld, 2016).
Revised: Their storage capacity can be considerably increased bypolynomial terms in the energy function (Chen et al., 1986; Psaltis & Cheol, 1986; Baldi & Venkatesh,1987; Gardner, 1987; Abbott & Arian, 1987; Horn & Usher, 1988; Caputo & Niemann, 2002; Krotov& Hopﬁeld, 2016).

Original: In contrast to these binary memory networks, we use continuous associativememory networks with very high storage capacity.
Revised: In contrast to these binary memory networks, we use continuous associativememory networks with very high storage capacity.

Original: These modern Hopﬁeld networks for deep learningarchitectures have an energy function with continuous states and can retrieve samples with only oneupdate (Ramsauer et al., 2021).
Revised: These modern Hopﬁeld networks for deep learningarchitectures have an energy function with continuous states and can retrieve samples with only oneupdate (Ramsauer et al., 2021).

Original: Modern Hopﬁeld networks have already been successfully appliedto immune repertoire classiﬁcation (Widrich et al., 2020), chemical reaction prediction (Seidl et al.
Revised: Modern Hopﬁeld networks have already been successfully appliedto immune repertoire classiﬁcation (Widrich et al., 2020), chemical reaction prediction (Seidl et al.

Original: ,2022) and reinforcement learning (Widrich et al., 2021).
Revised: ,2022) and reinforcement learning (Widrich et al., 2021).

Original: Modern Hopﬁeld networks are a novelconcept for contrastive learning to extract more covariance structure.
Revised: Modern Hopﬁeld networks are a novelconcept for contrastive learning to extract more covariance structure.

Original: However, modern Hopﬁeld networks lead to a higher similarity of retrieved samples.
Revised: However, modern Hopﬁeld networks lead to a higher similarity of retrieved samples.

Original: The increasedsimilarity exacerbates the saturation of CLIP’s InfoNCE objective (van den Oord et al., 2018).
Revised: The increasedsimilarity exacerbates the saturation of CLIP’s InfoNCE objective (van den Oord et al., 2018).

Original: InfoNCE saturates because it contains terms of the form a/ ( a + b ) .
Revised: InfoNCE saturates because it contains terms of the form a/ ( a + b ) .

Original: In analogy to Wang & Isola(2020), a is called the “alignment score” that measures the similarity of matched pairs and b is calledthe “uniformity penalty” that measures the similarity of unmatched pairs.
Revised: In analogy to Wang & Isola(2020), a is called the “alignment score” that measures the similarity of matched pairs and b is calledthe “uniformity penalty” that measures the similarity of unmatched pairs.

Original: The saturation problembecomes more severe for retrieved samples of the modern Hopﬁeld network since the alignment scorea increases.
Revised: The saturation problembecomes more severe for retrieved samples of the modern Hopﬁeld network since the alignment scorea increases.

Original: Saturation of InfoNCE hampers the decrease of the uniformity penalty b (see also (Yehet al., 2021)).
Revised: Saturation of InfoNCE hampers the decrease of the uniformity penalty b (see also (Yehet al., 2021)).

Original: Contrary to InfoNCE, the “InfoLOOB” (LOOB for “Leave One Out Bound”)
Revised: Contrary to InfoNCE, the “InfoLOOB” (LOOB for “Leave One Out Bound”)

Original: objective(Poole et al., 2019) contains only terms of the form a/b which do not saturate.
Revised: objective(Poole et al., 2019) contains only terms of the form a/b which do not saturate.

Original: Thus, even for a largealignment score a , learning still decreases the uniformity penalty b by distributing samples moreuniformly.
Revised: Thus, even for a largealignment score a , learning still decreases the uniformity penalty b by distributing samples moreuniformly.

Original: We introduce “Contrastive Leave One Out Boost” (CLOOB) which combines modern Hopﬁeldnetworks with the “InfoLOOB” objective.
Revised: We introduce “Contrastive Leave One Out Boost” (CLOOB) which combines modern Hopﬁeldnetworks with the “InfoLOOB” objective.

Original: CLOOB overcomes CLIP’s problem of “explaining away”by extracting more covariance structure from the original data.

Original: Our contributions are:
Revised: Our contributions are:

Original: (a) we propose CLOOB, a new contrastive learning method,
Revised: (a) we propose CLOOB, a new contrastive learning method,(b)

Original: (b) we propose modern Hopﬁeld networks to reinforce covariance structures,
Revised: we propose modern Hopﬁeld networks to reinforce covariance structures,(c)

Original: (c) we propose InfoLOOB as an objective to avoid saturation as observed with InfoNCE , and
Revised: we propose InfoLOOB as an objective to avoid saturation as observed with InfoNCE, and

Original: provide theoretical justiﬁcations for
Revised: provide theoretical justiﬁcations for

Original: optimizing InfoLOOB
Revised: optimizing InfoLOOB

Original:  CLOOB: Modern Hopﬁeld Networks with InfoLOOB
Revised:  CLOOB: Modern Hopﬁeld Networks with InfoLOOB

Original: Our novel  method CLOOB combines modern Hopﬁeld networks (see next section) with the InfoLOOB objective (see next but one section). Modern Hopﬁeld networks reinforce the covariancestructure in the data. InfoLOOB does not suffer from saturation as InfoNCE does.
Revised: Our novel contrastive learning method CLOOB can be seen as a replacement of CLIP and thereforebe used in any method which builds upon CLIP.

Original: Figure 1 sketchesthe CLOOB architecture for image-text pairs.
Revised: Figure 1 sketches the CLOOB architecture forimage-text pairs.

Original: The training set consists of N pairs of embeddings { ( x 1 , y 1 ) , . . .
Revised: The training set consists of N pairs of embeddings { ( x 1 , y 1 ) , . . .

Original: , ( x N , y N ) } with X =( x 1 , . . .
Revised: , ( x N , y N ) } with X =( x 1 , . . .

Original: , x N ) and Y =
Revised: , x N ) and Y =

Original: ( y 1 , . . .
Revised: ( y 1 , . . .

Original: , y N ) , M stored embeddings U
Revised: , y N ) , M stored embeddings U

Original: =
Revised: =

Original: ( u 1 , . . .
Revised: ( u 1 , . . .

Original: , u M ) , and Kstored embeddings V = ( v 1 , . .
Revised: , u M ) , and Kstored embeddings V = ( v 1 , . .

Original: . , v K ) .
Revised: . , v K ) .

Original: The state or query embeddings x i and y i retrieve U x
Revised: The state or query embeddings x i and y i retrieve U x

Original: iand
Revised: iand

Original: U
Revised: U

Original: y
Revised: y

Original: i , respectively, from U — analog for retrievals from V .
Revised: i , respectively, from U — analog for retrievals from V .

Original: The samples are normalized:(cid:107)
Revised: The samples are normalized:(cid:107)

Original: x
Revised: x

Original: i (cid:107) =
Revised: i (cid:107) =

Original: (cid:107) y
Revised: (cid:107) y

Original: i (cid:107) =
Revised: i (cid:107) =

Original: (cid:107) u
Revised: (cid:107) u

Original: i (cid:107) =
Revised: i (cid:107) =

Original: (cid:107) v i (cid:107) = 1 .
Revised: (cid:107) v i (cid:107) = 1 .

Original: U x
Revised: U x

Original: i denotes an image-retrieved image embedding,
Revised: i denotes an image-retrieved image embedding,

Original: U y
Revised: U y

Original: i atext-retrieved image embedding, V x i an image-retrieved text embedding and V y i a text-retrieved textembedding.
Revised: i atext-retrieved image embedding, V x i an image-retrieved text embedding and V y i a text-retrieved textembedding.

Original: These retrievals from modern Hopﬁeld networks are computed as follows (Ramsaueret al., 2021):
Revised: These retrievals from modern Hopﬁeld networks are computed as follows (Ramsaueret al., 2021):

Original: [Equation]
Revised: [Equation]

Original: The hyperparameter β corresponds to the inverse temperature: β = 0 retrieves the average of thestored pattern, while large β retrieves the stored pattern that is most similar to the state pattern (query).
Revised: The hyperparameter β corresponds to the inverse temperature: β = 0 retrieves the average of thestored pattern, while large β retrieves the stored pattern that is most similar to the state pattern (query).

Original: In the InfoLOOB loss Eq.
Revised: In the InfoLOOB loss Eq.

Original: (8), CLOOB substitutes the embedded samples x
Revised: (8), CLOOB substitutes the embedded samples x

Original: i and y i by the normalizedretrieved embedded samples.
Revised: i and y i by the normalizedretrieved embedded samples.

Original: In the ﬁrst term, x i and y i are substituted by U x i and U y i , respectively,while in the second term they are substituted by V x i and V y i .
Revised: In the ﬁrst term, x i and y i are substituted by U x i and U y i , respectively,while in the second term they are substituted by V x i and V y i .

Original: After retrieval, the samples arere-normalized to ensure (cid:107)
Revised: After retrieval, the samples arere-normalized to ensure (cid:107)

Original: U x i (cid:107)
Revised: U x i (cid:107)

Original: =
Revised: =

Original: (cid:107)
Revised: (cid:107)

Original: U
Revised: U

Original: y
Revised: y

Original: i
Revised: i

Original: (cid:107) =
Revised: (cid:107) =

Original: (cid:107) V x i
Revised: (cid:107) V x i

Original: (cid:107) =
Revised: (cid:107) =

Original: (cid:107) V y
Revised: (cid:107) V y

Original: i
Revised: i

Original: (cid:107) = 1 .
Revised: (cid:107) = 1 .

Original: xx 2 ··· x N
Revised: xx 2 ··· x N

Original: U x 1 U x 2 ··· U x N xx 2 ··· x N
Revised: U x 1 U x 2 ··· U x N xx 2 ··· x N

Original: V x 1 V x 2 ··· V x N yy 2 ··· y N
Revised: V x 1 V x 2 ··· V x N yy 2 ··· y N

Original: U y 1 U y 2 ··· U y N yy 2 ··· y N
Revised: U y 1 U y 2 ··· U y N yy 2 ··· y N

Original: V y 1 V y 2 ··· V y N
Revised: V y 1 V y 2 ··· V y N

Original: Hopﬁeld retrieval with U
Revised: Hopﬁeld retrieval with U

Original: Hopﬁeld retrieval with V
Revised: Hopﬁeld retrieval with V

Original: Hopﬁeld retrieval with U
Revised: Hopﬁeld retrieval with U

Original: Hopﬁeld retrieval with V (cid:39) (cid:39) (cid:39) (cid:39) (cid:39) (cid:39) imageencoder
Revised: Hopﬁeld retrieval with V (cid:39) (cid:39) (cid:39) (cid:39) (cid:39) (cid:39) imageencoder

Original: [Figure]
Revised: [Figure]

Original: dog is playing in the snow.
Revised: dog is playing in the snow.

Original: (cid:39) legend similarity to anchor positive sample negative sample
Revised: (cid:39) legend similarity to anchor positive sample negative sample

Original: We obtain the CLOOB loss function:
Revised: We obtain the CLOOB loss function:

Original: [Equation]
Revised: [Equation]

Original: By default, we store the minibatch in the modern Hopﬁeld networks, that is, U = X and V = Y .
Revised: By default, we store the minibatch in the modern Hopﬁeld networks, that is, U = X and V = Y .

Original: Thus, in Eq.
Revised: Thus, in Eq.

Original: (1) x i can retrieve itself from U =
Revised: (1) x i can retrieve itself from U =

Original: X , but in Eq.
Revised: X , but in Eq.

Original: (3) it can not retrieve itself from
Revised: (3) it can not retrieve itself from

Original: V = Y .
Revised: V = Y .

Original: Analogously, in Eq.
Revised: Analogously, in Eq.

Original: (4) y i can retrieve itself from V
Revised: (4) y i can retrieve itself from V

Original: = Y , but in Eq.
Revised: = Y , but in Eq.

Original: (2) it can not retrieveitself from U =
Revised: (2) it can not retrieveitself from U =

Original: X .
Revised: X .

Original: By storing the embeddings of the mini-batch examples in the Hopﬁeld memory,we do not require to compute the embeddings of additional samples via text and image encoders.
Revised: By storing the embeddings of the mini-batch examples in the Hopﬁeld memory,we do not require to compute the embeddings of additional samples via text and image encoders.

Original: However, the modern Hopﬁeld networks can also store prototypes, templates, or proprietary samplesto amplify particular embedding features via the stored samples.
Revised: However, the modern Hopﬁeld networks can also store prototypes, templates, or proprietary samplesto amplify particular embedding features via the stored samples.

Original: Either the original embeddings xand y or the retrieved embeddings U x , U y , V x , and V y may serve for the downstream tasks, e.g. forzero-shot transfer learning.
Revised: Either the original embeddings xand y or the retrieved embeddings U x , U y , V x , and V y may serve for the downstream tasks, e.g.for zero-shot transfer learning.

Original: Pseudocode 1 shows CLOOB in a PyTorch-like style.
Revised: Pseudocode 1 shows CLOOB in a PyTorch-like style.

Original: CLOOB has two major components: (i) modern Hopﬁeld networks that alleviate CLIP’s problemof insufﬁciently exploiting the covariance structure in the data and (ii) the InfoLOOB objective thatdoes not suffer from InfoNCE’s saturation problem.
Revised: CLOOB hastwo major components: (i) modern Hopﬁeld networks that alleviate CLIP’s problem of insufﬁcientlyexploiting the covariance structure in the data and (ii) the InfoLOOB objective that does not sufferfrom InfoNCE’s saturation problem.

Original: The next two sections analyze CLOOB’s majorcomponents.
Revised: The next two sections analyze CLOOB’s major components.

Original:  Modern Hopﬁeld Networks for Enriching the Covariance Structure
Revised:  Modern Hopﬁeld Networks for Enriching the Covariance Structure

Original: We use modern Hopﬁeld networks to amplify co-occurrences and the covariance structure.
Revised: We use modern Hopﬁeld networks to amplify co-occurrences and the covariance structure.

Original: Replacingthe original embeddings by retrieved embeddings reinforces features that frequently occur together instored embeddings.
Revised: Replacingthe original embeddings by retrieved embeddings reinforces features that frequently occur together instored embeddings.

Original: Additionally, spurious co-occurrences that are peculiar to a sample are averagedout.
Revised: Additionally, spurious co-occurrences that are peculiar to a sample are averagedout.

Original: By this means, the covariance structure is reinforced by the retrieved embeddings U T x
Revised: By this means, the covariance structure is reinforced by the retrieved embeddings U T x

Original: i U y iand V T x
Revised: i U y iand V T x

Original: i V y
Revised: i V y

Original: i .
Revised: i .

Original: The Jacobian J of the softmax p = softmax( β a ) is J( β a )
Revised: The Jacobian J of the softmax p = softmax( β a ) is J( β a )

Original: = β (cid:0) diag( p ) − pp T (cid:1) .
Revised: = β (cid:0) diag( p ) − pp T (cid:1) .

Original: We deﬁne the weighted covariance Cov( U ) , where sample u i is drawn with probability p i , as[Cov( U )]
Revised: We deﬁne the weighted covariance Cov( U ) , where sample u i is drawn with probability p i , as[Cov( U )]

Original: kl = (cid:2)
Revised: kl = (cid:2)

Original: U J( β a ) U T (cid:3) kl =
Revised: U J( β a ) U T (cid:3) kl =

Original: β ( (cid:80) Mi =1 p i u ik u il − (cid:80) Mi
Revised: β ( (cid:80) Mi =1 p i u ik u il − (cid:80) Mi

Original: =1 p i u
Revised: =1 p i u

Original: ik (cid:80) Mi
Revised: ik (cid:80) Mi

Original: =1 p i u il ) .
Revised: =1 p i u il ) .

Original: The formula of theweighted covariance differs from the standard empirical covariance, since the factor 1 /M is replacedby p i .
Revised: The formula of theweighted covariance differs from the standard empirical covariance, since the factor 1 /M is replacedby p i .

Original: Thus u i is sampled with probability p i insteadof being sampled uniformly with probability1 /M .
Revised: Thus, u i is sampled with probability p i instead with probability 1 /M (uniformly).

Original: [Table] We apply the mean value theorem to the softmax function with mean Jacobian matrix J m ( β a ) = 111 (cid:82)0 J( λβ a ) d λ .
Revised: We apply the mean value theorem to the softmax function with mean Jacobian matrix J m ( β a ) = 107 (cid:82)0 J( λβ a ) d λ .

Original: The mean Jacobian J m ( β a ) is a symmetric, diagonally dominant, positive semideﬁnite matrix with one eigenvalue of zero for eigenvector 1 and spectral norm bounded by (cid:107) J m (cid:107) 2 (cid:54)0 .
Revised: The mean Jacobian J m ( β a ) is a symmetric, diagonally dominant, positive semideﬁnite matrix with one eigenvalue of zero for eigenvector 1 and spectral norm bounded by (cid:107) J m (cid:107) 2 (cid:54) [Table]

Original: 5 β (see Appendix Lemma A1).
Revised: 5 β (see Appendix Lemma A1).

Original: According to Appendix Theorem A3, we can express U T x
Revised: According to Appendix Theorem A3, we can express U T x

Original: i U
Revised: i U

Original: y
Revised: y

Original: i as:
Revised: i as:

Original: [Equation]
Revised: [Equation]

Original: where the mean is ¯ u = 1 /M
Revised: where the mean is ¯ u = 1 /M

Original: U 

Revised: U

Original: and the weighted covariances are Cov( U , x i ) =
Revised: and the weighted covariances are Cov( U , x i ) =

Original: U J m ( β U T x i )
Revised: U J m ( β U T x i )

Original: U
Revised: U

Original: Tand Cov( U , y i )
Revised: Tand Cov( U , y i )

Original: =
Revised: =

Original: U J m ( β U
Revised: U J m ( β U

Original: T y i ) U T .
Revised: T y i ) U T .

Original: The weighted covariance Cov( U , . ) is the covariance
Revised: The weighted covariance Cov( U , . ) is the covariance

Original: if thestored pattern u
Revised: if thestored pattern u

Original: i is drawn according to an averaged p i given by J m ( . ) .
Revised: i is drawn according to an averaged p i given by J m ( . ) .

Original: Maximizing the dot product
Revised: Maximizing the dot product

Original: U T x i
Revised: U T x i

Original: U
Revised: U

Original: y
Revised: y

Original: i forces the normalized vectors x i and y i to agree on drawing the patterns u i with the sameprobability p i in order to generate similar weighted covariance matrices
Revised: i forces the normalized vectors x i and y i to agree on drawing the patterns u i with the sameprobability p i in order to generate similar weighted covariance matrices

Original: Cov( U , . ) .
Revised: Cov( U , . ) .

Original: If subsets of Uhave a strong covariance structure, then it can be exploited to produce large weighted covariancesand, in turn, large dot products of U T x
Revised: If subsets of Uhave a strong covariance structure, then it can be exploited to produce large weighted covariancesand, in turn, large dot products of U T x

Original: i U y
Revised: i U y

Original: i .
Revised: i .

Original: Furthermore, for a large dot product U T x
Revised: Furthermore, for a large dot product U T x

Original: i
Revised: i

Original: U
Revised: U

Original: y
Revised: y

Original: i , x i and y ihave to be similar to each other to extract the same direction from the covariance matrices.
Revised: i , x i and y ihave to be similar to each other to extract the same direction from the covariance matrices.

Original: The aboveconsiderations for U T x
Revised: The aboveconsiderations for U T x

Original: i
Revised: i

Original: U
Revised: U

Original: y i analogously apply to V T x i V y
Revised: y i analogously apply to V T x i V y

Original: i .
Revised: i .

Original: We did not use a loss function that contains dot products like U T x i V
Revised: We did not use a loss function that contains dot products like U T x i V

Original: y i , because these dot productshave higher variance than the ones we have used.
Revised: y i , because they have highervariance than the ones we have used.

Original: The dot product U T x i V y
Revised: The dot product U T x i V y

Original: i has higher variance,since it uses M + K stored patterns, whereas U T x i U y i andV T
Revised: i has higher variance, since it uses M + K stored patterns, whereas U T x i U y i and

Original:  x i V y
Revised: V T x i V y

Original: i use M and K stored patterns,respectively.
Revised: i use M and K stored patterns, respectively.

Original: Modern Hopﬁeld networks enable to extract more covariance structure.
Revised: Modern Hopﬁeld networks enable to extract more covariance structure.

Original: To demonstrate theeffect of modern Hopﬁeld networks, we computed the eigenvalues of the covariance matrix of theimage and text embeddings.
Revised: To demonstrate theeffect of modern Hopﬁeld networks, we computed the eigenvalues of the covariance matrix of theimage and text embeddings.

Original: We counted the number of effective eigenvalues, that is, the numberof eigenvalues needed to obtain 99% of the total sum of eigenvalues.
Revised: We counted the number of effective eigenvalues, that is, the numberof eigenvalues needed to obtain 99% of the total sum of eigenvalues.

Original: Figure 2 shows the relativechange of the number of effective eigenvalues compared to the respective reference epoch (the epochbefore the ﬁrst learning rate restart).
Revised: Figure 2 shows the relativechange of the number of effective eigenvalues compared to the respective reference epoch (the epochbefore the ﬁrst learning rate restart).

Original: Modern Hopﬁeld networks consistently increase the number ofeffective eigenvalues during learning.
Revised: Modern Hopﬁeld networks consistently increase the number ofeffective eigenvalues during learning.

Original: Consequently, modern Hopﬁeld networks enable to extractmore covariance structure during learning, i.e. enrich the embeddings by covariances that are alreadyin the raw multi-modal data.
Revised: Consequently, modern Hopﬁeld networks enable to extractmore covariance structure during learning, i.e. enrich the embeddings by covariances that are alreadyin the raw multi-modal data.

Original: This enrichment of embeddings mitigates explaining away.
Revised: This enrichment of embeddings mitigates explaining away.

Original: Furtherdetails can be found in Appendix Section A.2.6.
Revised: Furtherdetails can be found in Appendix Section A.2.7.

Original:  InfoLOOB for Contrastive Learning
Revised:  InfoLOOB for Contrastive Learning

Original: Modern Hopﬁeld networks lead to a higher similarity of retrieved samples.
Revised: Modern Hopﬁeld networks lead to a higher similarity of retrieved samples.

Original: The increased similarityexacerbates the saturation of the InfoNCE objective.
Revised: The increased similarityexacerbates the saturation of the InfoNCE objective.

Original: To avoid the saturation of InfoNCE, CLOOBuses the “InfoLOOB” objective.
Revised: To avoid the saturation of InfoNCE, CLOOBuses the “InfoLOOB” objective.

Original: The “InfoLOOB” objective is called “Leave one out upper bound” in
Revised: The “InfoLOOB” objective is called “Leave one out upper bound” in

Original: Poole et al.
Revised: Poole et al.

Original: and “L1Out” in Cheng et al.
Revised: and “L1Out” in Cheng et al.

Original: InfoLOOB is not established as a contrastiveobjective, although it is a known bound.
Revised: InfoLOOB is not established as a contrastiveobjective, although it is a known bound.

Original: Recently, InfoLOOB was independently introduced asobjective for image-to-image contrastive learning (Yeh et al., 2021).[Figure]
Revised: Recently, InfoLOOB was independently introduced asobjective for image-to-image contrastive learning (Yeh et al., 2021).

Original: R e l a t i v e i n c r e a s e o f nu m b e r o f e

Original: i g e n v a l u e s

Original: InfoNCE

Original: InfoNCE - Hopfield

Original: InfoLOOB

Original: InfoLOOB - Hopfield

Original: InfoNCE and InfoLOOB loss functions.
Revised: InfoNCE and InfoLOOB loss functions.

Original: N samples are drawn iid from p ( x , y ) giving the trainingset { ( x 1 , y 1 ) , . . .
Revised: N samples are drawn iid from p ( x , y ) giving the trainingset { ( x 1 , y 1 ) , . . .

Original: , ( x N , y N ) } .
Revised: , ( x N , y N ) } .

Original: For the sample y 1 , InfoNCE uses for the matrix of negative samples
Revised: For the sample y 1 , InfoNCE uses for the matrix of negative samples

Original: X = ( x 1 , . . .
Revised: X = ( x 1 , . . .

Original: , x N ) , while InfoLOOB uses ˜ X
Revised: , x N ) , while InfoLOOB uses ˜ X

Original: =
Revised: =

Original: ( x 2 , . .
Revised: ( x 2 , . .

Original: , x N ) .
Revised: , x N ) .

Original: The matrices differ by the positivesample x 1 .
Revised: The matrices differ by the positivesample x 1 .

Original: For the score function f ( x , y ) , we use f ( x , y ) =
Revised: For the score function f ( x , y ) , we use f ( x , y ) =

Original: exp( τ − 1 sim( x , y )) with the similaritysim( x , y ) =
Revised: exp( τ − 1 sim( x , y )) with the similarity [Figure] a t i v [Figure]

Revised: i

Revised: [Figure]

Original: y T x and τ as the temperature.
Revised: c [Figure] nu m b e r o f e i g [Figure]

Revised: v

Revised: [Figure]

Revised: l

Revised: [Figure]

Revised: sim( x , y ) =

Revised: y T x and τ as the temperature.

Original: We have the InfoNCE and InfoLOOB loss functions:
Revised: We have the InfoNCE and InfoLOOB loss functions:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: For simplicity, we abbreviatein the following y =
Revised: We abbreviate y =

Original: y 1 leading to the pair ( x 1 , y ) and the negatives˜ X
Revised: y 1 leading to the pair ( x 1 , y ) and the negatives ˜ X

Original: =
Revised: =

Original: ( x 2 , . .
Revised: ( x 2 , . .

Original: , x N ) .
Revised: , x N ) .

Original: In the second sum of the losses in Eq.
Revised: In thesecond sum of the losses in Eq.

Original: 7 and Eq. 8, we consider only the ﬁrstterm, respectively:
Revised: 7 and Eq. 8, we consider only the ﬁrst term, respectively:

Original: [Equation]
Revised: [Equation]

Original: In analogy to Wang & Isola (2020), a is called the “alignment score” that measures the similarity ofmatched pairs and b the “uniformity penalty” that measures the similarity of unmatched pairs.
Revised: In analogy to Wang & Isola (2020), a is called the “alignment score” that measures the similarity ofmatched pairs and b the “uniformity penalty” that measures the similarity of unmatched pairs.

Original: Gradients of InfoNCE and InfoLOOB loss functions.
Revised: Gradients of InfoNCE and InfoLOOB loss functions.

Original: Eq.
Revised: Eq.

Original: (9) and Eq.
Revised: (9) and Eq.

Original: (10) are equal to
Revised: (10) are equal to

Original: [Equation]
Revised: [Equation]

Original: ˜ X T y ) , where lse is the log-sum-exp function (see Eq.
Revised: where lse is the log-sum-exp function (see Eq.

Original: (A73) in the Appendix).
Revised: (A73) in the Appendix).

Original: [Figure]

Original: InfoNCE

Original: InfoNCE - Hopfield

Original: InfoLOOB

Original: InfoLOOB - Hopfield

Original: The gradients of Eq.
Revised: The gradients of Eq.

Original: (9) and Eq.
Revised: (9) and Eq.

Original: (10) with respect to y are
Revised: (10) with respect to y are

Original: [Equation]
Revised: [Equation]

Original: Using p = ( p 1 , . .
Revised: Using p = ( p 1 , . .

Original: . , p N ) T = softmax( τ − 1 X T y ) , the gradient of InfoNCE with respect to y is∂ L InfoNCE ( y )
Revised: . , p N ) T = softmax( τ − 1 X T y ) , the gradient of InfoNCE with respect to y is∂ L InfoNCE ( y )

Original: [Equation]
Revised: [Equation]

Original: By and large, the gradient of InfoNCE is scaled by (1 − p 1 ) compared to the gradient of InfoLOOB,where p 1 is the softmax similarity between the anchor y and the positive sample x 1 .
Revised: By and large, the gradient of InfoNCE is scaled by (1 − p 1 ) compared to the gradient of InfoLOOB,where p 1 is the softmax similarity between the anchor y and the positive sample x 1 .

Original: Consequently,
Revised: Consequently,

Original: InfoNCE is saturating with increasing similarity between the anchor and the positive sample.
Revised: InfoNCE is saturating with increasing similarity between the anchor and the positive sample.

Original: Formore details we refer to Appendix Section A.1.4.
Revised: For [Figure]

Original: This saturation of InfoNCE motivated the use of the InfoLOOB objective in order to decrease theuniformity penalty even for large alignment scores.
Revised: This saturation of InfoNCE motivated the use of the InfoLOOB objective in order to decrease theuniformity penalty even for large alignment scores.

Original: The uniformity penalty decreases since learningdoes not stall and the most prominent features become down-scaled which makes negative examplesless similar to the anchor sample.
Revised: The uniformity penalty decreases since learningdoes not stall and the most prominent features become down-scaled which makes negative examplesless similar to the anchor sample.

Original: The InfoNCE objective Eq.
Revised: The InfoNCE objective Eq.

Original: 9 has the form a/ ( a + b ) , while the
Revised: 9 has the form a/ ( a + b ) , while the

Original: InfoLOOB objective Eq.
Revised: InfoLOOB objective Eq.

Original: 10 has the form a/b .
Revised: 10 has the form a/b .

Original: InfoLOOB does not saturate and keeps decreasingthe uniformity penalty b .
Revised: InfoLOOB does not saturate and keeps decreasingthe uniformity penalty b .

Original: Figure 3 shows how InfoLOOB leads to an increase in the uniformity ofimage and text embeddings on the sphere, which is described by the statisticsof theuniformity testof Ajne that was extended by Prentice (Ajne, 1968; Prentice, 1978).
Revised: Figure 3 shows how InfoLOOB leads to an increase in the uniformity ofimage and text embeddings on the sphere, which is described by the statistics the testof Ajne that was extended by Prentice (Ajne, 1968; Prentice, 1978).

Original: Higher uniformity on the spherecorrelates with alower uniformity penalty b .
Revised: Higher uniformity on the spherecorrelates with a uniformity penalty b .

Original: For more details we refer to Appendix Section A.2.6.
Revised: For more details we refer to Appendix Section A.2.7.

Original: Theoretical justiﬁcation for optimizing InfoLOOB.
Revised: Theoretical justiﬁcation for optimizing InfoLOOB.

Original: The InfoNCE information is a lower boundon the mutual information, which was proven by Poole et al.
Revised: The InfoNCE information is a lower boundon the mutual information, which was proven by Poole et al.

Original: In the Appendix Section A.1,we elaborate more on theoretical properties of the bounds and properties of the objective functions.
Revised: In the Appendix Section A.1,we elaborate more on theoretical properties of the bounds and properties of the objective functions.

Original: Speciﬁcally, we show that InfoLOOB with neural networks is not an upper bound on the mutualinformation.
Revised: Speciﬁcally, we show that InfoLOOB with neural networks is not an upper bound on the mutualinformation.

Original: Thus, unlike hitherto approaches to contrastive learning we use InfoLOOB as anobjective, since it does not suffer from saturation effects as InfoNCE.
Revised: Thus, unlike hitherto approaches to contrastive learning we use InfoLOOB as anobjective, since it does not suffer from saturation effects as InfoNCE.

Original:  Experiments
Revised:  Experiments

Original: Our new CLOOB is compared to CLIP with respect to zero-shot transfer learning performance on twopre-training datasets.
Revised: CLOOB is compared to CLIP with respect to zero-shot transfer learning performance on two pretraining datasets.

Original: The ﬁrst dataset, Conceptual Captions (CC) (Sharma et al., 2018), has a veryrich textual description of images but only three million image-text pairs.
Revised: The ﬁrst dataset, Conceptual Captions (CC) (Sharma et al., 2018), has a very richtextual description of images but only three million image-text pairs.

Original: The second dataset, a subsetof YFCC100M (Thomee et al., 2016), has 15 million image-text pairs but the textual description isless rich than for CC and often vacuous.
Revised: The second dataset, a subset of YFCC100M (Thomee et al., 2016), has 15 million image-text pairs but the textual description is lessrich than for CC and often vacuous.

Original: For both pre-training datasets, the downstream zero-shottransfer learning performance is tested on seven image classiﬁcation datasets.{+datasets.5.
+}
Revised: For both pre-training datasets, the downstream zero-shot transferlearning performance is tested on seven image classiﬁcation {+datasets.5.
+}

Original: [Table]

Original: 5.1 Conceptual Captions Pre-training
Revised: Conceptual Captions Pre-training

Original: Pre-training dataset.
Revised: Pre-training dataset.

Original: The Conceptual Captions (CC) (Sharma et al., 2018) dataset contains 2.9million images with high-quality captions.
Revised: The Conceptual Captions (CC) (Sharma et al., 2018) dataset contains 2.9million images with high-quality captions.

Original: Images and their captions have been gathered from theweb via an automated process and have a wide variety of content.
Revised: Images and their captions have been gathered from theweb via an automated process and have a wide variety of content.

Original: Raw descriptions of images arefrom the alt-text HTML attribute.
Revised: Raw descriptions of images arefrom the alt-text HTML attribute.

Original: Methods compared.
Revised: Methods compared.

Original: The methods CLIP and our new CLOOB are compared (Radford et al., 2021).
Revised: The methods CLIP and our new CLOOB are compared (Radford et al., 2021).

Original: The CLOOB implementation is based on OpenCLIP (Ilharco et al., 2021), which achieves resultsequivalent to CLIP on the YFCC dataset (see Section 5.2).
Revised: The CLOOB implementation is based on OpenCLIP (Ilharco et al., 2021), which achieves resultsequivalent to CLIP on the YFCC dataset (see Section 5.2).

Original: OpenCLIP also reports results on the
Revised: OpenCLIP also reports results on the

Original: CC dataset.
Revised: CC dataset.

Original: As CLIP does not train models on CC, we report results from this reimplementation asbaseline.
Revised: As CLIP does not train models on CC, we report results from this reimplementation asbaseline.

Original: Analogously to Radford et al.
Revised: Analogously to Radford et al.

Original: (2021, Section 2.4), we used the modiﬁed ResNet (He et al.,2016) and BERT (Devlin et al., 2018, 2019) architectures to encode image and text input.
Revised: (2021, Section 2.4), we used the modiﬁed ResNet (He et al.,2016) and BERT (Devlin et al., 2018, 2019) architectures to encode image and text input.

Original: We usedthe ResNet encoder ResNet-50 for experiments on CC.
Revised: We usedthe ResNet encoder ResNet-50 for experiments on CC.

Original: Hyperparameter selection and learning schedule.
Revised: Hyperparameter selection and learning schedule.

Original: The hyperparameter values of OpenCLIP wereused as default, concretely, a learning rate of 1 × 10 − 3 and a weight decay of 0 .
Revised: The hyperparameter values of OpenCLIP wereused as default, concretely, a learning rate of 1 × 10 − 3 and a weight decay of 0 .

Original: 1 for the Adamoptimizer (Kingma et al., 2014) with decoupled weight decay regularization (Loshchilov & Hutter,2019).
Revised: 1 for the Adamoptimizer (Kingma et al., 2014) with decoupled weight decay regularization (Loshchilov & Hutter,2019).

Original: Deviating from OpenCLIP, we used a batch size of 512 due to computational restraints, whichdid not change the performance.
Revised: Deviating from OpenCLIP, we used a batch size of 512 due to computational restraints, which [Table] did not change the performance.

Original: The learning rate scheduler for all experiments was cosine annealingwith warmup and hard restarts (Loshchilov & Hutter, 2017).
Revised: The learning rate scheduler for all experiments was cosine annealingwith warmup and hard restarts (Loshchilov & Hutter, 2017).

Original: We report the hyperparameter τ (default0 . 07 ) from CLIP as τ − 1 of 14 .
Revised: We report the hyperparameter τ (default0 . 07 ) from CLIP as τ − 1 of 14 .

Original: 3 to be in the same regime as the hyperparameter β for the modern
Revised: 3 to be in the same regime as the hyperparameter β for the modern

Original: Hopﬁeld networks.
Revised: Hopﬁeld networks.

Original: The main hyperparameter search for CLOOB (also for YFCC pre-training in thenext section) was done with ResNet-50 as the vision encoder.
Revised: The main hyperparameter search for CLOOB (also for YFCC pre-training in thenext section) was done with ResNet-50 as the vision encoder.

Original: Learnable τ − 1 in combination withthe InfoLOOB loss results in undesired learning behavior (see Appendix Section A.1.4).
Revised: Learnable τ − 1 in combination withthe InfoLOOB loss results in undesired learning behavior (see Appendix Section A.1.4).

Original: Therefore,we set τ − 1 to a ﬁxed value of 30, which was determined via hyperparameter search (see Appendix
Revised: Therefore,we set τ − 1 to a ﬁxed value of 30, which was determined via hyperparameter search (see Appendix

Original: Section A.2.2).
Revised: Section A.2.2).

Original: For modern Hopﬁeld networks, the hyperparameter β was set to 8 .
Revised: For modern Hopﬁeld networks, the hyperparameter β was set to 8 .

Original: Further we scaledthe loss L InfoLOOB with τ to remove the factor τ − 1 from the gradients (see Appendix Section A.1.4)resulting in the loss function τ L InfoLOOB .
Revised: Further we scaledthe loss L InfoLOOB with τ to remove the factor τ − 1 from the gradients (see Appendix Section A.1.4)resulting in the loss function τ L InfoLOOB .

Original: Evaluation metrics: Zero-shot transfer learning.
Revised: Evaluation metrics: Zero-shot transfer learning.

Original: We evaluated and compared both CLIP and
Revised: We evaluated and compared both CLIP and

Original: CLOOB on their zero-shot transfer learning capabilities on the following downstream image classiﬁcation tasks.
Revised: CLOOB on their zero-shot transfer learning capabilities on the following downstream image classiﬁcation tasks.

Original: Birdsnap (Berg et al., 2014) contains images of 500 different North American birdspecies.
Revised: Birdsnap (Berg et al., 2014) contains images of 500 different North American birdspecies.

Original: The Country211 (Radford et al., 2021) dataset consists of photos across 211 countries andis designed to test the geolocalization capability of visual representations.
Revised: The Country211 (Radford et al., 2021) dataset consists of photos across 211 countries andis designed to test the geolocalization capability of visual representations.

Original: Flowers102 (Nilsback &
Revised: Flowers102 (Nilsback &

Original: Zisserman, 2008) is a dataset containing images of 102 ﬂower species.
Revised: Zisserman, 2008) is a dataset containing images of 102 ﬂower species.

Original: GTSRB (Stallkamp et al.
Revised: GTSRB (Stallkamp et al.

Original: ,2011) contains images for classiﬁcation of German trafﬁc signs.
Revised: ,2011) contains images for classiﬁcation of German trafﬁc signs.

Original: UCF101 (Soomro et al., 2012) is avideo dataset with short clips for action recognition.
Revised: UCF101 (Soomro et al., 2012) is avideo dataset with short clips for action recognition.

Original: For UCF101 we followed the procedure reportedin CLIP and extract the middle frame of every video to assemble the dataset.
Revised: For UCF101 we followed the procedure reportedin CLIP and extract the middle frame of every video to assemble the dataset.

Original: Stanford Cars (Krauseet al., 2013) contains images of 196 types of cars.
Revised: Stanford Cars (Krauseet al., 2013) contains images of 196 types of cars.

Original: ImageNet (Deng et al., 2009) is a large scale imageclassiﬁcation dataset with images across 1,000 classes.
Revised: ImageNet (Deng et al., 2009) is a large scale imageclassiﬁcation dataset with images across 1,000 classes.

Original: ImageNetv2 (Recht et al., 2019) consists ofthree new test sets with 10,000 images each for the ImageNet benchmark.
Revised: ImageNetv2 (Recht et al., 2019) consists ofthree new test sets with 10,000 images each for the ImageNet benchmark.

Original: For further details see
Revised: For further details see

Original: Appendix Section A.2.3.
Revised: Appendix Section A.2.3.

Original: Results.
Revised: Results.

Original: We employed the same evaluation strategy and used the prompt templates as published in
Revised: We employed the same evaluation strategy and used the prompt templates as published in

Original: CLIP (see Appendix Section A.2.3).
Revised: CLIP (see Appendix Section A.2.3).

Original: We report zero-shot results from two checkpoints in Table 1.
Revised: We report zero-shot results from two checkpoints in Table 1.

Original: CLIP and CLOOB were trained for a comparable number of epochs used in CLIP (see Appendix
Revised: CLIP and CLOOB were trained for a comparable number of epochs used in CLIP (see Appendix

Original: Section A.2.2) while CLIP* and CLOOB* were trained until evaluation performance plateaued (epoch128).
Revised: Section A.2.2) while CLIP* and CLOOB* were trained until evaluation performance plateaued (epoch128).

Original: In both cases CLOOB signiﬁcantly outperforms CLIP on the majority of tasks or matches itsperformance.
Revised: In both cases CLOOB signiﬁcantly outperforms CLIP on the majority of tasks or matches itsperformance.

Original: Statistical signiﬁcance of these results was assessed by an unpaired Wilcoxon test on a5 % level.
Revised: Statistical signiﬁcance of these results was assessed by an unpaired Wilcoxon test on a5 % level.

Original: 5.2 YFCC Pre-training
Revised: 5.2 YFCC Pre-training

Original: Pre-training dataset.
Revised: Pre-training dataset.

Original: To be comparable to the CLIP results, we used the same subset of 15 millionsamples from the YFCC100M dataset (Thomee et al., 2016) as in Radford et al.
Revised: To be comparable to the CLIP results, we used the same subset of 15 millionsamples from the YFCC100M dataset (Thomee et al., 2016) as in Radford et al.

Original: (2021), which werefer to as YFCC.
Revised: (2021), which werefer to as YFCC.

Original: YFCC was created by ﬁltering YFCC100M for images which contain naturallanguage descriptions and/or titles in English.
Revised: YFCC was created by ﬁltering YFCC100M for images which contain naturallanguage descriptions and/or titles in English.

Original: It was not ﬁltered by quality of the captions, thereforethe textual descriptions are less rich and contain superﬂuous information.
Revised: It was not ﬁltered by quality of the captions, thereforethe textual descriptions are less rich and contain superﬂuous information.

Original: The dataset with 400million samples used to train the CLIP models in Radford et al.
Revised: The dataset with 400million samples used to train the CLIP models in Radford et al.

Original: (2021) has not been released and,thus, is not available for comparison.
Revised: (2021) has not been released and,thus, is not available for comparison.

Original: Due to limited computational resources we were unable tocompare
Revised: Due to limited computational resources we were unable tocompare

Original: CLOOB to CLIP on other datasets of this size.
Revised: CLOOB to CLIP on other datasets of this size.

Original: Methods compared and evaluation.
Revised: Methods compared and evaluation.

Original: Besides experiments with a ResNet-50 image encoder, weadditionally conducted experiments with the larger ResNet variants ResNet-101 and ResNet-50x4.
Revised: Besides experiments with a ResNet-50 image encoder, weadditionally conducted experiments with the larger ResNet variants ResNet-101 and ResNet-50x4.

Original: Inaddition to the comparison of CLOOB and CLIP based on the OpenCLIP reimplementation (Ilharcoet al., 2021), we include the original CLIP results (Radford et al., 2021, Table 12).
Revised: Inaddition to the comparison of CLOOB and CLIP based on the OpenCLIP reimplementation (Ilharcoet al., 2021), we include the original CLIP results (Radford et al., 2021, Table 12).

Original: Hyperparameter selection.
Revised: Hyperparameter selection.

Original: Hyperparameters were the same as for the Conceptual Captions dataset,except learning rate, batch size, and β .
Revised: Hyperparameters were the same as for the Conceptual Captions dataset,except learning rate, batch size, and β .

Original: For modern Hopﬁeld networks, the hyperparameter β was setto 14 .
Revised: For modern Hopﬁeld networks, the hyperparameter β was setto 14 .

Original: 3 , which is default for τ − 1 in Radford et al.
Revised: 3 , which is default for τ − 1 in Radford et al.

Original: Furthermore, the learning rate was set to5 × 10 − 4 and the batch size to 1024 as used in OpenCLIP of Ilharco et al.
Revised: Furthermore, the learning rate was set to5 × 10 − 4 and the batch size to 1024 as used in OpenCLIP of Ilharco et al.

Original: All models weretrained for 28 epochs.
Revised: All models weretrained for 28 epochs.

Original: For further details see Appendix Section A.2.2.
Revised: For further details see Appendix Section A.2.2.

Original: Evaluation metrics.
Revised: Evaluation metrics.

Original: As in the previous experiment, methods were again evaluated at their zero-shottransfer learning capabilities on downstream tasks.
Revised: As in the previous experiment, methods were again evaluated at their zero-shottransfer learning capabilities on downstream tasks.

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: Results.
Revised: Results.

Original: Table 2 provides results of the original CLIP and CLOOB trained on YFCC.
Revised: Table 2 provides results of the original CLIP and CLOOB trained on YFCC.

Original: Results onzero-shot downstream tasks show that CLOOB outperforms CLIP on all 7 tasks (ImageNet V2 resultshave not been reported in Radford et al.
Revised: Results onzero-shot downstream tasks show that CLOOB outperforms CLIP on all 7 tasks (ImageNet V2 resultshave not been reported in Radford et al.

Original: Similarly, CLOOB outperforms CLIP on 6 out of 7tasks for linear probing.
Revised: Similarly, CLOOB outperforms CLIP on 6 out of 7tasks for linear probing.

Original: Results of CLOOB and the CLIP reimplementation of OpenCLIP are givenin Table 3.
Revised: Results of CLOOB and the CLIP reimplementation of OpenCLIP are givenin Table 3.

Original: CLOOB exceeds the CLIP reimplementation in 7 out of 8 tasks for zero-shot classiﬁcationusing ResNet-50 encoders.
Revised: CLOOB exceeds the CLIP reimplementation in 7 out of 8 tasks for zero-shot classiﬁcationusing ResNet-50 encoders.

Original: With larger ResNet encoders, CLOOB outperforms CLIP on all tasks.
Revised: With larger ResNet encoders, CLOOB outperforms CLIP on all tasks.

Original: Furthermore, the experiments with larger vision encoder networks show that CLOOB performanceincreases with network size. Predictions of CLOOB zero-shot classiﬁers for all datasets are shown in Appendix Section A.2.4. [Table]
Revised: Furthermore the experiments with larger vision encoder networks show that CLOOB performanceincreases with network size zero-

Original: [Table]

Original:  Ablation studies
Revised: 5.3 Ablation studies

Original: CLOOB has two new major components compared to CLIP: (1) modern Hopﬁeld networks and(2) the InfoLOOB objective instead of the InfoNCE objective.
Revised: CLOOB has two new major components compared to CLIP: (1) modern Hopﬁeld networks and(2) the InfoLOOB objective instead of the InfoNCE objective.

Original: To assess effects of the new majorcomponents of CLOOB, we performed ablation studies.
Revised: To assess effects of the new majorcomponents of CLOOB, we performed ablation studies.

Original: Modern Hopﬁeld networks.
Revised: Modern Hopﬁeld networks.

Original: Modern Hopﬁeld networks amplify the covariance structure in thedata via the retrievals.

Original: Ablation studies conﬁrm this ampliﬁcation as modern Hopﬁeld networksconsistently increase the number of effective eigenvalues of the embedding covariance matricesduring learning.
Revised: Ablation studies conﬁrm this ampliﬁcation as modern Hopﬁeld networks consistently increase the number of effective eigenvalues of the embedding covariance matricesduring learning.

Original: Figure 2 shows the relative change of the number of effective eigenvalues comparedto the respective reference epoch, which is the epoch before the ﬁrst learning rate restart.
Revised: Figure 2 shows the relative change of the number of effective eigenvalues comparedto the respective reference epoch, which is the epoch before the ﬁrst learning rate restart.

Original: Theseresults indicate that modern Hopﬁeld networks steadily extract more covariance structure duringlearning.
Revised: Theseresults indicate that modern Hopﬁeld networks steadily extract more covariance structure duringlearning.

Original: Modern Hopﬁeld networks induce higher similarity of retrieved samples, which in turnleads to stronger saturation of the InfoNCE objective.
Revised: Modern Hopﬁeld networks induce higher similarity of retrieved samples, which in turnleads to stronger saturation of the InfoNCE objective.

Original: As a result, we observe low uniformity (see
Revised: As a result, we observe low uniformity (see

Original: Figure 3) and a small number of effective eigenvalues (see Appendix Figure A1).
Revised: Figure 3) and a small number of effective eigenvalues (see Appendix Figure A1).

Original: Modern Hopﬁeld networks with InfoLOOB.
Revised: Modern Hopﬁeld networks with InfoLOOB.

Original: CLOOB counters the saturation of InfoNCE by usingthe InfoLOOB objective.
Revised: CLOOB counters the saturation of InfoNCE by usingthe InfoLOOB objective.

Original: The effectiveness of InfoLOOB manifests in an increased uniformitymeasure of image and text embeddings on the sphere, as shown in Figure 3.
Revised: The effectiveness of InfoLOOB manifests in an increased uniformitymeasure of image and text embeddings on the sphere, as shown in Figure 3.

Original: The ablation studyveriﬁes that modern Hopﬁeld networks together with InfoLOOB have a strong synergistic effect.
Revised: The ablation studyveriﬁes that modern Hopﬁeld networks together with InfoLOOB have a strong synergistic effect.

Original: InfoLOOB.
Revised: InfoLOOB.

Original: However, using solely InfoLOOB results in overﬁtting of the alignment score.
Revised: However, using solely InfoLOOB results in overﬁtting of the alignment score.

Original: Thisoverﬁtting occasionally leads to high similarities of unmatched pairs (seeAppendix Figure A3),which may decreases the zero-shot downstream performance.
Revised: Thisoverﬁtting occasionally leads to high similarities of unmatched pairs (see Figure 4), which maydecreases the zero-shot downstream performance.

Original: The reason for this is that the top-1evaluation metric is very sensitive to occasionally high similarities of prompts of the incorrect class.
Revised: The reason for this is that the top-1 evaluationmetric is very sensitive to occasionally high similarities of prompts of the incorrect class.

Original: Yeh et al. (2021) and Zhang et al.
Revised: Yeh et al.(2021) and Zhang et al.

Original: (2022) reported similar observations of overﬁtting. 
Revised: (2022) reported similar observations of overﬁtting. [Figure]

Original: CLOOB balances the overﬁtting of InfoLOOB with the underﬁtting of modern Hopﬁeld networksand remains in effective learning regimes.
Revised: CLOOB balances the overﬁtting of InfoLOOB with the underﬁtting of modern Hopﬁeld networksand remains in effective learning regimes.

Original: For more details and further ablation studies see Appendix
Revised: For more details and further ablation studies see Appendix

Original: Section A.2.1.
Revised: Section A.2.1.

Original:  Conclusion
Revised:  Conclusion

Original: We have introduced “Contrastive Leave One Out Boost” (CLOOB), which combines modern Hopﬁeldnetworks with the InfoLOOB objective.
Revised: We have introduced “Contrastive Leave One Out Boost” (CLOOB), which combines modern Hopﬁeldnetworks with the InfoLOOB objective.

Original: Modern Hopﬁeld networks enable CLOOB to extractadditional covariance structure in the data.
Revised: Modern Hopﬁeld networks enable CLOOB to extractadditional covariance structure in the data.

Original: This allows for building more relevant features in theembedding space, mitigating the explaining away problem.
Revised: This allows for building more relevant features in theembedding space, mitigating the explaining away problem.

Original: We show that InfoLOOB avoids thesaturation problem of InfoNCE.
Revised: We show that InfoLOOB avoids thesaturation problem of InfoNCE.

Original: Additionally, we theoretically justify the use of the InfoLOOBobjective for contrastive learning and suggest it as an alternative to InfoNCE.
Revised: Additionally, we theoretically justify the use of the InfoLOOBobjective for contrastive learning and suggest it as an alternative to InfoNCE.

Original: At seven zero-shottransfer learning tasks, the novel CLOOB was compared to CLIP after pre-training on the Conceptual
Revised: At seven zero-shottransfer learning tasks, the novel CLOOB was compared to CLIP after pre-training on the Conceptual

Original: Captions and the YFCC dataset.
Revised: Captions and the YFCC dataset.

Original: CLOOB consistently outperforms CLIP at zero-shot transfer learningacross all considered architectures and datasets. 
Revised: CLOOB consistently outperforms CLIP at zero-shot transfer learningacross all considered architectures and datasets. Checklist

Revised: For all authors...

Original: A Appendix
Revised: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope? [Yes] (b) Did you describe the limitations of your work?

Revised: [Yes] Our method is currently limited

Revised: to natural images and short text prompts as inputs, and, thus its use for other types of

Revised: images, such as medical or biological images, is unexplored.

Revised: While we hypothesize

Revised: that our approach could also be useful for similar data in other domains, this has not

Revised: been assessed.

Revised: (c) Did you discuss any potential negative societal impacts of your work?

Revised: [Yes] One

Revised: potential danger arises from users that overly rely on systems built on our method.

Revised: For

Revised: example in the domain of self-driving cars, users might start

Revised: paying

Revised: less attention to

Revised: the trafﬁc because

Revised: of

Revised: the AI-based driving system.

Revised: Finally

Revised: , our method

Revised: might also

Revised: beused

Revised: to

Revised: automate

Revised: various simple tasks,

Revised: which might lead to reduced need

Revised: for particular

Revised: jobs in production systems

Revised: As for almost all machine

Revised: learning methods,

Revised: our proposed

Revised: method relies

Revised: on human-annotated

Revised: training

Revised: data

Revised: and thereby human decisions, which

Revised: are

Revised: usually

Revised: strongly biased

Revised: Therefore, the

Revised: responsible

Revised: use of our method requires

Revised: the

Revised: careful selection of the training data and awareness of potential biases within those.

Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Revised: them?

Revised: [Yes]

Revised: If you are including theoretical results...

Revised: (a) Did you state the full set of assumptions of all theoretical results?

Revised: [Yes]

Revised: (b) Did you include complete proofs of all theoretical results?

Revised: [Yes]

Revised: If you ran experiments...

Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Revised: mental results (either in the supplemental material or as a URL)?

Revised: [Yes] We provide the

Revised: code as supplementary material.

Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Revised: how they

Revised: were chosen)?

Revised: [Yes] See Section 5.1, Section 5.2 and Appendix Section A.2.2.

Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experi-

Revised: ments multiple times)?

Revised: [Yes] We added error bars for all experiments for which this

Revised: was computationally feasible (see Table 1).

Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type

Revised: of GPUs, internal cluster, or cloud provider)?

Revised: [Yes] We used several different servers

Revised: equipped with GPUs of different types, such as V100 and A100.

Revised: The total amount of

Revised: compute is roughly 11 , 000 GPU hours (with A100).

Revised: If you are using existing assets (e

Revised: g., code, data, models) or curating/releasing new assets...

Revised: (a) If your work uses existing assets, did you cite the creators?

Revised: [Yes] For the model

Revised: implementations we used PyTorch (Paszke et al., 2017, BSD license) and for monitoring

Revised: the runs we used Weights & Biases (Biewald, 2020, MIT license).

Revised: (b) Did you mention the license of the assets?

Revised: [Yes] See above.

Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Revised: [Yes]

Revised: We provide the code as supplementary material.

Revised: (d) Did you discuss whether and how consent was obtained from people whose data

Revised: you’re using/curating?

Revised: [Yes] We only use public datasets that can be used for research

Revised: purposes, such as the YFCC dataset which was published under the Creative Commons

Revised: licence.

Revised: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable

Revised: information or offensive content?

Revised: [Yes] As almost all computer vision and natural

Revised: language datasets, the data suffers from many biases including social biases.

Revised: We refer

Revised: to Yang et al.

Revised: (2020) for a detailed analysis of biases in such datasets.

Revised: If you used crowdsourcing or conducted

Revised: with subjects...

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original:  This appendix consists of four sections (A.1–A.4).
Revised: [Table] [Table] A Appendix This appendix consists of four sections (A.1–A.4).

Original: Section A.1 provides the theoretical propertiesof the InfoLOOB and InfoNCE.
Revised: Section A.1 provides the theoretical propertiesof the InfoLOOB and InfoNCE.

Original: It is shown how to derive that InfoNCE is a lower bound onmutual information.
Revised: It is shown how to derive that InfoNCE is a lower bound onmutual information.

Original: Further it is shown how to derive that InfoLOOB is an upper bound on mutualinformation.
Revised: Further it is shown how to derive that InfoLOOB is an upper bound on mutualinformation.

Original: The proposed loss function and its gradients are discussed.
Revised: The proposed loss function and its gradients are discussed.

Original: Section A.2 provides detailson the experiments for Section 5.
Revised: Section A.2 provides detailson the experiments for Section 5.

Original: Section A.3 brieﬂy reviews continuous modern Hopﬁeld networks.
Revised: Section A.3 brieﬂy reviews continuous modern Hopﬁeld networks.

Original: Section A.4 discusses further related work.
Revised: Section A.4 discusses further related work.

Original: Contents of the appendix
Revised: Contents of the appendix

Original: A.1 InfoLOOB vs. InfoNCE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A.1 InfoLOOB vs. InfoNCE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A.1.1 InfoNCE: Lower Bound on Mutual Information . . . . . . . . . . . . . . .
Revised: A.1.1 InfoNCE: Lower Bound on Mutual Information . . . . . . . . . . . . . . .

Original: A.1.2 InfoLOOB:
Revised: A.1.2 InfoLOOB:

Original: Upper Bound on Mutual Information . . . . . . . . . . . . . .
Revised: Upper Bound on Mutual Information . . . . . . . . . . . . . .

Original: A.1.3 InfoLOOB Analysis of the Objective . . . . . . . . . . . . . . . . . . . .
Revised: A.1.3 InfoLOOB: Analysis of the Objective . . . . . . . . . . . . . . . . . . . .

Original: A.1.4 InfoNCE and InfoLOOB: Gradients . . . . . . . . . . . . . . . . . . . . .
Revised: A.1.4 InfoNCE and InfoLOOB: Gradients . . . . . . . . . . . . . . . . . . . . .

Original: A.1.5 InfoLOOB and InfoNCE: Probability Estimators . .
Revised: A.1.5 InfoLOOB and InfoNCE: Probability Estimators . .

Original: A.1.6 InfoLOOB and InfoNCE: Losses . . . . . . . . . . . . . . . . . . . . . . .
Revised: A.1.6 InfoLOOB and InfoNCE: Losses . . . . . . . . . . . . . . . . . . . . . . .

Original: A.2 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A.2 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A.2.1 Ablation studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A.2.1 Ablation studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A.2.2 Hyperparameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A.2.2 Hyperparameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A.2.3 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A.2.3 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A.2.4 Zero-shot evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A.2.4 Zero-shot evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A.2.5 Linear probing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A.2.5 Linear probing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A.2.6  Analysis of the image and text embeddings . . . . . . . . . . . . . . . . .
Revised: A.2.6 Image-Text retrieval . . . A.2.7 Analysis of the image and text embeddings . . . . . . . . . . . . . . . . .

Revised: A.2.8 Training time and memory consumption . .

Original: A.3 Review of Modern Hopﬁeld Networks . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A.3 Review of Modern Hopﬁeld Networks . . . . . . . . . . . . . . . . . . . . . . . .

Original: A.4 Further Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 
Revised: A.4 Further Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: List of theorems
Revised: of theorems

Original: A
Revised: A

Original: Theorem (InfoNCE lower bound) . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Theorem (InfoNCE lower bound) . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A
Revised: A

Original: Theorem (InfoLOOB upper bound) . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Theorem (InfoLOOB upper bound) . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A
Revised: A

Original: Theorem (Weighted Covariances) . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Theorem (Weighted Covariances) . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A
Revised: A

Original: Theorem (Modern Hopﬁeld Networks: Retrieval with One Update) . . . . . . . . .
Revised: Theorem (Modern Hopﬁeld Networks: Retrieval with One Update) . . . . . . . . .

Original: A
Revised: A

Original: Theorem (Modern Hopﬁeld Networks: Exponential Storage Capacity) . . . . . . .
Revised: Theorem (Modern Hopﬁeld Networks: Exponential Storage Capacity) . . . . . . .

Original: List of deﬁnitions
Revised: List of deﬁnitions

Original: A
Revised: A

Original: Deﬁnition (Pattern Stored and Retrieved) . . . . . . . . . . . . . . . . . . . . . . .
Revised: Deﬁnition (Pattern Stored and Retrieved) . . . . . . . . . . . . . . . . . . . . . . .

Revised: List of ﬁgures

Original: List of tables
Revised: List of tables

Original: A
Revised: A

Original: CC) . . . . . . . . . . . . . . . A A A
Revised: A

Original: A
Revised: A

Original: A.1 InfoLOOB vs. InfoNCEA.1.1 InfoNCE: Lower Bound on Mutual Information
Revised: A.1 InfoLOOB vs. InfoNCEA.1.1 InfoNCE: Lower Bound on Mutual Information

Original: We derive a lower bound on the mutual information between random variables X and Y distributedaccording to p ( x , y ) .
Revised: We derive a lower bound on the mutual information between random variables X and Y distributedaccording to p ( x , y ) .

Original: The mutual information I( X ; Y ) between random variables X
Revised: The mutual information I( X ; Y ) between random variables X

Original: and Y is
Revised: and Y is

Original: I( X ; Y ) =
Revised: I( X ; Y ) =

Original: E p ( x , y ) (cid:20)
Revised: E p ( x , y ) (cid:20)

Original: [Equation]
Revised: [Equation]

Original: p(
Revised: p

Original: [Equation]
Revised: [Equation]

Original: ) p ( x ) p ( y
Revised: p ( [Equation] p ( y

Original: [Equation]
Revised: [Equation]

Original: (cid:21)
Revised: (cid:21)

Original: [Equation]
Revised: [Equation]

Original: (cid:20)
Revised: (cid:20)

Original: [Equation]
Revised: [Equation]

Original: p (
Revised: p (

Original: [Equation]
Revised: [Equation]

Original: (cid:21)
Revised: (cid:21)

Original: [Equation]
Revised: [Equation]

Original: (cid:20)
Revised: (cid:20)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: p ( [Equation] ) (cid:21) .
Revised: p ( y ) (cid:21) .

Original: (A1) “InfoNCE” has been introduced in van den Oord et al.
Revised: (A1) “InfoNCE” has been introduced in van den Oord et al.

Original: and is a multi-sample bound .
Revised: and is a multi-sample bound .

Original: Inthe setting introduced in van den Oord et al.
Revised: Inthe setting introduced in van den Oord et al.

Original: (2018), we have an anchor sample y given.
Revised: (2018), we have an anchor sample y given.

Original: For theanchor sample y we draw a positive sample x 1 according to p ( x 1 | y ) .
Revised: For theanchor sample y we draw a positive sample x 1 according to p ( x 1 | y ) .

Original: Next, we draw a set˜ X = { x 2 , . . .
Revised: Next, we draw a set˜ X = { x 2 , . . .

Original: , x N } according to p ( ˜ X ) , which are n − 1 negative samples drawn iid according top ( x ) .
Revised: , x N } according to p ( ˜ X ) , which are n − 1 negative samples drawn iid according top ( x ) .

Original: We have drawn a set X
Revised: We have drawn a set X

Original: = { x 1 , x 2 , . . .
Revised: = { x 1 , x 2 , . . .

Original: , x N } according to p ( X | y ) , which is one positivesample x 1 drawn by p ( x 1 | y ) and N − 1 negative samples { x 2 , . . .
Revised: , x N } according to p ( X | y ) , which is one positivesample x 1 drawn by p ( x 1 | y ) and N − 1 negative samples { x 2 , . . .

Original: , x N } drawn iid according top ( x ) .
Revised: , x N } drawn iid according top ( x ) .

Original: The InfoNCE with probabilities is
Revised: The InfoNCE with probabilities is

Original: I InfoNCE ( X 1 ; Y ) =
Revised: I InfoNCE ( X 1 ; Y ) =

Original: E p ( y ) (cid:34) 
Revised: E p ( y ) (cid:34) E

Original: [Equation]
Revised: [Equation]

Original:  (cid:34)
Revised: y ) (cid:34)

Original: [Equation]
Revised: [Equation]

Original:  p 
Revised: (cid:32) p [Equation]

Original: [Equation]
Revised: [Equation]

Original: ) (cid:33)(cid:35)(cid:35) , (A2) where we inserted the factor 1 N in contrast to the original version in van den Oord et al.
Revised: ) (cid:33)(cid:35)(cid:35) , (A2) where we inserted the factor 1 N in contrast to the original version in van den Oord et al.

Original: (2018), wherewe followed Poole et al.
Revised: (2018), wherewe followed Poole et al.

Original: ; Tschannen et al.
Revised: ; Tschannen et al.

Original: ; Cheng et al.
Revised: ; Cheng et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: The InfoNCE with score function f ( x , y ) is
Revised: The InfoNCE with score function f ( x , y ) is

Original: [Equation]

Original: A

Original: [Equation]
Revised: [Equation]

Original: The InfoNCE with probabilities can be rewritten as:
Revised: The InfoNCE with probabilities can be rewritten as:

Original: [Equation]
Revised: [Equation]

Original: This is the InfoNCE with f ( x , y ) =
Revised: This is the InfoNCE with f ( x , y ) =

Original: p ( y | x ) .
Revised: p ( y | x ) .

Original: Set of pairs.
Revised: Set of pairs.

Original: The InfoNCE can be written in a different setting Poole et al.
Revised: The InfoNCE can be written in a different setting Poole et al.

Original: (2019), which isused in most implementations.
Revised: (2019), which isused in most implementations.

Original: We sample N pairs independently from p ( x , y ) , which gives Z ={ ( x 1 , y 1 ) , ( x 2 , y 2 ) , . . .
Revised: We sample N pairs independently from p ( x , y ) , which gives Z ={ ( x 1 , y 1 ) , ( x 2 , y 2 ) , . . .

Original: , ( x N , y N ) } .
Revised: , ( x N , y N ) } .

Original: The InfoNCE is then
Revised: The InfoNCE is then

Original: [Equation]
Revised: [Equation]

Original: Following van den Oord et al.
Revised: Following van den Oord et al.

Original: we have
Revised: we have

Original: I InfoNCE
Revised: I InfoNCE

Original: [Equation]
Revised: [Equation]

Original: where p ( i = 1 |
Revised: where p ( i = 1 |

Original: X, y ) is the probability that sample x 1 is the positive sample if we know there existsexactly one positive sample in X .
Revised: X, y ) is the probability that sample x 1 is the positive sample if we know there existsexactly one positive sample in X .

Original: The InfoNCE is a lower bound on the mutual information.
Revised: The InfoNCE is a lower bound on the mutual information.

Original: The following inequality is from van den
Revised: The following inequality is from van den

Original: Oord et al.
Revised: Oord et al.

Original: [Equation]
Revised: [Equation]

Original: y ) (cid:20) − ln (cid:18) p ( x 1 ) p ( x 1 | y ) (cid:19)(cid:21)(cid:21) ≥ E p ( y ) (cid:20)
Revised: ) (cid:20) − ln (cid:18) p ( x 1 ) p ( x 1 | y ) (cid:19)(cid:21)(cid:21) ≥ E p ( y ) (cid:20)

Original: E p ( x 1 | y ) (cid:20) [Equation] (cid:18)
Revised: E p ( x 1 | y ) (cid:20) − ln (cid:18)

Original: N
Revised: N

Original: [Equation]
Revised: [Equation]

Original: p ( x 1 ) p ( x 1 | y ) (cid:19)(cid:21)(cid:21) ≈ E p ( y ) (cid:34)
Revised: p ( x 1 ) p ( x 1 | y ) (cid:19)(cid:21)(cid:21) ≈ E p ( y ) (cid:34)

Original: E p ( X | y ) (cid:34)
Revised: E p ( X | y ) (cid:34)

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: (cid:33)(cid:35)(cid:35)
Revised: (cid:33)(cid:35)(cid:35)

Original: [Equation]
Revised: [Equation]

Original:   
Revised:   

Original:  
Revised:  

Original:  =
Revised:  =

Original: I InfoNCE ( X 1 ; Y ) , where the " ≥ " is obtained by bounding ln(1 /N + a ) by ln( a ) , which gives a bound that is not verytight, since a = p ( x 1 ) p ( x 1 | y ) can become small.
Revised: I InfoNCE ( X 1 ; Y ) , where the " ≥ " is obtained by bounding ln(1 /N + a ) by ln( a ) , which gives a bound that is not verytight, since a = p ( x 1 ) p ( x 1 | y ) can become small.

Original: However for the " ≈ " van den Oord et al.
Revised: However for the " ≈ " van den Oord et al.

Original: (2018) have toassume
Revised: (2018) have toassume

Original: N
Revised: N

Original: [Equation]
Revised: [Equation]

Revised: N

Original: [Equation]
Revised: [Equation]

Original: which is unclear how to ensure.
Revised: which is unclear how to ensure.

Original: For a proof of this bound see Poole et al.
Revised: For a proof of this bound see Poole et al.

Original: We assumed that for the anchor sample y a positive sample x 1 has been drawn according to p ( x 1 | y ) .
Revised: We assumed that for the anchor sample y a positive sample x 1 has been drawn according to p ( x 1 | y ) .

Original: A set ˜ X
Revised: A set ˜ X

Original: = { x 2 , . . .
Revised: = { x 2 , . . .

Original: , x N } of negative samples is drawn according to p ( x ) .
Revised: , x N } of negative samples is drawn according to p ( x ) .

Original: Therefore, we have aset X = { x 1 , x 2 , . . .
Revised: Therefore, we have aset X = { x 1 , x 2 , . . .

Original: , x N } that is drawn with one positive sample x 1 and N − 1 negative samples˜ X
Revised: , x N } that is drawn with one positive sample x 1 and N − 1 negative samples˜ X

Original: = { x 2 , . . .
Revised: = { x 2 , . . .

Original: , x N } .
Revised: , x N } .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Next, we present a theorem that shows this bound, where we largely follow Poole et al.
Revised: Next, we present a theorem that shows this bound, where we largely follow Poole et al.

Original: (2019) in theproof.
Revised: (2019) in theproof.

Original: In contrast to Poole et al.
Revised: In contrast to Poole et al.

Original: (2019), we do not use the NWJ bound Nguyen et al.
Revised: (2019), we do not use the NWJ bound Nguyen et al.

Original: Themutual information is
Revised: Themutual information is

Original: [Equation]
Revised: [Equation]

Original: Theorem A1 (InfoNCE lower bound) .
Revised: Theorem A1 (InfoNCE lower bound) .

Original: InfoNCE with score function f ( x , y ) according to Eq.
Revised: InfoNCE with score function f ( x , y ) according to Eq.

Original: (A3)is a lower bound on the mutual information.
Revised: (A3)is a lower bound on the mutual information.

Original: [Equation]
Revised: [Equation]

Original: InfoNCE with probabilities according to Eq.
Revised: InfoNCE with probabilities according to Eq.

Original: (A2) is a lower bound on the mutual information.
Revised: (A2) is a lower bound on the mutual information.

Original: [Equation]
Revised: [Equation]

Original: The second bound Eq.
Revised: The second bound Eq.

Original: (A14) is a special case of the ﬁrst bound Eq.
Revised: (A14) is a special case of the ﬁrst bound Eq.

Original: (A13) .
Revised: (A13) .

Original: Proof.
Revised: Proof.

Original: Part (I) : Lower bound with score function f ( x , y ) .
Revised: Part (I) : Lower bound with score function f ( x , y ) .

Original: For each set ˜ X
Revised: For each set ˜ X

Original: = { x 2 , . . .
Revised: = { x 2 , . . .

Original: , x N } , we deﬁne as data-dependent (depending on ˜ X ) score functiong ( x 1 , y , ˜ X ) that is based on the score function f ( x , y ) .
Revised: , x N } , we deﬁne as data-dependent (depending on ˜ X ) score functiong ( x 1 , y , ˜ X ) that is based on the score function f ( x , y ) .

Original: Therefore we have for each ˜ X a differentdata-dependent score function g based on f .
Revised: Therefore we have for each ˜ X a differentdata-dependent score function g based on f .

Original: We will derive a bound on the InfoNCE, which is theexpectation of a lower bond on the mutual information over the score functions.
Revised: We will derive a bound on the InfoNCE, which is theexpectation of a lower bond on the mutual information over the score functions.

Original: For score functiong ( x 1 , y , ˜ X ) , we deﬁne a variational distribution q ( x 1 | y , ˜ X ) over x 1 :
Revised: For score functiong ( x 1 , y , ˜ X ) , we deﬁne a variational distribution q ( x 1 | y , ˜ X ) over x 1 :

Original: [Equation]
Revised: [Equation]

Original:  which ensures
Revised: ) which ensures

Original: [Equation]
Revised: [Equation]

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: For the function g , we set
Revised: For the function g , we set

Original: [Equation]
Revised: [Equation]

Original: For the function f we use
Revised: For the function f we use

Original: [Equation]
Revised: [Equation]

Original: where sim( x , y ) is typically the cosine similarity.
Revised: where sim( x , y ) is typically the cosine similarity.

Original: We next show that InfoNCE is a lower bound on the mutual information.
Revised: We next show that InfoNCE is a lower bound on the mutual information.

Original: I( X 1 ; Y ) =
Revised: I( X 1 ; Y ) =

Original: E p ( ˜ X )
Revised: E p ( ˜ X )

Original: [I( X 1 ; Y )]
Revised: [I( X 1 ; Y )]

Original: =
Revised: =

Original: E p ( ˜ X ) (cid:20)
Revised: E p ( ˜ X ) (cid:20)

Original: E p ( x 1 , y ) (cid:20) ln p ( x 1 | y ) p ( x 1 ) (cid:21)(cid:21) (A21) =
Revised: E p ( x 1 , y ) (cid:20) ln p ( x 1 | y ) p ( x 1 ) (cid:21)(cid:21) (A21) =

Original: E p ( ˜ X ) (cid:34)
Revised: E p ( ˜ X ) (cid:34)

Original: E p ( x 1 , y ) (cid:34) ln (cid:32) p ( x 1 | y ) q ( x 1 | y , ˜ X ) q ( x 1 | y , ˜ X ) p ( x 1 ) (cid:33)(cid:35)(cid:35) =
Revised: E p ( x 1 , y ) (cid:34) ln (cid:32) p ( x 1 | y ) q ( x 1 | y , ˜ X ) q ( x 1 | y , ˜ X ) p ( x 1 ) (cid:33)(cid:35)(cid:35) =

Original: E p ( ˜ X ) (cid:34)
Revised: E p ( ˜ X ) (cid:34)

Original: E p ( x 1 , y ) (cid:34) ln q ( x 1 | y , ˜ X ) p ( x 1 ) (cid:35) + E p ( y ) (cid:104) KL( p ( x 1 | y ) (cid:107) q
Revised: E p ( x 1 , y ) (cid:34) ln q ( x 1 | y , ˜ X ) p ( x 1 ) (cid:35) + E p ( y ) (cid:104) KL( p ( x 1 | y ) (cid:107) q

Original: x 1 | y , ˜ X ))
Revised: x 1 | y , ˜ X ))

Original: (cid:105)(cid:35) ≥ E p ( ˜ X ) (cid:34)
Revised: (cid:105)(cid:35) ≥ E p ( ˜ X ) (cid:34)

Original: E p ( x 1 , y ) (cid:34) ln q ( x 1 | y , ˜ X ) p ( x 1 ) (cid:35)(cid:35) =
Revised: E p ( x 1 , y ) (cid:34) ln q ( x 1 | y , ˜ X ) p ( x 1 ) (cid:35)(cid:35) =

Original: E p ( ˜ X ) (cid:34)
Revised: E p ( ˜ X ) (cid:34)

Original: E p ( x 1 , y ) (cid:34) ln g ( x 1 , y , ˜ X )
Revised: E p ( x 1 , y ) (cid:34) ln g ( x 1 , y , ˜ X )

Original: Z ( y , ˜ X ) (cid:35)(cid:35) =
Revised: Z ( y , ˜ X ) (cid:35)(cid:35) =

Original: E p ( ˜ X ) (cid:104) E p ( x 1 , y ) (cid:104)
Revised: E p ( ˜ X ) (cid:104) E p ( x 1 , y ) (cid:104)

Original: ln g ( x 1 , y , ˜ X ) − ln (cid:16) E p ( x 1 ) (cid:104) g
Revised: ln g ( x 1 , y , ˜ X ) − ln (cid:16) E p ( x 1 ) (cid:104) g

Original: ( x 1 , y , ˜ X ) (cid:105)(cid:17)(cid:105)(cid:105) =
Revised: ( x 1 , y , ˜ X ) (cid:105)(cid:17)(cid:105)(cid:105) =

Original: E p ( ˜ X ) (cid:104) E p ( y ) (cid:104)
Revised: E p ( ˜ X ) (cid:104) E p ( y ) (cid:104)

Original: E p ( x 1 | y ) (cid:104)
Revised: E p ( x 1 | y ) (cid:104)

Original: ln g ( x 1 , y , ˜ X ) (cid:105) − ln (cid:16) E p ( x 1 ) (cid:104) g
Revised: ln g ( x 1 , y , ˜ X ) (cid:105) − ln (cid:16) E p ( x 1 ) (cid:104) g

Original: ( x 1 , y , ˜ X ) (cid:105)(cid:17)(cid:105)(cid:105) =
Revised: ( x 1 , y , ˜ X ) (cid:105)(cid:17)(cid:105)(cid:105) =

Original: E p ( ˜ X ) (cid:104) E p ( y ) (cid:104)
Revised: E p ( ˜ X ) (cid:104) E p ( y ) (cid:104)

Original: E p ( x 1 | y ) (cid:104)
Revised: E p ( x 1 | y ) (cid:104)

Original: ln g ( x 1 , y , ˜ X ) (cid:105)(cid:105)(cid:105) − E p ( ˜ X ) (cid:104) E p ( y ) (cid:104) ln (cid:16) E p ( x 1 ) (cid:104) g
Revised: ln g ( x 1 , y , ˜ X ) (cid:105)(cid:105)(cid:105) − E p ( ˜ X ) (cid:104) E p ( y ) (cid:104) ln (cid:16) E p ( x 1 ) (cid:104) g

Original: ( x 1 , y , ˜ X ) (cid:105)(cid:17)(cid:105)(cid:105) ≥ E p ( y ) p
Revised: ( x 1 , y , ˜ X ) (cid:105)(cid:17)(cid:105)(cid:105) ≥ E p ( y ) p

Original: ( X | y ) (cid:104)
Revised: ( X | y ) (cid:104)

Original: ln g ( x 1 , y , ˜ X ) (cid:105) − E p ( ˜ X ) (cid:104) E p ( y ) (cid:104)
Revised: ln g ( x 1 , y , ˜ X ) (cid:105) − E p ( ˜ X ) (cid:104) E p ( y ) (cid:104)

Original: E p ( x 1 ) (cid:104) g
Revised: E p ( x 1 ) (cid:104) g

Original: ( x 1 , y , ˜ X ) (cid:105) − 1 (cid:105)(cid:105) =
Revised: ( x 1 , y , ˜ X ) (cid:105) − 1 (cid:105)(cid:105) =

Original: E p ( y ) p ( X | y ) (cid:34) ln
Revised: E p ( y ) p ( X | y ) (cid:34) ln

Original: f
Revised: f

Original: ( x 1 , y ) 1 N (cid:80) Ni =1 f
Revised: ( x 1 , y ) 1 N (cid:80) Ni =1 f

Original: ( x i , y ) (cid:35) − E p ( y ) (cid:34)
Revised: ( x i , y ) (cid:35) − E p ( y ) (cid:34)

Original: E p ( X ) (cid:34) f
Revised: E p ( X ) (cid:34) f

Original: ( x 1 , y ) 1 N (cid:80) Ni =1 f
Revised: ( x 1 , y ) 1 N (cid:80) Ni =1 f

Original: ( x i , y ) (cid:35) −(cid:35) =
Revised: ( x i , y ) (cid:35) −(cid:35) =

Original: E p ( y ) p ( X | y ) (cid:34) ln
Revised: E p ( y ) p ( X | y ) (cid:34) ln

Original: f
Revised: f

Original: ( x 1 , y ) 1 N (cid:80) Ni =1 f
Revised: ( x 1 , y ) 1 N (cid:80) Ni =1 f

Original: ( x i , y ) (cid:35) − E p ( y ) (cid:34)
Revised: ( x i , y ) (cid:35) − E p ( y ) (cid:34)

Original: N
Revised: N

Original: N (cid:88) i =
Revised: N (cid:88) i =

Original: E p ( X ) (cid:34) f
Revised: E p ( X ) (cid:34) f

Original: ( x i , y ) 1 N (cid:80) Ni =1 f ( x i , y ) (cid:35) −(cid:35) =
Revised: ( x i , y ) 1 N (cid:80) Ni =1 f ( x i , y ) (cid:35) −(cid:35) =

Original: E p ( y ) p ( X | y ) (cid:34) ln
Revised: E p ( y ) p ( X | y ) (cid:34) ln

Original: f
Revised: f

Original: ( x 1 , y ) 1 N (cid:80) Ni =1 f
Revised: ( x 1 , y ) 1 N (cid:80) Ni =1 f

Original: ( x i , y ) (cid:35) − E p ( y ) (cid:34)
Revised: ( x i , y ) (cid:35) − E p ( y ) (cid:34)

Original: E p ( X ) (cid:34) 1 N (cid:80) Ni =1 f ( x i , y ) 1 N (cid:80)
Revised: E p ( X ) (cid:34) 1 N (cid:80) Ni =1 f ( x i , y ) 1 N (cid:80)

Original: Ni =1 f ( x i , y ) (cid:35) −(cid:35) =
Revised: Ni =1 f ( x i , y ) (cid:35) −(cid:35) =

Original: E p ( y ) p ( X | y ) (cid:34) ln
Revised: E p ( y ) p ( X | y ) (cid:34) ln

Original: f
Revised: f

Original: ( x 1 , y ) 1 N (cid:80) Ni =1 f
Revised: ( x 1 , y ) 1 N (cid:80) Ni =1 f

Original: ( x i , y ) (cid:35) =
Revised: ( x i , y ) (cid:35) =

Original: I InfoNCE ( X 1 ; Y ) .
Revised: I InfoNCE ( X 1 ; Y ) .

Original: For the ﬁrst " ≥ " we used that the Kullback-Leibler divergence is non-negative.
Revised: For the ﬁrst " ≥ " we used that the Kullback-Leibler divergence is non-negative.

Original: For the second " ≥ "we used the inequality ln a (cid:54) a − 1 for a > 0 .
Revised: For the second " ≥ "we used the inequality ln a (cid:54) a − 1 for a > 0 .

Original: Part (II) : Lower bound with probabilities.
Revised: Part (II) : Lower bound with probabilities.

Original: If the score function f isf ( x , y ) =
Revised: If the score function f isf ( x , y ) =

Original: p ( y | x ) , (A22) then the bound is
Revised: p ( y | x ) , (A22) then the bound is

Original: I( X 1 ; Y ) ≥ E p ( y ) p ( X | y ) (cid:34) ln (cid:32) f ( x 1 , y ) 1 N (cid:80) Ni =1 f ( x i , y ) (cid:33)(cid:35) =
Revised: I( X 1 ; Y ) ≥ E p ( y ) p ( X | y ) (cid:34) ln (cid:32) f ( x 1 , y ) 1 N (cid:80) Ni =1 f ( x i , y ) (cid:33)(cid:35) =

Original: E p ( y ) p ( X | y ) (cid:34) ln (cid:32) p ( y | x 1 ) 1 N (cid:80) Ni
Revised: E p ( y ) p ( X | y ) (cid:34) ln (cid:32) p ( y | x 1 ) 1 N (cid:80) Ni

Original: =1 p ( y | x i ) (cid:33)(cid:35) (A23) =
Revised: =1 p ( y | x i ) (cid:33)(cid:35) (A23) =

Original: E p ( y ) p ( X | y )  
Revised: E p ( y ) p ( X | y )  

Original: ln  p
Revised: ln 

Original: [Equation]
Revised: [Equation]

Original:  y [Equation] x 
Revised: p ( y | x 1 ) p ( y

Original: [Equation]
Revised: [Equation]

Original: y [Equation] N 
Revised: 1 N (cid:80)

Original: [Equation]
Revised: [Equation]

Original:  
Revised: p ( y | x i ) p ( y ) 

Original: [Equation]
Revised: [Equation]

Original: ( X [Equation] Y ) .
Revised: ( X 1 ; Y ) .

Original: This is the bound with probabilities in the theorem.
Revised: This is the bound with probabilities in the theorem.

Original: A.1.2 InfoLOOB: Upper Bound on Mutual Information
Revised: A.1.2 InfoLOOB: Upper Bound on Mutual Information

Original: We derive an upper bound on the mutual information between random variables X and Y distributedaccording to p ( x , y ) .
Revised: We derive an upper bound on the mutual information between random variables X and Y distributedaccording to p ( x , y ) .

Original: The mutual information I( X ; Y ) between random variables X
Revised: The mutual information I( X ; Y ) between random variables X

Original: and Y is
Revised: and Y is

Original: [Equation]
Revised: [Equation]

Original: X
Revised: X

Original: [Equation]
Revised: [Equation]

Original: In Poole et al.
Revised: In Poole et al.

Original: (2019) Eq.
Revised: (2019) Eq.

Original: (13) introduces a variational upper bound on the mutual information,which has been called "Leave one out upper bound" (called "L1Out" in Cheng et al.
Revised: (13) introduces a variational upper bound on the mutual information,which has been called "Leave one out upper bound" (called "L1Out" in Cheng et al.

Original: Forsimplicity, we call this bound "InfoLOOB", where LOOB is an acronym for "Leave One Out Bound".
Revised: Forsimplicity, we call this bound "InfoLOOB", where LOOB is an acronym for "Leave One Out Bound".

Original: In contrast to InfoNCE, InfoLOOB is an upper bound on the mutual information.
Revised: In contrast to InfoNCE, InfoLOOB is an upper bound on the mutual information.

Original: InfoLOOB is analogto InfoNCE except that the negative samples do not contain a positive sample.
Revised: InfoLOOB is analogto InfoNCE except that the negative samples do not contain a positive sample.

Original: Fig.
Revised: Fig.

Original: 1 and Fig.
Revised: 1 and Fig.

Original: 2 in
Revised: 2 in

Original: Cheng et al.
Revised: Cheng et al.

Original: (2020) both show that InfoLOOB is a better estimator for the mutual information than
Revised: (2020) both show that InfoLOOB is a better estimator for the mutual information than

Original: InfoNCE (van den Oord et al., 2018), MINE (Belghazi et al., 2018), and NWJ (Nguyen et al., 2010).
Revised: InfoNCE (van den Oord et al., 2018), MINE (Belghazi et al., 2018), and NWJ (Nguyen et al., 2010).

Original: The InfoLOOB with score function f ( x , y ) is deﬁned as
Revised: The InfoLOOB with score function f ( x , y ) is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: The InfoLOOB with probabilities is deﬁned as
Revised: The InfoLOOB with probabilities is deﬁned as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: This is the InfoLOOB Eq.
Revised: This is the InfoLOOB Eq.

Original: (A25) with f ( x , y )
Revised: (A25) with f ( x , y )

Original: =
Revised: =

Original: p ( y | x ) .
Revised: p ( y | x ) .

Original: The InfoLOOB with probabilities can be written in different forms:
Revised: The InfoLOOB with probabilities can be written in different forms:

Original: [Equation]
Revised: [Equation]

Original: ) =
Revised: =

Original: E p
Revised: E p

Original: [Equation]
Revised: [Equation]

Original: y )
Revised: y )

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original:     
Revised:     

Original:  .
Revised: [Equation]

Original: Set of pairs.
Revised: Set of pairs.

Original: The InfoLOOB can we written in a different setting (Poole et al., 2019), which willbe used in our implementations.
Revised: The InfoLOOB can we written in a different setting (Poole et al., 2019), which willbe used in our implementations.

Original: We sample N pairs independently from p ( x , y ) , which gives
Revised: We sample N pairs independently from p ( x , y ) , which gives

Original: X = { ( x 1 , y 1 ) , ( x 2 , y 2 ) , . . .
Revised: X = { ( x 1 , y 1 ) , ( x 2 , y 2 ) , . . .

Original: , ( x N , y N ) } .
Revised: , ( x N , y N ) } .

Original: The InfoLOOB is then
Revised: The InfoLOOB is then

Original: [Equation]
Revised: [Equation]

Revised: X

Revised: [Equation]

Original: We assume that an anchor sample y is given.
Revised: We assume that an anchor sample y is given.

Original: For the anchor sample y we draw a positive samplex 1 according to p ( x 1 | y ) .
Revised: For the anchor sample y we draw a positive samplex 1 according to p ( x 1 | y ) .

Original: Next, we draw a set ˜ X
Revised: Next, we draw a set ˜ X

Original: = { x 2 , . . .
Revised: = { x 2 , . . .

Original: , x N } of negative samples accordingto ˜ p
Revised: , x N } of negative samples accordingto ˜ p

Original: ( x | y ) .
Revised: ( x | y ) .

Original: For a given y , the x that have a large p ( x | y ) are drawn with a lower probability˜ p ( x | y ) compared to random drawing via p ( x ) .
Revised: For a given y , the x that have a large p ( x | y ) are drawn with a lower probability˜ p ( x | y ) compared to random drawing via p ( x ) .

Original: The negatives are indeed negatives.
Revised: The negatives are indeed negatives.

Original: We havedrawn ﬁrst anchor sample y and then X = { x 1 , . . .
Revised: We havedrawn ﬁrst anchor sample y and then X = { x 1 , . . .

Original: , x N } , where x 1 is drawn according to p ( x 1 | y )and ˜ X
Revised: , x N } , where x 1 is drawn according to p ( x 1 | y )and ˜ X

Original: = { x 2 , . . .
Revised: = { x 2 , . . .

Original: , x N } are drawn iid according to ˜ p ( x | y ) .
Revised: , x N } are drawn iid according to ˜ p ( x | y ) .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: We assume for score function f ( x , y )
Revised: We assume for score function f ( x , y )∀

Original: [Equation]
Revised: [Equation]

Original: We ensure this by using for score function f
Revised: ∀

Original: [Equation]
Revised: [Equation]

Original: (A
Revised: f ( [Equation] [Equation] We ensure this by using for score function ff (

Original: [Equation]
Revised: [Equation]

Original:  where sim( x , y ) is typically the cosine similarity.
Revised: (A33) where sim( x , y ) is typically the cosine similarity.

Original: InfoLOOB with score function f ( x , y ) and with undersampling via ˜ p
Revised: InfoLOOB with score function f ( x , y ) and with undersampling via ˜ p

Original: ( X | y ) is (compare thedeﬁnition of InfoLOOB Eq.
Revised: ( X | y ) is (compare thedeﬁnition of InfoLOOB Eq.

Original: (A25) without undersampling):
Revised: (A25) without undersampling):

Original: [Equation]
Revised: [Equation]

Original: The reference constant Z ( y ) gives the average score f
Revised: The reference constant Z ( y ) gives the average score f

Original: ( x , y ) , if the negatives for y are selected withlower probability via ˜ p
Revised: ( x , y ) , if the negatives for y are selected withlower probability via ˜ p

Original: ( x | y ) than with random drawing according to p ( x ) .
Revised: ( x | y ) than with random drawing according to p ( x ) .

Original: [Equation]
Revised: [Equation]

Original: We deﬁne the variational distribution
Revised: We deﬁne the variational distribution

Original: [Equation]
Revised: [Equation]

Original: With the variational distribution q ( x | y ) , we express our main assumption.
Revised: With the variational distribution q ( x | y ) , we express our main assumption.

Original: The main assumptionfor the bound is:
Revised: The main assumptionfor the bound is:

Original: [Equation]
Revised: [Equation]

Original: This assumption can be written as
Revised: This assumption can be written as

Original: [Equation]
Revised: [Equation]

Original: This assumption ensures that the x with large p ( x | y ) ) are selected with lower probability via˜ p ( x | y ) than with random drawing according to p ( x ) .
Revised: This assumption ensures that the x with large p ( x | y ) ) are selected with lower probability via˜ p ( x | y ) than with random drawing according to p ( x ) .

Original: The negatives are ensured to be real negatives,that is, p ( x | y ) is small and so is f ( x , y ) .
Revised: The negatives are ensured to be real negatives,that is, p ( x | y ) is small and so is f ( x , y ) .

Original: Consequently, we make sure that we draw x withsufﬁcient small f ( x , y ) .
Revised: Consequently, we make sure that we draw x withsufﬁcient small f ( x , y ) .

Original: The Kullback-Leibler gives the minimal required gap between drawingf ( x , y ) via p ( x ) and drawing f ( x , y ) via ˜ p
Revised: The Kullback-Leibler gives the minimal required gap between drawingf ( x , y ) via p ( x ) and drawing f ( x , y ) via ˜ p

Original: ( x | y ) .
Revised: ( x | y ) .

Original: EXAMPLE.
Revised: EXAMPLE.

Original: With h ( y ) > 0 , we consider the setting
Revised: With h ( y ) > 0 , we consider the setting

Original: [Equation]
Revised: [Equation]

Original: The main assumption becomes
Revised: The main assumption becomes

Original: [Equation]
Revised: [Equation]

Original: The main assumption holds since
Revised: The main assumption holds since

Original: [Equation]
Revised: [Equation]

Original: where we used for the (cid:54) Jensen’s inequality with the function f ( a ) = 1 /a , which is convex fora > 0 .
Revised: where we used for the (cid:54) Jensen’s inequality with the function f ( a ) = 1 /a , which is convex fora > 0 .

Original: For score function f ( x , y ) and distribution ˜ p ( x | y ) for sampling the negative samples, we havedeﬁned:
Revised: For score function f ( x , y ) and distribution ˜ p ( x | y ) for sampling the negative samples, we havedeﬁned:

Original: [Equation]
Revised: [Equation]

Original: Next theorem gives the upper bound of the InfoLOOB on the mutual information, which is
Revised: Next theorem gives the upper bound of the InfoLOOB on the mutual information, which is

Original: [Equation]
Revised: [Equation]

Original: Theorem A2 (InfoLOOB upper bound) .
Revised: Theorem A2 (InfoLOOB upper bound) .

Original: If ˜ X = { x 2 , . . .
Revised: If ˜ X = { x 2 , . . .

Original: , x N } are drawn iid according to ˜ p ( x | y )and if the main assumption holds:
Revised: , x N } are drawn iid according to ˜ p ( x | y )and if the main assumption holds:

Original: [Equation]
Revised: [Equation]

Original: Then InfoLOOB with score function f ( x , y ) and undersampling positives by ˜ p
Revised: Then InfoLOOB with score function f ( x , y ) and undersampling positives by ˜ p

Original: ( X | y ) is an upperbound on the mutual information:
Revised: ( X | y ) is an upperbound on the mutual information:

Original: [Equation]
Revised: [Equation]

Original: If the negative samples ˜ X
Revised: If the negative samples ˜ X

Original: = { x 2 , . . .
Revised: = { x 2 , . . .

Original: , x N } are drawn iid according to p ( x ) , then InfoLOOB withprobabilities according to Eq.
Revised: , x N } are drawn iid according to p ( x ) , then InfoLOOB withprobabilities according to Eq.

Original: (A26) is an upper bound on the mutual information:
Revised: (A26) is an upper bound on the mutual information:

Original: [Equation]
Revised: [Equation]

Original: The second bound Eq.
Revised: The second bound Eq.

Original: (A49) is a special case of the ﬁrst bound Eq.
Revised: (A49) is a special case of the ﬁrst bound Eq.

Original: (A48) .
Revised: (A48) .

Original: Proof.
Revised: Proof.

Original: Part (I) : Upper bound with score function f ( x , y ) .
Revised: Part (I) : Upper bound with score function f ( x , y ) .

Original: [Equation]
Revised: [Equation]

Original: 1 N − 1 (cid:80) Ni =
Revised: 1 N − 1 (cid:80) Ni =

Original: 2 f ( x i , y ) (cid:105) 
Revised: 2 f ( x i , y ) (cid:105) 

Original:  
Revised:  

Original:  =
Revised:  =

Original: E p ( x 1 , y )
Revised: E p ( x 1 , y )

Original: [ln f ( x 1 , y )]
Revised: [ln f ( x 1 , y )]

Original: − E p ( y ) (cid:34) ln (cid:32)
Revised: − E p ( y ) (cid:34) ln (cid:32)

Original: E ˜ p
Revised: E ˜ p

Original: ( X | y ) (cid:34)
Revised: ( X | y ) (cid:34)

Original: N −
Revised: N −

Original: N (cid:88) i = f
Revised: N (cid:88) i = f

Original: ( x i , y ) (cid:35)(cid:33)(cid:35) (cid:54) E p ( x 1 , y )
Revised: ( x i , y ) (cid:35)(cid:33)(cid:35) (cid:54) E p ( x 1 , y )

Original: [ln f ( x 1 , y )]
Revised: [ln f ( x 1 , y )]

Original: − E p ( y ) (cid:34)
Revised: − E p ( y ) (cid:34)

Original: E ˜ p
Revised: E ˜ p

Original: ( X | y ) (cid:34) ln
Revised: ( X | y ) (cid:34) ln

Original: (cid:32)
Revised: (cid:32)

Original: N −
Revised: N −

Original: N (cid:88) i = f
Revised: N (cid:88) i = f

Original: ( x i , y ) (cid:33)(cid:35)(cid:35) =
Revised: ( x i , y ) (cid:33)(cid:35)(cid:35) =

Original: E p ( y ) (cid:34)
Revised: E p ( y ) (cid:34)

Original: E ˜ p
Revised: E ˜ p

Original: ( X | y ) (cid:34) ln
Revised: ( X | y ) (cid:34) ln

Original: (cid:32) f
Revised: (cid:32) f

Original: ( x 1 , y ) 1 N − 1 (cid:80) Ni
Revised: ( x 1 , y ) 1 N − 1 (cid:80) Ni

Original: =2 f ( x i , y ) (cid:33)(cid:35)(cid:35)
Revised: =2 f ( x i , y ) (cid:33)(cid:35)(cid:35)

Original: [Equation]
Revised: [Equation]

Original: , where the ﬁrst " (cid:54) " uses assumption Eq.
Revised: , where the ﬁrst " (cid:54) " uses assumption Eq.

Original: (A37), while Jensens’s inequality was used for the second" (cid:54) " by exchanging the expectation and the " ln ".
Revised: (A37), while Jensens’s inequality was used for the second" (cid:54) " by exchanging the expectation and the " ln ".

Original: We also used
Revised: We also used

Original: E ˜ p
Revised: E ˜ p

Original: ( X | y ) (cid:34)
Revised: ( X | y ) (cid:34)

Original: N −
Revised: N −

Original: N (cid:88) i = f
Revised: N (cid:88) i = f

Original: ( x i , y ) (cid:35) =
Revised: ( x i , y ) (cid:35) =

Original: N −
Revised: N −

Original: N (cid:88) i =
Revised: N (cid:88) i =

Original: E ˜ p
Revised: E ˜ p

Original: ( x i | y )
Revised: ( x i | y )

Original: [ f ( x i , y )]
Revised: [ f ( x i , y )]

Original: =
Revised: =

Original: N −
Revised: N −

Original: N (cid:88) i =
Revised: N (cid:88) i =

Original: Z ( y ) =
Revised: Z ( y ) =

Original: Z ( y ) .
Revised: Z ( y ) .

Original: (A51)
Revised: (A51)

Original: Part (II) : Upper bound with probabilities.
Revised: Part (II) : Upper bound with probabilities.

Original: If the score function f is
Revised: If the score function f is

Original: [Equation]
Revised: [Equation]

Original: A52) and˜ p
Revised: A52) and˜ p

Original: [Equation]
Revised: [Equation]

Revised: x

Original: [Equation]
Revised: [Equation]

Revised: p

Revised: [Equation]

Revised: x ) ,

Revised: [Equation]

Original: A53) then
Revised: A53) then

Original: [Equation]
Revised: [Equation]

Original: A54)
Revised: A54)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Therefore, the main assumption holds, since
Revised: Therefore, the main assumption holds, since

Original: [Equation]
Revised: [Equation]

Original: The bound becomes
Revised: The bound becomes

Original: [Equation]
Revised: [Equation]

Original: An alternative proof is as follows:
Revised: An alternative proof is as follows:

Original: [Equation]
Revised: [Equation]

Original: x i 
Revised: x i ) p

Original: [Equation]
Revised: [Equation]

Original: p
Revised: y

Original: [Equation]
Revised: [Equation]

Original: (cid:33)(cid:35)(cid:35) 
Revised: (cid:33)(cid:35)(cid:35) =

Original: [Equation]
Revised: [Equation]

Original: (cid:19)(cid:21)(cid:21)
Revised: (cid:19)(cid:21)(cid:21)

Original: [Equation]
Revised: [Equation]

Original: p (
Revised: p (

Original: [Equation]
Revised: [Equation]

Original: (cid:34)
Revised: (cid:34)

Original: E p ( [Equation] ) (cid:34) ln (cid:32)
Revised: E p ( X | y ) (cid:34) ln (cid:32)

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: N (cid:88) i = p ( x i | y ) p ( x i ) (cid:33)(cid:35)(cid:35)
Revised: N (cid:88) i = p ( x i | y ) p ( x i ) (cid:33)(cid:35)(cid:35)

Original: [Equation]
Revised: [Equation]

Original: where we applied Jensens’s inequality for the exchanging the expectation and the " ln " to obtain the" (cid:54) " inequality.
Revised: where we applied Jensens’s inequality for the exchanging the expectation and the " ln " to obtain the" (cid:54) " inequality.

Original: Experiments that compare upper and lower bounds as mutual information estimates are providedin Cheng et al.
Revised: Experiments that compare upper and lower bounds as mutual information estimates are providedin Cheng et al.

Original: (2020) and in Poole et al.
Revised: (2020) and in Poole et al.

Original: In Fig.
Revised: In Fig.

Original: 2 in Cheng et al.
Revised: 2 in Cheng et al.

Original: (2020) it is shown that
Revised: (2020) it is shown that

Original: InfoLOOB is a good estimator of the mutual information.
Revised: InfoLOOB is a good estimator of the mutual information.

Original: A.1.3 InfoLOOB: Analysis of the Objective
Revised: A.1.3 InfoLOOB: Analysis of the Objective

Original: This subsection justiﬁes the maximization of the InfoLOOB bound for contrastive learning.
Revised: This subsection justiﬁes the maximization of the InfoLOOB bound for contrastive learning.

Original: Maximizing the InfoLOOB bound is not intuitive as it was introduced as an upper bound on the mutualinformation in the previous subsection.
Revised: Maximizing the InfoLOOB bound is not intuitive as it was introduced as an upper bound on the mutualinformation in the previous subsection.

Original: Still maximizing the InfoLOOB bound leads to a goodapproximation of the mutual information, in particular for high mutual information.
Revised: Still maximizing the InfoLOOB bound leads to a goodapproximation of the mutual information, in particular for high mutual information.

Original: InfoLOOB with a neural network as a scoring function is not an upper bound on the mutual information when not under-sampling.
Revised: InfoLOOB with a neural network as a scoring function is not an upper bound on the mutual information when not under-sampling.

Original: As we use InfoLOOB on training data for which we do not knowthe sampling procedure, we cannot assume under-sampling.
Revised: As we use InfoLOOB on training data for which we do not knowthe sampling procedure, we cannot assume under-sampling.

Original: Therefore, we elaborate more on therationale behind the maximization of the InfoLOOB bound.
Revised: Therefore, we elaborate more on therationale behind the maximization of the InfoLOOB bound.

Original: (I)
Revised: (I)

Original: We show that InfoLOOB with neuralnetworks as scoring function is bounded from above.
Revised: We show that InfoLOOB with neuralnetworks as scoring function is bounded from above.

Original: Therefore, there exists a maximum and theoptimization problem is well deﬁned.
Revised: Therefore, there exists a maximum and theoptimization problem is well deﬁned.

Original: (II)
Revised: (II)

Original: We show that InfoLOOB with neural networks as scoringfunction differs by two terms from the mutual information.
Revised: We show that InfoLOOB with neural networks as scoringfunction differs by two terms from the mutual information.

Original: The ﬁrst term is the Kullback-Leiblerdivergence between the variational q ( x | y ) and the posterior p ( x | y ) .
Revised: The ﬁrst term is the Kullback-Leiblerdivergence between the variational q ( x | y ) and the posterior p ( x | y ) .

Original: This divergence is minimalfor q ( x | y )
Revised: This divergence is minimalfor q ( x | y )

Original: =
Revised: =

Original: p ( x | y ) , which implies f ( y | x ) = p ( y | x ) .
Revised: p ( x | y ) , which implies f ( y | x ) = p ( y | x ) .

Original: The second term is governed bythe difference between the mean E[ f ( x , y )] and the empirical mean 1 / ( N − 1) (cid:80) i f
Revised: The second term is governed bythe difference between the mean E[ f ( x , y )] and the empirical mean 1 / ( N − 1) (cid:80) i f

Original: ( x i , y ) .
Revised: ( x i , y ) .

Original: The
Revised: The

Original: Hoeffding bound can be applied to this difference.
Revised: Hoeffding bound can be applied to this difference.

Original: Therefore, the second term is negligible for large
Revised: Therefore, the second term is negligible for large

Original: N .
Revised: N .

Original: In contrast, the KL term is dominant and the relevant term, therefore maximizing InfoLOOBleads to f ( y | x ) ≈ p
Revised: In contrast, the KL term is dominant and the relevant term, therefore maximizing InfoLOOBleads to f ( y | x ) ≈ p

Original: ( y | x ) .
Revised: ( y | x ) .

Original: We assume that an anchor sample y is given.
Revised: We assume that an anchor sample y is given.

Original: For the anchor sample y , we draw a positive sample x
Revised: For the anchor sample y , we draw a positive sample x

Original: 1according to p ( x 1 | y ) .
Revised: 1according to p ( x 1 | y ) .

Original: We deﬁne the set ˜ X
Revised: We deﬁne the set ˜ X

Original: = { x 2 , . . .
Revised: = { x 2 , . . .

Original: , x N } of negative samples, which are drawniid according to p ( x ) .
Revised: , x N } of negative samples, which are drawniid according to p ( x ) .

Original: We deﬁne the set X
Revised: We deﬁne the set X

Original: = { x 1 , . . .
Revised: = { x 1 , . . .

Original: , x N } .
Revised: , x N } .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: We use the score function
Revised: We use the score function

Original: [Equation]
Revised: [Equation]

Original: where sim( x , y ) is typically the cosine similarity.
Revised: where sim( x , y ) is typically the cosine similarity.

Original: The InfoLOOB with score function f ( x , y ) is deﬁned asI InfoLOOB
Revised: The InfoLOOB with score function f ( x , y ) is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: (A66)

Original: We deﬁne the variational distribution
Revised: We deﬁne the variational distribution

Original: [Equation]
Revised: [Equation]

Original: A67)
Revised: A67)

Original: [Equation]
Revised: [Equation]

Original: (A68)
Revised: (A68)

Original: The next inequality shows the relation between I( X 1 ; Y ) and I InfoLOOB ( X 1 ; Y ) for randomvariables X 1 and Y .
Revised: The next inequality shows the relation between I( X 1 ; Y ) and I InfoLOOB ( X 1 ; Y ) for randomvariables X 1 and Y .

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: N
Revised: N

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: i = f
Revised: i = f

Revised: [Equation]

Revised: x i

Original: [Equation]
Revised: [Equation]

Revised: y ) (cid:35)(cid:33)(cid:35)

Revised: [Equation]

Original: y
Revised: E

Revised: [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  y ) (cid:107) q
Revised: 1 | y ) (cid:107) q

Original: ( x 1 | y ))]
Revised: x 1 | y ))]

Original: =
Revised: =

Original: E p ( x 1 , y )
Revised: E p ( x 1 , y )

Original: [ln f ( x 1 , y )]
Revised: [ln f ( x 1 , y )]

Original: − E p ( y ) (cid:34)
Revised: − E p ( y ) (cid:34)

Original: E p ( X | y ) (cid:34) ln (cid:32)
Revised: E p ( X | y ) (cid:34) ln (cid:32)

Original: N −
Revised: N −

Original: N (cid:88) i = f
Revised: N (cid:88) i = f

Original: ( x i , y ) (cid:33)(cid:35)(cid:35) + E p ( y ) (cid:34)
Revised: ( x i , y ) (cid:33)(cid:35)(cid:35) + E p ( y ) (cid:34)

Original: E p ( X | y ) (cid:34) ln (cid:32)
Revised: E p ( X | y ) (cid:34) ln (cid:32)

Original: N −
Revised: N −

Original: N (cid:88) i = f
Revised: N (cid:88) i = f

Original: ( x i , y ) (cid:33)(cid:35)(cid:35) − E p
Revised: ( x i , y ) (cid:33)(cid:35)(cid:35) − E p

Original: ( y ) (cid:34) ln
Revised: ( y ) (cid:34) ln

Original: (cid:32)
Revised: (cid:32)

Original: E p ( X | y ) (cid:34)
Revised: E p ( X | y ) (cid:34)

Original: N −
Revised: N −

Original: N (cid:88) i = f
Revised: N (cid:88) i = f

Original: ( x i , y ) (cid:35)(cid:33)(cid:35) + E p ( y )
Revised: ( x i , y ) (cid:35)(cid:33)(cid:35) + E p ( y )

Original: [KL( p ( x 1 | y ) (cid:107) q
Revised: [KL( p ( x 1 | y ) (cid:107) q

Original: x 1 | y ))]
Revised: x 1 | y ))]

Original: =
Revised: =

Original: E p ( y ) (cid:34)
Revised: E p ( y ) (cid:34)

Original: E p ( X | y ) (cid:34) ln (cid:32) f
Revised: E p ( X | y ) (cid:34) ln (cid:32) f

Original: ( x 1 , y ) 1 N − 1 (cid:80) Ni
Revised: ( x 1 , y ) 1 N − 1 (cid:80) Ni

Original: =2 f ( x i , y ) (cid:33)(cid:35)(cid:35)
Revised: =2 f ( x i , y ) (cid:33)(cid:35)(cid:35)

Original: + E p ( y ) (cid:34)
Revised: + E p ( y ) (cid:34)

Original: E p ( X | y ) (cid:34) ln (cid:32)
Revised: E p ( X | y ) (cid:34) ln (cid:32)

Original: N −
Revised: N −

Original: N (cid:88) i = f
Revised: N (cid:88) i = f

Original: ( x i , y ) (cid:33)(cid:35)(cid:35) − E p
Revised: ( x i , y ) (cid:33)(cid:35)(cid:35) − E p

Original: ( y ) (cid:34) ln
Revised: ( y ) (cid:34) ln

Original: (cid:32)
Revised: (cid:32)

Original: E p ( X | y ) (cid:34)
Revised: E p ( X | y ) (cid:34)

Original: N −
Revised: N −

Original: N (cid:88) i = f
Revised: N (cid:88) i = f

Original: ( x i , y ) (cid:35)(cid:33)(cid:35) + E p ( y )
Revised: ( x i , y ) (cid:35)(cid:33)(cid:35) + E p ( y )

Original: [KL( p ( x 1 | y ) (cid:107) q
Revised: [KL( p ( x 1 | y ) (cid:107) q

Original: x 1 | y ))]
Revised: x 1 | y ))]

Original: = I InfoLOOB ( X 1 ; Y )
Revised: = I InfoLOOB ( X 1 ; Y )

Original: + E p
Revised: + E p

Original: ( y ) (cid:34)
Revised: ( y ) (cid:34)

Original: E p ( X | y ) (cid:34) ln (cid:32)
Revised: E p ( X | y ) (cid:34) ln (cid:32)

Original: N −
Revised: N −

Original: N (cid:88) i = f
Revised: N (cid:88) i = f

Original: ( x i , y ) (cid:33)(cid:35)(cid:35) − E p
Revised: ( x i , y ) (cid:33)(cid:35)(cid:35) − E p

Original: ( y ) (cid:34) ln
Revised: ( y ) (cid:34) ln

Original: (cid:32)
Revised: (cid:32)

Original: E p ( X | y ) (cid:34)
Revised: E p ( X | y ) (cid:34)

Original: N −
Revised: N −

Original: N (cid:88) i = f
Revised: N (cid:88) i = f

Original: ( x i , y ) (cid:35)(cid:33)(cid:35) + E p ( y )
Revised: ( x i , y ) (cid:35)(cid:33)(cid:35) + E p ( y )

Original: [KL( p ( x 1 | y ) (cid:107) q
Revised: [KL( p ( x 1 | y ) (cid:107) q

Original: x 1 | y ))]
Revised: x 1 | y ))]

Original: = I InfoLOOB ( X 1 ; Y )
Revised: = I InfoLOOB ( X 1 ; Y )

Original: + E p
Revised: + E p

Original: ( y ) (cid:34)
Revised: ( y ) (cid:34)

Original: E p ( X | y ) (cid:34) ln (cid:32)
Revised: E p ( X | y ) (cid:34) ln (cid:32)

Original: N −
Revised: N −

Original: N (cid:88) i = f
Revised: N (cid:88) i = f

Original: ( x i , y ) (cid:33)(cid:35)(cid:35) − E p
Revised: ( x i , y ) (cid:33)(cid:35)(cid:35) − E p

Original: ( y ) (cid:2)
Revised: ( y ) (cid:2)

Original: ln (cid:0) E p ( x 1 )
Revised: ln (cid:0) E p ( x 1 )

Original: [ f ( x 1 , y )]
Revised: [ f ( x 1 , y )]

Original: (cid:1)(cid:3)
Revised: (cid:1)(cid:3)

Original: + E p
Revised: + E p

Original: y )
Revised: y )

Original: [KL( p ( x 1 | y ) (cid:107) q
Revised: [KL( p ( x 1 | y ) (cid:107) q

Original: x 1 | y ))]
Revised: x 1 | y ))]

Original: =
Revised: =

Original: I InfoLOOB ( X 1 ; Y ) − E p
Revised: I InfoLOOB ( X 1 ; Y ) − E p

Original: ( y ) (cid:34)
Revised: ( y ) (cid:34)

Original: E p ( ˜ X ) (cid:34) ln (cid:32)
Revised: E p ( ˜ X ) (cid:34) ln (cid:32)

Original: E p ( x 1 ) [ f ( x 1 , y )]
Revised: E p ( x 1 ) [ f ( x 1 , y )]

Original: 1 N − 1 (cid:80) Ni =
Revised: 1 N − 1 (cid:80) Ni =

Original: 2 f ( x i , y ) (cid:33)(cid:35)(cid:35)
Revised: 2 f ( x i , y ) (cid:33)(cid:35)(cid:35)

Original: [Equation]

Original: =

Original: [Equation]
Revised: [Equation]

Original:  InfoLOOB
Revised: = I InfoLOOB

Original: [Equation]
Revised: [Equation]

Original:  where we used
Revised: ] , where we used

Original: [Equation]
Revised: [Equation]

Original: A
Revised: A

Original: [Equation]
Revised: [Equation]

Original: ( DE for difference of expectations) and
Revised: ( DE for difference of expectations) and

Original: [Equation]
Revised: [Equation]

Original: Since both KL and DE are non-negative (for DE see below), to increase InfoLOOB we have eitherto decrease KL or to increase DE .
Revised: Since both KL and DE are non-negative (for DE see below), to increase InfoLOOB we have eitherto decrease KL or to increase DE .

Original: Bounding DE .
Revised: Bounding DE .

Original: Next we bound DE .
Revised: Next we bound DE .

Original: We deﬁne
Revised: We deﬁne

Original: [Equation]
Revised: [Equation]

Original: The log-sum-exp function ( lse ) is
Revised: The log-sum-exp function ( lse ) is

Original: [Equation]
Revised: [Equation]

Original: for β > 0 and vector
Revised: for β > 0 and vector

Original: a = ( a 1 , . . . , a N ) .
Revised: a = ( a 1 , . . . , a N ) .

Original: The lse is a convex function (Lemma 4 in (Gao & Pavel, 2017)).
Revised: The lse is a convex function (Lemma 4 in (Gao & Pavel, 2017)).

Original: We obtain via Jensen’s inequalityand the convex lse :
Revised: We obtain via Jensen’s inequalityand the convex lse :

Original: E p
Revised: E p

Original: [Equation]
Revised: [Equation]

Original: y
Revised: y

Original: [Equation]
Revised: [Equation]

Original: Again using Jensen’s inequality and the concave ln , we get
Revised: Again using Jensen’s inequality and the concave ln , we get

Original: [Equation]
Revised: [Equation]

Original: = 0 .
Revised: = 0 .

Original: If we combine both previous inequalities, we obtain
Revised: If we combine both previous inequalities, we obtain

Original: [Equation]
Revised: [Equation]

Original: DE
Revised: DE

Original: [Equation]
Revised: [Equation]

Original: A
Revised: A

Original: [Equation]
Revised: [Equation]

Original: In particular, for bounded sim( x 1 , y ) , we get
Revised: In particular, for bounded sim( x 1 , y ) , we get

Original: [Equation]
Revised: [Equation]

Original: while Hoeffding’s lemma gives
Revised: while Hoeffding’s lemma gives

Original: [Equation]
Revised: [Equation]

Original: Thus, for bounded sim( x 1 , y ) , DE is bounded, therefore also InfoLOOB.
Revised: Thus, for bounded sim( x 1 , y ) , DE is bounded, therefore also InfoLOOB.

Original: Next, we show that DE is small.
Revised: Next, we show that DE is small.

Original: The Hoeffding bound (Proposition 2.5 in Wainwright (2019))
Revised: The Hoeffding bound (Proposition 2.5 in Wainwright (2019))

Original: statesthat if the random variable S f
Revised: statesthat if the random variable S f

Original: = f ( X 1 , y ) with X 1 ∼ p ( x 1 ) is σ 2 f -sub
Revised: = f ( X 1 , y ) with X 1 ∼ p ( x 1 ) is σ 2 f -sub

Original: Gaussian then with randomvariables S if iid distributed like S fp
Revised: Gaussian then with randomvariables S if iid distributed like S fp

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:33)
Revised: (cid:33)

Original: [Equation]
Revised: [Equation]

Original: (cid:15)
Revised: (cid:15)

Original: [Equation]
Revised: [Equation]

Original: (cid:33)
Revised: (cid:33)

Original: [Equation]
Revised: [Equation]

Original: (A80)
Revised: (A80)

Original: If S f ∈ [ a, b ] (e.g. if f ( x , y ) ∈ [ a, b ] ) then we can set σ f = ( b − a ) / 2 .
Revised: If S f ∈ [ a, b ] (e.g. if f ( x , y ) ∈ [ a, b ] ) then we can set σ f = ( b − a ) / 2 .

Original: For
Revised: For

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: ) we have [Equation] (A82)
Revised: )] −

Original: [Equation]
Revised: [Equation]

Original: x
Revised: i =

Original: [Equation]
Revised: [Equation]

Original: , y ) (cid:54) (cid:15) 
Revised: (cid:15) (A81) we haveln

Original: Z − (cid:15) , where we used ln a (cid:54) a − 1 for 0 < a .
Revised: [Equation]

Original: Analog for
Revised: [Equation]

Original: N − N (cid:88) i = f ( x i , y ) − E p ( x 1 ) [ f ( x 1 , y )](cid:54) (cid:15) (A83) we haveln (cid:32)
Revised: , y )]

Original: [Equation]
Revised: [Equation]

Original: x

Original: [Equation]

Original: [Equation]

Original: ( x 1 , y )]

Original: [Equation]

Original: [Equation]

Original: ) (cid:33)
Revised: (cid:33)

Original: [Equation]
Revised: [Equation]

Original: (cid:18) E p ( x 1 ) [ f ( x [Equation] y )]
Revised: where we used ln a (cid:54) a − 1 for 0 < a .

Original: E p ( x 1 ) [ f ( x

Original: [Equation] y )] [Equation] (cid:15) (cid:19) (A84) =
Revised: Analog for

Original: [Equation] (cid:18) [Equation] p ( x 1 ) [ f ( x
Revised: N −

Original: [Equation]

Original: y )] + (cid:15) E p ( x
Revised: N (cid:88) i = f

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: f ( x

Original: [Equation]
Revised: [Equation]

Original: ] (cid:19)
Revised: A83) we have

Original: [Equation]
Revised: [Equation]

Original: − (cid:15)
Revised: (cid:15)

Original: E p ( x 1 ) [ f ( x 1 , y )]

Original: = − (cid:15)

Original: Z , where we used − ln a ≥ 1 − a for 0 < a .
Revised: Z , where we used − ln a ≥ 1 − a for 0 < a .

Original: In summary, if for all y
Revised: In summary, if for all y

Original: [Equation]

Original: [Equation]

Original: N

Original: [Equation]
Revised: [Equation]

Original: i = [Equation] f [Equation]
Revised: ) then we have

Original: (cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: ) then we have

Original: [Equation]
Revised: [Equation]

Original: It follows that
Revised: It follows that

Original: [Equation]
Revised: [Equation]

Original: Consequently, for large N , the Hoeffding bound Eq.
Revised: Consequently, for large N , the Hoeffding bound Eq.

Original: (A79) holds with high probability, if (cid:15) is chosenreasonably.
Revised: (A79) holds with high probability, if (cid:15) is chosenreasonably.

Original: Therefore, with high probability the term DE is small.
Revised: Therefore, with high probability the term DE is small.

Original: Next, we show that DE is governed by the variance of sim( x , y ) for unmatched pairs.
Revised: Next, we show that DE is governed by the variance of sim( x , y ) for unmatched pairs.

Original: In Eq.
Revised: In Eq.

Original: (A76) the term DE is upper bounded:
Revised: (A76) the term DE is upper bounded:

Original: [Equation]
Revised: [Equation]

Original: We express these equations via the random variable S
Revised: We express these equations via the random variable S

Original: = sim( X 1 , y ) with s
Revised: = sim( X 1 , y ) with s

Original: = sim( x 1 , y ) , whichreplaces the random variable X 1 and its realization x 1 .
Revised: = sim( x 1 , y ) , whichreplaces the random variable X 1 and its realization x 1 .

Original: [Equation]
Revised: [Equation]

Original: where we used the Dirac delta-distribution δ and for s = sim( x 1 , y ) we deﬁned
Revised: where we used the Dirac delta-distribution δ and for s = sim( x 1 , y ) we deﬁned

Original: [Equation]
Revised: [Equation]

Original: Eq.
Revised: Eq.

Original: (A76) can be written as
Revised: (A76) can be written as

Original: [Equation]
Revised: [Equation]

Original: with ¯ s = E p ( s )
Revised: with ¯ s = E p ( s )

Original: [ s ] .
Revised: [ s ] .

Original: We assume that the random variable S with realization s = sim( x 1 , y ) is sub-Gaussian, where yis given and x 1 is drawn independently from y according to p ( x 1 ) .
Revised: We assume that the random variable S with realization s = sim( x 1 , y ) is sub-Gaussian, where yis given and x 1 is drawn independently from y according to p ( x 1 ) .

Original: Therefore, we assume that thesimilarity of a random matching is sub-Gaussian.
Revised: Therefore, we assume that thesimilarity of a random matching is sub-Gaussian.

Original: This assumption is true for bounded similarities(like the cosine similarity) and for almost sure bounded similarities.
Revised: This assumption is true for bounded similarities(like the cosine similarity) and for almost sure bounded similarities.

Original: The assumption is true if usingvectors that are retrieved from a continuous modern Hopﬁeld network since they are bounded bythe largest stored vector.
Revised: The assumption is true if usingvectors that are retrieved from a continuous modern Hopﬁeld network since they are bounded bythe largest stored vector.

Original: This assumption is true for a continuous similarity function if x , y , andparameters are bounded, since the bounded x , y , and parameters can be embedded in a compact seton which the similarity has a maximum.
Revised: This assumption is true for a continuous similarity function if x , y , andparameters are bounded, since the bounded x , y , and parameters can be embedded in a compact seton which the similarity has a maximum.

Original: This assumption is true for learned similarities if the input isbounded.
Revised: This assumption is true for learned similarities if the input isbounded.

Original: For a random variable S that is σ 2 -sub-Gaussian
Revised: For a random variable S that is σ 2 -sub-Gaussian

Original: (Deﬁnition 2.2 in Wainwright (2019))
Revised: (Deﬁnition 2.2 in Wainwright (2019))

Original: the constantσ 2 is called a proxy variance .
Revised: the constantσ 2 is called a proxy variance .

Original: The minimal proxy variance σ 2opt is called the optimal proxy variancewith Var[ S ] (cid:54) σ 2opt (Arbel et al., 2019).
Revised: The minimal proxy variance σ 2opt is called the optimal proxy variancewith Var[ S ] (cid:54) σ 2opt (Arbel et al., 2019).

Original: S is strictly sub-Gaussian, if σ 2opt =
Revised: S is strictly sub-Gaussian, if σ 2opt =

Original: Var[ S ] .
Revised: Var[ S ] .

Original: Proposition 2.1in Arbel et al.
Revised: Proposition 2.1in Arbel et al.

Original: (2019) states
Revised: (2019) states

Original: [Equation]
Revised: [Equation]

Original: with µ = E[ S ] .
Revised: with µ = E[ S ] .

Original: The supremum is attained for almost surely bounded random variables S .
Revised: The supremum is attained for almost surely bounded random variables S .

Original: Eq.
Revised: Eq.

Original: (4) in
Revised: (4) in

Original: Arbel et al.
Revised: Arbel et al.

Original: (2019) states
Revised: (2019) states

Original: [Equation]
Revised: [Equation]

Original: Thus, for S being sub-Gaussian, we have
Revised: Thus, for S being sub-Gaussian, we have

Original: [Equation]
Revised: [Equation]

Original: where σ 2opt is the optimal proxy variance of S .
Revised: where σ 2opt is the optimal proxy variance of S .

Original: For example, bounded random variables S ∈ [ a, b ]are sub-Gaussian with σ opt (cid:54) ( b − a ) / 

Revised: For example, bounded random variables S ∈ [ a, b ]are sub-Gaussian with σ opt (cid:54) ( b − a ) /

Original: (Exercise 2.4 in Wainwright (2019)).
Revised: (Exercise 2.4 in Wainwright (2019)).

Original: KL is decreased by making the variation distribution q ( x 1 | y ) more similar to the posterior p ( x 1 | y ) .
Revised: KL is decreased by making the variation distribution q ( x 1 | y ) more similar to the posterior p ( x 1 | y ) .

Original: The value DE only depends on the marginal distributions p ( y ) and p ( x ) , since p ( ˜ X )
Revised: The value DE only depends on the marginal distributions p ( y ) and p ( x ) , since p ( ˜ X )

Original: = (cid:81)
Revised: = (cid:81)

Original: Ni =2 p ( x i ) .
Revised: Ni =2 p ( x i ) .

Original: The value DE can be changed by adding an offset to f ( x , y ) .
Revised: The value DE can be changed by adding an offset to f ( x , y ) .

Original: However, scaling f ( x , y ) by a factordoes not change DE .
Revised: However, scaling f ( x , y ) by a factordoes not change DE .

Original: Consequently, DE is difﬁcult to change.
Revised: Consequently, DE is difﬁcult to change.

Original: Therefore, increasing InfoLOOB is most effective by making q ( x 1 | y ) more similar to the posteriorp ( x 1 | y ) .
Revised: Therefore, increasing InfoLOOB is most effective by making q ( x 1 | y ) more similar to the posteriorp ( x 1 | y ) .

Original: Gradient of InfoLOOB expressed by gradients of KL and DE .
Revised: Gradient of InfoLOOB expressed by gradients of KL and DE .

Original: Assume that the similarity isparametrized by w giving sim( x , y ; w ) .
Revised: Assume that the similarity isparametrized by w giving sim( x , y ; w ) .

Original: [Equation]
Revised: [Equation]

Original: where C is independent of w .
Revised: where C is independent of w .

Original: Next, we compute the derivative of KL with respect to parameters w .∂ KL
Revised: Next, we compute the derivative of KL with respect to parameters w .∂ KL

Original: [Equation]
Revised: [Equation]

Original: The derivative is the average difference between the posterior distribution p ( x 1 | y ) and the variationaldistribution q ( x 1 | y ) multiplied by the derivative of the similarity function.
Revised: The derivative is the average difference between the posterior distribution p ( x 1 | y ) and the variationaldistribution q ( x 1 | y ) multiplied by the derivative of the similarity function.

Original: If both distributionmatch, then the derivative vanishes.
Revised: If both distributionmatch, then the derivative vanishes.

Original: Next, we compute the derivative of DE with respect to parameters w .∂ DE
Revised: Next, we compute the derivative of DE with respect to parameters w .∂ DE

Original: [Equation]
Revised: [Equation]

Original: y ; w ) ∂ w d x(cid:21) − τ − 1 E p ( y ) (cid:34)
Revised: y ; w ) ∂ w d x(cid:21) − τ − 1 E p ( y ) (cid:34)

Original: E p ( ˜ X ) (cid:34)
Revised: E p ( ˜ X ) (cid:34)

Original: N −
Revised: N −

Original: N (cid:88) i = f
Revised: N (cid:88) i = f

Original: ( x i , y ) 1 N − 1 (cid:80) Nj =2 f ( x j , y ) ∂
Revised: ( x i , y ) 1 N − 1 (cid:80) Nj =2 f ( x j , y ) ∂

Original: sim( x i , y ; w ) ∂ w (cid:35)(cid:35) =
Revised: sim( x i , y ; w ) ∂ w (cid:35)(cid:35) =

Original: τ − 1 E p ( y ) (cid:20)
Revised: τ − 1 E p ( y ) (cid:20)

Original: E p ( x 1 ) (cid:20) f
Revised: E p ( x 1 ) (cid:20) f

Original: ( x 1 , y )
Revised: ( x 1 , y )

Original: E p ( x ) [ f ( x , y )] ∂ sim( x 1 , y ; w ) ∂ w (cid:21)(cid:21) − τ − 1 E p ( y ) (cid:34)
Revised: E p ( x ) [ f ( x , y )] ∂ sim( x 1 , y ; w ) ∂ w (cid:21)(cid:21) − τ − 1 E p ( y ) (cid:34)

Original: E p ( ˜ X ) (cid:34)
Revised: E p ( ˜ X ) (cid:34)

Original: N −
Revised: N −

Original: N (cid:88) i = f
Revised: N (cid:88) i = f

Original: ( x i , y ) 1 N − 1 (cid:80) Nj =2 f ( x j , y ) ∂
Revised: ( x i , y ) 1 N − 1 (cid:80) Nj =2 f ( x j , y ) ∂

Original: sim( x i , y ; w ) ∂ w (cid:35)(cid:35) =
Revised: sim( x i , y ; w ) ∂ w (cid:35)(cid:35) =

Original: τ − 1 E p ( y ) (cid:34)
Revised: τ − 1 E p ( y ) (cid:34)

Original: N −
Revised: N −

Original: N (cid:88) i =
Revised: N (cid:88) i =

Original: E p ( x i ) (cid:20) f
Revised: E p ( x i ) (cid:20) f

Original: ( x i , y )
Revised: ( x i , y )

Original: E p ( x ) [ f ( x , y )] ∂ sim( x i , y ; w ) ∂ w (cid:21)(cid:35) − τ − 1 E p ( y ) (cid:34)
Revised: E p ( x ) [ f ( x , y )] ∂ sim( x i , y ; w ) ∂ w (cid:21)(cid:35) − τ − 1 E p ( y ) (cid:34)

Original: E p ( ˜ X ) (cid:34)
Revised: E p ( ˜ X ) (cid:34)

Original: N −
Revised: N −

Original: N (cid:88) i = f
Revised: N (cid:88) i = f

Original: ( x i , y ) 1 N − 1 (cid:80) Nj =2 f ( x j , y ) ∂
Revised: ( x i , y ) 1 N − 1 (cid:80) Nj =2 f ( x j , y ) ∂

Original: sim( x i , y ; w ) ∂ w (cid:35)(cid:35) =
Revised: sim( x i , y ; w ) ∂ w (cid:35)(cid:35) =

Original: τ − 1 E p ( y ) (cid:34)
Revised: τ − 1 E p ( y ) (cid:34)

Original: E p ( ˜ X ) (cid:34)
Revised: E p ( ˜ X ) (cid:34)

Original: N −
Revised: N −

Original: N (cid:88) i = f
Revised: N (cid:88) i = f

Original: ( x i , y )
Revised: ( x i , y )

Original: E p ( x ) [ f ( x , y )] ∂ sim( x i , y ; w ) ∂ w (cid:35)(cid:35) −
Revised: E p ( x ) [ f ( x , y )] ∂ sim( x i , y ; w ) ∂ w (cid:35)(cid:35) −

Original: τ − 1 E p ( y ) (cid:34)
Revised: τ − 1 E p ( y ) (cid:34)

Original: E p ( ˜ X ) (cid:34)
Revised: E p ( ˜ X ) (cid:34)

Original: N −
Revised: N −

Original: N (cid:88) i = f
Revised: N (cid:88) i = f

Original: ( x i , y ) 1 N − 1 (cid:80) Nj =2 f ( x j , y ) ∂
Revised: ( x i , y ) 1 N − 1 (cid:80) Nj =2 f ( x j , y ) ∂

Original: sim( x i , y ; w ) ∂ w (cid:35)(cid:35) =
Revised: sim( x i , y ; w ) ∂ w (cid:35)(cid:35) =

Original: τ − 1 E p ( y ) (cid:34)
Revised: τ − 1 E p ( y ) (cid:34)

Original: E p ( ˜ X ) (cid:34)
Revised: E p ( ˜ X ) (cid:34)

Original: N −
Revised: N −

Original: N (cid:88) i = (cid:32)
Revised: N (cid:88) i = (cid:32)

Original: E p ( x ) [ f ( x , y )] − 1 N − 1 (cid:80) Nj =2 f ( x j , y ) (cid:33) f
Revised: E p ( x ) [ f ( x , y )] − 1 N − 1 (cid:80) Nj =2 f ( x j , y ) (cid:33) f

Original: ( x i , y ) ∂
Revised: ( x i , y ) ∂

Original: sim( x i , y ; w ) ∂ w (cid:35)(cid:35) =
Revised: sim( x i , y ; w ) ∂ w (cid:35)(cid:35) =

Original: τ − 1 E p ( y ) (cid:34)
Revised: τ − 1 E p ( y ) (cid:34)

Original: E p ( ˜ X ) (cid:34)
Revised: E p ( ˜ X ) (cid:34)

Original: N −
Revised: N −

Original: N (cid:88) i = (cid:32)
Revised: N (cid:88) i = (cid:32)

Original: Z − 1 N − 1 (cid:80) Nj =2 f ( x j , y ) (cid:33) f
Revised: Z − 1 N − 1 (cid:80) Nj =2 f ( x j , y ) (cid:33) f

Original: ( x i , y ) ∂
Revised: ( x i , y ) ∂

Original: sim( x i , y ; w ) ∂ w (cid:35)(cid:35) .
Revised: sim( x i , y ; w ) ∂ w (cid:35)(cid:35) .

Original: The derivative is the average of 1 Z − 1 1 N − 1 (cid:80) Nj =2 f ( x j , y ) multiplied by the score function and thederivative of the similarity function.
Revised: The derivative is the average of 1 Z − 1 1 N − 1 (cid:80) Nj =2 f ( x j , y ) multiplied by the score function and thederivative of the similarity function.

Original: The average is over y and ˜ X , therefore the whole derivativebecomes even smaller.
Revised: The average is over y and ˜ X , therefore the whole derivativebecomes even smaller.

Original: Consequently, for small b − a and large N , the derivative of DE is small.
Revised: Consequently, for small b − a and large N , the derivative of DE is small.

Original: Note that for
Revised: Note that for

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: therefore
Revised: therefore

Original: [Equation]
Revised: [Equation]

Original: If the expectation Z is well approximated by the average
Revised: If the expectation Z is well approximated by the average

Original: [Equation]
Revised: [Equation]

Original: Nj
Revised: Nj

Original: [Equation]
Revised: [Equation]

Original: x j , y ) , then both DE and itsgradient are small.
Revised: x j [Equation] y ) , then both DE and itsgradient are small.

Original: Derivative of InfoLOOB via KL and DE :
Revised: Derivative of InfoLOOB via KL and DE :

Original: [Equation]
Revised: [Equation]

Original: In this gradient, the KL term is dominating, therefore f ( x , y ) is pushed to approximate the conditionalprobability p ( y | x ) .
Revised: In this gradient, the KL term is dominating, therefore f ( x , y ) is pushed to approximate the conditionalprobability p ( y | x ) .

Original: Modern Hopﬁeld networks lead to larger values of p ( y | x ) as the mutualinformation becomes larger, therefore modern Hopﬁeld networks help to push f ( x , y ) to large values.
Revised: Modern Hopﬁeld networks lead to larger values of p ( y | x ) as the mutualinformation becomes larger, therefore modern Hopﬁeld networks help to push f ( x , y ) to large values.

Original: Furthermore, modern Hopﬁeld networks increase Z , which is in the denominator of the bound on
Revised: Furthermore, modern Hopﬁeld networks increase Z , which is in the denominator of the bound on

Original: DE and its derivative.
Revised: DE and its derivative.

Original: A.1.4 InfoNCE and InfoLOOB: Gradients
Revised: A.1.4 InfoNCE and InfoLOOB: Gradients

Original: We consider the InfoNCE and the InfoLOOB loss function.
Revised: We consider the InfoNCE and the InfoLOOB loss function.

Original: For computing the lossfunction, we sample N pairs independently from p ( x , y ) , which gives the training set{ ( x 1 , y 1 ) , ( x 2 , y 2 ) , . . .
Revised: For computing the lossfunction, we sample N pairs independently from p ( x , y ) , which gives the training set{ ( x 1 , y 1 ) , ( x 2 , y 2 ) , . . .

Original: , ( x N , y N ) } .
Revised: , ( x N , y N ) } .

Original: InfoNCE and InfoLOOB only differ in using the positiveexample in the negatives.
Revised: InfoNCE and InfoLOOB only differ in using the positiveexample in the negatives.

Original: More precisely, for the sample y 1 , InfoNCE uses for the matrix of negativesamples X = ( x 1 , . . .
Revised: More precisely, for the sample y 1 , InfoNCE uses for the matrix of negativesamples X = ( x 1 , . . .

Original: , x N ) , while InfoLOOB uses ˜ X
Revised: , x N ) , while InfoLOOB uses ˜ X

Original: =
Revised: =

Original: ( x 2 , . .
Revised: ( x 2 , . .

Original: , x N ) .
Revised: , x N ) .

Original: InfoNCE.
Revised: InfoNCE.

Original: The InfoNCE loss is
Revised: The InfoNCE loss is

Original: [Equation]
Revised: [Equation]

Original: where we used
Revised: where we used

Original: [Equation]
Revised: [Equation]

Original: For the score function f ( x , y ) , we use
Revised: For the score function f ( x , y ) , we use

Original: [Equation]
Revised: [Equation]

Original: A
Revised: A

Original: [Equation]
Revised: [Equation]

Original: with τ as the temperature.
Revised: with τ as the temperature.

Original: The loss function for this score function is
Revised: The loss function for this score function is

Original: [Equation]
Revised: [Equation]

Original: A [Equation] where lse is the log-sum-exp function ( lse ): lse
Revised: where lse is the log-sum-exp function ( lse ):lse( β, a )

Original: [Equation] , a [Equation]
Revised: =

Original:  1 log [Equation] [Equation] βa
Revised: β − 1 log (cid:32) N (cid:88)

Original: [Equation] ) (cid:33) [Equation] (A108) for β > 0 and vector a
Revised: i = exp( βa i ) (cid:33) , (A108) for β > 0 and vector a

Original: =
Revised: =

Original: ( a 1 , . . .
Revised: ( a 1 , . . .

Original: The gradient with respect to y is
Revised: The gradient with respect to y is

Original: [Equation]
Revised: [Equation]

Original: which is the positive example x 1 that ﬁts to the anchor example y minus the Hopﬁeld network updatewith state pattern y and stored patterns X
Revised: which is the positive example x 1 that ﬁts to the anchor example y minus the Hopﬁeld network updatewith state pattern y and stored patterns X

Original: and then this difference multiplied by τ − 1 .
Revised: and then this difference multiplied by τ − 1 .

Original: This gradient can be simpliﬁed, since the positive example x 1 is also in the negative examples.
Revised: This gradient can be simpliﬁed, since the positive example x 1 is also in the negative examples.

Original: Usingp =
Revised: Usingp =

Original: ( p 1 , . . .
Revised: ( p 1 , . . .

Original: , p N ) T = softmax (cid:0) τ − 1 X T y (cid:1) , we obtain
Revised: , p N ) T = softmax (cid:0) τ − 1 X T y (cid:1) , we obtain

Original: [Equation]
Revised: [Equation]

Original: InfoNCE (
Revised: y

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: is the softmax average over the negatives x i for 2 (cid:54) i (cid:54) N without x 1 .
Revised: is the softmax average over the negatives x i for 2 (cid:54) i (cid:54) N without x 1 .

Original: It can be easily seen that1 1 − p 1 (cid:80) Ni =2 p i =
Revised: It can be easily seen that1 1 − p 1 (cid:80) Ni =2 p i =

Original: − p 1 1 − p 1 = 1 .
Revised: − p 1 1 − p 1 = 1 .

Original: For the derivative of the InfoLOOB see below.
Revised: For the derivative of the InfoLOOB see below.

Original: The gradient with respect to x 1 is
Revised: The gradient with respect to x 1 is

Original: [Equation]
Revised: [Equation]

Original: Consequently, the learning rate is scaled by (1 − p 1 ) .
Revised: Consequently, the learning rate is scaled by (1 − p 1 ) .

Original: The sum of gradients with respect to x 1 and x i is
Revised: The sum of gradients with respect to x 1 and x i is

Original: [Equation]
Revised: [Equation]

Original: InfoNCE

Original: [Equation]

Original: y
Revised: y

Original: [Equation]
Revised: [Equation]

Original: where 1 is the vector with ones.
Revised: where 1 is the vector with ones.

Original: However, the derivatives with respect to the weights are not zerosince the x
Revised: However, the derivatives with respect to the weights are not zerosince the x

Original: i are differently computed.
Revised: i are differently computed.

Original: InfoLOOB.
Revised: InfoLOOB.

Original: The InfoLOOB loss is
Revised: The InfoLOOB loss is

Original: [Equation]
Revised: [Equation]

Original: where we used
Revised: where we used

Original: [Equation]
Revised: [Equation]

Original: For the score function f ( x , y ) , we use
Revised: For the score function f ( x , y ) , we use

Original: [Equation]
Revised: [Equation]

Original: with τ as the temperature.
Revised: with τ as the temperature.

Original: The loss function for this score function is
Revised: The loss function for this score function is

Original: [Equation]
Revised: [Equation]

Original: where lse is the log-sum-exponential function.
Revised: where lse is the log-sum-exponential function.

Original: The gradient with respect to y is
Revised: The gradient with respect to y is

Original: [Equation]
Revised: [Equation]

Original: which is the positive example x 1 that ﬁts to the anchor example y minus the Hopﬁeld network updatewith state pattern y and stored patterns ˜ X
Revised: which is the positive example x 1 that ﬁts to the anchor example y minus the Hopﬁeld network updatewith state pattern y and stored patterns ˜ X

Original: and then this difference multiplied by τ − 1 .
Revised: and then this difference multiplied by τ − 1 .

Original: The gradient with respect to x 1 is
Revised: The gradient with respect to x 1 is

Original: [Equation]
Revised: [Equation]

Original: The sum of gradients with respect to x 1 and x i is
Revised: The sum of gradients with respect to x 1 and x i is

Original: [Equation]
Revised: [Equation]

Original: where 1 is the vector with ones.
Revised: where 1 is the vector with ones.

Original: However, the derivatives with respect to the weights are not zerosince the x
Revised: However, the derivatives with respect to the weights are not zerosince the x

Original: i are differently computed.
Revised: i are differently computed.

Original: Gradients with respect to τ − 1 .
Revised: Gradients with respect to τ − 1 .

Original: The gradient of the InfoNCE loss Eq.
Revised: The gradient of the InfoNCE loss Eq.

Original: (A103) using the similarity
Revised: (A103) using the similarity

Original: Eq.
Revised: Eq.

Original: (A105) with respect to τ − 1 is
Revised: (A105) with respect to τ − 1 is

Original: [Equation]
Revised: [Equation]

Original: which is the similarity of the anchor y with the difference of the positive example x 1 and the Hopﬁeldnetwork update with state pattern y and stored patterns X .
Revised: which is the similarity of the anchor y with the difference of the positive example x 1 and the Hopﬁeldnetwork update with state pattern y and stored patterns X .

Original: The gradient of the InfoLOOB loss
Revised: The gradient of the InfoLOOB loss

Original: Eq.
Revised: Eq.

Original: (A115) using the similarity Eq.
Revised: (A115) using the similarity Eq.

Original: (A117) with respect to τ − 1 is
Revised: (A117) with respect to τ − 1 is

Original: [Equation]
Revised: [Equation]

Original: ) with the difference compared to Eq.
Revised: with the difference compared to Eq.

Original: (A123) that the Hopﬁeld network update is done withstored patterns ˜ X instead of X .
Revised: (A123) that the Hopﬁeld network update is done withstored patterns ˜ X instead of X .

Original: Without the positive example x 1 in the stored patterns˜ X , the term x 1 − ˜ X softmax (cid:16) τ − 1 ˜ X
Revised: Without the positive example x 1 in the stored patterns˜ X , the term x 1 − ˜ X softmax (cid:16) τ − 1 ˜ X

Original: T y (cid:17) in Eq.
Revised: T y (cid:17) in Eq.

Original: (A125) will not decrease like the termx 1 − X softmax (cid:0) τ − 1 X T y (cid:1) in Eq.
Revised: (A125) will not decrease like the termx 1 − X softmax (cid:0) τ − 1 X T y (cid:1) in Eq.

Original: (A123) but grow even larger with better separation of thepositive and negative examples.
Revised: (A123) but grow even larger with better separation of thepositive and negative examples.

Original: A.1.5 InfoLOOB and InfoNCE: Probability Estimators
Revised: A.1.5 InfoLOOB and InfoNCE: Probability Estimators

Original: In McAllester & Stratos (2018, 2020) it was shown that estimators of the mutual information by lowerbounds have problems as they come with serious statistical limitations.
Revised: In McAllester & Stratos (2018, 2020) it was shown that estimators of the mutual information by lowerbounds have problems as they come with serious statistical limitations.

Original: Statistically more justiﬁed forrepresenting the mutual information is a difference of entropies, which are estimated by minimizingthe cross-entropy loss.
Revised: Statistically more justiﬁed forrepresenting the mutual information is a difference of entropies, which are estimated by minimizingthe cross-entropy loss.

Original: Both InfoNCE and InfoLOOB losses can be viewed as cross-entropy losses.
Revised: Both InfoNCE and InfoLOOB losses can be viewed as cross-entropy losses.

Original: We sample
Revised: We sample

Original: N pairs independently from p ( x , y ) , which gives
Revised: N pairs independently from p ( x , y ) , which gives

Original: Z ={ ( x 1 , y 1 ) , ( x 2 , y 2 ) , . .
Revised: Z ={ ( x 1 , y 1 ) , ( x 2 , y 2 ) , . .

Original: , ( x N , y N ) } .
Revised: , ( x N , y N ) } .

Original: We set X
Revised: We set X

Original: = { x 1 , x 2 , . . .
Revised: = { x 1 , x 2 , . . .

Original: , x N } and Y = { y 1 , y 2 , . .
Revised: , x N } and Y = { y 1 , y 2 , . .

Original: , y N } ,so that, Z = X × Y .
Revised: , y N } ,so that, Z = X × Y .

Original: The score function f ( x , y ) is an estimator for p ( x , y ) .
Revised: The score function f ( x , y ) is an estimator for p ( x , y ) .

Original: Then we obtain estimators ˆ q for the conditional probabilities.
Revised: Then we obtain estimators ˆ q for the conditional probabilities.

Original: ˆ q
Revised: ˆ q

Original: ( y i | x i , Y \ { y i } ) is an estimator for p ( y i | x i ) andˆ q
Revised: ( y i | x i , Y \ { y i } ) is an estimator for p ( y i | x i ) andˆ q

Original: ( x i |
Revised: ( x i |

Original: y i , X \ { x i } ) an estimator for p ( x i |
Revised: y i , X \ { x i } ) an estimator for p ( x i |

Original: y i ) .
Revised: y i ) .

Original: Each estimator ˆ q uses beyond ( x i , y i ) additionalsamples to estimate the normalizing constant.
Revised: Each estimator ˆ q uses beyond ( x i , y i ) additionalsamples to estimate the normalizing constant.

Original: For InfoNCE these estimators are
Revised: For InfoNCE these estimators are

Original: [Equation]
Revised: [Equation]

Original: The cross-entropy losses for the InfoNCE estimators are
Revised: The cross-entropy losses for the InfoNCE estimators are

Original: [Equation]
Revised: [Equation]

Original: For InfoLOOB these estimators are
Revised: For InfoLOOB these estimators are

Original: [Equation]
Revised: [Equation]

Original: The cross-entropy losses for the InfoLOOB estimators are
Revised: The cross-entropy losses for the InfoLOOB estimators are

Original: [Equation]
Revised: [Equation]

Original: The InfoLOOB estimator uses for normalization
Revised: The InfoLOOB estimator uses for normalization

Original: [Equation]
Revised: [Equation]

Original: in contrast to InfoNCE
Revised: in contrast to InfoNCE

Original: [Equation]
Revised: [Equation]

Original: which uses
Revised: which uses

Original: [Equation]
Revised: [Equation]

Original:  j ) .
Revised: y j ) .

Original: (A138)
Revised: (A138)

Original: If InfoNCE estimates the normalizing constant separately, then it would be biased.
Revised: If InfoNCE estimates the normalizing constant separately, then it would be biased.

Original: ( x i , y i ) is drawnaccording to p ( x i , y i ) instead of p ( x i ) p ( y i ) .
Revised: ( x i , y i ) is drawnaccording to p ( x i , y i ) instead of p ( x i ) p ( y i ) .

Original: In contrast, if InfoLOOB estimated the normalizingconstant separately, then it would be unbiased.
Revised: In contrast, if InfoLOOB estimated the normalizingconstant separately, then it would be unbiased.

Original: A.1.6 InfoLOOB and InfoNCE: Losses
Revised: A.1.6 InfoLOOB and InfoNCE: Losses

Original: We have N pairs drawn iid from p ( x , y ) , where we assume that a pair ( x i , y i ) is alreadyan embedding of the original drawn pair.
Revised: We have N pairs drawn iid from p ( x , y ) , where we assume that a pair ( x i , y i ) is alreadyan embedding of the original drawn pair.

Original: These build up the embedding training set Z =
Revised: These build up the embedding training set Z =

Original: { ( x 1 , y 1 ) , ( x 2 , y 2 ) , . . .
Revised: { ( x 1 , y 1 ) , ( x 2 , y 2 ) , . . .

Original: , ( x N , y N ) } that allows to construct the matrices X
Revised: , ( x N , y N ) } that allows to construct the matrices X

Original: =
Revised: =

Original: ( x 1 , x 2 , . . .
Revised: ( x 1 , x 2 , . . .

Original: , x N )of N embedding samples x i and Y =
Revised: , x N )of N embedding samples x i and Y =

Original: ( y 1 , y 2 , . . .
Revised: ( y 1 , y 2 , . . .

Original: , y N ) of N embedding samples
Revised: , y N ) of N embedding samples

Original: y i .
Revised: y i .

Original: We also have
Revised: We also have

Original: M stored patterns U
Revised: M stored patterns U

Original: = ( u 1 , . .
Revised: = ( u 1 , . .

Original: . , u M ) and K stored patterns V = ( v 1 , . .
Revised: . , u M ) and K stored patterns V = ( v 1 , . .

Original: . , v K ) .
Revised: . , v K ) .

Original: The state vectors x i and y i are the queries for the Hopﬁeld networks, which retrieve some vectorsfrom U or V .
Revised: The state vectors x i and y i are the queries for the Hopﬁeld networks, which retrieve some vectorsfrom U or V .

Original: We normalize vectors (cid:107) x i (cid:107)
Revised: We normalize vectors (cid:107) x i (cid:107)

Original: =
Revised: =

Original: (cid:107) y
Revised: (cid:107) y

Original: i (cid:107) =
Revised: i (cid:107) =

Original: (cid:107) u
Revised: (cid:107) u

Original: i (cid:107) =
Revised: i (cid:107) =

Original: (cid:107) v i (cid:107) = 1 .
Revised: (cid:107) v i (cid:107) = 1 .

Original: The following vectors areretrieved from modern Hopﬁeld networks (Ramsauer et al., 2021):
Revised: The following vectors areretrieved from modern Hopﬁeld networks (Ramsauer et al., 2021):

Original: [Equation]
Revised: [Equation]

Original: where U x i denotes an image-retrieved image embedding,
Revised: where U x i denotes an image-retrieved image embedding,

Original: U y i
Revised: U y i

Original: a text-retrieved image embedding,
Revised: a text-retrieved image embedding,

Original: V x i an image-retrieved text embedding and V y i a text-retrieved text embedding.
Revised: V x i an image-retrieved text embedding and V y i a text-retrieved text embedding.

Original: The hyperparameterβ corresponds to the inverse temperature:
Revised: The hyperparameterβ corresponds to the inverse temperature:

Original: β = 0 retrieves the average of the stored pattern, whilelarge β retrieve the stored pattern that is most similar to the state pattern (query).
Revised: β = 0 retrieves the average of the stored pattern, whilelarge β retrieve the stored pattern that is most similar to the state pattern (query).

Original: We consider the loss functions
Revised: We consider the loss functions

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where for InfoLOOB the sum (cid:80) j (cid:54)
Revised: where for InfoLOOB the sum (cid:80) j (cid:54)

Original: =
Revised: =

Original: i in the denominator contains only negative examples j .
Revised: i in the denominator contains only negative examples j .

Original: We donot consider the loss function L H − UVUV InfoLOOB because of the high variance in the dot product U T x
Revised: We donot consider the loss function L H − UVUV InfoLOOB because of the high variance in the dot product U T x

Original: i V y
Revised: i V y

Original: i aselaborated in the following.
Revised: i as elaborated in the following.

Original: Let us consider the dot product between the anchor retrieval with the positive pattern retrieval for theloss functions with Hopﬁeld.
Revised: Let us consider the dot product between the anchor retrieval with the positive pattern retrieval for theloss functions with Hopﬁeld.

Original: In the ﬁrst term of the loss function Eq.
Revised: In the ﬁrst term of the loss function Eq.

Original: (A145), U x
Revised: (A145), U x

Original: i is the anchor with
Revised: i is the anchor with

Original: V y i as the positive sample and V y
Revised: V y i as the positive sample and V y

Original: i with U x i as the positive sample for the second term, since theanchor also appears in each term of the denominator.
Revised: i with U x i as the positive sample for the second term, since theanchor also appears in each term of the denominator.

Original: Equivalently the same is valid for Eq.
Revised: Equivalently the same is valid for Eq.

Original: (A146),but with positive samples V x i
Revised: (A146),but with positive samples V x i

Original: and U y i respectively.
Revised: and U y i respectively.

Original: These dot products can be written as
Revised: These dot products can be written as

Original: [Equation]
Revised: [Equation]

Original: High variance of U T x i V y
Revised: High variance of U T x i V y

Original: i .
Revised: i .

Original: To compute the dot product U T x
Revised: To compute the dot product U T x

Original: i V y
Revised: i V y

Original: i , M + K stored patterns are required( M of the u j and K of the v j ).
Revised: i , M + K stored patterns are required( M of the u j and K of the v j ).

Original: In contrast, the dot products U T x i
Revised: In contrast, the dot products U T x i

Original: U y
Revised: U y

Original: i and V T x i V y
Revised: i and V T x i V y

Original: i require only Mor respectively K stored patterns.
Revised: i require only Mor respectively K stored patterns.

Original: Therefore, U T x i V y
Revised: Therefore, U T x i V y

Original: i has higher variance than both U T x
Revised: i has higher variance than both U T x

Original: i U
Revised: i U

Original: y i
Revised: y i

Original: and
Revised: and

Original: V T x i V y i .
Revised: V T x i V y i .

Original: Covariance structure extracted by U T x
Revised: Covariance structure extracted by U T x

Original: i U
Revised: i U

Original: y i
Revised: y i

Original: and
Revised: and

Original: V T x i V y
Revised: V T x i V y

Original: i .
Revised: i .

Original: The Jacobian J of the softmax p =softmax( β a ) is
Revised: The Jacobian J of the softmax p =softmax( β a ) is

Original: [Equation]
Revised: [Equation]

Original: which is a symmetric, positive semi-deﬁnite matrix with one eigenvalue of zero for eigenvector 1 .
Revised: which is a symmetric, positive semi-deﬁnite matrix with one eigenvalue of zero for eigenvector 1 .

Original: J( β a ) is diagonally dominant since | p i (1 − p i ) | − (cid:80)
Revised: J( β a ) is diagonally dominant since | p i (1 − p i ) | − (cid:80)

Original: j (cid:54)
Revised: j (cid:54)

Original: =
Revised: =

Original: i | p i p
Revised: i | p i p

Original: j | =
Revised: j | =

Original: p
Revised: p

Original: i − (cid:80) j p i p
Revised: i − (cid:80) j p i p

Original: j = p
Revised: j = p

Original: i − p i = 0 .
Revised: i − p i = 0 .

Original: Next we give upper bounds on the norm of J .
Revised: Next we give upper bounds on the norm of J .

Original: Lemma A1.
Revised: Lemma A1.

Original: For a softmax p = softmax( β x ) with m = max i p i (1 − p i ) , the spectral norm of the
Revised: For a softmax p = softmax( β x ) with m = max i p i (1 − p i ) , the spectral norm of the

Original: Jacobian J of the softmax is bounded:
Revised: Jacobian J of the softmax is bounded:

Original: [Equation]
Revised: [Equation]

Original: In particular everywhere holds
Revised: In particular everywhere holds

Original: [Equation]
Revised: [Equation]

Original: If p max = max i p i ≥ 1 − (cid:15) ≥ 0 .
Revised: If p max = max i p i ≥ 1 − (cid:15) ≥ 0 .

Original: 5 , then for the spectral norm of the Jacobian holds
Revised: 5 , then for the spectral norm of the Jacobian holds

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We consider the maximum absolute column sum norm
Revised: We consider the maximum absolute column sum norm

Original: [Equation]
Revised: [Equation]

Original: and the maximum absolute row sum norm
Revised: and the maximum absolute row sum norm

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We have for A
Revised: We have for A

Original: =
Revised: =

Original: J = β (cid:0) diag( p ) − pp T (cid:1)(cid:88)
Revised: J = β (cid:0) diag( p ) − pp T (cid:1)(cid:88)

Original: j | a ij | = β   p i (1 − p i ) + (cid:88)
Revised: j | a ij | = β   p i (1 − p i ) + (cid:88)

Original: [Equation]
Revised: [Equation]

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: The last inequality is a direct consequence of Hölder’s inequality.
Revised: The last inequality is a direct consequence of Hölder’s inequality.

Original: For 0 (cid:54) p i (cid:54) 1 , we have p i (1 − p i ) (cid:54) 0 .
Revised: For 0 (cid:54) p i (cid:54) 1 , we have p i (1 − p i ) (cid:54) 0 .

Original: Therefore, m (cid:54) 0 .
Revised: Therefore, m (cid:54) 0 .

Original: 25 for all values of p i .
Revised: 25 for all values of p i .

Original: If p max ≥ 1 − (cid:15) ≥ 0 .
Revised: If p max ≥ 1 − (cid:15) ≥ 0 .

Original: 5 ( (cid:15) (cid:54) 0 . 5 ), then 1 − p max (cid:54) (cid:15) and for p
Revised: 5 ( (cid:15) (cid:54) 0 . 5 ), then 1 − p max (cid:54) (cid:15) and for p

Original: i (cid:54) =
Revised: i (cid:54) =

Original: p max p i (cid:54) (cid:15) .
Revised: p max p i (cid:54) (cid:15) .

Original: The derivative∂x (1 − x ) /∂x
Revised: The derivative∂x (1 − x ) /∂x

Original: = 1 − 2 x > 0 for x < 0 .
Revised: = 1 − 2 x > 0 for x < 0 .

Original: 5 , therefore x (1 − x ) increases with x for x < 0 .
Revised: 5 , therefore x (1 − x ) increases with x for x < 0 .

Original: Usingx = 1 − p max and for p i (cid:54) =
Revised: Usingx = 1 − p max and for p i (cid:54) =

Original: p max x = p
Revised: p max x = p

Original: i , we obtain p
Revised: i , we obtain p

Original: i (1 − p i ) (cid:54) (cid:15) (1 − (cid:15) ) for all i .
Revised: i (1 − p i ) (cid:54) (cid:15) (1 − (cid:15) ) for all i .

Original: Consequently,we have m (cid:54) (cid:15) (1 − (cid:15) ) .
Revised: Consequently,we have m (cid:54) (cid:15) (1 − (cid:15) ) .

Original: For the softmax p = softmax( β a ) with Jacobian ∂ J /∂ a = J( β a ) =
Revised: For the softmax p = softmax( β a ) with Jacobian ∂ J /∂ a = J( β a ) =

Original: β (cid:0) diag( p ) − pp T (cid:1) and forarbitrary N -dimensional vectors b and c , we have
Revised: β (cid:0) diag( p ) − pp T (cid:1) and forarbitrary N -dimensional vectors b and c , we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: (A163)

Original: Therefore, b T J( β a ) c is β times the covariance between b and c if component i is drawn withprobability p i of the multinomial distribution p .
Revised: Therefore, b T J( β a ) c is β times the covariance between b and c if component i is drawn withprobability p i of the multinomial distribution p .

Original: In our case the component i is sample i .
Revised: In our case the component i is sample i .

Original: Using the mean ˆ u = 1 /M (cid:80) Mi =1 u i , the empirical covariance of data U is
Revised: Using the mean ˆ u = 1 /M (cid:80) Mi =1 u i , the empirical covariance of data U is

Original: [Equation]
Revised: [Equation]

Original: The weighted covariance (samples u i are drawn according to p i )
Revised: The weighted covariance (samples u i are drawn according to p i )

Original: [Equation]
Revised: [Equation]

Original: which replaces 1 /M from equal sampling by the p i , that is, u i is sampled with probability p i .
Revised: which replaces 1 /M from equal sampling by the p i , that is, u i is sampled with probability p i .

Original: The next theorem states how to express the dot product U T x
Revised: The next theorem states how to express the dot product U T x

Original: i U
Revised: i U

Original: y i by weighted covariances of the data
Revised: y i by weighted covariances of the data

Original: U .
Revised: U .

Original: Theorem A3 (Weighted Covariances) .
Revised: Theorem A3 (Weighted Covariances) .

Original: Using the weighted covariances
Revised: Using the weighted covariances

Original: [Equation]
Revised: [Equation]

Original: A169) where the mean Jacobian J m is symmetric, diagonally dominant, and positive semi-deﬁnite withspectral norm bounded by (cid:107) J m (cid:107) 2 (cid:54) 0 .
Revised: where the mean Jacobian J m is symmetric, diagonally dominant, and positive semi-deﬁnite withspectral norm bounded by (cid:107) J m (cid:107) 2 (cid:54) 0 .

Original: 5 β .
Revised: 5 β .

Original: The dot product U T x i U y i can be expressed by the weighted covariances
Revised: The dot product U T x i U y i can be expressed by the weighted covariances

Original: [Equation]
Revised: [Equation]

Original: where the mean is ¯ u = 1 /M
Revised: where the mean is ¯ u = 1 /M

Original: U 1 .
Revised: U 1 .

Original: Proof.
Revised: Proof.

Original: We apply the mean value theorem to the softmax with the symmetric, diagonally dominant,positive semi-deﬁnite Jacobian matrix J m 
Revised: We apply the mean value theorem to the softmax with the symmetric, diagonally dominant,positive semi-deﬁnite Jacobian matrix J m =

Original: [Equation]
Revised: [Equation]

Original: We set a (cid:48)
Revised: We set a (cid:48)

Original: = 

Revised: =

Original: and use β a instead of a , which gives:
Revised: and use β a instead of a , which gives:

Original: [Equation]
Revised: [Equation]

Original: which is exact.
Revised: which is exact.

Original: We obtain
Revised: We obtain

Original: [Equation]
Revised: [Equation]

Original: The spectral norm of J m is bounded by (cid:107) J m (cid:107) 2 (cid:54) 0 .
Revised: The spectral norm of J m is bounded by (cid:107) J m (cid:107) 2 (cid:54) 0 .

Original: 5 β , since this bound holds for every J( λβ a ) in
Revised: 5 β , since this bound holds for every J( λβ a ) in

Original: J m ( β a ) =
Revised: J m ( β a ) =

Original: (cid:82) 1 0 J( λβ a ) d λ according to Lemma A1.
Revised: (cid:82) 1 0 J( λβ a ) d λ according to Lemma A1.

Original: The dot product between the anchor retrieval and the positive sample is:
Revised: The dot product between the anchor retrieval and the positive sample is:

Original: [Equation]
Revised: [Equation]

Original: where we used the mean ¯ u = 1 /M
Revised: where we used the mean ¯ u = 1 /M

Original: U 1 and the weighted covariances
Revised: U 1 and the weighted covariances

Original: [Equation]
Revised: [Equation]

Original: A
Revised: A

Original: [Equation]
Revised: [Equation]

Original: The Jacobian J m is symmetric, diagonally dominant, and positive semi-deﬁnite.
Revised: The Jacobian J m is symmetric, diagonally dominant, and positive semi-deﬁnite.

Original: The weightedcovariance Cov( U , . ) is the covariance if the stored pattern u i is drawn according to an averagedp i given by J m ( . ) .
Revised: The weightedcovariance Cov( U , . ) is the covariance if the stored pattern u i is drawn according to an averagedp i given by J m ( . ) .

Original: Analog for weighted covariance Cov( V , . ) .
Revised: Analog for weighted covariance Cov( V , . ) .

Original: When maximizing the dot product
Revised: When maximizing the dot product

Original: U T x i
Revised: U T x i

Original: U
Revised: U

Original: y
Revised: y

Original: i , the normalized vectors x i and y i are encouraged to agree on drawing the patterns
Revised: i , the normalized vectors x i and y i are encouraged to agree on drawing the patterns

Original: u
Revised: u

Original: i withthe same probability p i to generate similar weighted covariance matrices
Revised: i withthe same probability p i to generate similar weighted covariance matrices

Original: Cov( U , . ) .
Revised: Cov( U , . ) .

Original: If subsets of Uhave a strong covariance structure, then it can be exploited to produce large weighted covariancesand, in turn, large dot products of U T x
Revised: If subsets of Uhave a strong covariance structure, then it can be exploited to produce large weighted covariancesand, in turn, large dot products of U T x

Original: i U y
Revised: i U y

Original: i .
Revised: i .

Original: Furthermore, for a large dot product U T x
Revised: Furthermore, for a large dot product U T x

Original: i
Revised: i

Original: U
Revised: U

Original: y
Revised: y

Original: i , x i andy i have to be similar to one another to extract the same direction from the covariance matrices.
Revised: i , x i andy i have to be similar to one another to extract the same direction from the covariance matrices.

Original: Allconsiderations are analog for V T x
Revised: Allconsiderations are analog for V T x

Original: i
Revised: i

Original: V
Revised: V

Original: y
Revised: y

Original: i .
Revised: i .

Original: A.2 Experiments
Revised: A.2 ExperimentsA.2.1 Ablation studies

Original: A.2.1 Ablation studies

Original: As detailed in the main part of the paper, CLOOB has two new main components compared to CLIP:(1)
Revised: As detailed in the main part of the paper, CLOOB has two new main components compared to CLIP:(1)

Original: the modern Hopﬁeld networks and (2) the InfoLOOB objective instead of the InfoNCE objective.
Revised: the modern Hopﬁeld networks and (2) the InfoLOOB objective instead of the InfoNCE objective.

Original: To assess effects of the new major components of CLOOB, we performed ablation studies on the CCand YFCC datasets.
Revised: To assess effects of the new major components of CLOOB, we performed ablation studies on the CCand YFCC datasets.

Original: The results are reported in Table A1 for models pre-trained on CC for 31 and128 epochs and in Table A2 for models pre-trained on YFCC for 28 epochs.
Revised: The results are reported in Table A1 for models pre-trained on CC for 31 and128 epochs and in Table A2 for models pre-trained on YFCC for 28 epochs.

Original: 128) compared to all other models
Revised: 128) compared to all other models

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: For the ablation studies above we use a ﬁxed inverse temperature parameter τ − 1 of 30 for allcompared models.
Revised: For the ablation studies above we use a ﬁxed inverse temperature parameter τ − 1 of 30 for allcompared models.

Original: The value of τ − 1 was determined via hyperparameter search (see Section A.2.2).
Revised: The value of τ − 1 was determined via hyperparameter search (see Section A.2.2).

Original: In contrast to CLIP, we use a learning rate scheduler with restarts (Loshchilov & Hutter, 2017) to bemore ﬂexible regarding the number of total training epochs and enable training up to a plateau.
Revised: In contrast to CLIP, we use a learning rate scheduler with restarts (Loshchilov & Hutter, 2017) to bemore ﬂexible regarding the number of total training epochs and enable training up to a plateau.

Original: Toinvestigate the inﬂuence of the learning rate scheduler, we performed experiments with and without restarts.
Revised: Toinvestigate the inﬂuence of the learning rate scheduler, we performed experiments with and without restarts.

Original: Table A3 shows the zero-shot performance for the different downstream tasks for CLIP and
Revised: Table A3 shows the zero-shot performance for the different downstream tasks for CLIP and

Original: CLOOB respectively.
Revised: CLOOB respectively.

Original: For both CLIP and CLOOB, the performance at the majority of the tasks eitherincreases or remains roughly the same with restarts.
Revised: For both CLIP and CLOOB, the performance at the majority of the tasks eitherincreases or remains roughly the same with restarts.

Original: [Table]
Revised: [Table]

Original: A.2.2 Hyperparameters
Revised: A.2.2 Hyperparameters

Original: [Table]

Original: The hyperparameter search was done on a validation split of CC with about 15,000 samples.
Revised: The hyperparameter search was done on a validation split of CC with about 15,000 samples.

Original: For thehyperparameter τ − 1 several values were considered ( 14 . 3 , 30 , 50 , 70 ), where 30 leads to the bestresults for both YFCC and CC.
Revised: For thehyperparameter τ − 1 several values were considered ( 14 . 3 , 30 , 50 , 70 ), where 30 leads to the bestresults for both YFCC and CC.

Original: Analogously to CLIP, we use the Adam optimizer (Kingma et al.
Revised: Analogously to CLIP, we use the Adam optimizer (Kingma et al.

Original: with decoupled weight decay regularization (Loshchilov & Hutter, 2019).
Revised: with decoupled weight decay regularization (Loshchilov & Hutter, 2019).

Original: The weight decay isonly applied to weights that are not gains or biases.
Revised: The weight decay isonly applied to weights that are not gains or biases.

Original: As proposed in OpenCLIP (Ilharco et al.
Revised: As proposed in OpenCLIP (Ilharco et al.

Original: , 2021)weight decay was set to 0 .
Revised: , 2021)weight decay was set to 0 .

Original: Different choices of weight decay ( 0 . 2 or 0 . 05 ), did not lead to a relevantperformance change.
Revised: Different choices of weight decay ( 0 . 2 or 0 . 05 ), did not lead to a relevantperformance change.

Original: We use the same learning rate of 1 × 10 − 3 for CC and 5 × 10 − 4 for YFCC asused in OpenCLIP.
Revised: We use the same learning rate of 1 × 10 − 3 for CC and 5 × 10 − 4 for YFCC asused in OpenCLIP.

Original: For the hyperparameter β we considered values in the range of 5 to 20 .
Revised: For the hyperparameter β we considered values in the range of 5 to 20 .

Original: A valueof 8 resulted in the best performance for CC and 14 . 3 for YFCC.
Revised: A valueof 8 resulted in the best performance for CC and 14 . 3 for YFCC.

Original: The batch size for CC was reducedto 512 due to computational restraints which did not result in performance losses.
Revised: The batch size for CC was reducedto 512 due to computational restraints which did not result in performance losses.

Original: The batch size for
Revised: The batch size for

Original: YFCC was kept at 1024 as reported by OpenCLIP since a reduction resulted in a signiﬁcant drop inperformance.
Revised: YFCC was kept at 1024 as reported by OpenCLIP since a reduction resulted in a signiﬁcant drop inperformance.

Original: The learning rate scheduler for all experiments is cosine annealing with warmup andhard restarts (Loshchilov & Hutter, 2017) with a cycle length of 7 epochs.
Revised: The learning rate scheduler for all experiments is cosine annealing with warmup andhard restarts (Loshchilov & Hutter, 2017) with a cycle length of 7 epochs.

Original: For models trained on
Revised: For models trained on

Original: YFCC the warmup was set to 10000 steps and for models trained on CC to 20000 steps.
Revised: YFCC the warmup was set to 10000 steps and for models trained on CC to 20000 steps.

Original: A.2.3 Datasets
Revised: A.2.3 Datasets

Original: [Table]

Original: For pre-training we considered two datasets, Conceptual Captions (CC) (Sharma et al., 2018) and
Revised: For pre-training we considered two datasets, Conceptual Captions (CC) (Sharma et al., 2018) and

Original: YFCC100M (Thomee et al., 2016).
Revised: YFCC100M (Thomee et al., 2016).

Original: The CC dataset consists of 2.9 million images and correspondinghigh-quality captions.
Revised: The CC dataset consists of 2.9 million images and correspondinghigh-quality captions.

Original: Images and their corresponding notations for CC have been gathered via anautomated process from the web and therefore represent a  of  images are collected from the alt-text HTML attribute.
Revised: Images and their corresponding notations for CC have been gathered via anautomated process from the web and therefore represent a wide variety of styles. Raw descriptionsof images are collected from the alt-text HTML attribute.

Original: Both images and texts were ﬁltered suchthat only image-text pairs above a certain quality threshold are part of this dataset.
Revised: Both images and texts were ﬁltered suchthat only image-text pairs above a certain quality threshold are part of this dataset.

Original: The dataset werefer to as YFCC is a subset of the Yahoo Flickr Creative Commons 100 Million (YFCC100M)dataset.
Revised: The dataset werefer to as YFCC is a subset of the Yahoo Flickr Creative Commons 100 Million (YFCC100M)dataset.

Original: It was created by ﬁltering for images which contain natural language descriptions and/ortitles in English resulting in 15 million image-caption pairs.
Revised: It was created by ﬁltering for images which contain natural language descriptions and/ortitles in English resulting in 15 million image-caption pairs.

Original: The textual descriptions contain lessuseful information than CC because they are not ﬁltered by quality.
Revised: The textual descriptions contain lessuseful information than CC because they are not ﬁltered by quality.

Original: Occasionally they also containmetadata like camera settings or web addresses.
Revised: Occasionally they also containmetadata like camera settings or web addresses.

Original: We evaluate and compare our method on several downstream classiﬁcation tasks.
Revised: We evaluate and compare our method on several downstream classiﬁcation tasks.

Original: We evaluateon the same set of datasets as CLIP reported for a model trained on YFCC.
Revised: We evaluateon the same set of datasets as CLIP reported for a model trained on YFCC.

Original: This set contains
Revised: This set contains

Original: Birdsnap (Berg et al., 2014), Country211 (Radford et al., 2021), Flowers102 (Nilsback & Zisserman,2008), GTSRB (Stallkamp et al., 2011), UCF101 (Soomro et al., 2012), Stanford Cars (Krause et al.,2013) and ImageNet (Deng et al., 2009).
Revised: Birdsnap (Berg et al., 2014), Country211 (Radford et al., 2021), Flowers102 (Nilsback & Zisserman,2008), GTSRB (Stallkamp et al., 2011), UCF101 (Soomro et al., 2012), Stanford Cars (Krause et al.,2013) and ImageNet (Deng et al., 2009).

Original: Additionally, we include ImageNet V2 in our analysis(Recht et al., 2019).
Revised: Additionally, we include ImageNet V2 in our analysis(Recht et al., 2019).

Original: Table A4 shows an overview of training and test set sizes, number of classes andthe applied evaluation metric.
Revised: Table A4 shows an overview of training and test set sizes, number of classes andthe applied evaluation metric.

Original: In the case of several test sets per dataset the metric is calculated forevery set individually and the average performance is reported.
Revised: In the case of several test sets per dataset the metric is calculated forevery set individually and the average performance is reported.

Original: The set size in Table A4 correspondsto the total number of samples across all test and training sets of a dataset respectively. 
Revised: The set size in Table A4 correspondsto the total number of samples across all test and training sets of a dataset respectively. [Table]

Original: Birdsnap contains images of North American bird species, however our dataset is smaller thanreported in CLIP as some samples are no longer available.
Revised: Birdsnap contains images of North American bird species, however our dataset is smaller thanreported in CLIP as some samples are no longer available.

Original: The Country211 dataset was publishedin CLIP and is a small subset of the YFCC100m dataset.
Revised: The Country211 dataset was publishedin CLIP and is a small subset of the YFCC100m dataset.

Original: It consists of photos that can be assignedto 211 countries via GPS coordinates.
Revised: It consists of photos that can be assignedto 211 countries via GPS coordinates.

Original: For each country 200 photos are sampled for the training setand 100 for testing.
Revised: For each country 200 photos are sampled for the training setand 100 for testing.

Original: For the Flowers102 images of 102 ﬂower categories commonly occuring in the
Revised: For the Flowers102 images of 102 ﬂower categories commonly occuring in the

Original: United Kingdom were collected.
Revised: United Kingdom were collected.

Original: Several classes are very similar and there is a large variation in scale,pose and lighting.
Revised: Several classes are very similar and there is a large variation in scale,pose and lighting.

Original: The German Trafﬁc Sign Recognition Benchmark ( GTSRB ) was a challenge heldat the IJCNN 2011.
Revised: The German Trafﬁc Sign Recognition Benchmark ( GTSRB ) was a challenge heldat the IJCNN 2011.

Original: The dataset contains images of german trafﬁc signs from more than 40 classes.
Revised: The dataset contains images of german trafﬁc signs from more than 40 classes.

Original: Note that two versions of this dataset exist, one used for the challenge and an ofﬁcial dataset releasedafter the competition.
Revised: Note that two versions of this dataset exist, one used for the challenge and an ofﬁcial dataset releasedafter the competition.

Original: For CLIP the linear probing classiﬁers were trained using the competitiontraining set but tested on the ofﬁcial test set.
Revised: For CLIP the linear probing classiﬁers were trained using the competitiontraining set but tested on the ofﬁcial test set.

Original: Stanford Cars contains images of 196 car models atthe level of make, model and year (e.g. Tesla Model S Sedan 2012).
Revised: Stanford Cars contains images of 196 car models atthe level of make, model and year (e.g. Tesla Model S Sedan 2012).

Original: UCF101 (Soomro et al., 2012)is
Revised: UCF101 (Soomro et al., 2012)is

Original: a video dataset with short clips for action recognition consisting of three training sets and threetest sets.
Revised: a video dataset with short clips for action recognition consisting of three training sets and threetest sets.

Original: We follow the procedure reported in CLIP and extract the middle frame of every video toassemble the dataset.
Revised: We follow the procedure reported in CLIP and extract the middle frame of every video toassemble the dataset.

Original: The ImageNet Large Scale Visual Recognition Challenge was held from 2012through 2017 and is one of the most widely used benchmarks for object detection and localization.
Revised: The ImageNet Large Scale Visual Recognition Challenge was held from 2012through 2017 and is one of the most widely used benchmarks for object detection and localization.

Original: Several years later ImageNet V2 assembled three new test sets with images from the same 1,000classes to test for generalization of models optimized for the original ImageNet benchmark.
Revised: Several years later ImageNet V2 assembled three new test sets with images from the same 1,000classes to test for generalization of models optimized for the original ImageNet benchmark.

Original: Everytest set comprises 10,000 samples.
Revised: Everytest set comprises 10,000 samples.

Original: A.2.4 Zero-shot evaluation
Revised: A.2.4 Zero-shot evaluation

Original: Class names for all downstream tasks were adopted from CLIP, that is, among other changesspecial characters like hyphens or apostrophes were removed.
Revised: Class names for all downstream tasks were adopted from CLIP, that is, among other changesspecial characters like hyphens or apostrophes were removed.

Original: Furthermore, some class namesof the datasets were slightly changed (e.g. “ kite ” to “ kite (bird of prey) ” in ImageNet).
Revised: Furthermore, some class namesof the datasets were slightly changed (e.g. “ kite ” to “ kite (bird of prey) ” in ImageNet).

Original: For zero-shot evaluation, we use the same prompt templates as published in CLIP.
Revised: For zero-shot evaluation, we use the same prompt templates as published in CLIP.

Original: Depending onthe dataset the number of prompts can vary from one prompt (e.g. “ a photo of a {label},a type of bird. ”
Revised: Depending onthe dataset the number of prompts can vary from one prompt (e.g. “ a photo of a {label},a type of bird. ”

Original: for Birdsnap) up to 80 prompts for ImageNet covering various settings (e.g.“ a cropped photo of a {label}. ”, “ a origami {label}. ”).
Revised: for Birdsnap) up to 80 prompts for ImageNet covering various settings (e.g.“ a cropped photo of a {label}. ”, “ a origami {label}. ”).

Original: In case of several prompts anaverage embedding over all prompt embeddings is calculated.
Revised: In case of several prompts anaverage embedding over all prompt embeddings is calculated.

Original: Figure A4 shows the zero-shot resultsfor all evaluation tasks with the ResNet-50x4 model reported in Table 3. 
Revised: Figure A4 shows the zero-shot resultsfor all evaluation tasks with the ResNet-50x4 model reported in Table 3. [Table]

Revised: In addition to the results of the main paper

Revised: [Table]

Revised: the zeroshot performance of the models was tested onadditional datasets.

Revised: For details about the additional datasets we refer the reader to Section A.1 of

Original:  A.2.5 Linear probing
Revised: Radford et al. Table A5 shows the results for models trained on CC. Table A6 shows theresults for models trained on YFCC. A.2.5 Linear probing

Original: We tried to follow the evaluation procedure in Radford et al.
Revised: We tried to follow the evaluation procedure in Radford et al.

Original: (2021) as closely as possible.
Revised: (2021) as closely as possible.

Original: We noteone difference with respect to the implementation: Instead of scikit-learn’s logistic regression usingthe L-BFGS solver, we use cuML’s logistic regression classiﬁer with L-BFGS algorithm to utilize
Revised: We noteone difference with respect to the implementation: Instead of scikit-learn’s logistic regression usingthe L-BFGS solver, we use cuML’s logistic regression classiﬁer with L-BFGS algorithm to utilize

Original: GPUs for efﬁciency.
Revised: GPUs for efﬁciency.

Original: All hyperparameters are the same as described in Radford et al.
Revised: All hyperparameters are the same as described in Radford et al.

Original: (2021), themaximum number of iterations was set to 1000 , and the L2 regularization strength λ was determinedby using a parametric binary search.
Revised: (2021), themaximum number of iterations was set to 1000 , and the L2 regularization strength λ was determinedby using a parametric binary search.

Original: We tried to reproduce the CLIP results with the correspondingly published models, however, failed toproduce the exact numbers.
Revised: We tried to reproduce the CLIP results with the correspondingly published models, however, failed toproduce the exact numbers.

Original: This could be due to several factors:
Revised: This could be due to several factors:

Original: • The train and validation split.
Revised: • The train and validation split.

Original: Same as in Radford et al.
Revised: Same as in Radford et al.

Original: (2021) , we use the providedvalidation set to perform the hyperparameter search.
Revised: (2021) , we use the providedvalidation set to perform the hyperparameter search.

Original: When there is none provided, we use arandom half of the training dataset for validation.•
Revised: When there is none provided, we use arandom half of the training dataset for validation.

Original:  In case of a tie in the validation score, we use the maximal λ for the strongest regularization.
Revised: • In case of a tie in the validation score, we use the maximal λ for the strongest regularization.

Original: We note though that we came closer to reproducing the results published in CLIP whenusing the mean λ over all ties when these exist.
Revised: We note though that we came closer to reproducing the results published in CLIP when using the mean λ over all ties when these exist.

Original: • For the Birdsnap dataset, the resources that we have got online at the time of this writingcould be different from the resources that CLIP’s authors obtained at the time.
Revised: • For the Birdsnap dataset, the resources that we have got online at the time of this writing could be different from the resources that CLIP’s authors obtained at the time.

Original: Linear probing evaluation of YFCC pre-trained models is shown in Table A5.
Revised: Linear probing evaluation of YFCC pre-trained models is shown in Table A7.

Original: Comparing ourreimplementation of CLIP and CLOOB with different ResNet encoders, we observe mixed results.
Revised: Comparing ourreimplementation of CLIP and CLOOB with different ResNet encoders, we observe mixed results.

Original: The reason for this effect might be attributed to the observed task-dependence of multimodal models(Devillers et al., 2021).
Revised: The reason for this effect might be attributed to the observed task-dependence of multimodal models(Devillers et al., 2021).

Original: Another potential reason is that the beneﬁt of the restrictions to more reliablepatterns that occur in both modalities does not directly translate to an evaluation of just the encodingpart of one modality.
Revised: Another potential reason is that the beneﬁt of the restrictions to more reliable [Table] patterns that occur in both modalities does not directly translate to an evaluation of just the encodingpart of one modality.

Original: Again, as expected in self-supervised training, increasing the capacity of the
Revised: Again, as expected in self-supervised training, increasing the capacity of the

Original: CLOOB models beneﬁts accuracy.
Revised: CLOOB models beneﬁts accuracy.

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Stanford Cars 36.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: ImageNet

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: ImageNet V 60.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: A.2.6 Image-Text retrieval

Revised: In addition to zero-shot and linear probing, we tested the models trained on CC in image-to-textretrieval and text-to-image retrieval.

Revised: The task is to ﬁnd the matching image to a given text (image-totext) or, respectively, ﬁnding the matching text to a given image (text-to-image).

Revised: The dataset used forthis task is the validation set of CC, which contains 13330 image-text pairs.

Revised: We report the results in

Revised: Table A5.

Revised: CLOOB signiﬁcantly outperforms CLIP in both image-to-text and text-to-image retrieval.

Revised: [Table]

Original: A.2.6 Analysis of the image and text embeddings
Revised: A.2.7 Analysis of the image and text embeddings

Original: Following our ablation studies, we use models trained on CC to disentangle the effects of InfoLOOBand modern Hopﬁeld networks.
Revised: Following our ablation studies [Table] we use models trained on CC to disentangle the effects of InfoLOOBand modern Hopﬁeld networks.

Original: To track the behaviour during learning, we calculate the embeddingsof the validation set of CC (consisting of 13,338 image-caption pairs) for all epochs before a restartof the learning rate scheduler.
Revised: To track the behaviour during learning, we calculate the embeddingsof the validation set of CC (consisting of 13,338 image-caption pairs) for all epochs before a restartof the learning rate scheduler.

Original: We apply the extended uniformity test A n of Ajne (Ajne, 1968; Prentice, 1978) to the respectiveembeddings of the image and text encoders.
Revised: We apply the extended uniformity test A n of Ajne (Ajne, 1968; Prentice, 1978) to the respectiveembeddings of the image and text encoders.

Original: Let X =
Revised: Let X =

Original: ( x 1 , . . .
Revised: ( x 1 , . . .

Original: , x n ) be the embeddings of onemodality (text or image), consisting of n samples of dimension d .
Revised: , x n ) be the embeddings of onemodality (text or image), consisting of n samples of dimension d .

Original: All samples x are of length 1.
Revised: All samples x are of length 1.

Original: As speciﬁed in Eq.
Revised: As speciﬁed in Eq.

Original: (A177), A n calculates the difference between a uniform distribution, where allsamples on the hypersphere are orthogonal to each other, and the actual distribution of the embedding
Revised: (A177), A n calculates the difference between a uniform distribution, where allsamples on the hypersphere are orthogonal to each other, and the actual distribution of the embedding

Original: X .
Revised: X .

Original: Consequently, an embedding with low uniformity results in a high Ajne A n test statistic:
Revised: Consequently, an embedding with low uniformity results in a high Ajne A n test statistic:

Original: [Equation]
Revised: [Equation]

Original: To understand the inﬂuence of modern Hopﬁeld networks, we analyzed the covariance structure ofthe image and caption embeddings.
Revised: To understand the inﬂuence of modern Hopﬁeld networks, we analyzed the covariance structure ofthe image and caption embeddings.

Original: Similar to Jing et al.
Revised: Similar to Jing et al.

Original: (2022), we calculated the sorted eigenvaluesof the covariance matrices of image embeddings.
Revised: (2022), we calculated the sorted eigenvaluesof the covariance matrices of image embeddings.

Original: Figure A1 shows the sorted eigenvalues of InfoNCEand InfoLOOB with and without Hopﬁeld retrievals during training.
Revised: Figure A1 shows the sorted eigenvalues of InfoNCEand InfoLOOB with and without Hopﬁeld retrievals during training.

Original: Both models without modern Hopﬁeld networks struggle to increase the number of effective eigenvalues which contribute to the variance of the embeddings.
Revised: Both models without modern Hopﬁeld networks struggle to increase the number of effective eigenvalues which contribute to the variance of the embeddings.

Original: InfoNCE with modern Hopﬁeld networksstarts with a small number of effective eigenvalues.
Revised: InfoNCE with modern Hopﬁeld networksstarts with a small number of effective eigenvalues.

Original: We attribute this to the saturating effect of
Revised: We attribute this to the saturating effect of

Original: InfoNCE, which impedes the modern Hopﬁeld network to extract more covariance.
Revised: InfoNCE, which impedes the modern Hopﬁeld network to extract more covariance.

Original: During training [Figure] the effective eigenvalues steadily increase at a consistent rate.
Revised: During trainingthe effective eigenvalues steadily increase at a consistent rate.

Original: InfoLOOB with Hopﬁeld starts with ahigh number of effective eigenvalues and strongly improves them during training.
Revised: InfoLOOB with Hopﬁeld starts with ahigh number of effective eigenvalues and strongly improves them during training.

Original: Additionally, we looked at the distribution of similarity scores (dot product) of matched pairs andunmatched pairs over the validation set of CC of embeddings from models trained for 128 epochs.
Revised: Additionally, we looked at the distribution of similarity scores (dot product) of matched pairs andunmatched pairs over the validation set of CC of embeddings from models trained for 128 epochs.

Original: In Figure A2, we contrast the distributions of similarity scores of matched pairs with the distributionsof the similarity score for the 1000 unmatched pairs that have the highest similarity score with theanchor.
Revised: In Figure A2, we contrast the distributions of similarity scores of matched pairs with the distributionsof the similarity score for the 1000 unmatched pairs that have the highest similarity score with theanchor.

Original: InfoNCE with Hopﬁeld results in a moderate increase of the similarity scores of matchedpairs, as well as in an increase of the similarity score of unmatched pairs.
Revised: InfoNCE with Hopﬁeld results in a moderate increase of the similarity scores of matchedpairs, as well as in an increase of the similarity score of unmatched pairs.

Original: The latter is an undesiredside effect of Hopﬁeld networks as unmatched pairs get also more similar to one another.
Revised: The latter is an undesiredside effect of Hopﬁeld networks as unmatched pairs get also more similar to one another.

Original: Comparedto InfoNCE, InfoLOOB does not saturate.
Revised: Comparedto InfoNCE, InfoLOOB does not saturate.

Original: Therefore, it considerably increases the similarity betweenmatched pairs and also reduces the average similarity of the top-1000 unmatched pairs.
Revised: Therefore, it considerably increases the similarity betweenmatched pairs and also reduces the average similarity of the top-1000 unmatched pairs.

Original: InfoLOOBattributes a cosine similarity of one to many pairs, which is not plausible for multi-modal pairs.
Revised: InfoLOOBattributes a cosine similarity of one to many pairs, which is not plausible for multi-modal pairs.

Original: Clearly, this is an overﬁtting problem of InfoLOOB.
Revised: Clearly, this is an overﬁtting problem of InfoLOOB.

Original: Noteworthy, an observed increase in alignmentbetween epoch 31 and epoch 128 does not beneﬁt the downstream performance.
Revised: Noteworthy, an observed increase in alignmentbetween epoch 31 and epoch 128 does not beneﬁt the downstream performance.

Original: The combination of
Revised: The combination of

Original: Hopﬁeld and InfoLOOB increases the similarity score of matched pairs and simultaneously reducesthe average similarity of unmatched pairs compared to InfoNCE without Hopﬁeld (the CLIP setting).
Revised: Hopﬁeld and InfoLOOB increases the similarity score of matched pairs and simultaneously reducesthe average similarity of unmatched pairs compared to InfoNCE without Hopﬁeld (the CLIP setting).

Original:  Figure A3 illustrates the overﬁtting problem of InfoLOOB.
Revised: [Figure] Figure A3 illustrates the overﬁtting problem of InfoLOOB.

Original: The distributions of unmatched pairstakes into account the ten unmatched pairs per anchor with the highest similarity score.
Revised: The distributions of unmatched pairstakes into account the ten unmatched pairs per anchor with the highest similarity score.

Original: In particularin the case of InfoLOOB without Hopﬁeld, high similarity scores of the matched pairs correlate withhigh similarity scores of the top-10 unmatched pairs.
Revised: In particularin the case of InfoLOOB without Hopﬁeld, high similarity scores of the matched pairs correlate withhigh similarity scores of the top-10 unmatched pairs.

Original: In contrast, InfoLOOB with Hopﬁeld does notsuffer from this overﬁtting problem.
Revised: In contrast, InfoLOOB with Hopﬁeld does notsuffer from this overﬁtting problem.

Revised: A.2.8 Training time and memory consumption

Revised: To elaborate on the memory and time consumption of CLOOB, Table A9 compares the experimentsdone in Section 5.1.

Revised: The memory consumption of CLOOB is the same as CLIP since only embeddings from the mini-batch are used both in the objective and in the Hopﬁeld memories.