CLOOB: M ODERN H OPFIELD N ETWORKS WITH I NFO LOOB O UTPERFORM CLIP
Anonymous authors
Paper under double-blind review
A BSTRACT
Contrastive learning with the InfoNCE objective is exceptionally successful in various self-supervised learning tasks.
Recently, the CLIP model yielded impressive results on zero-shot transfer learning when using InfoNCE for learning visual representations from natural language supervision.
However, InfoNCE as a lower bound on the mutual information has been shown to perform poorly for high mutual information.
In contrast, the InfoLOOB upper bound (leave one out bound) works well for high mutual information but suffers from large variance and instabilities.
We introduce “Contrastive Leave One Out Boost” (CLOOB), where modern Hopﬁeld networks boost learning with the InfoLOOB objective.
Modern
Hopﬁeld networks replace the original embeddings by retrieved embeddings in the
InfoLOOB objective.
The retrieved embeddings give InfoLOOB two assets.
Firstly, the retrieved embeddings stabilize InfoLOOB, since they are less noisy and more similar to one another than the original embeddings.
Secondly, they are enriched by correlations, since the covariance structure of embeddings is reinforced through retrievals.
We compare CLOOB to CLIP after learning on the Conceptual Captions and the YFCC dataset with respect to their zero-shot transfer learning performance on other datasets.
CLOOB consistently outperforms CLIP at zero-shot transfer learning across all considered architectures and datasets.
I NTRODUCTION
With the advent of large corpora of unlabeled data in vision and language, self-supervised learning via contrastive learning has become highly successful.
Some contrastive learning objectives, such as those of BYOL (Grill et al., 2020) and SimSiam
(Chen & He, 2021), do not require negative samples.
However, the most popular objective for contrastive learning is InfoNCE (van den Oord et al., 2018), in which for an anchor sample, a positive sample is contrasted with negative samples.
The idea to use objectives with negative samples is well known in deep learning (Gutmann & Hyvärinen, 2010; Chen et al., 2017; Mikolov et al., 2013).
For contrastive learning, the most successful objective is InfoNCE, which has been introduced as Contrastive Predictive Coding (CPC) (van den Oord et al., 2018).
InfoNCE has been applied to transfer learning (Hénaff et al., 2019), to natural language response suggestion (Henderson et al., 2017), to learning sentence representations from unlabelled data (Logeswaran & Lee, 2018), and to unsupervised feature learning by maximizing distinctions between instances (Wu et al., 2018).
InfoNCE has been used for learning visual representations in Pretext-Invariant Representation Learning (PIRL) (Misra & vanDerMaaten, 2020), in
Momentum Contrast (MoCo) (He et al., 2020), and in SimCLR (Chen et al., 2020).
SimCLR became well known as is was highly effective for transfer learning.
Zero-shot transfer learning (Lampert et al., 2009) is one of the most ambitious goals in vision, since it would improve various real-world downstream applications.
Current models in natural language processing and vision perform very well on standard benchmarks, but they fail at new data, new applications, deployments in the wild, and stress tests (D’Amour et al., 2020; Recht et al., 2019; Taori et al., 2020; Lapuschkin et al., 2019;
Geirhos et al., 2020).
A model with high zero-shot transfer learning performance will not fail on such data, therefore will be trusted by practitioners.
Contrastive Language-Image Pre-training (CLIP) based on the InfoNCE objective yielded very impressive results at zero-shot transfer learning (Radford et al., 2021).
CLIP learns expressive image embeddings directly from raw text, thereby leverages a much richer source of supervision than just labels.
A plethora of CLIP follow-up work has already been published (see Appendix Section A.5). The CLIP model is considered as an important foundation model (Bommasani et al., 2021).
Though CLIP excels at zero-shot transfer learning, it can be improved.
CLIP training suffers from an “explaining away” problem (Wellman & Henrion, 1993), which leads to “shortcut learning” (Geirhos et al., 2020) or the Clever Hans phenomenon (Lapuschkin et al., 2019).
Explaining away impedes the increase of the similarity between a text and a corresponding image, since learning focuses on only one common aspect and does not exploit the full covariance structure of the data.
If one common aspect is sufﬁcient for high similarity, the InfoNCE objective saturates, since it has the form a/ ( a + b ) with a giving the similarity of a matched pair and b giving the average similarity of unmatched pairs.
For a large similarity a , the objective saturates and increasing a has a small effect.
Contrary to InfoNCE, the leave-one-out (“InfoLOOB”) bound (Poole et al., 2019) is of the form a/b which does not saturate. However, so far the InfoLOOB bound was not used as an objective in contrastive learning. We justify the maximization of the InfoLOOB bound for contrastive learning in Appendix Section A.1.3. We show that maximizing the InfoLOOB bound leads to a good approximation of the mutual information, in particular for high mutual information.
A problem of InfoLOOB is that it has high variance for small b .
Even when InfoLOOB avoids saturation, CLIP insufﬁciently extracts the covariance structure in the data.
The covariance originates from co-occurrences of related words in text or from co-occurrences of objects, textures, or colors in images.
CLIP’s problem of insufﬁciently extracting the covariance structure of the data is tackled by modern Hopﬁeld networks. Hopﬁeld networks are energy-based, binary associative memories, which popularized artiﬁcial neural networks in the 1980s (Hopﬁeld, 1982; 1984). Associative memory networks have been designed to store and retrieve samples. Their storage capacity can be considerably increased by polynomial terms in the energy function (Chen et al., 1986; Psaltis & Cheol, 1986; Baldi & Venkatesh, 1987; Gardner, 1987; Abbott & Arian, 1987;
Horn & Usher, 1988; Caputo & Niemann, 2002; Krotov & Hopﬁeld, 2016). In contrast to these binary memory networks, we use continuous associative memory networks with very high storage capacity. These modern Hopﬁeld networks for deep learning architectures have an energy function with continuous states and can retrieve samples with only one update (Ramsauer et al., 2021; 2020).
Modern Hopﬁeld Networks have already been successfully applied to immune repertoire classiﬁcation (Widrich et al., 2020) and chemical reaction prediction (Seidl et al., 2021).
Modern Hopﬁeld networks reinforce the covariance structure in the data and stabilize the InfoLOOB objective by increasing b . The covariance structure of retrieved embeddings is ampliﬁed through co-occurrences of embedding features in the memory.
Additionally, the retrieved embeddings are less noisy and more similar to one another which leads to a larger b .
We introduce “Contrastive Leave One Out Boost” (CLOOB) which overcomes CLIP’s problems of (i) “explaining away” with saturation and (ii) insufﬁciently extracting the covariance structure of the data. CLOOB uses the leave-one-out (“InfoLOOB”) bound (Poole et al., 2019) as the objective in combination with modern Hopﬁeld networks.
Our contributions are:
(a) we introduce a new contrastive learning method called CLOOB, (b) we propose InfoLOOB as an objective for contrastive learning, (c) we propose to use modern Hopﬁeld networks to reinforce covariance structures, (d) we show theoretical properties of the InfoLOOB objective and loss function.
I NFO LOOB VS . I NFO NCE
We discuss and analyse known bounds on the mutual information I( X ; Y ) between random variables X and Y , which are distributed according to p ( x , y ) :
I( X [Equation]
Y
[Equation] =
[Equation] x
[Equation]
We consider the multi-sample lower bound “InfoNCE” (van den Oord et al., 2018).
A pair of an anchor sample y and a positive sample x 1 is drawn via the joint distribution p ( x 1 , y ) .
The negative samples ˜ X
= { x 2 , . . .
, x N } are drawn iid according to the marginal distribution p ( x ) . Using
X = { x 1 , x 2 , . . .
, x N } , the probabilities of the datasets are p ( ˜ X ) =
(cid:81)
Ni =2 p ( x i ) , p ( X | y ) =
p ( x 1 | y ) (cid:81)
Ni =2 p ( x i ) , and p ( X ) = (cid:81) Ni =1 p ( x i ) .
The InfoNCE with score function f ( x , y ) is I InfoNCE ( X 1 ; Y ) = E p ( y )
[Equation]
[Equation] ) using the factor 1 /N as in Poole et al.
; Tschannen et al.
; Cheng et al.
; Chen et al.
For f ( x , y )
=
p ( y | x ) , we obtain the InfoNCE with probabilities.
The InfoNCE is a lower bound on the mutual information (Poole et al., 2019), which is stated in the next theorem.
Theorem 1 (InfoNCE lower bound) .
InfoNCE with score function f ( x , y ) is a lower bound on the mutual information:
[Equation]
In particular, the bound holds for InfoNCE with probabilities, i.e. for f ( x , y ) =
p ( y | x ) .
For a proof see Poole et al.
(2019) and the proof of Theorem A1 in the Appendix.
The “Leave one out upper bound” (Poole et al., 2019) on the mutual information was called “L1Out” in Cheng et al.
(2020), while we call it “InfoLOOB” (LOOB for “Leave One Out Bound”).
InfoLOOB is the same as InfoNCE (Eq.
(3)), but without the positive sample x 1 in the denominator.
Contrastive
Log-ratio Upper Bound (CLUB), another upper bound on the mutual information, was only used for minimizing it (Cheng et al., 2020).
Maximizing CLUB failed in experiments, because the embedding distribution was not uniform as known for similar objectives (Wang & Liu, 2021).
Uniform embedding distributions are required for successful contrastive learning (Wang & Isola, 2020).
We use InfoLOOB as an objective, since it approximates high mutual information better than InfoNCE.
Maximizing an upper bound on the mutual information might be counter-intuitive. Therefore, we justify the maximization of the InfoLOOB bound for contrastive learning in Appendix Section A.1.3. We show that maximizing the InfoLOOB bound approximates the mutual information, the better the higher it is. Recently, InfoLOOB was independently introduced for and successfully applied to image-to-image contrastive learning (Yeh et al., 2021).
The InfoLOOB with score function f ( x , y ) is deﬁned in the following, where we obtain the InfoLOOB with probabilities for f ( x , y ) = p ( y | x ) :
[Equation]
Before we show that InfoLOOB with a score function is an upper bound on the mutual information, we need some deﬁnitions.
˜ p
( x | y ) draws the positives for y with lower probability than p ( x ) , that is, the positives are under-sampled.
Z ( y ) =
E ˜ p ( x | y )
[ f ( x , y )] gives the average score f
( x , y ) , if under-sampling via ˜ p ( x | y ) , while Z ∗ ( y )
=
E p ( x ) [ f ( x , y )] average score f ( x , y ) if sampling from p ( x ) .
We deﬁne the variational distribution q ( x | y )
=
p ( x ) f ( x , y )
Z ∗ ( y ) .
Our main assumption is expressed by the log-ratio of the averages Z ( y ) and Z ∗ ( y ) :
[Equation]
which ensures that the positives x are sufﬁciently under-sampled via p ( x | y )
The Kullback-Leibler divergence gives the minimal difference between averaging f ( x , y ) via p ( x ) and via ˜ p
( x | y ) .
The next theorem shows that InfoLOOB is an upper bound on the mutual information.
Theorem 2 (InfoLOOB upper bound) .
If ˜ X = { x 2 , . . .
, x N } are drawn iid according to ˜ p ( x | y ) and if the main assumption Eq.
(5) holds, then InfoLOOB with score function f ( x , y ) is an upper bound on the mutual information:
[Equation]
The bound is valid for InfoLOOB with probabilities (without under-sampling), where the negative samples ˜
X
= { x 2 , . . .
, x N } are drawn iid according to p ( x ) and f ( x , y )
=
p ( y | x ) .
The proof for this theorem is given as proof for Theorem A2 in the Appendix.
Loss functions and their gradients.
The training set { ( x 1 , y 1 ) , ( x 2 , y 2 ) , . . . , ( x N , y N ) } consists of N samples that are drawn iid from p ( x , y ) .
InfoNCE uses the matrix X =
( x 1 , . .
, x N ) , while
InfoLOOB uses ˜ X
=
( x 2 , . .
, x N ) . The matrices differ by the positive sample x 1 .
For the score function f ( x , y ) , we use f ( x , y ) =
exp( τ − 1 sim( x , y )) with the similarity sim
( x , y ) =
y T x and τ as the temperature.
We have the InfoNCE and InfoLOOB loss functions: [Equation] In the second sum of the losses in Eq.
7 and Eq. 8, we consider only the ﬁrst term.
For simplicity, we abbreviate y =
y 1 leading to the pair ( x 1 , y ) and the negatives ˜ X = ( x 2 , . .
, x N ) . [Equation] InfoNCE
[Equation]
These loss terms can be simpliﬁed to L InfoNCE ( y ) =
− τ − 1 y T x 1 + τ − 1 lse( τ − 1 , X T y ) and L InfoLOOB ( y ) = − τ − 1 y T x 1 + τ − 1 lse( τ − 1 , ˜ X T y ) , where lse is the log-sum-exp function (see Eq.
(A103) in the Appendix).
The gradient of the InfoNCE loss with respect to y is − τ − 1 x 1 + τ − 1 X softmax( τ − 1 X T y ) and the gradient of the InfoLOOB loss is − τ − 1 x 1 + τ − 1 ˜ X softmax( τ − 1 ˜ X T y ) .
Using p = ( p 1 , . .
. , p N ) T = softmax( τ − 1 X T y ) , the gradient of
InfoNCE with respect to y is − τ − 1 (1 − p 1 )( x 1 − ˜ X softmax( τ − 1 ˜ X T y )) and its gradient with respect to x 1 is − τ − 1 (1 − p 1 )
y (see Appendix Subsection A.1.4).
By and large, the gradient of InfoNCE is scaled by (1 − p 1 ) compared to the gradient of InfoLOOB, where p 1 is softmax similarity between the anchor y and positive sample x 1 .
Consequently, InfoNCE saturates and learning stalls when anchor and positive sample become similar to each other.
CLOOB: I NFO LOOB WITH M ODERN H OPFIELD N ETWORKS
[Equation]
y N
V y 1 V y 2 ··· V y N
Hopﬁeld retrieval with U
Hopﬁeld retrieval with V
Hopﬁeld retrieval with U
Hopﬁeld retrieval with V (cid:39) (cid:39) (cid:39) (cid:39) (cid:39) (cid:39) [Figure] textencoder
[Figure]
in the snow. (cid:39) legend similarity to anchor positive sample [Figure] sample
CLOOB for contrastive learning. Our novel Contrastive Leave One Out Boost (CLOOB) combines the InfoLOOB objective with modern Hopﬁeld networks. Modern Hopﬁeld networks substitute the original by retrieved embeddings, thereby reduce the variance of InfoLOOB and reinforce the covariance structure in the data. Figure 1 sketches the CLOOB architecture for image-text pairs.
The training set consists of N pairs of embeddings { ( x 1 , y 1 ) , . . . , ( x N , y N ) } , M stored embeddings
U = ( u 1 , . . .
, u M ) , and K stored embeddings V = ( v 1 , . .
. , v K ) .
The state or query embeddings x i and y i retrieve U
x i and U y i , respectively, from U — analogous notation for retrievals from V .
All samples are normalized: (cid:107) x i (cid:107) =
(cid:107) y
i (cid:107) =
(cid:107) u
i (cid:107) =
(cid:107) v i (cid:107) = 1 .
The following vectors are retrieved from modern Hopﬁeld networks (Ramsauer et al., 2021):
[Equation]
where U x i denotes an image-retrieved image embedding, U y i
a text-retrieved image embedding, V x i an image-retrieved text embedding and V y i a text-retrieved text embedding. The hyperparameter β corresponds to the inverse temperature: β = 0 retrieves the average of the stored pattern, while large β retrieves the stored pattern that is most similar to the state pattern (query).
In InfoLOOB, CLOOB substitutes the embedded samples x
i and y i by the retrieved embedded samples.
In the ﬁrst term, x i and y i are substituted by U x i and U y i , respectively, while in the second term by V x i and V y i .
All retrieved samples are normalized, (cid:107) U
x
i
[Equation]
We obtain the InfoLOOB loss function that is used by CLOOB:
[Equation]
Modern Hopﬁeld Networks reduce high variance of InfoLOOB. CLOOB uses InfoLOOB as objective, since it estimates the mutual information (MI) better than InfoNCE, in particular, for large
MI.
Cheng et al. (2020, Fig. 1 and Fig. 2) show that InfoLOOB is a better estimator for the MI than InfoNCE (van den Oord et al., 2018), MINE (Belghazi et al., 2018), and NWJ (Nguyen et al., 2010).
We experimentally conﬁrmed that InfoLOOB better estimates the mutual information than InfoNCE. [Figure]
[Figure] n o r [Figure] o n
[Figure] Modern Hopﬁeld networks reduce the variance of the InfoLOOB loss. However, InfoLOOB has higher variance than lower bounds on MI like InfoNCE, which considerably hampers learning (Cheng et al., 2020, Fig. 1 and Fig. 2), see also Appendix Section A.2.
The
InfoNCE objective has the form a/ ( a + b ) while InfoLOOB has the form a/b with a giving the anchor-to-positive similarity and b the average anchor-to-negative similarity. For small b , we observe high variance and instability of InfoLOOB. Modern Hopﬁeld networks (Ramsauer et al., 2021) are a remedy for the high variance.
Modern Hopﬁeld networks substitute the original patterns by retrieved patterns, which are an average over the stored patterns. We tested the variance of MI estimators/bounds on toy tasks, with samples drawn from Gaussian distributions following (Belghazi et al., 2018; Poole et al., 2019; Cheng et al., 2020). With the InfoLOOB objective, we train deep learning architectures with and without modern Hopﬁeld networks on top, where the current learning batch is stored in the modern Hopﬁeld networks.
We used training data with mutual information of 10 and 14, where the parameters were optimized for the best performance on a validation set. We test the ﬁnal model on different levels of mutual information.
Figure 2 shows that modern Hopﬁeld networks reduce the variance of the model. The average variances are reduced from 0 . 67 to 0 . 33 for
{+MI 10 and from 1 . 
+} to 0 . 48 for MI 14 (more details in Appendix A.2). Modern Hopﬁeld Networks amplify the covariance structure in the data.
The covariance structure is extracted by the retrieved embeddings U T x
i U
y i
and
V T x i V y
i .
The Jacobian J of the softmax p = softmax( β a ) is J( β a )
= β (cid:0) diag( p ) − pp T (cid:1) .
We deﬁne the weighted covariance Cov( U ) , where sample u i is drawn with probability p i , as [Cov( U )]
kl = (cid:2)
U J( β a ) U T (cid:3) kl =
β ( (cid:80) Mi =1 p i u ik u il − (cid:80) Mi
=1 p i u
ik (cid:80) Mi
=1 p i u il ) .
The formula of the weighted covariance differs from the standard empirical covariance, since the factor 1 /M is replaced by p i .
Thus u i is sampled with probability p i instead of being sampled uniformly with probability 1 /M .
We apply the mean value theorem to the softmax function with mean Jacobian matrix J m ( β a ) =
(cid:82)0 J( λβ a ) d λ .
The mean Jacobian J m ( β a ) is a symmetric, diagonally dominant, positive semideﬁnite matrix with one eigenvalue of zero for eigenvector 1 and spectral norm bounded by (cid:107) J m (cid:107) 2 (cid:54) 0 .
5 β (see Appendix Lemma A1).
We can express U T x
i U
y
i as (see Appendix Theorem A3):
[Equation] [Equation]
where the mean is ¯ u = 1 /M
U
and the weighted covariances are Cov( U , x i ) =
U J m ( β U
T x i ) U T and Cov( U , y i ) =
U J m ( β U
T y i ) U T .
The weighted covariance Cov( U , . ) is the covariance if the stored pattern u i is drawn according to an averaged p i given by J m ( . ) .
When maximizing the dot product U T x
i
U
y
i , the normalized vectors x i and y i are encouraged to agree on drawing the patterns
u
i with the same probability p i in order to generate similar weighted covariance matrices
Cov( U , . ) .
If subsets of U have a strong covariance structure, then it can be exploited to produce large weighted covariances and, in turn, large dot products of U T x
i U y
i .
Furthermore, for a large dot product U T x
i
U
y
i , x i and y i have to be similar to each other to extract the same direction from the covariance matrices.
Above considerations for U T x
i
U
y i analogously apply to V T x i V y
i .
We did not use a loss function that contains dot products like U T x i V y i , because these dot products have higher variance than the ones we have used. The dot product U T x i V y i has higher variance, since it uses M + K stored patterns, whereas U T x i U y i and V T
x
i V y i use M and K , respectively. Modern Hopﬁeld Networks can reuse training samples as stored patterns. We use the training samples as the stored patterns in the modern Hopﬁeld network.
Hence, we set u i =
x
i and v i
=
y i , that is, U = X and V = Y .
Consequently, we store the learning batch in the modern Hopﬁeld networks as U and V .
In particular this means that x i can retrieve itself from U =
X
but not from V = Y .
Analogously, y i can retrieve itself from V =
Y but not from U
= X .
Modern Hopﬁeld networks allow the usage of retrieved embeddings. After learning, both the model embeddings x and y as well as the retrieved embeddings U x , U y , V x , and V y may serve for the downstream tasks, e.g. for zero-shot transfer learning.
When using the retrieved embeddings, the modern Hopﬁeld networks can store random samples, prototypes, templates, or proprietary samples. Therefore, particular embedding features can be ampliﬁed according to the task at hand. Modern Hopﬁeld networks is a new concept for contrastive learning.
In bioinformatics the covariance structure in a sequence is reinforced by ﬁrst retrieving similar sequences from a database and then aligning them.
Conserved regions are characterized by high local covariance in the alignment (Dickson & Gloor, 2012; Kreth & Fodor, 2014).
Modern Hopﬁeld networks detect high covariances of embedded features, which is conveyed by the retrieved sample that corresponds to an alignment.
E XPERIMENTS
On two pretraining datasets, we compare our new CLOOB to CLIP (Radford et al., 2021) with respect to their capability of zero-shot transfer learning.
The ﬁrst dataset, Conceptual Captions (CC) (Sharma et al., 2018), has a very rich textual description of images but only three million image-text pairs.
The second dataset, a subset of YFCC100M
(Thomee et al., 2016), has 15 million image-text pairs but the textual description is less rich than for CC and often vacuous.
For both pretraining datasets, the downstream zero-shot transfer learning performance is tested on seven image classiﬁcation datasets. [Table]
4. C ONCEPTUAL C APTIONS P RETRAINING
Pretraining dataset.
The Conceptual Captions (CC) (Sharma et al., 2018) dataset consists of 2. million images with high-quality captions.
Images and their captions have been gathered via an automated process from the web and therefore represent a wide variety of content.
Raw descriptions of images are collected from the
Methods compared.
We compare our new CLOOB to CLIP (Radford et al., 2021).
The CLOOB implementation is based on OpenCLIP (Ilharco et al., 2021), which achieves results equivalent to CLIP on the YFCC dataset (see Section 4.2).
OpenCLIP also reports results on the CC dataset.
As CLIP does not train models on CC we report results from this reimplementation as baseline.
Analogously to Radford et al. (2021, Section 2.4), we use the modiﬁed ResNet (He et al., 2016) and BERT (Devlin et al., 2018; 2019) architectures to encode image and text input.
We use the ResNet encoders ResNet-50, ResNet-101, and ResNet-50x4.
Hyperparameter selection and learning schedule.
We use the hyperparameter values of OpenCLIP, concretely, a learning rate of 1 × 10 − 3 and a weight decay of 0 .
1 for the Adam optimizer (Kingma et al., 2014) with decoupled weight decay regularization (Loshchilov & Hutter, 2019).
Deviating from OpenCLIP, we use a batch size of 512 due to computational restraints, which did not change the performance.
The learning rate scheduler for all experiments is cosine annealing with warmup and hard restarts (Loshchilov & Hutter, 2017).
We report the hyperparameter τ (default 0 . 07 ) from CLIP as τ − 1 of 14 .
3 to be in the same regime as the hyperparameter β for the modern Hopﬁeld networks.
The main hyperparameter search for CLOOB (also for YFCC pretraining in the next section) was done with ResNet-50 as the vision encoder.
Learnable τ − 1 in combination with the InfoLOOB loss results in undesired learning behavior (see Appendix Section A.1.4). Therefore, we set τ − 1 to a ﬁxed value of 30, which was determined via hyperparameter search (see Appendix Section A.3.2).
For modern Hopﬁeld networks,the hyperparameter β was set to 8 .
Further we scale the loss in Eq.
(11) with τ to remove the factor τ − 1 from the gradients (see Appendix Section A.1.4) resulting in the loss function τ L InfoLOOB .
Evaluation metrics: Zero-shot transfer learning.
We evaluate and compare both CLIP and
CLOOB on their zero-shot transfer learning capabilities on the following downstream image classiﬁcation tasks.
Birdsnap (Berg et al., 2014) contains images of 500 different North American bird species.
The Country211 (Radford et al., 2021) dataset consists of photos across 211 countries and is designed to test the geolocalization capability of visual representations.
Flowers102 (Nilsback & Zisserman, 2008) is a dataset containing images of 102 ﬂower species.
GTSRB (Stallkamp et al., 2011) contains images for classiﬁcation of German trafﬁc signs.
UCF101 (Soomro et al., 2012) is a video dataset with short clips for action recognition.
For UCF101 we follow the procedure reported in CLIP and extract the middle frame of every video to assemble the dataset.
Stanford Cars (Krause et al., 2013) contains images of 196 types of cars.
ImageNet (Deng et al., 2009) is a large scale image classiﬁcation dataset with images across 1,000 classes. ImageNetv2 (Recht et al., 2019) consists of
InfoLOOB increases the performance of CLIP in most of the tasks.
Hopﬁeld with InfoLOOB strongly improves the performance in 7 out of 8 datasets compared to both CLIP models. [Table] three new test sets with 10,000 images each for the ImageNet benchmark.
For further details see
Appendix Section A.3.3.
Results.
We employ the same evaluation strategy and use the prompt templates as published in CLIP (see Appendix Section A.3.3).
We report zero-shot results from two checkpoints in Table 1.
CLIP and
CLOOB were trained for a comparable number of epochs used in CLIP (see Appendix Section A.3.2) while CLIP* and CLOOB* were trained until evaluation performance plateaued (epoch 128).
In both cases CLOOB signiﬁcantly outperforms CLIP on the majority of tasks or matches its performance.
Statistical signiﬁcance of these results was assessed by an unpaired Wilcoxon test on a 5 % level.
Ablation studies.
CLOOB has two new major components compared to CLIP: (1) the InfoLOOB objective instead of the InfoNCE objective and (2) the modern Hopﬁeld networks.
To assess which of the new major components of CLOOB has led to the performance increase over CLIP, we performed ablation studies on CC.
First, we enhanced CLIP by replacing the InfoNCE objective with InfoLOOB.
Table 2 shows that the InfoLOOB objective increases the performance of CLIP in the majority of the datasets.
The reason is that InfoLOOB suffers less than InfoNCE from the “explaining away” problem. However, InfoLOOB is more effective for higher mutual information, that is, for a richer covariance structure.
Hopﬁeld networks amplify the covariance structure by retrieved embeddings. For InfoLOOB, however, this ampliﬁcation is disadvantageous as the saturation effect is increased by higher similarity between anchor and positive.
Thus, combining modern Hopﬁeld networks with InfoNCE leads to a performance drop.
Combining Hopﬁeld and InfoLOOB into CLOOB strongly improves the performance on 7 out of 8 zero-shot transfer learning tasks.
An additional ablation considers the learning rate scheduler.
For more details see in Appendix Section A.3.1.
4. YFCC P RETRAINING
Pretraining dataset.
To be comparable to the CLIP results, we use the same subset of 15 million samples from the YFCC100M dataset (Thomee et al., 2016) as in Radford et al. (2021), which we refer to as YFCC.
YFCC was created by ﬁltering YFCC100M for images which contain natural language descriptions and/or titles in English.
It was not ﬁltered by quality of the captions, therefore the textual descriptions are less rich and contain superﬂuous information.
The dataset withmillion samples used to train the CLIP models in Radford et al. (2021) has not been released and, thus, is not available for comparison.
Due to limited computational resources we are unable to compare CLOOB to CLIP on other datasets of this size.
Methods compared and evaluation.
In addition to the comparison of CLOOB and CLIP based on the OpenCLIP reimplementation (Ilharco et al., 2021), we include the original CLIP results (Radford et al., 2021, Table 12).
Hyperparameter selection.
We use the hyperparameters selected at the Conceptual Captions dataset, except learning rate, batch size, and β .
For modern Hopﬁeld networks, the hyperparameter β is set to 14 . [-
-]3 , which is the default parameter of τ − 1 for the InfoNCE objective in Radford et al.
Furthermore, the learning rate is set to 5 × 10 − 4 and a batch size of 1024 as in OpenCLIP of Ilharco et al.
For further details see Appendix Section A.3.2.
Evaluation metrics.
As in the previous experiment, methods are again evaluated at their zero-shot transfer learning capabilities on downstream tasks.
Results.
Table 3 provides results of the original CLIP and CLOOB trained on YFCC.
The results on zero-shot downstream tasks show that CLOOB outperforms the results of CLIP on all 7 tasks (ImageNet V2 results have not been reported in Radford et al.
Similarly, CLOOB outperforms
CLIP on 6 out of 7 tasks for linear probing.
Results of the comparison of CLOOB an the CLIP reimplementation of OpenCLIP are given in Table 4.
CLOOB exceeds the CLIP reimplementation in 7 out of 8 tasks for zero-shot classiﬁcation using ResNet-50 encoders. With larger ResNet encoders, CLOOB outperforms CLIP on all tasks.
Furthermore, the experiments with larger vision encoder networks show that CLOOB performance increases with network size.
Visualizations of predictions of CLOOB zero-shot classiﬁers from all datasets are shown in Appendix Section A.3.4.
[Table]
-shot results for the CLIP reimplementation and CLOOB using different ResNet architectures trained on YFCC.
CLOOB outperforms CLIP in 7 out of 8 tasks using ResNetencoders. With larger ResNet encoders CLOOB outperforms CLIP on all tasks.
The performance of CLOOB scales with increased encoder size.
[Table]
C ONCLUSION
For constrastive learning, we have introduced “Contrastive Leave One Out Boost” (CLOOB), for which modern Hopﬁeld networks boost learning with the InfoLOOB objective.
Modern Hopﬁeld networks both increase the stability of InfoLOOB and reinforce the covariance structure of the data.
We have shown theoretical properties of the InfoLOOB bound and objective.
Our results suggest InfoLOOB as an alternative to InfoNCE in contrastive learning.
An ablation study shows that both, the InfoLOOB objective and modern Hopﬁeld networks, are necessary to yield high performance.
At seven zero-shot transfer learning tasks, the novel CLOOB is compared to CLIP after pretraining on
Conceptual Captions and the YFCC dataset.
CLOOB consistently outperforms CLIP at zero-shot transfer learning across all considered architectures and datasets.
R EPRODUCIBILITY S TATEMENT
We will publish the source code after the reviewing period.
This will ensure that the results are reproducible in their entirety.
The datasets used for training our models as well as for the downstream tasks are publicly available.
E THICAL C ONSIDERATIONS
Impact on ML and related scientiﬁc ﬁelds.
Our research has the potential to positively impact a wide variety of ﬁelds of life due to its general applicability.
Most importantly, it has the potential to reduce the cost for training other AI systems, which could lead to a reduction of compute costs and carbon dioxide emissions.
However, any new development in machine learning can be applied for good or for bad.
Our system can be used for medical applications where it could save lives but might also be used for surveillance and malevolent systems.
Impact on society.
A potential danger could arise from an application of our approach in which users rely overly on the outcomes.
For example, in a medical setting, physicians might rely on the technical system and shift the liability towards the machine.
This might also happen in the domain of self-driving cars, when drivers start paying less attention to the trafﬁc because of an AI-based driving system.
Finally, our method may also be deployed in companies to automate various simple tasks, which might lead to a reduced need for particular jobs in production systems.
Consequences of failures of the method.
Depending on the application area, a failure of this method might be of lesser concern, such as a failed execution of a computer program.
If our method is employed within a larger automation system, a failure could result in damages such as a car accident or errors of a production system.
However, this holds for almost all machine learning methods, and their usage and testing depends on the application area.
Leveraging of biases in the data and potential discrimination.
Our proposed method relies on human-annotated data and thereby human decisions, which are usually strongly biased.
The undesirable biases contained in dataset are learned and may propagate to downstream applications.
Therefore, the responsible use of our method depends on a careful selection of the training data and awareness of the potential biases within those.
A A PPENDIX
This appendix consists of four sections (A.1–A.4).
Section A.1 provides the theoretical properties of the InfoLOOB and InfoNCE.
It is shown how to derive that InfoNCE is a lower bound on mutual information.
Further it is shown how to derive that InfoLOOB is an upper bound on mutual information.
The proposed loss function L InfoLOOB and its gradients are discussed.
In Section A. we discuss the estimation of mutual information for a toy example.
Section A.3 provides details on the experiments for Section 4.
Section A.4 brieﬂy reviews continuous modern Hopﬁeld networks.
Section A.5 discusses further related work.
C ONTENTS OF THE APPENDIX
A Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.1 InfoLOOB vs. InfoNCE
A.1.1 InfoNCE: Lower Bound on Mutual Information
A.1.2 InfoLOOB:
Upper Bound on Mutual Information
A.1.3 InfoLOOB: Analysis of the Objective
A.1.4 InfoNCE and InfoLOOB:
Gradients
A.1.5 InfoLOOB and InfoNCE: Probability Estimators
A.1.6 InfoLOOB and InfoNCE: Losses
A.2 Mutual Information Estimation . .
A.3 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.3.1 Ablation studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.3.2 Hyperparameters . .
A.3.3 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.3.4 Zero-shot evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.3.5 Linear probing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.4 Review of Modern Hopﬁeld Networks . . . . . . . . . . . . . . . . . . . . . . . . .
A.5 Further Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
L IST OF THEOREMS
A
Theorem (InfoNCE lower bound) . . . . . . . . . . . . . . . . . . . . . . . . . . .
A
Theorem (InfoLOOB upper bound) . . . . . . . . . . . . . . . . . . . . . . . . . .
A
Theorem (Weighted Covariances) . . . . . . . . . . . . . . . . . . . . . . . . . .
A
Theorem (Modern Hopﬁeld Networks: Retrieval with One Update) . . . . . . . . .
A
Theorem (Modern Hopﬁeld Networks: Exponential Storage Capacity) . . . . . . .
L IST OF DEFINITIONS
A Deﬁnition (Pattern Stored and Retrieved
L IST OF FIGURES
A1 Estimated mutual information of different objectives
A2 Visualization of zero-shot classiﬁcation of three examples from each dataset
L IST OF TABLES
A1 Inﬂuence of loss functions and Hopﬁeld retrieval
A2 Inﬂuence of learning rate scheduler
A3 Datasets used for zero-shot and linear probing
A4 Linear probing for CLIP (reimplementation) and CLOOB trained on YFCC
A. I NFO LOOB VS . I NFO NCE A.1. I NFO NCE: L OWER B OUND ON M UTUAL I NFORMATION
We derive a lower bound on the mutual information between random variables X and Y distributed according to p ( x , y ) .
The mutual information I( X ; Y ) between random variables X
and Y is
I( X
[Equation]
x
[Equation]
y
[Equation]
“InfoNCE” has been introduced in van den Oord et al.
and is a multi-sample bound .
In the setting introduced in van den Oord et al.
(2018), we have an anchor sample y given.
For the anchor sample y we draw a positive sample x 1 according to p ( x 1 | y ) .
Next, we draw a set ˜
X = { x 2 , . . .
, x N } according to p ( ˜ X ) , which are n − 1 negative samples drawn iid according to p ( x ) .
We have drawn a set X
= { x 1 , x 2 , . . .
, x N } according to p ( X | y ) , which is one positive sample x 1 drawn by p ( x 1 | y ) and N − 1 negative samples { x 2 , . .
, x N } drawn iid according to p ( x ) .
The InfoNCE with probabilities is
[Equation]
where we inserted the factor 1 N in contrast to the original version in van den Oord et al.
(2018), where we followed Poole et al.
; Tschannen et al.
; Cheng et al.
; Chen et al.
The InfoNCE with score function f ( x , y ) is
[Equation]
The InfoNCE with probabilities can be rewritten as:
[Equation]
This is the InfoNCE with f ( x , y ) =
p ( y | x ) .
Set of pairs.
The InfoNCE can be written in a different setting Poole et al.
(2019), which is used in most implementations.
We sample N pairs independently from p ( x , y ) , which gives Z = { ( x 1 , y 1 ) , ( x 2 , y 2 ) , . . .
, ( x N , y N ) } .
The InfoNCE is then
[Equation]
we have
[Equation]
where p ( i = 1 |
X, y ) is the probability that sample x 1 is the positive sample if we know there exists exactly one positive sample in X .
The InfoNCE is a lower bound on the mutual information.
The following inequality is from van den
Oord et al.
[Equation]
[Equation]
(cid:33)(cid:35)(cid:35)
[Equation]

 

[Equation]
=
I InfoNCE ( X 1 ; Y ) , where the " ≥ " is obtained by bounding ln(1 /N + a ) by ln( a ) , which gives a bound that is not very tight, since a = p ( x 1 ) p ( x 1 | y ) can become small.
However for the " ≈ " van den Oord et al.
(2018) have to assume
[Equation]
which is unclear how to ensure.
For a proof of this bound see Poole et al.
We assumed that for the anchor sample y a positive sample x 1 has been drawn according to p ( x 1 | y ) .
A set ˜ X
= { x 2 , . . .
, x N } of negative samples is drawn according to p ( x ) .
Therefore, we have a set X
= { x 1 , x 2 , . . .
, x N } that is drawn with one positive sample x 1 and N − 1 negative samples ˜
X = { x 2 , . . .
, x N } .
We have
[Equation]
Next, we present a theorem that shows this bound, where we largely follow Poole et al.
in the proof.
In contrast to Poole et al.
(2019), we do not use the NWJ bound Nguyen et al.
The mutual information is
[Equation]
Theorem A1 (InfoNCE lower bound) .
InfoNCE with score function f ( x , y ) according to Eq.
(A3) is a lower bound on the mutual information.
[Equation]
InfoNCE with probabilities according to Eq.
(A2) is a lower bound on the mutual information.
[Equation]
The second bound Eq.
(A14) is a special case of the ﬁrst bound Eq.
(A13) .
Proof.
Part (I) : Lower bound with score function f ( x , y ) .
For each set ˜ X
= { x 2 , . . .
, x N } , we deﬁne as data-dependent (depending on ˜ X ) score function g ( x 1 , y , ˜ X ) that is based on the score function f ( x , y ) .
Therefore we have for each ˜ X a different data-dependent score function g based on f .
We will derive a bound on the InfoNCE, which is the expectation of a lower bond on the mutual information over the score functions.
For score function g ( x 1 , y , ˜ X ) , we deﬁne a variational distribution q ( x 1 | y , ˜ X ) over x 1 :
[Equation]
which ensures
[Equation]
We have
[Equation]
For the function g , we set
[Equation]
For the function f we use
[Equation]
where sim( x , y ) is typically the cosine similarity.
We next show that InfoNCE is a lower bound on the mutual information.
I( X 1 ; Y ) =
E p ( ˜ X )
[I( X 1 ; Y )]
=
E p ( ˜ X ) (cid:20)
E p ( x 1 , y ) (cid:20) ln p ( x 1 | y ) p ( x 1 ) (cid:21)(cid:21) (A21) =
E p ( ˜ X ) (cid:34)
E p ( x 1 , y ) (cid:34) ln (cid:32) p ( x 1 | y ) q ( x 1 | y , ˜ X ) q ( x 1 | y , ˜ X ) p ( x 1 ) (cid:33)(cid:35)(cid:35) =
E p ( ˜ X ) (cid:34)
E p ( x 1 , y ) (cid:34) ln q ( x 1 | y , ˜ X ) p ( x 1 ) (cid:35) + E p ( y ) (cid:104) KL( p ( x 1 | y ) (cid:107) q
x 1 | y , ˜ X ))
(cid:105)(cid:35) ≥ E p ( ˜ X ) (cid:34)
E p ( x 1 , y ) (cid:34) ln q ( x 1 | y , ˜ X ) p ( x 1 ) (cid:35)(cid:35) =
E p ( ˜ X ) (cid:34)
E p ( x 1 , y ) (cid:34) ln g ( x 1 , y , ˜ X )
Z ( y , ˜ X ) (cid:35)(cid:35) =
E p ( ˜ X ) (cid:104) E p ( x 1 , y ) (cid:104)
ln g ( x 1 , y , ˜ X ) − ln (cid:16) E p ( x 1 ) (cid:104) g
( x 1 , y , ˜ X ) (cid:105)(cid:17)(cid:105)(cid:105) =
E p ( ˜ X ) (cid:104) E p (
[Equation] [Equation]
=
[Equation]
=
I InfoNCE ( X 1 ; Y ) .
For the ﬁrst " ≥ " we used that the Kullback-Leibler divergence is non-negative.
For the second " ≥ " we used the inequality ln a (cid:54) a − 1 for a > 0 .
Part (II) : Lower bound with probabilities.
If the score function f is
[Equation]
A22) then the bound is
[Equation]
This is the bound with probabilities in the theorem.
A.1. I NFO LOOB: U PPER B OUND ON M UTUAL I NFORMATION
We derive an upper bound on the mutual information between random variables X and Y distributed according to p ( x , y ) .
The mutual information I( X ; Y ) between random variables X
and Y is
I( X ; Y
[Equation]
In Poole et al.
(2019) Eq.
(13) introduces a variational upper bound on the mutual information, which has been called "Leave one out upper bound" (called "L1Out" in Cheng et al.
For simplicity, we call this bound "InfoLOOB", where LOOB is an acronym for "Leave One Out Bound".
In contrast to InfoNCE, InfoLOOB is an upper bound on the mutual information.
InfoLOOB is analog to InfoNCE except that the negative samples do not contain a positive sample.
Fig.
1 and Fig.
2 in
Cheng et al.
(2020) both show that InfoLOOB is a better estimator for the mutual information than
InfoNCE (van den Oord et al., 2018), MINE (Belghazi et al., 2018), and NWJ (Nguyen et al., 2010).
The InfoLOOB with score function f ( x , y ) is deﬁned as
[Equation]
The InfoLOOB with probabilities is deﬁned as
[Equation]
This is the InfoLOOB with f ( x , y ) =
p ( y | x ) .
The InfoLOOB with probabilities can be written in different forms:
[Equation]
Set of pairs.
The InfoLOOB can we written in a different setting (Poole et al., 2019), which will be used in our implementations.
We sample N pairs independently from p ( x , y ) , which gives
X = { ( x 1 , y 1 ) , ( x 2 , y 2 ) , . . .
, ( x N , y N ) } .
The InfoLOOB is then
I InfoLOOB ( X ; Y ) =
E p ( X | y )
[Equation]
We assume that an anchor sample y is given.
For the anchor sample y we draw a positive sample x 1 according to p ( x 1 | y ) .
Next, we draw a set ˜ X
= { x 2 , . . .
, x N } of negative samples according to ˜ p ( x | y ) .
For a given y , the x that have a large p ( x | y ) are drawn with a lower probability ˜ p ( x | y ) compared to random drawing via p ( x ) .
The negatives are indeed negatives.
We have drawn ﬁrst anchor sample y and then X = { x 1 , . . .
, x N } , where x 1 is drawn according to p ( x 1 | y ) and ˜ X
= { x 2 , . . .
, x N } are drawn iid according to ˜ p ( x | y ) .
We have
[Equation]
A
[Equation]
We assume for score function f ( x , y )
[Equation]
We ensure this by using for score function f
[Equation]
where sim( x , y ) is typically the cosine similarity.
InfoLOOB with score function f ( x , y ) is
[Equation]
The reference constant Z ( y ) gives the average score f
( x , y ) , if the negatives for y are selected with lower probability via ˜ p
( x | y ) than with random drawing according to p ( x ) .
[Equation]
We deﬁne the variational distribution
[Equation]
With the variational distribution q ( x | y )
[Equation]
we express our main assumption.
The main assumption for the bound is:
[Equation]
This assumption can be written as
[Equation]
This assumption ensures that the x with large p ( x | y ) ) are selected with lower probability via ˜ p
( x | y ) than with random drawing according to p ( x ) .
The negatives are ensured to be real negatives, that is, p ( x | y ) is small and so is f ( x , y ) .
Consequently, we make sure that we draw x with sufﬁcient small f ( x , y ) .
The Kullback-Leibler gives the minimal required gap between drawing f ( x , y ) via p ( x ) and drawing f ( x , y ) via ˜ p
( x | y ) .
EXAMPLE.
With h ( y ) > 0 , we consider the setting
[Equation]
The main assumption becomes
[Equation]
The main assumption holds since
[Equation]
p ( y | x ) h ( y ) p ( y ) (cid:19) − 1 (cid:35)(cid:33) −(cid:54) (cid:32)
E p ( x ) (cid:20) p ( y | x ) h ( y ) p ( y ) (cid:21) − 1 (cid:33) −= E p ( x ) (cid:20) p ( y | x ) h ( y ) p ( y ) (cid:21) = (cid:90) p ( y , x ) h ( y ) p ( y ) d x = h ( y ) , where we used for the (cid:54) Jensen’s inequality with the function f ( a ) = 1 /a , which is convex for a > 0 .
For score function f ( x , y ) and distribution ˜ p ( x | y ) for sampling the negative samples, we have deﬁned:
[Equation]
Next theorem gives the upper bound of the InfoLOOB on the mutual information, which is
[Equation]
Theorem A2 (InfoLOOB upper bound) .
If ˜ X = { x 2 , . . .
, x N } are drawn iid according to ˜ p ( x | y ) and if the main assumption holds:
[Equation]
Then InfoLOOB with score function f ( x , y ) as in Eq.
(A25) is an upper bound on the mutual information:
[Equation]
If the negative samples ˜ X
= { x 2 , . . .
, x N } are drawn iid according to p ( x ) , then InfoLOOB with probabilities according to Eq.
(A26) is an upper bound on the mutual information:
I
[Equation]
X 1 ; Y )
[Equation]
The second bound Eq.
(A49) is a special case of the ﬁrst bound Eq.
(A48) .
Proof.
Part (I) : Upper bound with score function f ( x , y ) .
I( X 1 ; Y ) =
E p ( x 1 , y ) (cid:20) ln p ( x 1 | y ) p ( x 1 ) (cid:21) (A50) =
E p ( x 1 , y ) (cid:20) ln (cid:18) p ( x 1 | y ) q ( x 1 | y ) q ( x 1 | y ) p ( x 1 ) (cid:19)(cid:21) =
E p ( x 1 , y ) (cid:20) ln
q
( x 1 | y ) p ( x 1 ) (cid:21) + E p
( y )
[KL( p ( x 1 | y ) (cid:107) q
x 1 | y ))] (cid:54) E p ( x 1 , y ) (cid:20) ln
q
( x 1 | y ) p ( x 1 ) (cid:21) + E p
( y ) (cid:2)
ln E p ( x 1 ) [ f ( x 1 , y )] − ln Z ( y ) (cid:3) =
E p ( x 1 , y ) (cid:20) ln
q
( x 1 | y ) p ( x 1 ) + ln E p ( x 1 )
[ f ( x 1 , y )]
Z ( y ) (cid:21) =
E p ( x 1 , y ) (cid:20) ln (cid:18) f
( x 1 , y )
E p ( x 1 ) [ f ( x 1 , y )] E p ( x 1 ) [ f ( x 1 , y )]
Z ( y ) (cid:19)(cid:21) =
E p ( x 1 , y ) (cid:20) ln
f
( x 1 , y )
Z ( y ) (cid:21) =
E p ( x 1 , y )  
ln   f ( x 1 , y )
E ˜ p
( X | y ) (cid:104) 1 N − 1 (cid:80)
[Equation]
where the ﬁrst " (cid:54) " uses assumption Eq.
(A37), while Jensens’s inequality was used for the second " (cid:54) " by exchanging the expectation and the " ln ".
We also used
E ˜ p
[Equation]
X | y )
[Equation]
Part (II) : Upper bound with probabilities.
If the score function f is
[Equation]
and
[Equation]
then
[Equation]
Therefore, the main assumption holds, since
[Equation]
The bound becomes
[Equation]
An alternative proof is as follows:
[Equation]
[Equation]
(cid:33)(cid:35)(cid:35)
[Equation]
(cid:34)
[Equation]
) (cid:34)
[Equation] (cid:32) [Equation] N (cid:88) i = p (
[Equation]
) p ( x i ) (cid:33)(cid:35)(cid:35)
[Equation]
where we applied Jensens’s inequality for the exchanging the expectation and the " ln " to obtain the " (cid:54) " inequality.
Experiments that compare upper and lower bounds as mutual information estimates are provided in Cheng et al.
(2020) and in Poole et al.
In Fig.
2 in Cheng et al.
(2020) it is shown that
InfoLOOB is a good estimator of the mutual information.
A.1. I NFO LOOB: A NALYSIS OF THE O BJECTIVE
This subsection justiﬁes the maximization of the InfoLOOB bound for contrastive learning.
Maximizing the InfoLOOB bound is not intuitive as it was introduced as an upper bound on the mutual information in the previous subsection.
Still maximizing the InfoLOOB bound leads to a good approximation of the mutual information, in particular for high mutual information.
InfoLOOB with a neural network as a scoring function is not an upper bound on the mutual information when not under-sampling.
As we use InfoLOOB on training data for which we do not know the sampling procedure, we cannot assume under-sampling.
Therefore, we elaborate more on the rationale behind the maximization of the InfoLOOB bound.
(I)
We show that InfoLOOB with neural networks as scoring function is bounded from above.
Therefore, there exists a maximum and the optimization problem is well deﬁned.
(II)
We show that InfoLOOB with neural networks as scoring function differs by two terms the mutual information.
The ﬁrst term is the Kullback-Leibler divergence between the variational q ( x | y ) and the posterior p ( x | y ) .
This divergence is minimal for q ( x | y )
=
p ( x | y ) , which implies f ( y | x ) = p ( y | x ) .
The second term is governed by the difference between the mean E[ f ( x , y )] and the empirical mean 1 / ( N − 1) (cid:80) i f ( x , y ) .
Hoeffding’s inequality bounds this difference as we demonstrate in this subsection.
Therefore, the second term is negligible for large N .
In contrast, the KL term is dominant and the relevant term, therefore maximizing InfoLOOB leads to f ( y | x ) ≈ p
( y | x ) .
We assume that an anchor sample y is given.
For the anchor sample y , we draw a positive sample xaccording to p ( x 1 | y ) .
We deﬁne the set ˜ X
= { x 2 , . . .
, x N } of negative samples, which are drawn iid according to p ( x ) .
We deﬁne the set X
= { x 1 , . . .
, x N } .
We have
[Equation]
We use the score function
[Equation]
(A
[Equation]
where sim( x , y ) is typically the cosine similarity.
The InfoLOOB with score function f ( x , y ) is deﬁned as
I InfoLOOB
[Equation]
X
[Equation]
(cid:33)(cid:35)(cid:35) .
(A66)
We deﬁne the variational distribution
[Equation]
(A67)
[Equation]
[Equation]
[Equation]
KL( p ( x 1 | y ) (cid:107) q
( x 1 | y ))]
=
E p ( x 1 , y )
[ln f ( x 1 , y )]
− E p ( y ) (cid:34)
E p ( X | y ) (cid:34) ln (cid:32)
N −
N (cid:88) i = f
( x i , y ) (cid:33)(cid:35)(cid:35) + E p ( y ) (cid:34)
E p ( X | y ) (cid:34) ln (cid:32)
N −
N (cid:88) i = f
( x i , y ) (cid:33)(cid:35)(cid:35) − E p
( y ) (cid:34) ln
(cid:32)
E p ( X | y ) (cid:34)
N −
N (cid:88) i = f
( x i , y ) (cid:35)(cid:33)(cid:35) + E p ( y )
[KL( p ( x 1 | y ) (cid:107) q
x 1 | y ))]
=
E p ( y ) (cid:34)
E p ( X | y ) (cid:34) ln (cid:32) f
( x 1 , y ) 1 N − 1 (cid:80) Ni
=2 f ( x i , y ) (cid:33)(cid:35)(cid:35)
+ E p ( y ) (cid:34)
E p ( X | y ) (cid:34) ln (cid:32)
N −
N (cid:88) i = f
( x i , y ) (cid:33)(cid:35)(cid:35) − E p
( y ) (cid:34) ln
(cid:32)
E p ( X | y ) (cid:34)
N −
N (cid:88) i = f
( x i , y ) (cid:35)(cid:33)(cid:35) + E p ( y )
[KL( p ( x 1 | y ) (cid:107) q
x 1 | y ))]
= I InfoLOOB ( X 1 ; Y )
+ E p
( y ) (cid:34)
E p ( X | y ) (cid:34) ln (cid:32)
N −
N (cid:88) i = f
( x i , y ) (cid:33)(cid:35)(cid:35) − E p
( y ) (cid:34) ln
(cid:32)
E p ( X | y ) (cid:34)
N −
N (cid:88) i = f
( x i , y ) (cid:35)(cid:33)(cid:35) + E p ( y )
[KL( p ( x 1 | y ) (cid:107) q
x 1 | y ))]
= I InfoLOOB ( X 1 ; Y )
+ E p
( y ) (cid:34)
E p ( X | y ) (cid:34) ln (cid:32)
N −
N (cid:88) i = f
( x i , y ) (cid:33)(cid:35)(cid:35) − E p
( y ) (cid:2)
ln (cid:0) E p ( x 1 )
[ f ( x 1 , y )]
(cid:1)(cid:3)
+ E p
y )
[KL( p ( x 1 | y ) (cid:107) q
x 1 | y ))]
=
I InfoLOOB ( X 1 ; Y ) − E p
( y ) (cid:34)
E p ( ˜ X ) (cid:34) ln (cid:32)
E p ( x 1 ) [ f ( x 1 , y )]
1 N − 1 (cid:80) Ni =
2 f ( x i , y ) (cid:33)(cid:35)(cid:35)
[Equation]
=
[Equation]
InfoLOOB
[Equation]
where we used
[Equation]
and
[Equation]
Since both KL and DE are non-negative (for DE see below), to increase InfoLOOB we have either to decrease KL or to increase DE .
Bounding DE .
Next we bound DE .
We deﬁne
[Equation]
The log-sum-exponential ( lse ) is the maximum of L on the N -dimensional simplex D with D
= { z | (cid:80)
i z i = 1 , 0 (cid:54) z
i } (Gao & Pavel, 2017):
[Equation]
For some z ∈ D we have
E
[Equation]
[Equation]
therefore
[Equation]
We obtain
E p ( y ) (cid:34)
E p ( ˜ X ) (cid:34)
[Equation]
We obtain via Jensen’s inequality
[Equation]
y ))
(cid:3)(cid:33)(cid:35)
= 0 .
If we combine both previous inequalities, we obtain
[Equation]
DE
[Equation]
A
[Equation]
In particular, for bounded sim( x 1 , y ) , we get
[Equation]
while Hoeffding’s lemma gives
[Equation]
Thus, for bounded sim( x 1 , y ) , DE is bounded, therefore also InfoLOOB.
For sub-exponential distributions with variance σ 2 , for which Bernstein’s condition with τ > b holds (Eq.
(2.16) in
Wainwright (2019)), we get (Proposition 2.3 in Wainwright (2019)):
[Equation]
Next, we show that DE is small.
Hoeffding’s inequality states that if f ( x , y ) ∈ [ a, b ] then
[Equation]
For
[Equation]
we have
[Equation]
where we used ln a (cid:54) a − 1 for 0 < a .
Analog for
[Equation]
we have
[Equation]
where we used − ln a ≥ 1 − a for 0 < a .
In summary, for
[Equation]
we have
[Equation]
It follows that
[Equation]
DE averages the ln -term over y and ˜ X , therefore it has an even smaller bound than the bound above on the ln -term.
Consequently, for small b − a and large N , the term DE is small.
KL is decreased by making the variation distribution q ( x 1 | y ) more similar to the posterior p ( x 1 | y ) .
The value DE only depends on the marginal distributions p ( y ) and p ( x ) , since p ( ˜ X )
= (cid:81)
Ni =2 p ( x i ) .
The value DE can be changed by adding an offset to f ( x , y ) .
However, scaling f ( x , y ) by a factor does not change DE .
Consequently, DE is difﬁcult to change.
Therefore, increasing InfoLOOB is most effective by making q ( x 1 | y ) more similar to the posterior p ( x 1 | y ) .
Gradient of InfoLOOB expressed by gradients of KL and DE .
Assume that the similarity is parametrized by w giving sim( x , y ; w ) .
[Equation]
where C is independent of w .
Next, we compute the derivative of KL with respect to parameters w .
∂ KL
[Equation]
The derivative is the average difference between the posterior distribution p ( x 1 | y ) and the variational distribution q ( x 1 | y ) multiplied by the derivative of the similarity function.
If both distribution match, then the derivative vanishes.
Next, we compute the derivative of DE with respect to parameters w
[Equation]
, y )] ∂ sim( x 1 , y ; w ) ∂ w d x(cid:21) − τ − 1 E p ( y ) (cid:34)
E p ( ˜ X ) (cid:34)
N −
N (cid:88) i = f
( x i , y ) 1 N − 1 (cid:80) Nj =2 f ( x j , y ) ∂
sim( x i , y ; w ) ∂ w (cid:35)(cid:35) =
τ − 1 E p ( y ) (cid:20)
E p ( x 1 ) (cid:20) f
( x 1 , y )
E p ( x ) [ f ( x , y )] ∂ sim( x 1 , y ; w ) ∂ w (cid:21)(cid:21) − τ − 1 E p ( y ) (cid:34)
E p ( ˜ X ) (cid:34)
N −
N (cid:88) i = f
( x i , y ) 1 N − 1 (cid:80) Nj =2 f ( x j , y ) ∂
sim( x i , y ; w ) ∂ w (cid:35)(cid:35) =
τ − 1 E p ( y ) (cid:34)
N −
N (cid:88) i =
E p ( x i ) (cid:20) f
( x i , y )
E p ( x ) [ f ( x , y )] ∂ sim( x i , y ; w ) ∂ w (cid:21)(cid:35) − τ − 1 E p ( y ) (cid:34)
E p ( ˜ X ) (cid:34)
N −
N (cid:88) i = f
( x i , y ) 1 N − 1 (cid:80) Nj =2 f ( x j , y ) ∂
sim( x i , y ; w ) ∂ w (cid:35)(cid:35) =
τ − 1 E p ( y ) (cid:34)
E p ( ˜ X ) (cid:34)
N −
N (cid:88) i = f
( x i , y )
E p ( x ) [ f ( x , y )] ∂ sim( x i , y ; w ) ∂ w (cid:35)(cid:35) −
τ − 1 E p ( y ) (cid:34)
E p ( ˜ X ) (cid:34)
N −
N (cid:88) i = f
( x i , y ) 1 N − 1 (cid:80) Nj =2 f ( x j , y ) ∂
sim( x i , y ; w ) ∂ w (cid:35)(cid:35) =
τ − 1 E p ( y ) (cid:34)
E p ( ˜ X ) (cid:34)
N −
N (cid:88) i = (cid:32)
E p ( x ) [ f ( x , y )] − 1 N − 1 (cid:80) Nj =2 f ( x j , y ) (cid:33) f
( x i , y ) ∂
sim( x i , y ; w ) ∂ w (cid:35)(cid:35) =
τ − 1 E p ( y ) (cid:34)
E p ( ˜ X ) (cid:34)
N −
N (cid:88) i = (cid:32)
Z − 1 N − 1 (cid:80) Nj =2 f ( x j , y ) (cid:33) f
( x i , y ) ∂
sim( x i , y ; w ) ∂ w (cid:35)(cid:35) .
The derivative is the average of 1 Z − 1 1 N − 1 (cid:80) Nj =2 f ( x j , y ) multiplied by the score function and the derivative of the similarity function.
The average is over y and ˜ X , therefore the whole derivative becomes even smaller.
Consequently, for small b − a and large N , the derivative of DE is small.
Note that for (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
we have
[Equation]
therefore
[Equation]
If the expectation Z is well approximated by the average
[Equation]
Nj
[Equation]
f ( x j , y ) , then both DE and its gradient are small.
Derivative of InfoLOOB via KL and DE :
[Equation]
In this gradient, the KL term is dominating, therefore f ( x , y ) is pushed to approximate the conditional probability p
( y | x ) .
Modern Hopﬁeld networks lead to larger values of p ( y | x ) as the mutual information becomes larger, therefore modern Hopﬁeld networks help to push f ( x , y ) to large values.
Furthermore, modern Hopﬁeld networks increase Z , which is in the denominator of the bound on
DE and its derivative.
A.1. I NFO NCE AND I NFO LOOB: G RADIENTS
We consider the InfoNCE and the InfoLOOB loss function.
For computing the loss function, we sample N pairs independently from p ( x , y ) , which gives the training set { ( x 1 , y 1 ) , ( x 2 , y 2 ) , . . .
, ( x N , y N ) } .
InfoNCE and InfoLOOB only differ in using the positive example in the negatives.
More precisely, InfoNCE uses for the matrix of negative samples
X = ( x 1 , . . .
, x N ) , while InfoLOOB uses ˜ X
=
( x 2 , . .
, x N ) .
InfoNCE.
The InfoNCE loss is
[Equation]
where we used
[Equation]
For the score function f ( x , y ) , we use
[Equation]
with τ as the temperature.
The loss function for this score function is
[Equation]
where lse is the log-sum-exp function ( lse ):
[Equation]
for β > 0 and vector
a = ( a 1 , . . . , a N ) .
The gradient with respect to y is
[Equation]
which is the positive example x 1 that ﬁts to the anchor example y minus the Hopﬁeld network update with state pattern y and stored patterns X
and then this difference multiplied by τ − 1 .
This gradient can be simpliﬁed, since the positive example x 1 is also in the negative examples.
Using p = ( p 1 , . .
. , p N ) T = softmax (cid:0) τ
[Equation]
1 X T y (cid:1) , we obtain
[Equation]
where
[Equation]
is the softmax average over the negatives x i for 2 (cid:54) i (cid:54) N without x 1 .
It can be easily seen that1 − p
[Equation]
Ni =2 p i
= 1 − p 1 1 − p 1 = 1 .
For the derivative of the InfoLOOB see below.
The gradient with respect to x 1 is
[Equation]
Consequently, the learning rate is scaled by (1 − p 1 ) .
The sum of gradients with respect to x 1 and x i is
[Equation]
where 1 is the vector with ones.
However, the derivatives with respect to the weights are not zero since the x i are differently computed.
InfoLOOB.
The InfoLOOB loss is
[Equation]
where we used
[Equation]
For the score function f ( x , y ) , we use
[Equation]
with τ as the temperature.
The loss function for this score function is
[Equation]
where lse is the log-sum-exponential function.
The gradient with respect to y is
[Equation]
which is the positive example x 1 that ﬁts to the anchor example y minus the Hopﬁeld network update with state pattern y and stored patterns ˜ X
and then this difference multiplied by τ − 1 .
The gradient with respect to x 1 is
[Equation]
The sum of gradients with respect to x 1 and x i is
[Equation]
where 1 is the vector with ones.
However, the derivatives with respect to the weights are not zero since the x i are differently computed.
Gradients with respect to τ − 1 .
The gradient of the InfoNCE loss Eq.
(A98) using the similarity Eq.
(A100) with respect to τ − 1 is
[Equation]
which is the similarity of the anchor y with the difference of the positive example x 1 and the Hopﬁeld network update with state pattern y and stored patterns X .
The gradient of the InfoLOOB loss
Eq.
(A110) using the similarity Eq.
(A112) with respect to τ − 1 is
[Equation]
with the difference that the Hopﬁeld network update is done with stored patterns ˜ X instead of X .
Without the positive example x 1 in the stored patterns ˜ X , the term x 1 − ˜ X softmax (cid:16) τ − 1 ˜ X T y (cid:17) in Eq.
(A120) will not decrease like the term x 1 − X softmax (cid:0) τ − 1 X T y (cid:1) in Eq.
(A118) but grow even larger with better separation of the positive and negative examples.
A.1. I NFO LOOB AND I NFO NCE: P ROBABILITY E STIMATORS
In McAllester & Stratos (2018; 2020) it was shown that estimators of the mutual information by lower bounds have problems as they come with serious statistical limitations.
Statistically more justiﬁed for representing the mutual information is a difference of entropies, which are estimated by minimizing the cross-entropy loss.
Both InfoNCE and InfoLOOB losses can be viewed as cross-entropy losses.
We sample
N pairs independently from p ( x , y ) , which gives
Z = { ( x 1 , y 1 ) , ( x 2 , y 2 ) , . .
, ( x N , y N ) } .
We set X
= { x 1 , x 2 , . . .
, x N } and Y = { y 1 , y 2 , . .
, y N } , so that, Z = X × Y .
The score function f ( x , y ) is an estimator for p ( x , y ) .
Then we obtain estimators ˆ q for the conditional probabilities.
ˆ q
( y i | x i , Y \ { y i } ) is an estimator for p ( y i | x i ) and ˆ q ( x i |
y i , X \ { x i } ) an estimator for p ( x i |
y i ) .
Each estimator ˆ q uses beyond ( x i , y i ) additional samples to estimate the normalizing constant.
For InfoNCE these estimators are
[Equation]
The cross-entropy losses for the InfoNCE estimators are
[Equation]
For InfoLOOB these estimators are
[Equation]
The cross-entropy losses for the InfoLOOB estimators are
[Equation]
The InfoLOOB estimator uses for normalization
[Equation]
in contrast to InfoNCE, which uses
[Equation]
If InfoNCE estimates the normalizing constant separately, then it would be biased.
( x i , y i ) is drawn according to p ( x i , y i ) instead of p ( x i ) p ( y i ) .
In contrast, if InfoLOOB estimated the normalizing constant separately, then it would be unbiased.
A.1. I NFO LOOB AND I NFO NCE: L OSSES
We have N pairs drawn iid from p ( x , y ) , where we assume that a pair ( x i , y i ) is already an embedding of the original drawn pair.
These build up the embedding training set Z = { ( x 1 , y 1 ) , ( x 2 , y 2 ) , . . .
, ( x N , y N ) } that allows to construct the matrices X
=
( x 1 , x 2 , . . .
, x N ) of N embedding samples x i and Y =
( y 1 , y 2 , . . .
, y N ) of N embedding samples
y i .
We also have
M stored patterns U
= ( u 1 , . .
. , u M ) and K stored patterns V = ( v 1 , . .
. , v K ) .
The state vectors x i and y i are the queries for the Hopﬁeld networks, which retrieve some vectors from U or V .
We normalize vectors (cid:107) x i (cid:107)
=
(cid:107) y
i (cid:107) =
(cid:107) u
i (cid:107) =
(cid:107) v i (cid:107) = 1 .
The following vectors are retrieved from modern Hopﬁeld networks (Ramsauer et al., 2021):
[Equation]
where U x i denotes an image-retrieved image embedding, U y i
a text-retrieved image embedding, V x i an image-retrieved text embedding and V y i a text-retrieved text embedding. The hyperparameter β corresponds to the inverse temperature: β = 0 retrieves the average of the stored pattern, while large β retrieve the stored pattern that is most similar to the state pattern (query).
We consider the loss functions
[Equation]
[Table] [Equation] [Table]
[Equation]
where for [Table] the sum (cid:80)
[Equation]
= i in the denominator contains only negative examples j .
We do not consider the loss function L H − UVUV InfoLOOB because of the high variance in the dot product U T x
i V y
i as elaborated in the following.
Let us consider the dot product between the anchor retrieval with the positive pattern retrieval for the loss functions with Hopﬁeld.
In the ﬁrst term of the loss function Eq.
(A138),
U x i is the anchor with
V y i as the positive sample and V y
i with U x i as the positive sample for the second term, since the anchor also appears in each term of the denominator.
Equivalently the same is valid for Eq.
(A139), but with positive samples V x i and U y i respectively.
These dot products can be written as
[Equation]
High variance of U T x i V y
i .
To compute the dot product U T x
i V y
i , M + K stored patterns are required ( M of the u j and K of the v j ).
In contrast, the dot products U T x i
U y
i and V T x i V y
i require only M or respectively K stored patterns.
Therefore, U T x i V y
i has higher variance than both U T x
i U
y i
and
V T x i V y i .
Covariance structure extracted by U T x
i U
y i
and
V T x i V y
i .
The Jacobian J of the softmax p = softmax( β a ) is
[Equation]
which is a symmetric, positive semi-deﬁnite matrix with one eigenvalue of zero for eigenvector 1 .
J( β a ) is diagonally dominant since | p i (1 − p i [Equation] | − (cid:80) j (cid:54)
[Equation]
Next we give upper bounds on the norm of J .
Lemma A1.
For a softmax p = softmax( β x ) with m = max i p i (1 − p i ) , the spectral norm of the
Jacobian J of the softmax is bounded:
[Equation]
In particular everywhere holds
[Equation]
If p max = max i p i ≥ 1 − (cid:15) ≥ 0 .
5 , then for the spectral norm of the Jacobian holds
[Equation]
Proof.
We consider the maximum absolute column sum norm
[Equation]
) and the maximum absolute row sum norm
[Equation]
A150)
We have for A
=
J = β (cid:0) diag( p )
[Equation]
pp T (cid:1) (cid:88)
[Equation]
Therefore, we have
[Equation]
The last inequality is a direct consequence of Hölder’s inequality.
For 0 (cid:54) p i (cid:54) 1 , we have p i (1 − p i ) (cid:54) 0 .
Therefore, m (cid:54) 0 .
25 for all values of p i .
If p max ≥ 1 − (cid:15) ≥ 0 .
5 ( (cid:15) (cid:54) 0 . 5 ), then 1 − p max (cid:54) (cid:15) and for p
i (cid:54) =
p max p i (cid:54) (cid:15) .
The derivative ∂x ( − x ) /∂x
= 1 − 2 x > 0 for x < 0 .
5 , therefore x (1 − x ) increases with x for x < 0 .
Using x = 1 − p max and for p
i (cid:54) =
p max x = p
i , we obtain p
i (1 − p i ) (cid:54) (cid:15) (1 − (cid:15) ) for all i .
Consequently, we have m (cid:54) (cid:15) (1 − (cid:15) ) .
For the softmax p = softmax( β a ) with Jacobian ∂ J /∂ a = J( β a ) =
β (cid:0) diag( p ) − pp T (cid:1) and for arbitrary N -dimensional vectors b and c , we have
[Equation]
Therefore, b T J( β a ) c is β times the covariance between b and c if component i is drawn with probability p
i of the multinomial distribution p .
In our case the component i is sample i .
Using the mean ˆ u = 1 /M (cid:80) Mi =1 u i , the empirical covariance of data U is [Equation]
[Equation]
The weighted covariance (samples u i are drawn according to p i )
[Equation]
[Equation] which replaces 1 /M from equal sampling by the p i , that is, u i is sampled with probability p i .
The next theorem states how to express the dot product U T x
i U
y i by weighted covariances of the data
U .
Theorem A3 (Weighted Covariances) .
Using the weighted covariances
[Equation]
where the mean Jacobian J m is symmetric, diagonally dominant, and positive semi-deﬁnite with spectral norm bounded by (cid:107) J m (cid:107) 2 (cid:54) 0 .
5 β .
The dot product U T x i U y i can be expressed by the weighted covariances
[Equation]
where the mean is ¯ u = 1 /M
U 1 .
Proof.
We apply the mean value theorem to the softmax with the symmetric, diagonally dominant, positive semi-deﬁnite Jacobian matrix
[Equation]
We set a (cid:48)
=
and use β a instead of a , which gives:
[Equation]
which is exact.
We obtain
[Equation]
The spectral norm of J m is bounded by (cid:107) J m (cid:107) 2 (cid:54) 0 .
5 β , since this bound holds for every J( λβ a ) in
J m ( β a ) =
(cid:82) 1 0 J( λβ a ) d λ according to Lemma A1.
The dot product between the anchor retrieval and the positive sample is:
[Equation]
where we used the mean ¯ u = 1 /M
U 1 and the weighted covariances
[Equation]
The Jacobian J m is symmetric, diagonally dominant, and positive semi-deﬁnite.
The weighted covariance Cov( U , . ) is the covariance if the stored pattern u i is drawn according to an averaged p i given by J m ( . ) .
Analog for weighted covariance Cov( V , . ) .
When maximizing the dot product
U T x i
U
y
i , the normalized vectors x i and y i are encouraged to agree on drawing the patterns
u
i with the same probability p i to generate similar weighted covariance matrices
Cov( U , . ) .
If subsets of U have a strong covariance structure, then it can be exploited to produce large weighted covariances and, in turn, large dot products of U T x
i U y
i .
Furthermore, for a large dot product U T x
i
U
y
i , x i and y i have to be similar to one another to extract the same direction from the covariance matrices.
All considerations are analog for V T x
i V y i .
A. M UTUAL I NFORMATION E STIMATION
We follow the toy experiment discussed in Poole et al.
, Belghazi et al.
and
Cheng et al.
(2020) and experimentally conﬁrm the superior quality of InfoLOOB for mutual information than InfoNCE.
The dataset consists of samples ( x i , y i ) drawn jointly from a multivariate Gaussian distribution with correlation ρ where the dimension of the samples x and y is set to d = 20 .
We examine the performance of InfoLoob with and without Hopﬁeld and InfoNCE at estimating mutual information of these samples.
Due to the Gaussian distribution, the true value of mutual information can be calculated as I ( x , y ) =
− d 2 log(1 − ρ 2 ) .
We set the mutual information true value to the values (2 . 0 , 4 . 0 , 6 . 0 , 8 . 0 , 10 . 0 , 14 . 0) by varying the value of ρ .
At each MI true value, we sample data batches 1024 times, with batch size equal to 64, for the training of variational MI estimators.
Figure 2 shows that modern Hopﬁeld networks reduce the variance of the model.
For models trained on data with mutual information of 10 we observe an average variance of approx.
67 for a model without Hopﬁeld and an average variance of approx.
33 for a model with Hopﬁeld.
For models trained on data with mutual information of 14 we observe an average variance of approx.
for a model without Hopﬁeld and an average variance of approx. 48 for a model with Hopﬁeld. In Figure A1 we show the performance of our method InfoLOOB with and without Hopﬁeld at estimating mutual information as well as InfoNCE.
As expected estimates of InfoNCE have estimates that saturate at log(batch size).
InfoLOOB without Hopﬁeld exhibits good estimates of high mutual information while InfoLOOB with Hopﬁeld accomplishes both - good estimates of high mutual information with a decreased variance.
A. E XPERIMENTS A.3. A BLATION STUDIES
As mentioned in the main paper, CLOOB has two new main components compared to CLIP: (1) the InfoLOOB objective instead of the InfoNCE objective and (2) the modern Hopﬁeld networks.
To assess which of the new main components of CLOOB have led to the performance increase over CLIP, we performed ablation studies on the CC dataset.
The results are reported in Table A1.
First, we enhanced CLIP by replacing the InfoNCE objective with InfoLOOB (see column CLIP InfoLOOB).
Next, we added modern Hopﬁeld networks to the CLIP architecture and used retrieved embeddings instead of the original embeddings, while keeping the InfoNCE objective (see column Hopﬁeld
InfoNCE).
Finally, we add modern Hopﬁeld networks to CLIP and replace the InfoNCE objective [Figure] CLIP [Table]
[Table] with InfoLOOB (see column Hopﬁeld InfoLOOB). As shown in Table A1 the InfoLOOB objective increases the performance of CLIP in the majority of the datasets.
We attribute this increase to the fact that InfoLOOB suffers less than InfoNCE from the “explaining away” problem. However, InfoLOOB is even more effective for higher mutual information, that is, a richer covariance structure.
Hopﬁeld networks amplify the covariance structure in their retrieved embeddings.
Though, this ampliﬁed covariance structure is disadvantageous for InfoNCE, as the saturation effect is stronger. The stronger saturation effect is caused by a richer covariance structure through Hopﬁeld networks, which in turn leads to higher similarity between anchor and positive. Therefore, we see a performance drop when combining modern Hopﬁeld networks with InfoNCE. Concluding, modern Hopﬁeld networks are a perfect match for InfoLOOB as they yield higher mutual information.
Therefore, CLOOB strongly improves the performance on 7 out of 8 zero-shot transfer learning tasks compared to CLIP.
For CLIP with InfoNCE, the hyperparameter τ − 1 is a learnable parameter.
For the other experiments, we use a ﬁxed τ − 1 of 30 .
The value for τ − 1 was determined via hyperparameter search (see
Section A.3.2).
In contrast to CLIP, we use a learning rate scheduler with restarts (Loshchilov & Hutter, 2017) to be more ﬂexible regarding the number of total training epochs and enable training up to a plateau.
To investigate the inﬂuence of the learning rate scheduler, we performed experiments with and without restarts.
Table A2 shows the zero-shot performance for the different downstream tasks for CLIP and
CLOOB respectively.
For both CLIP and CLOOB, the performance at the majority of the tasks either increases or remains roughly the same with restarts.
[Table] [Table]
A.3. H YPERPARAMETERS
The hyperparameter search was done on a validation split of CC with about 15,000 samples.
For the hyperparameter τ − 1 several values were considered ( 14 . 3 , 30 , 50 , 70 ), where 30 leads to the best results for both YFCC and CC.
Analogously to CLIP, we use the Adam optimizer (Kingma et al., 2014) with decoupled weight decay regularization (Loshchilov & Hutter, 2019).
The weight decay is only applied to weights that are not gains or biases.
As proposed in OpenCLIP (Ilharco et al., 2021) weight decay was set to 0 .
Different choices of weight decay ( 0 . 2 or 0 . 05 ), did not lead to a relevant performance change.
We use the same learning rate of 1 × 10 − 3 for CC and 5 × 10 − 4 for YFCC as used in OpenCLIP.
For the hyperparameter β we considered values in the range of 5 to 20 .
A value of 8 resulted in the best performance for CC and 14 . 3 for YFCC.
The batch size for CC was reduced to 512 due to computational restraints which did not result in performance losses.
The batch size for
YFCC was kept at 1024 as reported by OpenCLIP since a reduction resulted in a signiﬁcant drop in performance.
The learning rate scheduler for all experiments is cosine annealing with warmup and hard restarts (Loshchilov & Hutter, 2017) with a cycle length of 7 epochs.
For models trained on
YFCC the warmup was set to 10000 steps and for models trained on CC to 20000 steps.
A.3. D ATASETS
For pretraining we consider two datasets, Conceptual Captions (CC) (Sharma et al., 2018) and
YFCC100M (Thomee et al., 2016).
The CC dataset consists of 2.9 million images and corresponding high-quality captions.
Images and their corresponding notations for CC have been gathered via an automated process from the web and therefore represent a wide variety of styles.
Raw descriptions of images are collected from the alt-text HTML attribute.
Both images and texts are ﬁltered such that only image-text pairs above a certain quality threshold are part of this dataset.
The dataset we refer to as YFCC is a subset of the Yahoo Flickr Creative Commons 100 Million (YFCC100M) dataset.
It was created by ﬁltering for images which contain natural language descriptions and/or titles in English resulting in 15 million image-caption pairs.
The textual descriptions contain less useful information than CC because they are not ﬁltered by quality.
Occasionally they also contain metadata like camera settings or web addresses.
We evaluate and compare our method on several downstream classiﬁcation tasks.
We evaluate on the same set of datasets as CLIP reported for a model trained on YFCC.
This set contains
Birdsnap (Berg et al., 2014), Country211 (Radford et al., 2021), Flowers102 (Nilsback & Zisserman, 2008), GTSRB (Stallkamp et al., 2011), UCF101 (Soomro et al., 2012), Stanford Cars (Krause et al., 2013) and ImageNet (Deng et al., 2009).
Additionally, we include ImageNet V2 in our analysis (Recht et al., 2019).
Table A3 shows an overview of training and test set sizes, number of classes and the applied evaluation metric.
In the case of several test sets per dataset the metric is calculated for every set individually and the average performance is reported.
The set size in Table A3 corresponds to the total number of samples across all test and training sets of a dataset respectively.
Birdsnap contains images of North American bird species, however our dataset is smaller than reported in CLIP as some samples are no longer available.
The Country211 dataset was published in CLIP and is a small subset of the YFCC100m dataset.
It consists of photos that can be assigned to 211 countries via GPS coordinates.
For each country 200 photos are sampled for the training set and 100 for testing.
For the Flowers102 images of 102 ﬂower categories commonly occuring in the
United Kingdom were collected.
Several classes are very similar and there is a large variation in scale, pose and lighting.
The German Trafﬁc Sign Recognition Benchmark ( GTSRB ) was a challenge held at the IJCNN 2011.
The dataset contains images of german trafﬁc signs from more than 40 classes.
Note that two versions of this dataset exist, one used for the challenge and an ofﬁcial dataset released after the competition.
For CLIP the linear probing classiﬁers were trained using the competition training set but tested on the ofﬁcial test set.
Stanford Cars contains images of 196 car models at the level of make, model and year (e.g. Tesla Model S Sedan 2012).
UCF101 (Soomro et al., 2012) is a video dataset with short clips for action recognition consisting of three training sets and three test sets.
We follow the procedure reported in CLIP and extract the middle frame of every video to assemble the dataset.
The ImageNet Large Scale Visual Recognition Challenge was held fromthrough 2017 and is one of the most widely used benchmarks for object detection and localization.
Several years later ImageNet V2 assembled three new test sets with images from the same 1, classes to test for generalization of models optimized for the original ImageNet benchmark.
Every test set comprises 10,000 samples.
A.3. Z ERO - SHOT EVALUATION
Class names for all downstream tasks were adopted from CLIP, that is, among other changes special characters like hyphens or apostrophes were removed.
Furthermore, some class names of the datasets were slightly changed (e.g. “ kite ” to “ kite (bird of prey) ” in ImageNet).
For zero-shot evaluation, we use the same prompt templates as published in CLIP.
Depending on the dataset the number of prompts can vary from one prompt (e.g. “ a photo of a {label}, a type of bird. ”
for Birdsnap) up to 80 prompts for ImageNet covering various settings (e.g. “ a cropped photo of a {label}. ”, “ a origami {label}. ”).
In case of several prompts an average embedding over all prompt embeddings is calculated.
Figure A2 shows the zero-shot results for all evaluation tasks with the ResNet-50x4 model reported in Table 4.
A.3. L INEAR PROBING
We try to follow the evaluation procedure in Radford et al. (2021) as closely as possible.
We note one difference with respect to the implementation: Instead of scikit-learn’s logistic regression using the L-BFGS solver, we use cuML’s logistic regression classiﬁer with L-BFGS algorithm to utilize GPUs for efﬁciency.
All hyperparameters are the same as described in Radford et al. (2021), the maximum number of iterations was set to 1000, and the L2 regularization strength λ was determined by using a parametric binary search.
We tried to reproduce the CLIP results with the correspondingly published models, however, failed to produce the exact numbers.
This could be due to several factors:
• The train and validation split.
Same as in Radford et al. (2021) , we use the provided validation set to perform the hyperparameter search.
When there is none provided, we use a random half of the training dataset for validation.
[Table]
• In case of a tie in the validation score
[Table]
we use the maximal λ for the strongest regularization.
We note though that we came closer to reproducing the results published in CLIP when using the mean λ over all ties when these exist.
• For the Birdsnap dataset, the resources that we have got online at the time of this writing could be different from the resources that CLIP’s authors obtained at the time.
Linear probing evaluation of YFCC-pretrained models is shown in Table A4.
Comparing our reimplementation of CLIP and CLOOB with ResNet-50 encoders, we observe mixed results.
The reason for this effect might be attributed to the observed task-dependence of multimodal models (Devillers et al., 2021).
Another potential reason is that the beneﬁt of the restrictions to more reliable patterns that occur in both modalities does not directly translate to an evaluation of just the encoding part of one modality.
Again, as expected in self-supervised training, increasing the capacity of the
CLOOB models beneﬁts accuracy.
A. R EVIEW OF M ODERN H OPFIELD N ETWORKS
We brieﬂy review continuous modern Hopﬁeld networks that are used for deep learning architectures.
They are continuous and differentiable, therefore they a work with gradient descent in deep architectures.
They retrieve with one update only, therefore they can be activated like other deep learning layers. They have exponential storage capacity, therefore they can tackle large problems.
Hopﬁeld networks are energy-based, binary associative memories, which popularized artiﬁcial neural networks in the 1980s (Hopﬁeld, 1982; 1984). Associative memory networks have been designed to store and retrieve samples. Their storage capacity can be considerably increased by polynomial terms in the energy function (Chen et al., 1986; Psaltis & Cheol, 1986; Baldi & Venkatesh, 1987; Gardner, 1987;
Abbott & Arian, 1987; Horn & Usher, 1988; Caputo & Niemann, 2002; Krotov & Hopﬁeld, 2016).
In contrast to these binary memory networks, we use continuous associative memory networks with very high storage capacity. These modern Hopﬁeld networks for deep learning architectures have an energy function with continuous states and can retrieve samples with only one update (Ramsauer et al., 2021; 2020). Modern Hopﬁeld Networks have been successfully applied to immune repertoire classiﬁcation (Widrich et al., 2020) and chemical reaction prediction (Seidl et al., 2021).
We assume a set of patterns { u 1 , . . .
, u N } ⊂ R d that are stacked as columns to the matrix U =
( u 1 , . . .
, u N ) and a state pattern (query) ξ ∈ R d that represents the current state.
The largest norm of a stored pattern is M = max i (cid:107) u i (cid:107) .
Continuous modern Hopﬁeld networks with state ξ have the energy [Equation] A
[Equation]
For energy E and state ξ , the update rule
[Equation]
[Figure]
DodgeCaliberWagon
[Figure]
[Figure]
ChryslerAspenSUV
[Figure]
[Figure]
redandwhitecirclered truckandblackcarnopassing correctrank:36/ magnolia correctrank:1/
[Figure]
[Figure]
GreatBlueHeron [Figure] [Figure] has been proven to converge globally to stationary points of the energy E , which are almost always local minima (Ramsauer et al., 2021).
The update rule Eq.
(A171) is also the formula of the wellknown transformer attention mechanism (Ramsauer et al., 2021), therefore Hopﬁeld retrieval and transformer attention coincide.
The separation ∆ i of a pattern u i is deﬁned as its minimal dot product difference to any of the other patterns:
∆ i = min j,j (cid:54) = i (cid:0)
u Ti u i − u Ti u j (cid:1) .
A pattern is well-separated from the data if ∆ i ≥ 2 βN + 1 β log (cid:0) 2( N − 1) NβM 2 (cid:1) .
If the patterns u i are well separated, the iterate Eq.
(A171) converges to a ﬁxed point close to a stored pattern.
If some patterns are similar to one another and, therefore, not well separated, the update rule Eq.
(A171) converges to a ﬁxed point close to the mean of the similar patterns.
This ﬁxed point is a metastable state of the energy function and averages over similar patterns.
The next theorem states that the update rule Eq.
(A171) typically converges after one update if the patterns are well separated.
Furthermore, it states that the retrieval error is exponentially small in the separation ∆ i .
Theorem A4 (Modern Hopﬁeld Networks: Retrieval with One Update) .
With query ξ , after one update the distance of the new point f ( ξ ) to the ﬁxed point u ∗ i is exponentially small in the separation ∆ i .
The precise bounds using the Jacobian J = ∂f ( ξ ) ∂ ξ and its value J m in the mean value theorem are: [Equation] A172)
[Equation]
For given (cid:15) and sufﬁcient large ∆ i , we have (cid:107) f ( ξ ) − u ∗
i (cid:107) < (cid:15) , that is, retrieval with one update.
The retrieval error (cid:107) f ( ξ ) − u i (cid:107) of pattern u i is bounded by
[Equation]
For a proof see (Ramsauer et al., 2021).
The main requirement of modern Hopﬁeld networks to be suited for contrastive learning is that they can store and retrieve enough embeddings if the batch size is large.
We want to store a potentially large set of embeddings.
We ﬁrst deﬁne what we mean by storing and retrieving patterns from a modern Hopﬁeld network.
Deﬁnition A1 (Pattern Stored and Retrieved) .
We assume that around every pattern u i a sphere
S i is given.
We say u i is stored if there is a single ﬁxed point u ∗ i ∈ S
i to which all points ξ ∈ S i converge, and S i ∩ S j = ∅ for i (cid:54)
=
j .
We say u i is retrieved for a given (cid:15) if iteration (update rule) Eq.
(A171) gives a point ˜
x i that is at least (cid:15) -close to the single ﬁxed point u ∗ i ∈ S i .
The retrieval error is (cid:107) ˜
x
i − u
i (cid:107) .
As with classical Hopﬁeld networks, we consider patterns on the sphere, i.e. patterns with a ﬁxed norm.
For randomly chosen patterns, the number of patterns that can be stored is exponential in the dimension d of the space of the patterns ( u i ∈ R d ).
Theorem A5 (Modern Hopﬁeld Networks: Exponential Storage Capacity) .
We assume a failure probability 0 < p (cid:54) 1 and randomly chosen patterns on the sphere with radius M :=
K √ d − 1 .
We deﬁne a := 2 d − 1 (1 + ln(2 βK 2 p ( d − 1))) , b := 2 K 2 β 5 , and c :=
b W 0 (exp( a +ln( b )) , where W 0 is the upper branch of the Lambert W function (Olver et al., 2010, (4.13)), and ensure c ≥ (cid:16) 2 √ p (cid:17) 4 d − 1 .
Then with probability 1 − p , the number of random patterns that can be stored is
[Equation]
[Equation]
Therefore it is proven for c ≥ 3 .
1546 with β = 1 , K = 3 , d = 20 and p = 0 .
001 ( a + ln( b ) > 1 . 27 ) and proven for c ≥ 1 . 3718 with β = 1 , K = 1 , d = 75 , and p = 0 .
001 ( a + ln( b ) < − 0 . 94 ).
For a proof see (Ramsauer et al., 2021).
This theorem justiﬁes to use continuous modern Hopﬁeld networks for using retrieved embeddings instead of the original embeddings for large batch sizes.
Even for hundreds of thousands of embeddings, the continuous modern Hopﬁeld network is able to retrieve the embeddings if the dimension of the embeddings is large enough.
A. F URTHER R ELATED W ORK
Multiple works have proposed improvements to InfoNCE.
Joint Contrastive Learning (JCL) studies the effect of sampling multiple positives for each anchor.
(Cai et al., 2020).
Sampling negatives around each positive leads to higher bias but lower variance than InfoNCE (Wu et al., 2021).
InfoNCE has been generalized to C-InfoNCE and WeaC-InfoNCE, which are conditional contrastive learning approaches to remove undesirable information in self-supervised representations (Tsai et al., 2021).
ProtoNCE is a generalized version of the InfoNCE, which pushes representations to be closer to their assigned prototypes (Li et al., 2021).
ProtoNCE combines contrastive learning with clustering.
SimCSE employs InfoNCE for contrastive learning to learn sentence embeddings (Gao et al., 2021).
InfoNCE has been extended to video representation learning (Han et al., 2020).
Many follow up works have been based on the CLIP model.
The CLIP model is used in Vision-andLanguage tasks (Shen et al., 2021).
The CLIP model guided generative models via an additional training objective (Bau et al., 2021;
Galatolo et al., 2021; Frans et al., 2021) and improved clustering of latent representations (Pakhomov et al., 2021).
It is used in studies of out of distribution performance (Devillers et al., 2021; Milbich et al., 2021; Miller et al., 2021), of ﬁne-tuning robustness (Wortsman et al., 2021), of zero-shot prompts (Zhou et al., 2021) and of adversarial attacks to uncurated datasets (Carlini & Terzis, 2021).
It stirred discussions about more holistic evaluation schemes in computer vision (Agarwal et al., 2021).
Multiple methods utilize the CLIP model in a straightforward way to perform text-to-video retrieval (Fang et al., 2021; Luo et al., 2021; Narasimhan et al., 2021).