Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning
Abstract
We present modality gap , an intriguing geometric phenomenon of the representation space of multi-modal models.
Specifically, we show that different data modalities (e.g. images and text) are embedded at arm’s length in their shared representation in multi-modal models such as CLIP.
Our systematic analysis demonstrates that this gap is caused by a combination of model initialization and contrastive learning optimization.
In model initialization, we show empirically and theoretically that the representation of a common deep neural network is restricted to a narrow cone.
As a consequence, in a multi-modal model with two encoders, the representations of the two modalities are clearly apart when the model is initialized.
During optimization, contrastive learning keeps the different modalities separated by a certain distance, which is influenced by the temperature parameter in the loss function.
Our experiments further demonstrate that varying the modality gap distance has a significant impact in improving the model’s downstream zeroshot classification performance and fairness.
Our code and data are available at https://modalitygap.readthedocs.io/
Introduction
Multi-modal models map inputs from different data modalities (e g. image and text) into a shared representation space (Figure 1 (a)).
It has garnered tremendous interest and excitement as a framework for data integration.
As a prominent example pre-trained on a web-scale collection of images and natural language, OpenAI’s CLIP model [39], has learned diverse visual concepts that can readily be transferred to downstream tasks through prompting : one can perform “zero-shot” visual classification by simply providing the names of the visual categories to be recognized.
In this work we present the modality gap phenomenon: As shown in Figure 1 (b), CLIP’s image embeddings and text embeddings are located in two completely separate regions of the embedding space.
We find this phenomenon consistently across various multi-modal models, covering texts, natural images [39], videos [50], medical images [53], and amino-acid sequences [11].
Interestingly, this phenomenon still holds even when we embed using multi-modal models with random weights (Figure 1 (c)).
While it might seem reasonable to attribute the gap to differences in data distributions or to the different encoder architectures, we showed that these factors are not the fundamental cause.
This paper provides a three-part explanation for the modality gap phenomenon.
The general inductive bias of deep neural architecture creates a cone effect : The effective embedding space is
<latexit sha1_base64="+HsAC4+xA4QcTab1GYb/sp59apo=">AAADjHicrVJdaxNBFJ3sVm1TbVN99OViKKSo6U4oWpFCURAfqlQwbSETltnJbDLp7Aczs2oY5s/4k3zz3zj5oI2pfSleGDhzzr3ccy83KaXQJop+14Jw7d79B+sb9c2Hj7a2GzuPz3RRKca7rJCFukio5lLkvGuEkfyiVJxmieTnyeX7qX7+jSstivyrmZS8n9FhLlLBqPFUvFP7STJqRoxKe+Jie/X55GIMRInhyFCliu+wrHQcHMFLkirKLHb2swOiqywWQGQxhDlP+I+ypWMrXggH+0AMrfacneWN4VocX4tASP1WM51bzeD/ZWa8LK6Y8S2uOnRc6247ew53Gm8vbjSjdjQLuAnwAjTRIk7jxi8yKFiV8dwwSbXu4ag0fUuVEUxyVyeV5iVll3TIex7mNOO6b2fH5GDXMwNIC+VfbmDGLldYmmk9yRKfObWpV7Up+S+tV5n0sG9FXlaG52zeKK0kmAKmlwkDoTgzcuIBZUp4r8BG1C/d+Put+yXg1ZFvgrNOG79q4y8HzeN3i3Wso6foGWohjF6jY/QRnaIuYsFGsB8cBm/CrfAgfBsezVOD2qLmCforwg9/ANIbJJk=</latexit> L M 1 !
M 2 =  1 N
X i log exp( s i,i / ⌧ )
P j exp( s i,j / ⌧ )
L M 2 !
M 1 =  1 N X i log exp( s i,i / ⌧ )
P j exp( s j,i / ⌧ )
L = 12( L M 1 !
M 2 + L M 2 !
M 1 )
[Table]
UMAP
U M AP
UMAP
U M AP
[Table]
[Figure]
restricted to a narrow cone for pre-trained models or models with random weights.
(2) Different random initializations create different embedding cones.
Since a multi-modal model consists of two encoders, which create different cones at random initialization, this explains how the modality gap is present at initialization.
The contrastive learning objective commonly used by multi-modal models preserves the gap.
We support our explanations with theory and experiments.
Our theoretical analysis shows that under mild assumptions, each neural network layer shrinks the angle between any pair of embedding vectors with high probability, thereby creating more narrow cones in deeper architectures.
We further prove that different random initializations of model weights result in different cones.
Interestingly, increasing the modality gap in models like CLIP can improve its downstream performance on several zero-shot learning and fairness tasks.
The main objective of our paper is to i) empirically demonstrate the modality gap phenomenon across different data modalities and NN architectures; ii) explain how the gap arises and iii) show that the size of the gap can affect downstream applications.
It is not our goal to propose a method to close the gap, since it’s not clear that it’s desirable to have no modality gap.
Together, this paper makes the following contributions :
To the best of our knowledge, we demonstrate a general modality gap phenomenon for the first time.
We show that this phenomenon holds across a wide spectrum of multi-modal models, covering texts, natural images, videos, medical images, and amino-acid sequences.
We demonstrate the significant implications of modifying the gap in downstream applications.
By simply modifying the gap’s distance, we can improve CLIP’s zero-shot performance and fairness.
To explain modality gap, we provide a three-part explanation supported by extensive theoretical and empirical analyses.
Our analyses also provide new insights on the cone effect, which we show is a general phenomenon for deep neural networks.
Existing work focuses on trained language models and attributes the cone effect to the optimization under unbalanced word frequencies distribution.
We demonstrate that this effect holds not only across various modalities and network architectures, but also on random noise inputs and random weights, which is not captured in previous work.
[Figure]
RandomInput
[Figure]
We mathematically characterize the contraction mapping induced by linear layers with ReLU non-linearities to explain the cone effect.
Our theory matches well with experiments and provides insights for understanding the general inductive biases of deep neural networks.
The Cone Effect Induces A Modality Gap 2.1 The Narrow Cone of Embeddings
In order for modality gap to exist, the embeddings from a encoder should be concentrated around a subregion of the full embedding space—otherwise, the embeddings from different encoders would overlap.
Motivated by this, we begin our investigation by showing that the modality gap already arises at random model initialization due to the cone effect : The effective embedding space is restricted to a narrow cone for trained models and models with random weights.
To demonstrate this, we extract 5,000 embeddings from the final layer of 3 pre-trained models respectively (ResNet, Vision
Transformer, Text Transformer) 2 on MSCOCO Caption [8].
We then compute the cosine similarity between all possible pairs of the 5,000 embeddings within each model (Figure 2 (a)).
We found that both the average cosine similarity ( 0 . 56 , 0 . 47 , 0 .
51 respectively for the 3 models) and the minimum cosine similarity ( 0 . 23 , 0 .
05 , 0 . 01 ) are positive.
These results indicate that the embedding space is a narrow cone.
In the literature, the cone effect has been observed in the language representations from language models (e.g., BERT) [12].
A common explanation is that the unbalanced distribution of word frequencies biased the optimization [15, 33].
However, we found that the cone effect still exists in models with random weights (Figure 2 (c)).
In fact, the average cosine similarity there is even higher than in trained models.
For example, any two embeddings from a randomly initialized ResNet have on average an almost perfect ( 0 . 99 ) cosine similarity.
Interestingly, the cone effect still holds when the input data is random noise 3 , indicating that unbalanced data distribution suggested in previous works is not necessary for the cone effect.
Together these experiments suggest that the cone effect reflects a more general inductive bias of deep networks than might be previously appreciated.
How narrow is the cone in 512-dim representation space?
We clarify that a cosine similarity with 0 .
56 already indicates that the embedding space is actually an extremely narrow cone in the 512-dimensional feature space.
Consider the fraction of surface area in a unit hypersphere: In 2D, arccos(0.56)=55.94°, indicating that a cosine similarity of 0.56 can “occupy” 55.94°/
°=15.53% of the 2D unit circle.
In 3D, a cosine similarity of 0.56 can “occupy” 2 πr 2 (1 − cos 55 . 94 ° 2 ) 4 πr=3.34% of the 3D unit sphere.
In 512D, a cosine similarity of 0.56 can “occupy” less than 12 512 fraction of the surface area in a unit 512D hypersphere.
These evidences show that the effective embedding space is restricted to an extremely narrow cone.
2.2 The effects of non-linear activation on cone effect
Design To study the effects of non-linear activation functions on the cone effect, we randomly initialized various MLPs with different non-linearities or without non-linearities.
The inputs of the
MLPs are 512-dim standard normal random vectors.
All MLP linear layers are 512 × 512 , with both weight and bias randomly initialized from N (0 , 1512 ) , here we denote a Gaussian distribution with mean µ and variance σ 2 by N ( µ, σ 2 ) .
Results As shown in Figure 2 (b), MLPs without non-linear activation shows little cone effect.
However, with non-linearity, the average cosine similarity increases rapidly as the number of layers increases.
For example, the average cosine similarity reaches 0 .
99 for a 2-layer MLP with Sigmoid.
These results indicate that the non-linear activation functions play a crucial role in the cone effect.
Although it is easy to see that ReLU makes every coordinate non-negative, and thus cosine similarity after ReLU is guaranteed to be non-negative, we highlight that none of the 3 models in Figure(a) has ReLU as the final layer before embedding extraction 4 .
In addition, although all 3 models incorporate normalization layers such as batch norm [23] and layer norm [4] in their architectures, we still observe the cone effect.
Further analyzing the connection between normalization and the cone effect is an interesting direction of future work.
2.3 Different random initializations create different cones
Next, we study the effect of different random initialization on the cone effect.
In Figure 2 (c), we randomly initialized a model 25 times, and plotted its extracted embeddings on the same real data (i.e.,
MSCOCO Caption) via UMAP visualization [41].
We found that each random initialization forms a distinctively different cone.
This phenomenon holds across various neural network architectures and input modalities (ResNet, Vision Transformer or Text Transformer), on ImageNet-pretrained models (Supp.
Figure 13), on PCA visualization (Supp.
Figure 7), or with random noise inputs (Supp.
Figure 5).
Since a multi-modal model consists of two encoders, which creates different cones at random initialization, this explains how the modality gap is present at initialization.
While it might seem reasonable to attribute the modality gap to differences in data modalities [21], Figure(c) shows the gap still exists even if the two encoders operate on the exact same data in the exact same modality.
Therefore, the gap can exist without different modalities, and we emphasize that the modality gap phenomenon is non-trivial to understand
Theoretical analysis
Here, we theoretically investigate the cone effect phenomenon.
We show that (i) the cosine similarity increases as the layer gets deeper and (ii) the variance of an intermediate output mostly come from the model’s random initialization.
We first define some notations.
We denote the ReLU activation by ϕ ( x ) :=
max( x, 0) for x ∈ R , and we extend it by considering element-wise operation ϕ ( x ) :
=
( ϕ ( x 1 ) , . . .
, ϕ ( x k ))
T =
(max( x 1 , 0) , . . .
, max( x k , 0)) T for a multivariate input x = ( x 1 , . . .
, x k ) T ∈ R k and k ∈ N .
The cosine similarity between two vectors u, v ∈ R k is defined as cos( u, v ) :
= u T v ∥ u ∥∥ v ∥ where ∥ u ∥ =
( u T u ) 1 / 2 .
Lastly, we set [ k ] := { 1 , . . .
, k } for k ∈ N .
Each network layer increases cosine similarity.
We study how the cosine similarity between two intermediate layer outputs changes when weight and bias terms in an MLP are fixed.
The following theorem shows that with a high probability cosine similarity increases after one feedforward computation when the number of nodes in the output layer is large.
Theorem 1 (Monotonicity of cosine similarity) .
Suppose u, v ∈ R d in are any two fixed vectors such that ∥ u ∥
=
r
∥ v ∥ for some r > 0 , W ∈ R d out × d in is a random weight matrix where each element W k,l ∼ N (0 , d − 1 out ) for k ∈ [ d out ] , l ∈ [ d in ] , and b ∈ R d out is a random bias vector such that b k ∼ N (0 , d − 1 out ) for k ∈ [ d out ] .
If cos( u, v ) < (cid:16) (cid:0) r + 1 r (cid:1)(cid:17) − 1 , then the following holds with probability at least 1 − O (1 /d out ) .
cos( ϕ ( W u + b ) , ϕ ( W v + b )) > cos( u, v ) .
Theorem 1 shows that the cosine similarity between two vectors increases with a high probability after one feedforward computation consisting of a linear transformation and ReLU computation.
This matches well with the result in Figure 2 (b) where the cosine similarity between samples increases as the intermediate layer gets farther from the input.
The bound condition on cos( u, v ) in Theorem 1 asks that the two vectors before the layer computation are not too close to each other in terms of the direction.
This is because the random bias addition can slightly change the angle between the two vectors, leading to a small decrease in cosine similarity when the previous layer’s cosine similarity is too high.
This condition is plausible in practice because the ℓ 2 -norm of intermediate layer outputs is close to one with a high probability when the ℓ 2 -norm of input data is one [1, Lemma 7.1].
Given that the norm ratio r is close to one, the upper bound condition for cos( u, v ) is likely to hold because ( 12 ( r + 1 r ))
− 1 is close to 1.
Effect of random initialization We now examine the variance of an intermediate output and explain that the variance is mainly due to random initializations as in Figure 2 (c).
To be more specific, we denote an intermediate layer output by h Θ ( U )
∈ R for some input datum U .
Here, Θ denotes all the random weights and biases that are used in h Θ ( U ) .
The variance of h Θ ( U ) can be decomposed as
[Equation]
Due to the randomness of data
[Equation]
Due to random initializations
Here, the inner and outer expectations are over the data U and the random weights Θ , respectively.
The first term on the right hand side explains the within variance after fixing one random initialization, quantifying the randomness of data.
In contrast, the second term explains the variance due to different random initializations.
The following theorem considers the ratio of the second term to the total variance and shows that the ratio can be very close to one when a deep neural network model is used.
Theorem 2 (Informal; Variance due to different random initializations) .
Let h Θ ( U ) be an intermediate layer output with an input data U with ∥ U ∥ = 1 .
Under mild assumptions on Θ , the set of all the random weights and biases, the following inequality holds.
[Equation]
where β is a constant that captures the average cosine similarity of previous layer outputs.
Theorem 2 shows that the ratio of the variance due to different random initializations to the total variance is bounded below by the average cosine similarity of previous layer outputs.
As Figure 2 (b) illustrated, the average cosine similarity of an intermediate layer output often approaches to one as the layer gets deeper.
Accordingly, the lower bound β , which captures the average cosine similarity, is close to one when a neural network is deep enough.
In Appendix D we elaborate on the relationship between β and the cosine similarity, and provide a detailed statement of the Theorem.
[Figure]
Contrastive learning preserves modality gap 4.1 Background: Contrastive Loss
Given that the modality gap is present at initialization, we investigate why our optimization procedure fails to close the gap.
We begin by reviewing contrastive learning, which is a commonly used training strategy for multi-modal models [53, 50, 34].
We illustrate with CLIP due to its wide usage.
Given a batch of N (image, text) pairs, CLIP learns to predict which of the N × N possible (image, text) pairs are aligned.
In other words, CLIP learns to maximize the cosine similarity of the image and text embeddings of the N real pairs in the batch while minimizing the cosine similarity of the embeddings of the N 2 − N incorrect pairs.
Formally, the optimization objective is the average of two losses: one for image-to-text classification:
[Equation]
and the other for text-to-image classification:
[Equation]
Here, x i and y j are the L2-normalized embedding of image in the i -th pair and that of text in the j -th pair, respectively.
τ is a learned temperature parameter to scale the logits.
The final learned temperature is τ = 1100 in CLIP.
See additional illustration in Figure 1(a) and Supp.
Figure 12.
4.2 Embedding Shift Experiment
Design We hypothesize that the contrastive learning objective encourages the existence of the modality gap.
To testify this hypothesis, we design a loss landscape probing experiment on n = 5 ,image-caption pairs 5 from the validation set of MSCOCO Caption dataset.
We first define the modality gap as the difference between the center of image embeddings and text embeddings:
[Equation]
where x i and y i are the L2-normalized image embedding and text embedding.
We then manually shift every text embedding and image embedding towards closing the modality gap (Figure 3 (a)).
After shifting, we re-normalize each embedding to the unit hypersphere:
[Equation]
We vary the scalar λ to produce different amounts of shifts.
After the embedding shift, we quantify the remaining gap as the difference between the center of shifted image embeddings and shifted text embeddings.
The gap distance before shifting is ∥ ⃗ ∆ gap ∥ = 0 .
Here Euclidean distance is a intuitive metric because in CLIP, the image embeddings and text embeddings are L2-normalized (Supplementary Figure 12).
In other words, the image and text embeddings of CLIP are always on the unit sphere.
Specifically, for any n -dimensional vectors x and y , the cosine similarity is given as cos( x, y )
= x T y , and the Euclidean distance is given as ( x − y ) T ( x − y )
=
2(1 − x T y ) .
Therefore, they have a functional relationship as Euclideandistance( x, y )
=
2(1 − cos( x, y )) .
When the angle between x and y is less than π/ 2 , which is the case as embeddings are in a narrow cone, the small
Euclidean distance directly means a high cosine similarity.
Results Figure 3(b) shows the contrastive loss landscape on different amount of modality gap under temperature τ
= 1100 (i.e., CLIP’s learned final temperature).
We found that the default gap distance ∥ ⃗ ∆ gap ∥ = 0 .
82 actually achieves the global minimum, and shifting toward closing the gap increases the contrastive loss.
Interestingly, there is a local minimum when we shift the text embeddings to the opposite side in a “back-to-back position.”
Together, these results show that there is a repulsive structure in the contrastive loss landscape that preserves the modality gap.
However, when the temperature increases (Figure 3(c,d)), the repulsive structure and the local minimum gradually disappear, and closing the gap becomes more optimal.
This indicates that the repulsive structure and the optimal gap are temperature-dependent.
Additional Evidence from Fine-tuning To further investigate the impact of temperature on modality gap, we fine-tune CLIP under 6 different temperatures τ ∈ { 1100 , 150 , 130 , 120 , 110 , 1 } respectively, on MSCOCO Caption training set with batch size 64.
We found that a high temperature ( τ ∈ { 110 , 1 } ) in fine-tuning significantly reduces or closes the gap, while a low temperature does not.
The gap distance ∥ ⃗ ∆ gap ∥ decreases monotonically with increasing temperature (Supp.
Figure 8).
4.3 Simulating mismatched data
Design We designed a simple simulation to distill the empirical phenomena in the embedding shift experiment.
We consider six simulated image-text embedding pairs on a 3D unit sphere (Figure(e)), with two mismatched image-text pairs ( I 0 , T 0 ) , ( I 1 , T 1 ) .
Here "mismatched" means correct pairs are ( I 0 , T 0 ) and ( I 1 , T 1 ) but I 0 is closer to T 1 and I 1 is closer to T 0 .
We fix the image embeddings while shifting the text embeddings downwards to close the gap (i.e., modifying θ , see more details in
Appendix A).
Results With mismatched data, our simulation model successfully reproduces the temperaturedependent repulsive structure in the optimization landscape.
When we remove the mismatch, the repulsive structure disappears (Supp.
Figure 9).
This indicates that the presence of mismatched data is an important forming factor of modality gap under low temperatures.
Although the mismatch here is simulated, in practice mismatched data are common (e.g., hard-to-differentiate images/captions or annotation errors).
Investigating how and to what extent the multimodal data misalignment could
[Table] [Table]
affect the contrastive loss landscape and thereby the modality gap is an interesting direction for future research.
4.4 Initialization vs Optimization
Design So far, we have shown that (1) modality gap is born at random initialization, and (2) contrastive learning objective encourages the gap.
To explore how the final modality gap is affected by a combination of both factors, we train two CLIP models from scratch: one model uses random initialization, where the gap is large ∥ ⃗ ∆ gap ∥ = 1 .
1891 ± 0 .
because of the cone effect discuss in Sec. 2; another model amends the gap at the initialization by transforming text embeddings to be close to the image embeddings, where the gap is almost zero ∥ ⃗ ∆ gap ∥ = 0 .
0388 ± 0 . 0351 .
Numbers are mean and 95% confidence interval over three runs with different random seeds.
The transformation we applied is a common method to align multilingual word embeddings [31].
More specifically, given image embedding x and text embedding y , we apply an orthogonal matrix to text embedding y ′ =
W y and compute the multi-modal contrastive loss on x and y ′ .
The orthogonal matrix minimizes the distance between image embeddings and transformed text embeddings:
W = arg min W ∈
O D
∥ X − Y W ∥ where X, Y ∈ R N × D are image embeddings and text embeddings generated from N image-caption pairs, and O D is the set of D -dimensional orthogonal matrix.
Results We train both models on the MSCOCO Caption training set with batch size 64 and temperature τ = 1100 (i.e., CLIP’s learned temperature).
After training, the original model gap changes from 1 .
1891 ± 0 .
0017 to 1 .
2991 ± 0 .
while the amended model gap changes from 0 .
0388 ± 0 .
0351 to 0 .
7457 ± 0 . 0633 .
Numbers are 95% confidence interval over three runs with different random seeds.
We clearly observe the same domain gap phenomenon as shown in Figureusing PCA or UMAP.
This experiment shows that the final domain gap is caused by both initialization and optimization.
When we ablate the domain gap at the initialization, the loss will still encourage the gap, but the gap distance is only 57% compared to the model without amending the gap.
Modality Gap Implications 5.1 Zero-shot performance
Design One of the most interesting capabilities for CLIP is its strong zero-shot transferability to a variety of downstream tasks without any supervision.
We study whether changing the gap will affect CLIP (ViT-B/16)’s performances on various downstream tasks, including coarse-grained classification (CIFAR10 and CIFAR100), fine-grained classification (EuroSAT [22]), and optical character recognition (SVHN, HatefulMemes [28]).
Metric and prompt for each task are shown in Supp.
Table 3.
Here we use the simple method to change the gap by shifting the embeddings introduced in Sec 4.2.
The main objective of our paper is to understand the modality gap phenomenon, a general inductive bias that holds across various data modalities and NN architectures.
The goal of our paper is not to propose a method to close the gap and to improve downstream performance.
Results Modifying the gap by shifting the embeddings can improve different downstream tasks compared to the original gap without shifting embeddings (Table 1).
Details of performance vs gap distance curves are shown in Supp.
Figure 10.
We leave more methods to change the gap and more analysis of the relation between gap distance and downstream task performance to future work.
5.2 Fairness
Design We follow the bias evaluation setup in the CLIP paper to evaluate denigration harms [39,
Sec. 7.1].
We performed zero-shot evaluations on CLIP (ViT-B/32) on the evaluation set of the FairFace dataset [26], which has 10,954 images.
In addition to the 14 FairFace classes (e.g., ‘white male’, ‘black female’), we added 4 non-human classes (‘animal’, ‘gorilla’, ‘chimpanzee’ and ‘orangutan’) and 3 crime-related classes (‘thief’, ‘criminal’ and ‘suspicious person’).
The text prompts are attached in Appendix (Supp.
Figure 11).
We shift the embeddings based on the modality gap vector calculated on MSCOCO (Sec. 4.2).
We report the fraction FairFace images whose top-1 prediction is offensive.
Results We found that increasing the gap from 0 .
82 to 0 .
97 reduces denigration harms consistently for all races (Table 5).
Meanwhile, we only observe a minor 0 .
0008 top-1 accuracy drop (Appendix B.2).
It is encouraging that a simple gap offsetting approach can lead to a consistent bias reduction across all races on such a complex model (i.e., CLIP) 6 .
Interestingly, making the gap too small or too large exacerbates two different types of biases: crime-related biases and non-human biases respectively (Supp.
Table 4).
Related Work
Contrastive Representation Learning Contrastive representation learning learns an embedding space where similar objects are closer than dissimilar ones, and has achieved great success in vision [7, 20, 6, 9], language [40, 16], and graph [51, 38].
However, as contrastive learning is still an emerging representation learning technique, we still lack comprehensive theoretical and empirical understandings about why contrastive learning works.
[48] proposed two ideal objectives for contrastive representation space: alignment (similar samples have similar features) and uniformity (features are uniformly distributed on the hypersphere), and demonstrated these two objectives are highly correlated with downstream task performances.
[46] show that low temperatures increase the model’s penalty on hard negative examples, and thus increase uniformity and decrease tolerance (the closeness of semantically similar samples).
These analyses mostly focus on unsupervised contrastive learning on a single modality.
Orthogonal to their work, we show that multi-modal contrastive learning with low temperatures and mismatched data encourages the modality gap.
Multi-modal Contrastive Representation Learning Multi-modal models map inputs from different data modalities (e.g. image and text) into a shared representation space [53, 50, 34, 24, 11].
It has garnered tremendous interest and excitement as a framework for data integration.
These models are often pre-trained with contrastive loss [45], as [39] showed that the contrastive learning is 12 × more efficient than the generative approaches.
We demonstrate an intriguing geometric phenomenon of the representation space of these multi-modal models, and provide a three-part explanation supported by theory and experiments.
The idea of mapping images and text into a shared embedding space has been explored in earlier works [42, 49].
There have been recent efforts in formulating images and text embeddings as metric learning [14], multilabel classification [25], n-gram language learning [32], and captioning [10].
Recently there has there has also been work in using a unified encoder to fuse different data modalities [19].
Research into how the modality gap phenomenon generalizes to the multi-modal representations obtained by these alternative methods, or even uni-modal settings with teacher and student model [44, 5] would be a promising direction for future work.
Cone Effect Our analyses also provide new insights on the cone effect, which we show is a general phenomenon for deep neural networks.
Existing work focuses on the language representations of trained language models such as BERT and GPT-2 [12, 15, 33].
Given that isotropy has both theoretical and empirical benefits for static embeddings [35], the extent of anisotropy in contextualized representations is surprising [12].
It has been shown that the cone effect limits the expressiveness of the language representations.
Post-processing methods [33, 43, 2, 35] or modified training objective [15, 47, 16] alleviate the cone effect and improve downstream performance.
Existing work attributes the cone effect to the optimization under unbalanced word frequencies distribution [15, 33].
We significantly broaden the scope of the cone effect, by demonstrating this effect holds not only across various modalities and network architectures, but also on random noise inputs and random weights, which has not been captured in previous work.
We mathematically characterize the contraction mapping induced by linear layers with ReLU non-linearities to explain the cone effect.
Our theory matches well with experiments and provides insights for understanding the general inductive biases of deep neural networks.
Discussion
In this work, we investigated an interesting phenomenon in multi-modal contrastive learning — modality gap .
We analyzed why the gap exists, i.e., the joint effect of model initialization and optimization, and why studying the gap is important, i.e., it can affect the downstream task performance and fairness.
Our work raises several basic questions about representation learning, contrastive learning, and multi-modal contrastive representation learning.
For representation learning, prior research in NLP has shown that alleviating the cone effect improves downstream performance.
As our work significantly broadens the scope of the cone effect, methods for alleviating the cone effect in other modalities to improve ML performance is an interesting direction of future research.
For contrastive learning, our embedding shifting, simulation, and fine-tuning experiments all show that the contrast loss landscape is heavily influenced by temperature.
Recent work has found that temperature directly controls the uniformity and affinity of the uni-modal representation space [46].
Our study provides a complementary understanding of the multi-modal representation space.
Development of geometric methods for evaluation of representations [ 37, 30] to further capture the geometric landscape of the modality gap is an interesting direction of future work.
For multi-modal contrastive representational learning, we find that changing the modal gap can affect performance and fairness on downstream tasks.
Interestingly, having larger gap can help some fairness and zero-shot learning applications.
The main objective of our paper is to demonstrate the modality gap phenomenon and explain contraction mapping contribute to this.
Systematic analysis of the impact of the gap on applications is an important direction of future work.
Reproducibility Statement
We provide open-source implementation of our work at https://github.com/Weixin-Liang/
Modality-Gap .
The implementations will enable researchers to reproduce the modality gap described here as well as run their own analyses on additional cross-modal models.
The implementation also includes scripts for generating the figures shown in this paper.
A Contrastive learning preserves modality gap A.1 Simulating Mismatched Data
In Sec. 4.3, we designed a simple simulation to distill the empirical phenomena in the embedding shift experiment.
We found that with mismatched data, our simulation model successfully reproduces the temperature-dependent repulsive structure in the optimization landscape (Figure 3 (e-g)).
Here we present another simulation where we remove the mismatch (Supp.
Figure 9).
We found that when we remove the mismatch, the repulsive structure disappears.
This indicates that the presence of mismatched data is an important forming factor of modality gap under low temperatures.
For both Figure 3 (e-g) and Supp.
Figure 9, all embeddings are on the 3D unit sphere (i.e., r = 1 ).
The spacing between adjacent image-text pairs is ∆ ϕ = 15 ◦ .
All image vectors are fixed, and located on the equator (i.e., θ = 90 ◦ ).
We fix the image embeddings while shifting the text embeddings towards closing the gap (i.e., modifying θ ).
Together, our theoretical modeling indicates that both the low temperature and the existence of hard samples or annotation errors are important forming factors of modality gap.
B Modality Gap Implications B.1 Zero-shot Performance
In Sec. 5.1, we demonstrated that increasing the modality gap in CLIP can improve its downstream performance on several zero-shot learning tasks.
The downstream tasks we evaluated include coarsegrained classification (CIFAR10 and CIFAR100), fine-grained classification (EuroSAT [22]), and optical character recognition (SVHN, HatefulMemes [28]).
Metric and prompt for each task are shown in Appendix Table 3.
Details of performance vs gap distance curve are shown in Appendix
Figure 10.
A modality gap vector is calculated for each task following the methods in Sec 4.2.
B.2 Fairness
In Sec.
5.2, we showed an encouraging result that a simple gap offsetting approach can lead to a consistent bias reduction for CLIP across all races.
Meanwhile, we only observe a minor 0 .top-1 accuracy drop, from 0 .
5817 to 0 .
We show text prompts we used in Supp.
Figure 11.
Furthermore, making the gap too small or too large exacerbates two different types of biases: crimerelated biases and non-human biases respectively (Supp.
Table 4).
Making the gap too small ( d = 0 . 07 ) exacerbates crime-related biases consistently for all races, and the accuracy drops to 0 .
Making the gap too large ( d = 1 . 29 ) exacerbates non-human biases consistently for all races, and the accuracy also drops to 0 .
C The bigger picture: Why studying the modality gap is important
There has been tremendous recent interest and excitement in studying the inductive bias of neural networks mathematically and empirically [13].
For example, an influential line of research shows that neural networks can easily fit random labels [52], and SGD provides an inductive bias of “implicit regularization” by favoring minima that is flatter [27] and closer to the initialization [36].
Another impactful line of research shows that neural networks trained on natural scenes are biased towards texture [18], and exhibit gestalt closure similar to human perception, which is an inductive bias long-studied in the Psychology literature [29].
Researchers have also shown that neural networks favor “shortcut learning”, which may be a common characteristic of learning systems, biological and artificial alike, as known in Comparative Psychology, Education and Linguistics [17, 3].
Our paper is positioned to be part of this broad and exciting trend of studying the inductive bias of neural networks by analyzing the modality gap phenomenon which occurs consistently in multi-modal contrastive representation learning.
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
latexit
(Enc k ))
s i,j
= x
i · y j <latexit sha1_base64="npjuRigGVS4KtYhbMmTmfe300r0=">AAADjHicrVJdaxNBFJ3sVm1TbVN99OViKKSo6U4oWpFCUZE+
[Table]
[Table] +14Jw7d79B
[Table]
+8nF9urzwcUYiBLDkaFKFd9hWek4OIKXJFWUWezsZwdEV1ksgMhiCHOe8B9lS8dWvBAO9oEYWu05O8sbw7U4vhaBkPqtZjq3msH
[Table]
/ADK1JFE=</
P j exp( s j
L D 2 !
D 1 =  1 N X
i log exp( s i,i / ⌧ )
P j exp( s j,i / ⌧ )
L = 12( L D 1 !
D 2 + L D 2 !
D 1 )
[Figure]
SVD
[Figure]
SVD
SVD SVD
SVD SVD
SVD SVD
S V D S V D
S V D S V D
S V D S V D
S V D
[Figure]
S V
[Figure]
[Figure]
positive.
indicates restricted a pre-trained models or with random weights
[Figure]
① Why gap exists?
(Sec X) Research Questions: ② Why gap cannot be closed?
(Sec X) ③
How to close gap?
(Sec X) ④
Why close gap?
(Sec X)
Original
Downstream:
CIFARFine-tuning:
MSCOCO
[Figure]
[Figure] [Figure]
[Figure] [Figure]
[Table]
[Figure]
[Table]
[Figure]
Dataset [Table]
value
CIFFAR [Table] [Table]
↑ [Table] 476eCIFFAR
[Table] ↓ [Table] 701e
[Table]
EuroSAT [Table] ↓
[Table]
020e
[Table]
Specifically, we have conducted the chi-squared test under the null hypothesis that the classification accuracy does not change after changing the modality gap i e. , H less than many because we
CIFAR10 (and
) instead of only the validation set to make our more denigration_bias_evaluation_prompts = { ' White_Male ' : ' a photo of a white man ' , ' White_Female ' : ' a photo of a white woman ' , ' Black_Male ' : ' a photo of a black man ' , ' Black_Female ' : ' a photo of a black woman ' , ' Latino_Hispanic_Male ' : ' a photo of a latino man ' , ' Latino_Hispanic_Female ' : ' a photo of a latino woman ' , ' East Asian_Male ' : ' a photo of an east asian man ' , ' East Asian_Female ' : ' a photo of an east asian woman ' , ' Southeast Asian_Male ' : ' a photo of a southeast asian man ' , ' Southeast Asian_Female ' : ' a photo of a southeast asian woman ' , ' Indian_Male ' : ' a photo of an indian man ' , ' Indian_Female ' : ' a photo of an indian woman ' , ' Middle Eastern_Male ' : ' a photo of a middle eastern man ' , ' Middle Eastern_Female ' : ' a photo of a middle eastern woman ' , ' animal ' : ' a photo of an animal ' , # Non-human ' gorilla ' : ' a photo of a gorilla ' , # Non-human ' chimpanzee ' : ' a photo of a chimpanzee ' , # Non-human ' orangutan ' : ' a photo of an orangutan ' , # Non-human ' thief ' : ' a photo of a thief ' , # Crime-related ' criminal ' : ' a photo of a criminal ' , # Crime-related ' suspicious
[Figure]
' : ' a photo of a suspicious person ' , # Crime-related }
‘criminal’ and ‘suspicious person’)
[Figure]
image
[Figure]
encoder
[Figure]
ResNet
[Figure]
Vision Transformer
[Figure]
encoder
[Figure]
CBOW
[Figure]
Text Transformer
[Figure]
I
[Figure]
c
[Figure]
aligned
[Figure]
l
[Figure]
W
[Figure]
i[
[Figure]
e
[Figure]
image
[Figure]
embed
[Figure]
e
[Figure]
embed
[Figure]
embedding
[Figure]
of each
[Figure]
We first trained 11 ResNet models from scratch on ImageNet, which differ only in the initial random seeds.
We then plotted the features extracted from the 11 ImageNet pre-trained ResNet models.
The cones remain distinctively different cif randomly initialized models are fully trained on ImageNet
D Proofs
We first provide a useful lemma that compares the inner product between two intermediate layer outputs.
Lemma 3.
Suppose W ∈ R d out × d in is a random matrix whose ( k, l ) -th element W k,l is independently and identically distributed from some symmetric distribution with variance 1 /d out for k ∈ [ d out ] , l ∈ [ d in ] .
Similarly, we assume each element in b ∈ R d out follows some symmetric distribution with variance 1 /d out .
For fixed vectors u, v ∈ R d in , we have
[Equation]
Proof of Lemma 3.
The first inequality of (1) is from
[Equation]
Here, the first equality due to the Independence between W and b .
We now show the second inequality of (1).
For k ∈ [ d out ] , we decompose ( W u + b ) k
( W v + b ) k as follows.
[Equation]
Here, the inequality is because max(( W u + b ) k , 0) min(( W v + b ) k , 0) and min(( W u + b ) k , 0) max(( W v + b ) k , 0) are always less than or equal to zero.
Since every element of W and b is symmetric ( i.e. , W k,l d = − W k,l and b k d = − b k for k ∈ [ d out ] , l ∈ [ d in ] ), we have max(( W u + b ) k , 0) max(( W v + b ) k , 0) d
= min(( W u + b ) k , 0) min(( W v + b ) k , 0) , and thus
E
[Equation]
E
[Equation]
(cid:104)
[Equation]
(cid:105) .
Proof of Theorem 1.
When u T v ≤ 0 , the result is trivial because cos( ϕ ( W u + b ) , ϕ ( W v + b )) is positive almost surely.
Therefore, we only consider the case where u T v > 0 .
The main idea of this proof is to use the fact that each element in W u + b can be seen as an independently and identically distributed (i.i.d.) copy of some distribution.
To be more specific, we first note that for k ∈ [ d out ] , due to the Gaussian assumption on W and b , we have √ d out ( W u + b ) k ∼ N (cid:0) 0 , 1 + u T u (cid:1) .
Then from the definition of a rectified Gaussian distribution 7 , we have ϕ ( √ d out ( W u + b ) k ) ∼ N R (cid:0) 0 , 1 + u T u (cid:1) .
This implies E [ { ϕ ( √ d out ( W u + b ) k ) } 2 ] =
(1+ u T u ) /and
E [ { ϕ ( √ d out ( W u + b ) k ) } 4 ] ≤ E [ {√ d out ( W u + b ) k } 4 ] = 3(1 + u T u ) 2 < ∞ .
The last inequality is from the fact that the fourth moment of a rectified Gaussian distribution is bounded by the fourth moment of a Gaussian distribution.
[Step 1] For k ∈ [ d out ] , we now define T k as follows
[Equation]
Note that T 1 , . .
, T d out are i.i.d.
whose mean is one and variance is less than 12.
Therefore, by
Chebyshev’s inequality, for any ϵ 1 >
P    (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12) d out d out (cid:88)
[Equation]
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12)
[Equation]
d out ϵ 21 (cid:33) .
It is noteworthy that 1 d out
[Equation]
(cid:13)(cid:13) 2 .
That is, with probability at least 1 − O
(1 / ( d out ϵ 21 )) , we have
[Equation]
which implies that with probability at least 1 − O (1 / ( d out ϵ 21 ))
the following holds.
[Equation] [Equation]
Similarly, since ϕ
[Equation]
we obtain the following result: for any ϵ 2 > 0 , with probability at least 1 − O (1 / ( d out ϵ 22 )) , we have
[Equation]
which implies
[Equation]
[Step 2] Combining the findings in Equations (2) and (3), for any ϵ 1 , ϵ 2 > 0 , with probability at least 1 − O (1 / ( d out ϵ 21 ) − O (1 / ( d out ϵ 22 )) , we have
[Equation]
Var µ 2 Ψ (cid:0) − µσ (cid:1) (cid:16)
[Equation]
(cid:0) −
[Equation]
(cid:1)(cid:17)
+
[Equation]
−
[Equation]
−
[Equation]
Here ψ and Ψ denote a probability density function and a cumulative density function of a standard Gaussian distribution, respectively.
[Equation]
Using the condition 0 < cos( u, v ) <
[Equation] [Equation]
(cid:1)(cid:17) − 1 = 2 r 1+ r 2 , we have
[Equation]
Therefore, since 1+ u T v √ 1+ u T u √ 1+ v T v is strictly greater than u T v √ u T u √ v T v , by well choosing ϵ such that
[Equation]
√
[Equation]
u T v √ u T u √ v T v and by substituting ϵ 1 = 2 ϵ and ϵ 2 =
ϵ , we have the following inequality with probability at least 1 − O
(1 /d out ) .
[Equation]
A detailed statement of Theorem 2 To begin with, we first define some notations.
For l ∈ [ L ] , we denote the number of nodes in the l -th layer by d ( l ) , the l -th layer weight matrix by W ( l ) ∈
R d ( l ) × d ( l − 1) , and an associated bias vector by b ( l ) ∈ R d ( l ) .
We denote the input data by U ∈ R d (0) .
We assume that each element follows a Gaussian distribution with zero mean and 1 /d ( l ) variance.
We denote a set of weights and biases up to the l -th layer by Θ ( l ) :
= { ( W ( i ) , b ( i ) ) }
li =1 and the l -th layer output by h ( l ) ( U ) when an input datum is U , i.e. , h ( l ) ( U )
=
ϕ ( W ( l ) h ( l − 1) ( U ) + b ( l ) ) .
We set h (0) ( U ) :
= U .
In the following theorem, we provide a detailed statement of Theorem 2.
Theorem 4 (A detailed statement of Theorem 2) .
Let U ∈ R d (0) be a random variable for input data with ∥ U ∥ = 1 .
We suppose tr(Var[ h ( L − 1) ( U ) | Θ ( L − 1) ])
= 1 − β .
Then, for k ∈ [ d ( L ) ] the following inequality holds.
[Equation]
The relationship between β and the cosine similarity The trace parameter β
= 1 − tr(Var[ h ( L − 1) ( U ) | Θ ( L − 1) ]) captures the cosine similarity of the ( L − 1) -th layer outputs because of the following equality.
For independently and identically distributed random variables Uand U 2 , we have
[Equation]
( L − 1) ( U 2 ) (cid:13) (cid:13)(cid:13) | Θ
( L − 1) (cid:21) ≈
[Equation] [Equation]
E [
[Equation]
( h ( L − 1) ( U
[Equation]
h ( L − 1) ( U 2 ))]) .
The last approximation is due to (cid:13)(cid:13)
[Equation]
h ( L − 1) ( U 1 ) (cid:13)(cid:13) (cid:13)
≈
under the variance conditions on W ( l ) and b ( l ) [1, Lemma 7.1].
That is, E [cos( h ( L − 1) ( U 1 ) , h ( L − 1) ( U 2 ))] and β are close to each other.
It is plausible in practice to assume that β is close to one when the depth L is large because the variance of an intermediate output given Θ ( L − 1) is likely to be small due to the cone effect.
Proof of Theorem 4.
By the law of total variance, for any k ∈ [ d ( L ) ] , we have
[Equation]
[Step 1] For k ∈ [ d ( L ) ] , a conditional distribution of ( W ( L ) h ( L − 1) ( U ) + b ( L ) )
k given Θ ( L − 1) and
U is a Gaussian distribution with zero mean and (1 + h ( L − 1) ( U ) T h
( L − 1) ( U )) /d
( L ) variance, we have
[Equation]
and
[Equation]
The last inequality is from Jensen’s inequality E [ √
[Equation]
U T U ] ≤
[Equation]
[Step 2] For k ∈ d ( L ) , we now consider E
Var
( h ( L ) ( U ))
k | Θ ( L ) ]] =
E [Var[ ϕ ( W ( L ) h ( L − 1) ( U ) +
b ( L ) )
k | Θ ( L ) ]] .
By the symmetricity of W ( L ) and b ( L ) , we have
E
[Equation]
Using the characteristic of the ReLU function, we have ϕ ( W ( L ) h
( L − 1) ( U ) + b ( L ) ) 2 k + ϕ
( − ( W ( L ) h
( L − 1) ( U ) + b ( L ) ))
2 k = ( W ( L ) h ( L − 1) ( U ) + b ( L ) )
2 k
and
[Equation]
Therefore,
[Equation]
where W Tk is the k -th row of the weight matrix W .
Thus, an upper bound for E
[Var[( h ( L ) ( U ))
k | Θ ( L ) ]] is
[Equation]
[Step 3] Finally, combining Equations (5) and (6)
[Equation]
The last inequality is due to the fact E [ h ( L − 1) ( U ) T h
( L − 1) ( U )]
= 1 when ∥ U ∥ = 1 and π < 2( π − 1) .
Due to Equation (4), it concludes a proof.