FILIP: F INE - GRAINED I NTERACTIVE L ANGUAGE -
I MAGE P RE -T RAINING
Anonymous authors
Paper under double-blind review
A BSTRACT
Unsupervised large-scale vision-language pre-training has shown promising advances on various downstream tasks.
Existing methods often model the crossmodal interaction either via the similarity of the global feature of each modality which misses sufﬁcient information, or ﬁner-grained interactions using cross/selfattention upon visual and textual tokens.
However, cross/self-attention suffers from inferior efﬁciency in both training and inference.
In this paper, we introduce a large-scale Fine-grained Interactive Language-Image Pre-training (FILIP) to achieve ﬁner-level alignment through a cross-modal late interaction mechanism, which uses a token-wise maximum similarity between visual and textual tokens to guide the contrastive objective.
FILIP successfully leverages the ﬁnergrained expressiveness between image patches and textual words by modifying only contrastive loss, while simultaneously gaining the ability to pre-compute image and text representations ofﬂine at inference, keeping both large-scale training and inference efﬁcient.
Furthermore, we construct a new large-scale image-text pair dataset called FILIP300M for pre-training.
Experiments show that FILIP achieves state-of-the-art performance on multiple downstream vision-language tasks including zero-shot image classiﬁcation and image-text retrieval.
The visualization on word-patch alignment further shows that FILIP can learn meaningful ﬁne-grained features with promising localization ability.
I NTRODUCTION
Large-scale Vision-Language Pre-training (VLP) models like CLIP (Radford et al., 2021) and
ALIGN (Jia et al., 2021) have recently demonstrated success across various downstream tasks.
They learn visual and textual representations from millions of image-text pairs collected from the Internet and show superior zero-shot ability and robustness.
The core technique of these models lies in the global contrastive alignment of the images and texts through a dual-stream model.
Such architecture is inference-efﬁcient for downstream tasks like retrieval because the encoders for the two modalities can be decoupled and the image or text representations can be pre-computed ofﬂine.
However, CLIP and ALIGN model the cross-modal interaction via solely the similarity of the global feature of each modality, lacking the ability of capturing ﬁner-level information like the relationship between visual objects and textual words.
In this paper, we develop a simple yet efﬁcient cross-modal ﬁner-grained interaction mechanism for large-scale VLP.
To achieve ﬁner-grained cross-modal interaction, previous methods mainly exploited two kinds of methods.
(1) One line of work (Chen et al., 2020;
Li et al., 2020b; Dong et al., 2021; Li et al., 2021b;
Zhang et al., 2021; Zhan et al., 2021) uses a pre-trained object detector to extract region-of-interest (ROI) features from images, and then fuses it with the paired text through a VLP model.
This design complicates the pre-training due to pre-computing and storing a large number of ROI features.
In addition, the zero-shot ability of these approaches is usually limited by the predeﬁned number of classes and their performance is also restricted by the quality of the detector.
(2) Another line of work (Li et al., 2021a;
Kim et al., 2021) enforces the token-wise or patch-wise representations from both modalities into the same space and models these ﬁner-grained interactions via cross-attention (Li et al., 2021a) or self-attention (Kim et al., 2021).
However, these methods are usually less efﬁcient in terms of both training and inference.
In particular, during training, cross-attention in (Li et al., 2021a) requires to be performed in an encoder-decoder structure, while the complexity of the self-attention (Kim et al., 2021) grows quadratically with the length of the prolonged concatenated sequences of both modalities.
During inference, the data from both modalities are intertwined to compute the cross-attention or self-attention, and can not be pre-computed ofﬂine as dual-stream models like CLIP and ALIGN.
This can be less efﬁcient for downstream tasks like image/text retrieval and image classiﬁcation.
In this paper, we propose a large-scale Fine-grained Interactive Language-Image Pre-training framework named FILIP to address these limitations.
Inspired by Khattab & Zaharia (2020), we model the ﬁne-grained semantic alignment through a novel cross-modal late interaction mechanism in the contrastive loss, instead of using cross or self-attention.
Speciﬁcally, our ﬁne-grained contrastive learning uses a token-wise maximum similarity between visual and textual tokens to guide the contrastive objective.
In this way, FILIP successfully leverages the ﬁner-grained expressiveness among image patches and textual words while simultaneously gaining the ability to pre-compute image and text representations ofﬂine.
Unlike Khattab & Zaharia (2020), we discard the padded tokens and use average instead summation of token-wise maximum similarities when computing the image-text alignment, which enhances the cross-modal representation learning and stabilizes training.
Furthermore, we construct a large-scale pre-training dataset named FILIP300M from the Internet.
Data cleaning and image-text data augmentation are also explored and proved useful in this work.
Extensive experiments show that by effectively learning ﬁne-grained representations, FILIP achieves state-of-the-art performance on multiple downstream tasks, including zero-shot image classiﬁcation and image-text retrieval.
For example, FILIP reaches 77.1% top-1 accuracy for zero-shot ImageNet classiﬁcation, surpassing CLIP with less training data.
Visualizations on word-patch alignment further show that FILIP learns meaningful ﬁner-grained features with promising localization ability.
R ELATED W ORK
Vision-Language Pre-training Models.
The pre-train-and-ﬁne-tune scheme has achieved great success in the domains of natural language processing (Devlin et al., 2019; Brown et al., 2020) and computer vision (Dosovitskiy et al., 2020).
It is then naturally extended to a joint cross-modal domain of Vision-and-Language Pre-training (VLP).
The pre-training datasets of recent VLP models include publically available datasets like YFCC100M
(Thomee et al., 2016) and CC12M (Changpinyo et al., 2021), as well as larger-scale datasets with more than 100M samples in CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021), which are shown to be even more powerful.
The pretraining tasks of VLP models can be categorized into two categories: image-text contrastive learning task and Language Modeling (LM) based tasks: (i) CLIP (Radford et al., 2021), ALIGN (Jia et al., 2021) and UNIMO (Li et al., 2021b) make use of cross-modal contrastive learning which aligns the textual and visual information into a uniﬁed semantic space; (ii) VisualBERT
(Li et al., 2019),
UNITER (Chen et al., 2020), M6 (Lin et al., 2021), and DALL-E (Ramesh et al., 2021) employ
LM-like objectives, including both masked LM (e.g., Masked Language/Region Modeling), and autoregressive LM (e.g., image captioning, text-grounded image generation).
On the other hand, some methods rely on a pre-trained object detection model such as Faster-RCNN (Ren et al., 2015) to extract image regional features ofﬂine, which requires extra labeled bounding-box data and makes the approach less scalable.
Recent efforts such as SOHO (Huang et al., 2021) and SimVLM (Wang et al., 2021) try to eliminate this burden via visual dictionary or PreﬁxLM (Raffel et al., 2020).
In this paper, we directly learn ﬁne-grained vision-language representations in an end-to-end and simpler manner while maintaining the beneﬁt of inference efﬁciency.
Multi-Modality Interaction Mechanism.
The core of vision-language pre-training models lies in modeling the interaction between the two modalities.
There are mainly two types of cross-modal interaction architectures: Single-stream models like VisualBERT (Li et al., 2019) and ViLT (Kim et al., 2021) directly concatenate the patch-wise or regional visual features and textual embeddings and feed them to the transformer-based model.
Dual-stream models such as ViLBERT (Lu et al., 2019) and CLIP (Radford et al., 2021) have separate encoders for different modalities.
This allows ﬂexible use of different models for different modalities, and efﬁcient inference for downstream tasks like image-text retrieval, through the ability of decoupling the encoders and pre-compute image/text features ofﬂine.
SCAN (Lee et al., 2018) considers latent alignments between image regions and words.
However, it is based on Triplet loss with a bottom-Up attention via a Faster-RCNN to extract object features while we try to directly learn to localize ﬁne-grained object from patches.
In this paper, while following the dual-stream approach for its ﬂexible and efﬁcient inference, we further propose a new multi-modal interaction mechanism to capture the ﬁne-grained representations.
M e a n 1 .
M a x 1.
Max 2.
Mean <BOS> cat in the basin <EOS> *
Visual Token
Textual Token
Token
Embedding
PositionalEmbedding
Cross-modal Late Interaction
Image Encoder
Text Encoder 𝑛 ! … … … … … …
𝑛 ! ×𝑛 " 𝐼 !
𝐼 " 𝐼 # 𝑇 !
𝑇 " 𝑇 # … … 𝑠 $,&’ 𝑇 !
𝑇 " 𝑇 # 𝐼 !
𝐼 " 𝐼 # … … 𝑠 $,&(
Image-to-text Contrastive
Text-to-image Contrastive
A
Token-wise Similarity
Image-to-textText-to-image
5    𝑛 "
Linear Projection
M ETHOD
In this paper, we propose a new cross-modal pre-training model that excels in ﬁne-grained interaction between image encoder and text encoder for mining more detailed semantic alignment, named as FILIP, as shown in Figure 1.
Particularly, FILIP is a dual-stream model with Transformer-based image and text encoders.
For the visual modality, the image encoder is a Vision Transformer (Dosovitskiy et al., 2020) which takes the concatenation of an extra [CLS] token embedding and linearly projected image patches as input.
For the textual modality, following Radford et al.
(2021), we use the lower-cased byte pair encoding (BPE) (Sennrich et al., 2016b) with a vocabulary size of 49, to tokenize the text.
Each text sequence starts with [BOS] token and ends with [EOS] token.
After the word embedding layer, the token embeddings are fed into a modiﬁed decoder-only Transformer model as in (Radford et al., 2019).
On top of the image and text encoders, the representations of textual tokens and visual tokens are linearly projected to the multi-modal common space, and are separately L2-normalized.
Different from existing dual-stream models (e.g., CLIP and ALIGN) which models cross-modal interaction via only the global features of the entire image and text sequence, we introduce a novel ﬁne-grained contrastive learning objective equipped with cross-modal late interaction which takes into account the ﬁne-grained interaction between image patches and textual tokens, detailed in Section 3.1.
3. F INE - GRAINED C ONTRASTIVE L EARNING
Contrastive representation learning has recently been found to learn better representations than its predictive counterpart in both visual (Tian et al., 2020) and vision-language cross-modal pre-training
(Radford et al., 2021).
Under a general formulation of cross-modal contrastive learning (Radford et al., 2021), we want to learn encoders f θ for image data I and g φ for text data T such that, given an image x I ∈ I , and a text x T ∈ T , the encoded representations f θ ( x I ) and g φ ( x T ) are close if they are related and far apart if not, under a distance metric.
In each training batch, we sample b image-text pairs { x Ik , x Tk } bk =1 , For image x
Ik in image-text pair { x Ik , x Tk } , x Tk is its positive, while the other texts will be used as in-batch negatives.
The image-to-text contrastive loss L Ik for x
Ik can then be formulated as
[Equation]
exp ( s Ik,j ) , where s Ik,j denotes the similarity of the k -th image to the j -th text.
Similarly, the text-to-image contrastive loss for x Tk is
[Equation]
The total loss of this mini-batch can be represented by
[Equation]
3.1. C ROSS - MODAL L ATE I NTERACTION
From the contrastive loss (1), the cross-modal interaction is reﬂected in how we compute the similarities s Ii,j and s Ti,j for the i -th image and j -th text.
Previous methods like CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021) simply encode each image or text separately to a global feature i.e., f θ ( x Ii ) ∈ R d and g φ ( x Tj ) ∈ R d , and compute these two similarities as
[Equation]
neglecting ﬁner-grained interactions (e.g., word-patch alignment) between the two modalities.
To alleviate this problem, while simultaneously maintain the training and inference efﬁciency of dualstream models, we apply a cross-modal late interaction inspired by Khattab & Zaharia (2020) to model the token-wise cross-modal interaction.
Speciﬁcally, denote n 1 and n 2 as the number of (non-padded) tokens of the i -th image and j -th text, respectively, and the corresponding encoded features are f θ ( x Ii ) ∈ R n 1 × d and g φ ( x Tj ) ∈ R n 2 × d .
For the k -th visual token, we compute its similarities with all textual tokens of x Tj , and use the largest one
[Equation]
as its token-wise maximum similarity with x Tj .
We then use the average token-wise maximum similarity of all non-padded tokens in the image (resp. text) as the similarity of an image to a text (resp.
a text to an image).
The similarity of the i -th image to the j -th text can thus be formulated as:
[Equation]
where m Ik = arg max 0 ≤ r<n
[ f θ ( x Ii )] (cid:62) k
[ g φ ( x Tj )] r .
Similarly, the similarity of the j -th text to the i -th image is
[Equation]
where m Tk = arg max 0 ≤ r<n 1 [ f θ ( x Ii )] (cid:62) r
[ g φ ( x Tj )] k .
Note that s Ii,j ( x Ii , x Tj ) in Equation (4) does not necessarily equal s Ti,j ( x Ii , x Tj ) in Equation (5).
Remark 1 Intuitively, the token-wise maximum similarity in Equation (3) means that for each image patch, we ﬁnd its most similar textual token.
Similarly, for each textual token, we also ﬁnd its closest image patch.
By applying this to the similarity calculation in (4) and (5) for contrastive loss (1), the dual-stream model learns ﬁne-grained alignment between image patches and textual tokens.
The original late interaction mechanism in (Khattab & Zaharia, 2020) computes the relevance score of a document to a query padded with mask tokens , as a sum of token-wise maximum similarities, and is optimized via a pairwise softmax cross-entropy loss.
Though inspired from Khattab & Zaharia (2020), our proposed cross-modal late interaction differs in several aspects.
Firstly, we exclude the padded textual tokens when computing the similarity, as they harm the performance.
We speculate that this is because these padded tokens also learn textual representations and will mislead the model to align image patches to these meaningless padded tokens rather than meaningful non-padded words.
Secondly, when computing similarities (4) and (5), we use the average of the token-wise maximum similarities instead of summation in (Khattab & Zaharia, 2020).
This is because the number of non-padded tokens varies from text to text, and this summation over all non-padded tokens can have quite different magnitudes, leading to less stabilized training and worse ﬁnal performance. These two modiﬁcations are crutial to not only the downstream tasks’ performance, but also the quality of the word-patch alignment.
A more detailed discussion can be found in Appendix A.6. Thirdly, we optimize the late interaction mechanism via a contrastive loss (1) which is found powerful vision-language pre-training (Radford et al., 2021) instead of the original pairwise loss in (Khattab & Zaharia, 2020).
Training Efﬁciency.
Though the cross-modal late interaction is able to capture ﬁner-grained features compared with the original loss, it relies on the token-wise representations of both modalities, and can be inefﬁcient in terms of communication, memory and computation, especially when the batch size is large.
To alleviate this problem, we utilize several methods.
Firstly, we reduce the embedding size to 256.
Besides, we reduce the precision of the last-layer features of both modalities from fp32 to fp16 before node communication in a distributed learning setting, and perform the multiplication in Equations (4) and (5) under the reduced precision.
In addition, since the complexity of similarity calculation scales with the sequence length of textual tokens and image patches, for each image (resp. text), we select the 25% tokens with the highest token-wise maximum similarity score (Equation (3)) among all texts (resp. images) in the same local worker before node communication, based on the intuition that each sample can be represented by a few of the most representative tokens.
Effects of these modiﬁcations are studied in Section 4.4.
3.1. P ROMPT E NSEMBLE AND T EMPLATES
Due to the problem of polysemy and inconsistency with the pre-training process, following Radford et al.
(2021), we also use prompt templates to augment the original label for some downstream tasks.
For visualizations, for simplicity, we use only one prompt template across the paper, i.e. “a photo of a { label } .”
as Radford et al.
For other experiments, we report results using prompt ensemble following Radford et al.
When multiple prompts are allowed, the token-wise representations of different prompt templates for the same class label are different, and can not be summed together to form a mean textual representation as in (Radford et al., 2021).
Thus, instead of ensembling different prompt templates by their mean textual representation, we ensemble them by their mean token-wise similarity.
Speciﬁcally, suppose there are C prompt templates, each label is augmented to C different texts x T 1 , x T 2 , · · · , x TC .
The similarity between an image x I and this label is computed as 1 C (cid:80)
Cc =1 s I · , · ( x I , x Tc ) , where s I · , · is deﬁned in Equation (4).
We use a uniﬁed rule-based method inspired by Radford et al.
(2018) to construct prompt templates for image classiﬁcation tasks.
Speciﬁcally, each template consists of four components:
[Equation]
Here, the “[preﬁx]” is an in-context description like “a photo of a” similar as Radford et al.
(2021); “label” is a class label of the dataset; “[category description]” describes the category which is found helpful for some ﬁne-grained image classiﬁcation datasets (Radford et al., 2021), e.g., “ a type of pet” for dataset Oxford-IIIT Pets.
An interesting ﬁnding is that, adding a sufﬁx that includes the reference word “it” (e.g., “I like it.”)
at the end of the prompt empirically improves the zero-shot classiﬁcation performance of the proposed model.
We speculate this is because the reference word “it” strengthens the ﬁne-grained cross-modal alignment, as it can also be aligned to image patches of the target object.
Detailed prompt templates for different datasets can be found in Appendix A.4.
3. I MAGE AND T EXT A UGMENTATION
To obtain better generalization and data-efﬁciency of the model, we perform data augmentation on both images and texts during the pre-training phase to construct more image-text pairs.
We apply
AutoAugment (Krizhevsky et al., 2012; Sato et al., 2015; Cubuk et al., 2019; Hoffer et al., 2020) for image augmentation, following the SOTA vision recognition methods (Touvron et al., 2021; Xie et al., 2020b).
To ensure the augmented texts are semantically similar as the original one, for text augmentation, we rewrite the original text using back-translation (Xie et al., 2020a; Sennrich et al., 2016a).
Speciﬁcally, the texts are ﬁrst translated to the target language and then translated back to the source language.
We choose German and Russian as the target language and get extra two texts for each image-text pair.
When constructing a batch of image-text pairs during the pre-training, the C I F A
R
C I F A R
C a lt ec h
S t a n f o r d C a r s F l o w e r s F ood S UN
[Table]
D T D
A i r c r a f t s
[Table]
x f
[Table]
r d P
[Table]
t s [Table] u r o S
[Table]
text of each image-text pair is randomly sampled from the three candidate texts,
i [Table]
e., the original text and two back-translated texts.
3. P RE - TRAINING D ATASET
A sufﬁciently large image-text dataset is a prerequisite for vision-language pre-training.
Recent
CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021) construct datasets with 400M and 1800M image-text pairs, respectively.
In this work, we also construct a large-scale dataset called
FILIP300M, which consists of 300M image-text pairs and covers board vision and language concepts.
Speciﬁcally, we collect image-text pairs from the Internet, and apply the following imageand text-based ﬁltering rules to clean data.
For image-based ﬁltering, we remove the images whose shorter dimension is smaller than 200 pixels and the aspect ratio is larger than 3.
For text-based ﬁltering, we keep only English texts, and exclude the meaningless ones, e.g., img 0.jpg.
We also discard image-text pairs whose texts are repeated for over 10 times.
Besides, we also use 3 public datasets, including Conceptual Captions 3M (CC3M) (Sharma et al., 2018),
Conceptual 12M (CC12M) (Changpinyo et al., 2021) and Yahoo Flickr Creative Commons 100M (YFCC100M) (Thomee et al., 2016).
We apply the same ﬁltering rules on YFCC100M. Finally, we use about 340M image-text pairs for pre-training.
Despite using a smaller training dataset than CLIP and
ALIGN, our models still outperform them in most down-steam tasks (see Section 4).
E XPERIMENTS 4. E XPERIMENTAL S ETUP
Model Architectures.
We train two models from scratch, i.e., FILIP base and FILIP large .
The model architectures follow CLIP (Radford et al., 2021), i.e., the image encoder is ViT-B/32 for FILIP base and ViT-L/14 for FILIP large .
More details can be found in Appendix A.2.
Pre-training Details.
To save memory and scale up the batch size, automatic mixed-precision (Micikevicius et al., 2018) and gradient checkpoint (Griewank & Walther, 2000; Chen et al., 2016) are used The input images are resized to 224 × 224 resolution during pre-training and the maximum length of the text is limited to 77 tokens following Radford et al.
The training is mainly conducted on Nvidia V100 GPUs and Ascend Cards.
FILIP base is trained on 128 cards aboutdays and FILIP large takes about 24 days to train on 192 cards.
Unless otherwise speciﬁed, we use
FILIP large to compare with other methods and FILIP base for ablation.
We train both models using the LAMB optimizer (You et al., 2020) and cosine learning rate schedule (Loshchilov & Hutter, 2016) with a linear warmup.
Weight decay regularization is applied to all parameters except bias, layer normalization, token embedding, positional embedding and temperature in contrastive loss.
Detailed values of hyperparameters for different datasets and models can be found in Appendix A.2.
4. Z ERO -S HOT I MAGE C LASSIFICATION
In this section, we evaluate our proposed FILIP on the zero-shot image classiﬁcation task.
We compare our FILIP with CLIP (Radford et al., 2021) on 12 downstream classiﬁcation datasets, using the same evaluation setting as in CLIP.
As described in Section 3.1.2, we apply a set of prompts for each dataset and ensemble them to get the ﬁnal results, see Appendix A.4 for details.
We only
[Table]
compare the zero-shot performance with CLIP here as ALIGN does not release its model and the related performances are not reported in their paper.
Table 1 shows the results on 12 datasets.
Despite using less training data (340M vs. 400M), both
FILIP base and FILIP large considerably outperform their CLIP counterparts in terms of average topaccuracy over 12 datasets, i.e., achieving absolute improvements of 5.6% and 3.0%, respectively.
In particular, our FILIP surpasses CLIP on ImageNet, the largest dataset among 12 datasets.
FILIP also achieves substantial performance gains on some domain-speciﬁc datasets, e.g., for Aircrafts, the two
FILIP models reach a 30% improvement over CLIP on average.
We speculate this is because, unlike
CLIP which aggregates the information of the whole image into the representation of the [CLS] token, our proposed FILIP model focuses more on the target object by directly aligning the image patches corresponding to the target object with the textual tokens corresponding to the class label (visualizations of word-patch alignment are in Section 4.5).
4. I MAGE -T EXT R ETRIEVAL
Image-text retrieval consists of two sub-tasks: image-to-text retrieval and text-to-image retrieval.
We evaluate our FILIP model on two retrieval benchmark datasets: Flickr30K (Plummer et al., 2015) and MSCOCO (Lin et al., 2014), under both zero-shot and ﬁne-tuned settings.
More details of experimental setting can be found in Appendix A.2.
Tables 2 and 3 show the results of zero-shot and ﬁne-tuned image-text retrieval, respectively.
We compare our FILIP model against methods with complex attention layers including Unicoder-VL (Li et al., 2020a), ImageBERT (Qi et al., 2020), UNITER (Chen et al., 2020), VILLA (Gan et al., 2020),
ERNIE-ViL (Yu et al., 2021), Oscar (Li et al., 2020b), VinVL (Zhang et al., 2021), ALBEF (Li et al., 2021a), and methods trained on larger-scale image-text datasets including CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021).
As we can see, FILIP achieves state-of-the-art performances under all metrics on both Flickr30K and MSCOCO datasets, except for zero-shot text-to-image retrieval on Flickr30K, where FILIP achieves competitive performance with SOTA.
For zero-shot image-to-text retrieval on MSCOCO dataset, the absolute R@1 of our proposed FILIP is 2.7% higher than ALIGN, which is trained on a much larger dataset.
4. A BLATION S TUDY
Effectiveness of Each Component.
We study the effectiveness of each component in FILIP, i.e., image/text augmentations and cross-modal late interaction.
Experiments are conducted on FILIP base , with a ﬁltered subset of YFCC100M as the training dataset (as described in Section 3.3), on both zero-shot retrieval and classiﬁcation tasks.
We measure models’ performance on MSCOCO zeroshot image-text retrieval and ImageNet zero-shot classiﬁcation, which are two effective indicators for the quality of the learned vision-language representations.
Table 4 reports the results.
As can be seen, all three components are beneﬁcial for both tasks.
Despite the simple design, cross-modal late interaction brings signiﬁcant performance improvements over the baseline (the vanilla CLIP ViT-B/32), with an absolute R@1 gain of 5.5% (resp.
3.8%) for image-to-text (resp.
text-to-image) retrieval on MSCOCO and an absolute top-1 accuracy gain [Table] [Table] Our FILIP base
[Table] [Table] of 3.9% for zero-shot classiﬁcation on ImageNet. Further improvements are observed when all components are combined together.
Efﬁciency Study of Cross-modal Late Interaction.
Since the late interaction mechanism in Section 3.1.1 requires to calculate the similarity between all visual and textual tokens, its efﬁciency can be a problem when employed in large-scale distributed training.
As described in Section 3.1.1, we make several attempts to address the issue.
Table 5 shows the efﬁciency improvement on zero-shot classiﬁcation on ImageNet when these attempts are applied.
As can be seen, these attempts improve the efﬁciency of late interaction without accuracy drop.
Combining all three attempts achieves only slightly slower training and larger memory consumption than the original loss in CLIP.
4. V ISUALIZATION OF F INE - GRAINED A LIGNMENT
In this section, we visualize FILIP’s capability of capturing ﬁne-grained cross-modal correspondence using the method of word-patch alignment.
To make a fair comparison, we use our FILIP base [Figure] trained on YFCC100M and CLIP’s ViT-B/32, which are of the same size, for visualization.
Each image is patchiﬁed to 7 × 7 image patches.
More visualization results can be found in Appendix A.3.
Visualization Method.
The word-patch alignment is performed based on the token-wise similarity between the image patches and textual tokens.
Speciﬁcally, for the k -th image patch, the location index of textual token with the largest similarity with it ( m Ik in Equation (4)) is considered as its predicted label, and is placed at the center of it.
Take class “balloon” as an example.
There are 8 tokens in the tokenized textual sequence “[BOS] a photo of a balloon.
[EOS]”, and the location index of the class label “balloon” is “5”.
Note that one class label may be tokenized to more than one token.
Location indices of textual tokens corresponding to the class label are highlighted in red, while the others are marked in white.
A desired model that learns ﬁne-grained representations would predict image patches of the target object to red indices.
Observations.
Figure 2 shows the word-patch alignment results for FILIP and CLIP on 4 classes from the ImageNet dataset.
As can be seen, FILIP exhibits the ﬁner-grained understanding of an image in the following aspects.
(i)
A single object: From the visualization of class “small white butterﬂy”, the image patches covering the object are all classiﬁed correctly; (ii) Same object in different shapes: From the visualizations of class “balloon” and “lifeboat”, image patches corresponding to all target objects with different shapes and locations are correctly classiﬁed; (iii) Key Components of an object: For class “electric locomotive”, there are two key components crucial to correctly classifying the image, i.e., “electric” and “locomotive”, whose corresponding textual token indices are “5” and “6”, respectively.
As can be seen, image patches matching these two key components are respectively correctly classiﬁed.
On the other hand, CLIP can not correctly align image patches with corresponding textual tokens.
Compared with Kim et al.
(2021) which uses an extra optimal transport to align the textual word and image patch distributions, the word-patch alignment can be simply automatically learned by our method.
C ONCLUSION AND F UTURE W ORK
This paper introduces FILIP, a simple yet generic framework towards ﬁne-grained vision-language pre-training.
By using a token-wise maximum similarity, our method learns ﬁne-grained representation for patches in the images and words in the sentences.
While it achieves competitive results against several large-scale multi-modal pre-training on various downstream tasks, both its architecture and training procedure can still be optimized to improve its performance.
In the future, a more advanced image encoder as well as a well-designed interaction layer can be used to boost the performance.
Furthermore, we can further add more masked language/image loss to support more generation tasks.
To this end, we hope to extend FILIP as a generic and uniﬁed interface for solving a large variety of vision-language tasks.
D ETAILED E XPERIMENTAL S ETTINGS
[Table]
Model Architectures.
We follow the same architecture design as CLIP, for both FILIP base and
FILIP large , except that we reduce the embedding dimension from 512/768 to 256 for the efﬁciency of loss computation.
Table 7 describes the details of architectures.
Details for Pre-training and Hyperparameters.
For the implementation of the contrastive loss, following CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021)
[Table]
we also set the temperature in the softmax function to be a learnable parameter and initialize it as 0.07.
For the pre-training, we use the LAMB optimizer implemented by the cybertronai’s open-source repository ( https: //github.com/cybertronai/pytorch-lamb ).
For the learning rate scheduler, we ﬁrst assign a base learning rate and then linearly warm it up to the peak learning rate according to the effective total batch size by a square root strategy, peak lr = base lr × (cid:113) total bs 512 .
We note that a large weight decay is crucial to stabilize training and improve generalization.
Speciﬁcally, we found that the training stability is a challenging issue when applying mix-precision training to large-scale models, i.e., the training is extremely unstable and the NaN loss easily happens.
Recent works
DALL-E (Ramesh et al., 2021) and Cogview (Ding et al., 2021) also notice this issue and provide their solutions.
However, we found that simply increasing the weight decay and applying the trick of removing the weight decay of speciﬁc parameters as described in Section 4.1 work for our case.
The base learning rate and weight decay are selected manually via observing the performance at the early training stage.
Table 8 summarizes the common hyperparameters and Table 9 shows the model- and dataset-speciﬁc hyperparameters for FILIP pre-training.
[Table]
[Table]
Details for Image-text Retrieval.
Following previous works (Jia et al., 2021; Li et al., 2021a), for Flickr30K, we test on the 1K test set with or without ﬁne-tuning on the 30K training set, while for MSCOCO, we test on the 5K test set with or without ﬁne-tuning on the 113K training set.
We use the similarity between image and text for ranking and use the contrastive loss for ﬁne-tuning.
Since there are multiple texts for each image in these two datasets, we change the ground-truth label of contrastive loss to consider multiple positives, by assigning a probability of 1/#positive to each positive following ALBEF (Li et al., 2021a).
Besides, we also use prompts during evaluation for both datasets, see Appendix A.4 for details.
Table 10 shows the hyperparameters for image-text retrieval ﬁne-tuning.
[Table]
A. M ORE VISUALIZATIONS OF W ORD - PATCH A LIGNMENT AND G RAD - CAM H EATMAPS
In Figure 3, we visualize the cross-modal alignment of the proposed method for more images, in terms of both word-patch alignment as described in Section 4.5 and Grad-CAM heatmaps (Selvaraju et al., 2017).
We compute the Grad-CAM heatmaps based on the average self-attention maps over the image patches classiﬁed to targeted textual tokens (i.e., the textual token(s) corresponding to the class label in the ImageNet dataset) in the last layer of the image encoder.
We average the heatmaps over all attention heads.
As can be seen, our proposed model learns meaningful alignment between image patches and textual tokens.
A. P ROMPT T EMPLATES FOR DOWNSTREAM TASKS
Image Classiﬁcation.
Table 11 shows the prompt templates for different image classiﬁcation datasets in the form of “ [preﬁx] { label } , [category description].
[sufﬁx]. ”
in Equation (6).
There are three components to be determined in the template, i.e., the preﬁx, the category description and the sufﬁx.
For each component, we select several well-performed ones for each dataset.
Then we use the full combinations of all three components as the set of prompt templates for ensemble.
For instance, we use 5 preﬁxes, no category descriptions, and 6 sufﬁxes for dataset ImageNet.
Then the total number of prompt templates for this dataset is: 5 × 1 × 6 = 30 .
Image-text Retrieval.
Following CLIP (Radford et al., 2021), we use prompt in zero-shot imagetext retrieval for both Flickr30K and MSCOCO datasets.
The prompt is selected by the same rule as described in Section 3.1.2, except that we do not use “[category description]” here.
Table 12 shows the prompt templates for zero-shot image-text retrieval on Flickr30K and MSCOCO datasets.
[Figure]
[Table]
“a photo of a”, “a jpeg photo of a”, “a painting of a”, “itap of a”, “grafﬁti of a”, “a cartoon”, “a doodle”
[Table]
life”
[Table]
“It’s
[Table]
“It’s
[Table]
, “It’s
[Table]
“Hope
[Table]
it”
CIFAR “a jpeg photo of a”, “a painting of a”, “a good photo of a”, “a bad photo of a”, “a photo of a”, “itap of a”, “a rendering of a”
[Table]
“It’s beautiful”, “It’s ugly”, “I like it”, “I take it today”
Caltech “a photo of a”, “a cropped photo of a”, “a good photo of a”, “a bad photo of a”
None
None
[Table]
“I
[Table]
it”
[Table]
“I hate it”, “It’s ugly”, “It’s
[Table]
Car “a photo of a”, “a close-up photo of a”, “a good photo of a”, “a bad photo of a” “a type of car”, “a type of automobile” “I like it”, “It belongs to my friend”, “It’s brand new”, “It’s popular recently”, “It’s important to me”, “I take it today”
Flowers “a photo of a (many) ”, “a rendering of a (many) ”, “itap of a (many) ” “a type of ﬂower”, “a type of bloom” “It’s beautiful”, “It’s from my best friend”, “It gives out a sweet perfume/fragrance”
ImageNet “a photo of a”, ”a good photo of a”, “a bad photo of a”, “a closeup photo of a”, “itap of a”
None “I like it”, “It’s common in daily life”, “It’s not common in daily life”, “It’s ugly”, “It’s cute”, “It’s beautiful”
Food “a photo of my”, “a close-up photo of my”, “itap of my” “a type of food”, “a type of nourishment” “I made it today”, “I like it”, “I hate it”, “It’s delicious”, “It’s with nice ﬂavour”, “It’s with terrible ﬂavour”, “It’s popular recently”
SUN “a photo of a”, “a good photo of a”, “a bad photo of a”, “a bright photo of a”, a dark photo of a”, “a black and white photo of a”, “a nice scene of a”, “a terrible scene of a”
None
None, “I like it”, “I hate it”, “It’s beautiful”, “It’s common in daily life”, “It’s important to me”
DTD “itap of a”, “a close-up photo of a” “texture”, “surface”, “material”
None, “It’s out of style”, “It’s popular in old days”, “It’s ugly”, “It’s beautiful”
Aircrafts “a photo of the”, “a close-up photo of the”, “a good photo of the ”, “a pixelated photo of the” “a type of plane”, “a type of aircraft”, “a type of airliner”
None,“I like it”, “It’s important to me”, “I take it today”, “Hope you like it”
[Table]
“a photo of my”, “a low resolution photo of my”, “a good photo of my” “a type of pet”, “a type of dog or cat”
None, “It’s cute”, “It’s important to me”, “I like it”, “It’s beautiful”
[Table]
“a photo of a”, “a painting of a”, “a cropped photo of a”, “a good photo of a”, “a blurry photo of a”
None, “an example of aerial or satellite images”
None, “I like it”, “It’s taken from an aircraft or some ﬂying object”, “It’s collected by imaging satellites”
[Table]
A. L INEAR P ROBE ON I MAGE C LASSIFICATION
In this section, we evaluate FILIP on the linear probe for image classiﬁcation.
Following common linear probe setting, we freeze the whole backbone network and only ﬁnetune the last linear classiﬁer.
Since we remove the “[CLS]” token in our vision encoder, we apply a mean pooling over all the other visual tokens to aggregate them into a global image representation which is then fed into the linear classiﬁer.
Setting.
Following CLIP, we train the logistic regression classiﬁer using scikit-learn’s L-BFGS implementation (Pedregosa et al., 2011), with maximum 1,000 iterations on those 11 datasets except
ImageNet.
For ImageNet, we use a pytorch-based codebase to accelerate the training with GPU.
Following Doersch et al.
, we adopt a Batch Normalization (Ioffe & Szegedy, 2015) layer before the linear classiﬁer which is beneﬁcial to stabilize the mixed-precision training.
Random resized crop and horizontal ﬂipping are used to augment training data.
We use the cosine learning rate scheduler with a linear warmup of 10 epochs.
More hyperparameters used in linear probe on
ImageNet are shown in Table 13.
[Table]
Results
Table 14 compares the linear probe performance of our proposed FILIP with CLIP over 12 datasets.
Our FILIP base (resp.
FILIP large ) achieves 85.7% (resp.
91.3%) average top1 accuracy over 12 downstream tasks, which provides noticeable improvements,
i
[Table]
e., 2% (resp.
1.5%) higher, compared to its CLIP’s counterpart.
This implies that our FILIP learns more powerful vision features which potentially facilitates border downstream vision tasks.
[Table]
A. C OMPARISON WITH K HATTAB & Z AHARIA (2020)
As is stated in Section 3.1, compared to Khattab & Zaharia (2020), besides being the ﬁrst to apply the late interaction to contrastive learning for vision-language pre-training, we make two other modiﬁcations, i.e., removing padded tokens and using average over non-padded tokens instead of summation).
In the following, we show that these two modiﬁcations are crucial to the performance, and the quality of ﬁner-granular word-patch alignment.
For comparison, we replace the proposed cross-modal late interaction in FILIP base with the original late interaction in Khattab & Zaharia (2020).
Following the setting in Section 4.4, we pre-train on the ﬁltered YFCC100M with mixed-precision using 8 V100 GPUs.
The batch size per GPU isand dimension of the token feature is 256.
We report results on using the top 25% tokens (selected using the method in Section 3.1) during training.
Note that the original late interaction in Khattab &
Zaharia (2020) is sensitive to the temperature in the softmax function, and we report the best result for it among several initialization values.
Effect to Performance.
Table 15 shows the comparison on zero-shot ImageNet classiﬁcation.
When these two modiﬁcations are removed, the zero-shot Top1 accuracy of ImageNet drops from 34.3 to 32.7.
[Table]
Effect to the Word-patch Alignment In Figure 4, we compare the word-patch alignment using the models trained with the proposed cross-modal late interaction and the late interaction in Khattab & Zaharia (2020).
According to the visualizations, using the original late interaction in Khattab &
Zaharia (2020) leads to less accurate word-patch alignment.
Speciﬁcally, the object patches are often aligned to the padded tokens instead of class names.
We speculate this is because the padded tokens learn similar representations as existing key textual tokens, similar to the ﬁnding in Section 3.2 of
Khattab & Zaharia (2020) that padding with masked tokens (which is called “query augmentation” in
Khattab & Zaharia (2020)) tend to “re-weigh existing terms based on their importance for matching the query”.
A. A BLATION ON THE F ULL P RE - TRAINING D ATASET
In Table 16, we compare the proposed cross-modal late interaction loss with the original CLIP loss
Radford et al.
(2019) on the full pre-training dataset introduced in Section 3.3.
CLIP is the results reported by CLIP paper, CLIP rep is our reproduced CLIP version with the its original contrastive loss using exactly the same architecture on the same pre-training dataset as FILIP base .
As can be seen, the FILIP base has 6.7 points higher average accuracy than the CLIP rep over 12 datasets.
This further veriﬁes that the performance gain come from the proposed cross-modal late interaction, rather than the data or architecture.
A. I NFERENCE T IME OF I MAGE - TEXT R ETRIEVAL
Setting.
In this section, we test the inference time of both image retrieval and text retrieval on the test set of Flickr30K and MSCOCO.
We compare our proposed model FILIP large against SCAN (Lee et al., 2018) and CLIP (ViT-L/14)
(Radford et al., 2021) .
We test the inference time of CLIP and SCAN using their released code.
For image retrieval, we precompute the image features and report the inference time for one text query, which contains contains (i) the time to extract the feature of one text query, and (ii) the time of similarity calculation with all images and ranking.
Similarly, for text retrieval, we precompute the text features and report the inference time for one image query, which contains (i) the time to extract the feature of one image query, and (ii) the time of similarity calculation with all texts and
[Figure]
C
I F
A R
C
I F
A R
C a lt ec h
S t a n f o r d
C a r s
F l o w e r s
F ood
S
UN
D T
D
[Table]
P
[Table]
[Table]
Results.
The inference time of retrieval is shown in Table 17.
Beneﬁtting from the efﬁciency optimizations (i.e., FP16 quantization and reduced feature dimension) in Section 3.1, the inference time of FILIP is close to CLIP.
In image retrieval, SCAN is slightly faster than FILIP on Flickr30K with 1000 images, because SCAN uses a lightweight GRU as text encoder.
However, SCAN is but much slower than FILIP (i.e., about 17ms slower per query) on MSCOCO with more (i.e., 5000) images because of the slower computation involved in the two-stage stacked cross-attention when computing the similarity.
For text retrieval, SCAN is much slower than FILIP and its own image retrieval, mainly due to three reasons: (i) the image encoder is a Faster RCNN which is more expensive than the lightweight GRU text encoder; (ii) the text candidates are 5 times more than the image candidates in image retrieval; and (ii)
The similarity computation of SCAN relies on the cross-attention computation, which is straightforward to be paralleled, even in their ofﬁcial code; while our FILIP’s similarity computation is simply a matrix multiplication and is readily optimized for most modern hardwares.