PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretraining
Abstract
Large-scale vision-language pre
-training has achieved promising results on downstream tasks.
Existing methods highly rely on the assumption that the image-text pairs crawled from the Internet are in perfect one-to-one correspondence.
However, in real scenarios, this assumption can be difficult to hold: the text description, obtained by crawling the affiliated metadata of the image, often suffers from the semantic mismatch and the mutual compatibility .
To address these issues, we introduce PyramidCLIP, which constructs an input pyramid with different semantic levels for each modality, and aligns visual elements and linguistic elements in the form of hierarchy via peer-level semantics alignment and cross-level relation alignment .
Furthermore, we soften the loss of negative samples (unpaired samples) so as to weaken the strict constraint during the pre-training stage, thus mitigating the risk of forcing the model to distinguish compatible negative pairs.
Experiments on five downstream tasks demonstrate the effectiveness of the proposed PyramidCLIP.
In particular, with the same amount of 15 million pre-training image-text pairs, PyramidCLIP exceeds CLIP on ImageNet zero-shot classification topaccuracy by 10.6%/13.2%/10.0% with ResNet50/ViT-B32/ViT-B16 based image encoder respectively.
When scaling to larger datasets, PyramidCLIP achieves the state-of-the-art results on several downstream tasks.
In particular, the results of
PyramidCLIP-ResNet50 trained on 143M image-text pairs surpass that of CLIP using 400M data on ImageNet zero-shot classification task, significantly improving the data efficiency of CLIP.
Introduction
Recently, vision-language pre-training (VLP) has achieved great success, which aims to improve the accuracy of downstream vision-language tasks by pre-training a model on large-scale image-text pairs harvested from the web without any manual annotation.
The mainstream VLP methods can be roughly categorized into two paradigms, single-stream (1; 2; 3; 4; 5) and dual-stream (6; 7; 8; 9).
Compared to the single-stream counterpart, the dual-stream paradigm decouples the image encoder and text encoder and extracts features for images and texts respectively.
Due to its simplicity and flexibility for downstream applications, the dual-stream paradigm dominates.
The representative dual-stream model
CLIP (6) performs contrastive vision-language pre-training on 400M image-text pairs collected from the Internet, which achieves astounding results.
Later, DeCLIP (10) and FILIP (8) improve CLIP by introducing more supervisions, and bringing in fine-grained cross-modal interaction.
Although existing CLIP-alike methods have achieved promising results on downstream tasks, they strongly rely on the assumption that the image-text pairs are of high quality, i.e. , the pairs are in perfect one-to-one correspondence and have no correlation with other unpaired samples.
However, this ideal assumption is hard to satisfy in practice as shown in Figure 1.
Firstly, semantic mismatch
[Figure]
between the visual modality and linguistic modality exists, e.g. , (a) Caption Redundancy: the affiliated text description is redundant and contains irrelevant information;
(b) Image Redundancy: the Regionof-Interest (ROI) corresponding to the text is only a sub-region of the image; and (c)
Cast Deficiency: text misses the descriptions of main objects in the image, while visual modelling needs to reason about the relationship among salient instances.
Secondly, captions might be compatible to some extent among pairs, as illustrated in
Figure 1(d).
Existing methods directly treat other pairs as negative samples regardless the potential correlation, which may confuse the model.
In order to tackle the issues mentioned above, we propose PyramidCLIP in this paper, which attempts to align image-text pairs more precisely in the form of hierarchy.
PyramidCLIP constructs two input pyramids with different semantic levels at both sides of the dual-stream network, i.e. , the global image, local image region, and
ROI features of the salient objects in the image for visual modelling; text summarization, the original caption and categories with attribute descriptions of salient objects for linguistic modelling.
Then we contrast visual elements and linguistic elements via peer-level semantics alignment and cross-level relation alignment , tackling the issues of mentioned in (a), (b) and (c).
Specifically, for peer-level semantics alignment, since the global image and text summarization both contain global semantic information, and the local region and original caption both contain more fine-grained semantic information, they are treated as two pairs of positive samples.
For cross-level relation alignment, to avoid the modelling of object relationship being overwhelmed by the semantics modelling, we explicitly align the fine-grained object relation with cross-layer elements in another modality.
Moreover, for the issue of the mutual compatibility , we soften the loss term of the negative unpaired samples during the contrast process to ease the strict constraint, alleviating the negative effect of unpaired similarities.
Extensive experiments demonstrate the effectiveness of our proposed PyramidCLIP.
For fair comparison, when trained on YFCC15M-V2 (10) dataset, with ResNet50 (11) / ViT-B32 (12) / ViT-B16 (12) as the image encoder and Transformer as the text encoder, our model achieves the state-of-the-art (SoTA) zero-shot classification on ImageNet (13) with 47.8% / 46.0% / 50.7% top-1 accuracy.
In comparison, the CLIP achieves 37.2% / 32.8% / 40.7% respectively.
Furthermore, when scaling to the large-scale dataset, the results of PyramidCLIP achieve SoTA on several downstream tasks, in particular, the results of PyramidCLIP-ResNet50 trained on 143M image-text pairs surpass that of
CLIP trained on 400M data on ImageNet zero-shot classification task, improving the data efficiency of CLIP significantly.
Our main contributions are summarized as follows: ( i )
We propose PyramidCLIP for more accurate image-text alignment for vision-language model pre-training, which effectively constructs two input pyramids at both sides of the visual encoder and linguistic encoder, and then align the visual elements and linguistic elements via peer-level semantics alignment and cross-level relation alignment.
( ii )
We soften the loss term of negative samples during the contrast process to ease the strict constraint, so as to alleviate the negative effect caused by local similarities.
( iii ) Extensive experiments demonstrate the effectiveness of PyramidCLIP, which achieves SoTA on several downstream tasks.
Related Work
Vision-Language Pre-training Vision-Language Pre-training (VLP) learns a strong joint representation between two modalities by pre-training models on large-scale image-text pairs.
In terms of the model architecture, the mainstream VLP models can be divided into two types: single-stream and dual-stream.
The former one uses a single transformer to model both image and text representations in a unified semantic space by concatenating image and text input embeddings, including VisualBERT (2), UNITER (1), UNICODER (4), OSCAR (3) and UNIMO (14).
The latter one encodes images and texts separately with decoupled image encoder and text encoder, such as ViLBERT (15),
LXMERT (9), ALIGN (7), CLIP (6), and DeCLIP (10).
From a different perspective, the pre-training objective mainly comprises two categories: image-text contrastive learning and masked token tasks based on Language Modeling (LM).
Among the methods mentioned above, UNIMO, ALIGN, CLIP and DeCLIP adopt contrastive learning to align the textual and visual representation in a unified semantic space.
In contrast, VisualBERT, UNITER, LXMERT and UNICODER use masked token tasks, including Masked Language/Region Modeling and autoregressive LM.
In this paper, we employ dual-stream architecture and contrastive learning for simplicity, flexibility, and relatively cheaper computation cost.
Fine-grained Alignment Due to the semantic gap between image and text, there may be some troubles in directly performing the alignment between these two modalities.
For example, some words or phrases in the descriptions may be irrelevant to the images, or the corresponding descriptions of the objects in an image may not always be available in the caption.
Thus, finer-grained alignment is indispensable, as it can provide more accurate and richer supervision signals of multiple granularity, improving the performance significantly.
FILIP (8) improves the contrastive objective to achieve finer-level alignment by using a token-wise maximum similarity between visual and textual tokens.
Methods in (3; 16; 17; 18) construct multi-level semantic concepts for finer-grained alignment.
OSCAR (3) first introduces multi-level semantics, capturing object region features and the corresponding tags with a pre-trained object detector, then concatenates text, object tags and region features together to learn the joint representations.
VinVL (16) enhances the visual representations of OSCAR by pre-trianing a more powerful object-attribute detector.
Both OSCAR and VinVL form the multi-level semantics only in the visual modality.
MVPTR (17) and X-VLM (18) obtain their multi-level semantics concepts in both visual and linguistic modalities.
MVPTR limits the interaction between object tags and textual tokens, and learns the object-tag alignment in an explicit manner.
It also models the nested property of language by learning phrase-level semantics.
X-VLM learns multi-level alignments by positioning vision concepts using given texts, and makes alignments between these two parts.
However, in addition to the image encoder and the text encoder, the two methods both have an additional cross-modal encoder, bringing computation overhead.
In this paper, we follow the dual-stream design of CLIP and construct three visual semantics levels and three linguistic semantics levels to form our PyramidCLIP.
Different from methods mentioned above, each level is input to the corresponding encoder individually, without concatenating.
The obtained three visual representations and three linguistic representations are used to compute six contrastive loss terms, which helps to achieve multi-level alignments.
Method
In this section, we introduce the proposed PyramidCLIP for more accurate alignment of image and text for vision-language model pre-training.
3.1 Overall Architecture
The entire framework of the proposed PyramidCLIP is presented in Figure 2.
PyramidCLIP is a dual-stream network, including a text encoder h and an image encoder f = f 2 ◦ f 1 , where f 1 and f 2 denote the front part and the rear part of the image encoder respectively.
Each encoder consists of a linear projector and a normalization operator in the end, which project the final class token into a unified dimension and then normalize it, obtaining the corresponding visual or linguistic representation vector in the same embedded space.
During the training, for each image-text pair ( I, T ) , the image I is transformed into two views, i.e. , local view L and global view G , through random crop with different ratios.
And text T is input to a summary extractor (19) to generate text summarization T S with higher level semantics.
The image global view G and text summarization T S both capture global context information, while the image local view L and original text T contain more detailed information.
Therefore, G and T S are regarded as a pair of positive samples, while L and T are regarded as another pair of positive samples, denoted as ( G, T S ) and ( L, T ) .
These two pairs are input to the dual-stream encoder to extract global and local representation pairs, ( v g , l s ) and ( v l , l t ) , where v g = f ( G ) , l s = h ( T S ) , v l = f ( L ) and l t =
h ( T ) .
Finally, ( v g , l s ) and ( v l , l t ) are pulled together through contrastive learning lossesand 2 respectively (refer to Figure 2), and other samples in the same batch are treated as negative samples.
We term this contrasting process as Peer-level Semantics Alignment .
Furthermore, in order to explicitly model the relationship between salient objects in the image, the
ROI feature sequence R = { o 1 , o 2 , ..., o M } of M detected salient objects in the image
I , with the category and attribute information for each object, are extracted through a pre-trained object-attribute detector.
Then a linear embedding module φ is used to transform R into the latent dimension of Multi-head Self-attention (MHSA) layer (12) in the image encoder.
The feature sequence is successively fed into the rear part f 2 , which contains one or more MHSA layers, to adaptively capture the relation between these salient instances, generating the final representation vector v r , i.e. , v r = f
( φ ( R )) .
For the object categories with attributes, we join them together, constructing another text description T OA , to provide a more granular, comprehensive and accurate caption for the image.
Then we feed it into the text encoder, generating the representation vector l a = h ( T OA ) .
To avoid the relation modelling being overwhelmed by the context semantic modelling, we have ( v g , l a ), ( v r , l s ), ( v l , l a ) and ( v r , l t ) as another four positive pairs, and the distances between which are narrowed through contrastive learning losses 3 , 4 , 5 and 6 respectively, termed as Cross-level Relation Alignment .
It is worth noting that at the inference stage, only the original image-text pair ( I, T ) is used, i.e. , the visual representation v i from I and the linguistic representation l t from T .
3.2 Peer-level Semantics Alignment
Now we present the details of the peer-level semantics alignment .
As mentioned above, the dualstream vision-language contrastive learning methods such as CLIP strongly rely on the assumption that the image-text pairs are of good quality of one-to-one correspondence.
However, semantic mismatch between images and text captions often occurs in the automatically harvested data.
Therefore, we construct an input pyramid with multi-level semantics on both sides of the dual-stream network, and then align image and text within the same semantics level.
Specifically, the image I is transformed to the global view G and the local view L through two random crops with different ratios.
For the text caption, besides the original caption T , text summarization T S with more compact semantics is extracted using a pre-trained text summarization extractor.
Coarse-grained Global Contrast We set the random crop ratio for generating global view G to be [0 . 9 , 1] , which basically contains all the information in the original image.
Text summarization T S condenses the original caption T , removing some redundant and overly detailed information in the T .
G and T S both capture global information and can be used as pairs of positive samples.
The projected embedding v g and l s of G and T S are pulled closer through contrastive learning.
Fine-grained Local Contrast Since the alignment of the global view G with the text summarization
T S described above is relatively coarse, finer-grained information is largely discarded.
Intuitively, some image sub-regions can be aligned with the original caption T .
To this end, we introduce fine-grained local contrast.
We set the random cropping ratio for generating local view L to be [0 . 5 , 1] , which focuses on the sub-region of the image I .
Then the projected embeddings v l and l t of
L and T are also brought together through contrastive loss (refer to Section 3.4).
Naturally, we have also tried to bring the finer-grained, peer-level ( v r , l a ) closer, but there is no further gain (see Appendix F).
3.3 Cross-level Relation Alignment
To further improve the alignment precision, we introduce the ROI feature of each salient object in the image and the corresponding description with category and attributes, using a pre-trained objectattribute detector, as the most fine-grained semantic level to provide more accurate supervisions.
Specifically, given an image I with M salient objects, the extracted visual semantics of each object region is formulated as [ o ′ m , z m ] , where m denotes the m th object, o ′ m is a 2048-dimensional feature vector and z m is a 4-dimensional normalized position vector indicating the coordinates of top-left and bottom-right corners.
By concatenating o ′ m and z m , we have the 2052-dimensional position-sensitive
ROI feature vector o m , forming the ROI feature sequence R = {
o 1 , o 2 , ..., o M } with the order organized from high confidence to low.
Then R ∈ R M × 2052 is transformed into R M × d using the projector in the embedding module φ , where d indicates the latent dimension of the MHSA layers in the image encoder.
A randomly initialized d -dimensional class token is additionally appended at the front, resulting F ∈ R (1+ M ) × d , which is further feed into the rear part f 2 of the image encoder to compute the normalized ROI relation embedding v r , i.e. , F = φ ( R ) and v r = f 2 ( F ) .
Note that feature sequence F is position-sensitive following R .
Thus, positional embedding is not applied before it enters into the MHSA layer.
Meanwhile, the detected names of category and attributes for each salient object form a phrase with one or more adjectives (attributes) modifying a noun (category), like “ gold round brown table ” in Figure 2.
Then all the M phrases from M salient objects are joint into a text description T OA with the same order as the ROI feature sequence, and the phrases are separated by commas.
Next, T OA is input to the text encoder to obtain the embedding l a = h ( T OA ) .
To enhance the relation modelling capacity of the text encoder, while avoiding weakening the reasoning ability of the image encoder, ( v g , l a ), ( v r , l s ), ( v l , l a ) and ( v r , l t ) are used as another four positive pairs, and the distance between each pair is minimized simultaneously.
Since the object-level inputs R and T OA are direct concatenation of feature vectors and phrases respectively, hence very fine-grained, while other inputs G , L , T S and T are complete images or sentences, we term this contrasting process cross-level relation alignment .
In the case that the visual model is a convolution neural network (CNN), the traditional pooling layer is replaced by attention pooling, which actually is a MHSA layer.
So the embedded ROI feature sequence F is input to the attention pooling layer, i.e. , f 2 , which indicates the final attention pooling layer, as shown in Figure 3(a).
For the transformer-based visual model (ViT), the sequence F can be directly input to a transformer layer.
Considering that F already encodes high-level visual semantics, we feed it into the rear part f 2 of the ViT encoder, see Figure 3(b).
Besides, as pointed out in (20; 21), the standard ViT may not fully leverage the local context information, which limits the visual representation capacity of ViT-based image encoder.
Following (21), we incorporate a depth-wise convolution into the Feed-Forward module of ViT, termed Locally-enhanced
Feed-Forward (LeFF), improving the patch-level local perception and interaction.
The structure of
LeFF is shown in Figure 3(c).
First, the patch tokens are projected into a higher dimension through a linear projection layer and reshaped.
Next, a 3 × 3 depth-wise convolution is utilized to capture the local information.
Then the feature maps are flattened to a token sequence and re-projected into the initial dimension.
While the class token is identical during the process and is concatenated with locally-enhanced patch tokens, generating the final output.
As depicted in Figure 3(b), LeFF is only applied in the front part
f 1 of the ViT-based image encoder, since it is clearly not suitable for the embedded ROI feature sequence F .
In Figure 3(b), L denotes the total number of transformer layers in ViT and L s is the number of transformer layers with LeFF in f 1 .
And L = 12 and L s = 9 in the experiments.
The influence of different settings of L s can be seen in Appendix F.
3.4 Softened Objective Function
For a batch of N image-text pairs { ( I i , T i ) } Ni =1 , where i indicates the i th pair, the normalized embedded vectors { v gi , v li , v ri , l si , l ti , l ai } Ni =1 of the same dimension are obtained by the dual-stream encoders.
In this formulation, v gi , v li and v ri are generated by the image encoder from global-crop image G , local-crop image L and ROI feature sequence R respectively, while l si , l ti and l ai are generated by the text encoder from text summarization T S , original text T and object-attribute description T OA respectively.
Then we use this vector group to construct six supervision signals L
GS , L LT , L GA , L RS , L LA and L RT for in-batch contrastive learning, which can be calculated with { ( v gi , l si ) } Ni =1 , { ( v li , l ti ) } Ni =1 , { ( v gi , l ai ) } Ni =1 , { ( v ri , l si ) } Ni =1 , { ( v li , l ai ) } Ni =1 and { ( v ri , l ti ) } Ni =1 respectively.
Our six contrastive losses, with the formulation of InfoNCE (22), are designed to achieve the alignment between visual representation and linguistic representation from disparate semantic levels.
Take the first loss term L GS from { ( v gi , l si ) } Ni =1 as an example.
For the i th pair, the normalized vision-to-language similarity p vi ( G ) =
{ p vij ( G ) } Nj =1 and language-to-vision similarity p li ( T S ) = { p lij ( T S ) } Nj =1 can be calculated through:
[Equation]
where τ is a learnable temperature parameter initialized with 0 .
07 and the function sim( · ) conducts dot product to measure the similarity scores.
In standard practice, for the i th pair, the corresponding one-hot label vectors of the ground-truth y vi ( G ) =
{ y vij ( G ) } Nj =
and y li ( T S )
= {
y lij ( T S ) } Nj =1 , with positive pair denoted by 1 and negatives by 0 , are used as the targets to calculate cross-entropy.
This kind of hard targets assumes there is absolutely no similarity between unpaired image and text.
However, within a large-size batch, unpaired image and text may have more or less local similarities, i.e. , some local regions in an image may be matched with some words or phrases in other unpaired texts.
To address this problem for better generalization, we use label smoothing to soften the hard targets.
The corresponding softened targets (cid:101) y vi ( G ) and (cid:101)
y li ( T S ) for the i th pair can be formulated as:
[Equation]
where α is the smoothing hyper-parameter set to 0 .
2 in our experiments.
Then L GS can be written as:
[Equation]
The other loss terms L LT , L GA , L RS , L LA and L RT can be calculated similarly.
We then divide them into three groups that are respectively the peer-level alignment L peer =
( L GS + L LT ) / 2 , the global-relation cross-level alignment L globalcross =
( L GA + L RS ) / 2 and the local-relation cross-level alignment L localcross =
( L LA + L RT ) / 2 .
Therefore, the overall objective function of PyramidCLIP is:
[Equation]
where the loss weights λ and µ are both set to 1 / 3 in our experiments.
Experiments 4.1 Implementation Details and Datasets
Pre-training Stage We experiment on three different architectures, PyramidCLIP-ResNet50/
ViTB32/ViT-B16, according to the choice of image encoder.
Their detailed architecture designs follow that of CLIP (6).
LAION99M contains 99M image-text pairs with the highest similarity selected from
LAION400M (23) according to the similarity scores provided by the producer.
We use the publicly
[Table]
released T5 model (19) to extract text summarization for all texts and utilize an object-attribute detector pre-trained by VinVL (16) to extract salient object features together with category and attribute information in the image.
Please refer to Appendix A for details.
Downstream Tasks We validate the effectiveness of our proposed method on five downstream tasks: zero-shot image classification, zero-shot image-text retrieval, linear probe, object detection and instance segmentation.
For classification, experiments are carried out on 11 datasets, such as ImageNet (13), CIFAR-100 (28).
For image-text retrieval, experiments are conducted on Flickr30K (29) and MS-COCO (30).
For object detection and instance segmentation, the proposed method is verified on PASCAL VOC (31) and MS-COCO.
More details can be found in Appendix B.
4.2 Fair Comparison against SoTA
[Table]
We first compare our method against other SoTA approaches on ImageNet zero-shot classification task using the same amount of pre-training data YFCC15M-V2 and the results are shown in Table 2.
It can be seen that compared to CLIP, PyramidCLIP improves the top-1 accuracy by 10.6% / 13.2% / 10.0% when the visual model is
ResNet50 / ViT-B32 / ViT-B16 respectively.
Furthermore,
PyramidCLIP outperforms all other SoTA approaches by a large margin.
In addition, since the distribution of different datasets can vary vastly, we also conduct experiments on YFCC15M-V1 and LAION15M, which is obtained by sampling 15 million image-text pairs from
LAION99M for fair comparison.
The results can be seen in Appendix C and our PyramidCLIP still shows great superiority.
4.3 Comparison on Large-scale Datasets
We further validate the effectiveness of our method on a large-scale dataset, i.e. , 143M image-text pairs, and downstream zero-shot image-text retrieval and image classification results are shown in
Table 3.
It can be seen that on image-text retrieval task, PyramidCLIP exceeds CLIP trained on 400M data and DECLIP by a large margin.
And on ImageNet classification task, with the same amount of pre-training data, PyramidCLIP significantly exceeds the results of CLIP by 6.1%/3.8%/3.5% using ResNet50/ViT-B32/ViT-B16 as image encoder.
Furthermore, it is worth noting that, when the vision model is ResNet50, PyramidCLIP trained on 143M data surpass CLIP using 400M data, which greatly improves data efficiency.
4.4 Transferability to Small-scale Classification Datasets
In this section, we validate the transferability of our method on 10 relatively small downstream classification datasets, both on zero-shot and linear probe tasks.
The results are shown in Table 4.
It can be seen that the average accuracy of PyramidCLIP on 10 datasets all exceed CLIP trained on 400M data on two kinds of tasks.
It is worth noting that our pre-training data is less than 36% of
CLIP, but the average accuracy is better, indicating higher data utilization.
4.5 Transferability to Object Detection and Instance Segmentation
In order to verify that our model can better exploit the relationship between objects in the image, we further validate our models on object detection and instance segmentation tasks.
Specifically, we take the visual model ResNet50 to initialize the backbone of Faster R-CNN and Mask R-CNN (34) and then all the parameters are fine-tuned.
The results are shown in the Table 6.
It can be seen that our model significantly outperforms random initialization and surpasses CLIP and DECLIP model.
[Table] [Table]
4.6 Ablation Study
[Table]
In this section, we verify the effectiveness of each component in PyramidCLIP on downstream zeroshot ImageNet classification task, and all the experiments are pre-trained for 8 epochs on YFCC15M-V1.
The results are listed in Table 5, which indicate that on the basis of the peer-level alignment, all the other components including cross-level global-relation and local-relation alignment, LeFF in ViT and softened objectives can bring significant gains individually.
More ablation results can be seen in Appendix F, including the detailed ablation on L s , the choice of peer-level alignment and the effectiveness each component on downstream object detection task.
4.7 Visualization
Semantic Features We utilize t -SNE (35) to visualize the learned semantic features of CIFAR-10 (28).
Each text feature of 10 categories is obtained using the ensemble of 80 prompt templates.
And image features are extracted with ResNet50 visual encoder.
As depicted in Figure 4, CLIP pre-trained on 143M data has a poor clustering performance, with the visual features of most categories overlapping heavily and the textual features of some two categories being very close, such as ( automobile , truck ) and ( frog , deer ).
Although CLIP pre-trained on 400M data (6) performs better than CLIP on 143M data, the visual features of some categories like dog and cat still have a large overlap.
In comparison, the semantic features extracted by PyramidCLIP on 143M data, are well separated with each textual feature surrounded by most visual features of the same category.
[Table]
[Figure]
Grad-CAM Heatmaps Grad-CAM (36) is also utilized to help understand why PyramidCLIP outperforms CLIP.
Specifically, we conduct this through text-to-image retrieval on MS-COCO, using
CLIP/PyramidCLIP-ResNet50 pre-trained on 143M data.
For each query text, the model retrieves top5 images with highest similarities.
Then for each of the five retrieved images, we use Grad-CAM to find which areas have the highest activation to the query text.
As shown in Figure 5, PyramidCLIP has better retrieval performance than CLIP.
Moreover, PyramidCLIP can accurately capture the complete object regions that are highly matched with the query texts, while CLIP captures regions with either components missing or additional noises, or even obtains completely irrelevant matches.
More visualization results can be found in Appendix G.
Conclusion
In this paper, we have proposed a hierarchical pre-training method, termed PyramidCLIP, to achieve improved alignment between visual and linguistic modalities.
It resolves the issue that the weblycrawled data is not in perfect one-to-one correspondence by explicitly constructing pyramidal semantic inputs at the both sides of dual-stream network.
We also show that softened peer-level semantics alignment and cross-level relation alignment can interact between two modalities and are beneficial.
PyramidCLIP achieves the state-of-the-art results on five downstream tasks, which demonstrates the superiority.
[Table]
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[No] (c) Did you discuss any potential negative societal impacts of your work?
[No] Our work does not have any potential negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] Our codes may be released in the future.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Appendix A. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] We did not run the same experiment for multiple times.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix A. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[No] (c) Did you include any new assets either in the supplemental material or as a URL?
[No] Our codes may be released in the future.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[No] Existing assets used in our work are all publicly released.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[No] Datasets used in our work are all publicly released without this problem.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Pre-training Stage Settings A.1 Datasets
In Section 4.1, we list all the subsets of the total 143M dataset, and here we introduce them in detail.
SBU (24) is a relatively small image-text dataset, which contains 1 million image-text pairs obtained from Flickr.
YFCC15M (27) is a commonly used subset of YFCC100M (27) and there are mainly two versions of YFCC15M, V1 and V2.
YFCC15M-V1 is obtained by applying the same filtering rule on
YFCC100M as CLIP (6), while YFCC15M-V2 is collected by DeCLIP (10) with a different filtering strategy.
In addition to a subset of YFCC, the V2 dataset also contains some additional data crawled from the Internet and is of higher quality than V1.
CC3M (25) and CC12M (26) conduct the same image-text filter pipeline on Internet webpage sources, and the difference is that the filtering method of the latter is more relaxed.
LAION400M (23) is one of the largest openly available image-text datasets.
We rank the image-text pairs by their similarity scores, which are pre-computed by the producer using a pre-trained CLIP model, and pick up a 99M subset with the highest similarity scores.
Combining all the above datasets, we finally have a dataset of 143M image-text pairs.
During the training process, we randomly shuffle the training sequence of datasets at the beginning of each training epoch, then train our model on these datasets one by one.
A.2 Implementation Details
Model Architectures We follow the same architecture design as CLIP (6) for PyramidCLIPResNet50/ViT-B32/ViT-B16, except that we incorporate a depth-wise convolution into the FeedForward module of ViT, namely, LeFF.
The input resolution of image encoder is 224 × 224 and the maximum context length of text encoder is 77.
The final image and text features are projected to the same dimension, which is 1024 for PyramidCLIP-ResNet50 and 512 for PyramidCLIP-ViT, followed by L2 normalization before interaction.
Details of the Object-attribute Detector The object-attribute detector, adopting the framework of
Faster R-CNN (37), is pre-trained by VinVL (16).
And image is resized to the resolution of 640 ×before entering the detector.
We take 10 objects with the highest confidence from the detector to obtain the corresponding ROI features and category descriptions with attribute information.
Pre-training Setup We train our PyramidCLIP using an AdamW (38) optimizer and the cosine learning rate scheduler with a linear warmup.
Specifically, the learning rate linearly increases fromto the peak value within 10% of the total steps, and then decreases with a cosine anneal strategy.
The weight decay rate of AdamW is set to 0.2.
To save GPU memory, automatic mixed-precision (39) is used.
The models are trained from scratch for either 8 or 32 epochs in our experiments, i.e. , 8 epochs for ablation and 32 epochs for comparison.
When training on 15M datasets, including YFCC15M-V1,
V2 and LAION15M, the batch size is set to 4096 and the peak learning rate is set to 2 × 10 − 3 .
While on the 143M large-scale dataset, the batch size is set to 8192 and the peak learning rate is set to 5 ×
10 − 4 .
Besides, PyramidCLIP-ResNet50/ViT-B32 takes 64 V100 GPUs to train on 143M data, while PyramidCLIP-ViT-B16 takes 128 A100 GPUs.
B Downstream Settings B.1 Datasets for Downstream Classification Task
Besides ImageNet (13), we also evaluate the transferability of our model on other 10 downstream classification datasets including Oxford-IIIT Pets (40), CIFAR-10 (28), CIFAR-100 (28), Describable
Textures (41), Stanford Cars(42), Food-101 (43), Oxford Flowers 102 (44), FGVC Aircraft (45),
SUN397 (46) and Caltech-101 (47).
The details of each dataset are listed in Table 7 and we follow the same data split and evaluation metric as CLIP for fair comparison.
B.2 Implementation Details
Downstream Zero-shot Image Classification Due to the fact that the labels of common classification datasets are mostly nouns rather than natural language descriptions, we adopt the same prompt setting as used in CLIP, that is, for every single class name, we generate 80 different textual descriptions
[Table]
with 80 prompt templates (such as “a photo of label”).
The ensembles of these textual representations are used in computing similarities between images and label names.
Downstream Zero-shot Image-text Retrieval
The image-text retrieval task can be split into two sub-tasks,
i
[Table]
e. , image retrieval and text retrieval, according to the target modality.
We evaluate the zero-shot image-text retrieval capabilities on Flickr30K and MS-COCO dataset, which is performed by ranking image-text pairs according to their similarity scores.
Downstream Image Linear Probe To implement linear probe evaluation, we follow CLIP (6) to train a logistic regression classifier on the frozen visual features extracted by the image encoder.
Specifically, we train the logistic regression classifier using L-BFGS algorithm provided by scikitlearn with maximum 1,000 iterations, and report the corresponding metric for each dataset.
Moreover, the L2 regularization strength C is determined using hyperparameter sweep on the validation sets.
Downstream Object Detection and Instance Segmentation Following (48; 49), for the downstream object detection and instance segmentation tasks, all the parameters are fine-tuned.
For detection task on PASCAL VOC, the detector is trained for 24k steps with a batch size of 40, and the initial learning rate is 0.02 with 100 warm-up iterations and decays by 10 at 18k, 22k steps.
The scale of image is randomly sampled from [480 , 800] with interval 32 during training and is set to 800 for inference.
For the detection and instance segmentation on COCO, the model is trained for 90k iterations with the initial learning rate 0.02, and the scales of images are randomly sampled from [600 , 800] during training and is also set to 800 for inference.
Downstream Image End-to-end
Fine-tuning
In addition to the five downstream tasks mentioned in the main text, following MAE (50), we also conduct the end-to-end fine-tuning experiments on downstream classification task.
Here, we elaborate the implementation details of this setting, and the corresponding results can be seen in Appendix E.
For downstream end-to-end fine-tuning, we first fine-tune the classifier layer alone while freezing the others for 8 epochs to endow the model a proper initialization.
Then we fine-tune all the parameters in the usual manner.
AdamW is used as optimizer, and the learning rate is set to 1e-3 and adjusted by a cosine learning rate scheduler without warmup.
We totally fine-tune the model for 128 epochs, including the initialization stage.
C Fair Comparison on YFCC15M-V1 and LAION15M
In this section, we compare our PyramidCLIP against CLIP under the same amount of pre-training data, i.e. , YFCC15M-V1 and LAION15M.
The results on downstream zero-shot image-text retrieval task and ImageNet classification are shown in Table 8 and Table 9.
It can be seen that PyramidCLIP surpasses CLIP regardless of the distribution of pre-training dataset, demonstrating the superiority of our pre-training method.
Furthermore, we validate the effectiveness of proposed method on downstream object detection and instance segmentation tasks, and the results are shown in Table 10.
It can be seen that the weights of our model exceeds that of CLIP model on both object detection task and instance segmentation task.
Noting, on object detection task, the improvement on PASCAL VOC is more obvious than that on MS-COCO, since the amount of PASCAL VOC is smaller than COCO and the effect of weight initialization is more conspicuous.
[Table]
[Table] [Table]
[Table]
D Downstream Task: Linear Probe
We first validate the transferability of our method on downstream classification task via linear probe.
The results are exhibited in Table 11.
It can be seen that when the image encoder is ViT-B/32, the average accuracy of PyramidCLIP pre-trained on 143M data exceeds that of CLIP using 400M data.
Furthermore, regardless of the image encoder used, our method outperforms CLIP on more than half of the small datasets, noting that the amount of pre-training data we used is only about 36% of that used by CLIP, further demonstrating the effectiveness of the proposed method.
E Downstream Task: End-to-end Fine-tuning
On the basis of linear probe, we further validate the transferability of our method via end-to-end finetuning.
The results are shown in Table 12.
We compare PyramidCLIP against CLIP and supervised counterpart.
It can be seen that PyramidCLIP pre-trained on 143M data exceeds both CLIP and supervised ResNet50.
Also, it is worth noting that compared to CLIP, we use only 36% pre-training
[Table] [Table]
[Table]
F More Ablation
In this section
[Table]
we supplement the ablation study of some important components in PyramidCLIP.
All the ablation experiments are conducted on YFCC15M-V1 and trained for 8 epochs.
F.1 Supplementary Ablation of PyramidCLIP Components
Ablation of Each Component on Other Downstream Tasks In Section 4.6, we only provide ablation study of each component on ImageNet zero-shot classification.
Here we list the corresponding ablation results on MS-COCO zero-shot image-text retrieval and PASCAL VOC object detection in Table 13, which indicate that on the basis of the peer-level alignment, all the other components in PyramidCLIP can still bring accuracy improvement individually on the two downstream tasks.
[Table]
✓ - 36.4(+7.9) 21.1(+4.5) 47.4(+1.7) 76.0(+1.7)
ViT
[Table]
B/ ✓ 24.
- - ✓ ✓ 27.9(+3.6) 16.0(+2.6) - - ✓ ✓ ✓ 29.3(+5.0) 17.7(+4.3) - - ✓ ✓ ✓ ✓ 31.4(+7.1) 18.2(+4.8) - - ✓ ✓ ✓ ✓ ✓ 31.7(+7.4) 18.8(+5.4) - -
The Influence of Different L s Settings We further probe into the influence of the transformer layers L s in the front part of ViT-based image encoder on zero-shot ImageNet classification task.
Note that the total number of transformer layers L in ViT is 12.
The corresponding results with different L s values are shown in Figure 6.
It can be found that L s = 9 achieves the best result, hence L s is set to 9 in our experiments.
Besides, L s = 0 represents that the feature sequence F is feed into the first transformer layer of ViT and all the 12 layers are without LeFF.
While
L s = 12 indicates that all the 12 layers are with LeFF and the raw sequence
F is directly input to the final projector without being processed by transformer, which is the reason why L s = 12 shows such poor performance.
F.2 Ablation of Other Possible Alignments
In addition to the current six loss terms of PyramidCLIP described in the main text, i.e. , the peer-level semantics alignment L GS and L LT , the global-relation cross-level alignment L GA and L RS , and the local-relation cross-level alignment L LA and L RT , there are three other possible losses that are L RA , L
GT and L LS , corresponding to ( v r , l a ) , ( v g , l t ) and ( v l , l s ) respectively, as depicted in Figure 7(a).
Note that L RA belongs to the peer-level alignment, and L GT and L LS are semantically mismatched.
Among them, L GT is actually the original CLIP loss, shown in Figure 7(b).
We will discuss the influence of these three losses in next two parts and explain why they are not incorporated into our
PyramidCLIP paradigm.
Granular Ablation of Peer-level Alignment In this part, we explore the influence of each subcomponent belonging to the peer-level alignment, including L GS , L LT and L RA , shown in Figure 7(c), and compared with original CLIP loss.
As listed in Table 14, reducing the image random crop ratio of
CLIP can improve the model performance, i.e. , using L LT rather than L GT (see yellow rows), since the local view statistically removes some redundant information in the image, i.e. , some sub-regions not described in the text.
That is why we use the image local view and the original text as a pair to construct the peer-level local contrast.
Besides, it can be seen that the addition of global contrast L GS also brings significant improvement.
However, there is no further gain when bringing into L RA (see blue rows), hence it is not included in PyramidCLIP.
And we attribute this to the inherent precise alignment between the ROI feature sequence R and the object-attribute description T OA , since the feature and category with attributes of each salient object in the image are extracted in pairs by the pre-trained powerful object-attribute detector.
Ablation of Semantically Mismatched Alignment On the basis of PyramidCLIP paradigm, we further study on the additional effect of L GT and L LS , termed as semantically mismatched alignment, shown in Figure 7(d).
The ablation results are listed in Table 15, which reveal that adding semantically mismatched alignment cannot bring stable benefits, even degrades the performance in most cases.
Therefore L GT and L LS are not attached to the PyramidCLIP paradigm, which is consistent with our motivation, i.e. addressing the semantics mismatch problem.
[Table] [Table]
G More Visualizations
In this section, more Grad-CAM heatmaps are visualized through text-to-image retrieval on MSCOCO.
We utilize codes provided in (51) to implement Grad-CAM visualization.
As shown in
Figure 8, CLIP is more likely to focus on the scene or background areas in the images corresponding to the scene descriptions in query texts, while PyramidCLIP pay more attention to salient objects, which benefits from the introduce of cross-level relation alignment.
For example, in Figure 8(b),
CLIP focuses on areas corresponding to " green field " in the query text, while PyramidCLIP on " horse ".
In Figure 8(d), CLIP focuses on areas corresponding to " mountain ", while PyramidCLIP on " skier with a red jacket ".
And the same phenomenon can also be seen in Figure 8(a)(c).