PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretraining
Abstract
Large-scale vision-language pre
-training has achieved promising results on downstream tasks.
Existing methods highly rely on the assumption that the image-text pairs crawled from the Internet are in perfect one-to-one correspondence.
However, in real scenarios, this assumption can be difficult to hold: the text description, obtained by crawling the affiliated metadata of the image, often suffers from the semantic mismatch and the mutual compatibility .
To address these issues, we introduce PyramidCLIP, which constructs an input pyramid with different semantic levels for each modality, and aligns visual elements and linguistic elements in the form of hierarchy via peer-level semantics alignment and cross-level relation alignment .
Furthermore, we soften the loss of negative samples (unpaired samples) so as to weaken the strict constraint during the pre-training stage, thus mitigating the risk of forcing the model to distinguish compatible negative pairs.
Experiments on five downstream tasks demonstrate the effectiveness of the proposed PyramidCLIP.
In particular, with the same amount of 15 million pre-training image-text pairs, PyramidCLIP exceeds CLIP on ImageNet zero-shot classification topaccuracy by 10.6%/13.2%/10.0% with ResNet50/ViT-B32/ViT-B16 based image encoder respectively.
When scaling to larger datasets, PyramidCLIP achieves the state-of-the-art results on several downstream tasks.
In particular, the results of
PyramidCLIP-ResNet50 trained on 143M image-text pairs surpass that of CLIP using 400M data on ImageNet zero-shot classification task, significantly improving the data efficiency of CLIP.
Introduction
Recently, vision-language pre-training (VLP) has achieved great success, which aims to improve the accuracy of downstream vision-language tasks by pre-training a model on large-scale image-text pairs harvested from the web without any manual annotation.
The mainstream VLP methods can be categorized into two paradigms, single-stream (1; 2; 3; 4; 5) and dual-stream (6; 7; 8; 9; 10).
Compared to the single-stream counterpart, the dual-stream paradigm decouples the image encoder and text encoder and extracts features for images and texts respectively.
Due to its simplicity and flexibility for downstream applications, the dual-stream paradigm dominates.
The representative dual-stream model CLIP (6) performs contrastive vision-language pre-training on 400M imagetext pairs collected from the Internet, which achieves astounding results.
Later, DeCLIP (10) and FILIP (8) improve CLIP by introducing more supervisions, and bringing in fine-grained cross-modal interaction.
Although existing CLIP-alike methods have achieved promising results on downstream tasks, they strongly rely on the assumption that the image-text pairs are of high quality, i.e. , the pairs are in perfect one-to-one correspondence and have no correlation with other unpaired samples.
However, this ideal assumption is hard to satisfy in practice as shown in Figure 1.
Firstly, semantic mismatch
[Figure]
between the visual modality and linguistic modality exists, e.g. , (a) Caption Redundancy: the affiliated text description is redundant and contains irrelevant information;
(b) Image Redundancy: the Regionof-Interest (ROI) corresponding to the text is only a sub-region of the image; and (c)
Cast Deficiency: text misses the descriptions of main objects in the image, while visual modelling needs to reason about the relationship among salient instances.
Secondly, captions might be compatible to some extent among pairs, as illustrated in
Figure 1(d).
Existing methods directly treat other pairs as negative samples regardless the potential correlation, which may confuse the model.
In order to tackle the issues mentioned above, we propose PyramidCLIP in this paper, which attempts to align image-text pairs more precisely in the form of hierarchy.
PyramidCLIP constructs two input pyramids with different semantic levels at both sides of the dual-stream network, i.e. , the global image, local image region, and
ROI features of the salient objects in the image for visual modelling; text summarization, the original caption and categories with attribute descriptions of salient objects for linguistic modelling.
Then we contrast visual elements and linguistic elements via peer-level semantics alignment and cross-level relation alignment , tackling the issues of mentioned in (a), (b) and (c).
Specifically, for peer-level semantics alignment, since the global image and text summarization both contain global semantic information, and the local region and original caption both contain more fine-grained semantic information, they are treated as two pairs of positive samples.
For cross-level relation alignment, to avoid the modelling of object relationship being overwhelmed by the semantics modelling, we explicitly align the fine-grained object relation with cross-layer elements in another modality.
Moreover, for the issue of the mutual compatibility , we soften the loss term of the negative unpaired samples during the contrast process to ease the strict constraint, alleviating the negative effect of unpaired similarities.
Extensive experiments demonstrate the effectiveness of our proposed PyramidCLIP.
For fair comparison, when trained on YFCC15M-V2 (10) dataset, with ResNet50 (11) / ViT-B32 (12) / ViT-B16 (12) as the image encoder and Transformer as the text encoder, our model achieves the state-of-the-art (SoTA) zero-shot classification on ImageNet (13) with 47.8% / 46.0% / 50.7% top-1 accuracy.
In comparison, the CLIP achieves 37.2% / 32.8% / 40.7% respectively.
Furthermore, when scaling to the large-scale dataset, the results of PyramidCLIP achieve SoTA on several downstream tasks, in particular, the results of PyramidCLIP-ResNet50 trained on 143M image-text pairs surpass that of
CLIP trained on 400M data on ImageNet zero-shot classification task, improving the data efficiency of CLIP significantly.
Our main contributions are summarized as follows: ( i )
We propose PyramidCLIP for more accurate image-text alignment for vision-language model pre-training, which effectively constructs two input pyramids at both sides of the visual encoder and linguistic encoder, and then align the visual elements and linguistic elements via peer-level semantics alignment and cross-level relation alignment.
( ii )
We soften the loss term of negative samples during the contrast process to ease the strict constraint, so as to alleviate the negative effect caused by local similarities.
( iii ) Extensive experiments demonstrate the effectiveness of PyramidCLIP, which achieves SoTA on several downstream tasks.
Related Work
Vision-Language Pre-training Vision-Language Pre-training (VLP) learns a strong joint representation between two modalities by pre-training models on large-scale image-text pairs.
In terms of the model architecture, the mainstream VLP models can be divided into two types: single-stream and dual-stream.
The former one uses a single transformer to model both image and text representations in a unified semantic space by concatenating image and text input embeddings, including VisualBERT (2), UNITER (1), UNICODER (4), OSCAR (3) and UNIMO (14).
The latter one encodes images and texts separately with decoupled image encoder and text encoder, such as ViLBERT (15),
LXMERT (9), ALIGN (7), CLIP (6), and DeCLIP (10).
From a different perspective, the pre-training objective mainly comprises two categories: image-text contrastive learning and masked token tasks based on Language Modeling (LM).
Among the methods mentioned above, UNIMO, ALIGN, CLIP and DeCLIP adopt contrastive learning to align the textual and visual representation in a unified semantic space.
In contrast, VisualBERT, UNITER, LXMERT and UNICODER use masked token tasks, including Masked Language/Region Modeling and autoregressive LM.
In this paper, we employ dual-stream architecture and contrastive learning for simplicity, flexibility, and relatively cheaper computation cost.
Fine-grained Alignment Due to the semantic gap between image and text, there may be some troubles in directly performing the alignment between these two modalities.
For example, some words or phrases in the descriptions may be irrelevant to the images, or the corresponding descriptions of the objects in an image may not always be available in the caption.
Thus, finer-grained alignment is indispensable, as it can provide more accurate and richer supervision signals of multiple granularity, improving the performance significantly.
FILIP (8) improves the contrastive objective to achieve finer-level alignment by using a token-wise maximum similarity between visual and textual tokens.
Methods in (3; 16; 17; 18) construct multi-level semantic concepts for finer-grained alignment.
OSCAR (3) first introduces multi-level semantics, capturing object region features and the corresponding tags with a pre-trained object detector, then concatenates text, object tags and region features together to learn the joint representations.
VinVL (16) enhances the visual representations of OSCAR by pre-trianing a more powerful object-attribute detector.
Both OSCAR and VinVL form the multi-level semantics only in the visual modality.
MVPTR (17) and X-VLM (18) obtain their multi-level semantics concepts in both visual and linguistic modalities.
MVPTR limits the interaction between object tags and textual tokens, and learns the object-tag alignment in an explicit manner.
It also models the nested property of language by learning phrase-level semantics.
X-VLM learns multi-level alignments by positioning vision concepts using given texts, and makes alignments between these two parts.
However, in addition to the image encoder and the text encoder, the two methods both have an additional cross-modal encoder, bringing computation overhead.
In this paper, we follow the dual-stream design of CLIP and construct three visual semantics levels and three linguistic semantics levels to form our PyramidCLIP.
Different from methods mentioned above, each level is input to the corresponding encoder individually, without concatenating.
The obtained three visual representations and three linguistic representations are used to compute six contrastive loss terms, which helps to achieve multi-level alignments.
Method
In this section, we introduce the proposed PyramidCLIP for more accurate alignment of image and text for vision-language model pre-training.
3.1 Overall Architecture
The entire framework of the proposed PyramidCLIP is presented in Figure 2.
PyramidCLIP is a dual-stream network, including a text encoder h and an image encoder f = f 2 ◦ f 1 , where f 1 and f 2 denote the front part and the rear part of the image encoder respectively.
Each encoder consists of a linear projector and a normalization operator in the end, which project the final class token into a unified dimension and then normalize it, obtaining the corresponding visual or linguistic representation vector in the same embedded space.
During the training, for each image-text pair ( I, T ) , the image I is transformed into two views, i.e. , local view L and global view G , through random crop with different ratios.
And text T is input to a summary extractor (19) to generate text summarization T S with higher level semantics.
The image global view G and text summarization T S both capture global context information, while the image local view L and original text T contain more detailed information.
Therefore, G and T S are regarded as a pair of positive samples, while L and T are regarded as another pair of positive samples, denoted as ( G, T S ) and ( L, T ) .
These two pairs are input to the dual-stream encoder to extract global and local representation pairs, ( v g , l s ) and ( v l , l t ) , where v g = f ( G ) , l s = h ( T S ) , v l = f ( L ) and l t =
h ( T ) .
Finally, ( v g , l s ) and ( v l , l t ) are pulled together through contrastive learning lossesand 2 respectively (refer to Figure 2), and other samples in the same batch are treated as negative samples.
We term this contrasting process as Peer-level Semantics Alignment .
Furthermore, in order to explicitly model the relationship between salient objects in the image, the
ROI feature sequence R = { o 1 , o 2 , ..., o M } of M detected salient objects in the image
I , with the category and attribute information for each object, are extracted through a pre-trained object-attribute detector.
Then a linear embedding module φ is used to transform R into the latent dimension of Multi-head Self-attention (MHSA) layer (12) in the image encoder.
The feature sequence is successively fed into the rear part f 2 , which contains one or more MHSA layers, to adaptively capture the relation between these salient instances, generating the final representation vector v r , i.e. , v r = f
( φ ( R )) .
For the object categories with attributes, we join them together, constructing another text description T OA , to provide a more granular, comprehensive and accurate caption for the image.
Then we feed it into the text encoder, generating the representation vector l a = h ( T OA ) .
To avoid the relation modelling being overwhelmed by the context semantic modelling, we have ( v g , l a ), ( v r , l s ), ( v l , l a ) and ( v r , l t ) as another four positive pairs, and the distances between which are narrowed through contrastive learning losses 3 , 4 , 5 and 6 respectively, termed as Cross-level Relation Alignment .
It is worth noting that at the inference stage, only the original image-text pair ( I, T ) is used, i.e. , the visual representation v i from I and the linguistic representation l t from T .
3.2 Peer-level Semantics Alignment
Now we present the details of the peer-level semantics alignment .
As mentioned above, the dualstream vision-language contrastive learning methods such as CLIP strongly rely on the assumption that the image-text pairs are of good quality of one-to-one correspondence.
However, semantic mismatch between images and text captions often occurs in the automatically harvested data.
Therefore, we construct an input pyramid with multi-level semantics on both sides of the dual-stream network, and then align image and text within the same semantics level.
Specifically, the image I is transformed to the global view G and the local view L through two random crops with different ratios.
For the text caption, besides the original caption T , text summarization T S with more compact semantics is extracted using a pre-trained text summarization extractor.
Coarse-grained Global Contrast We set the random crop ratio for generating global view G to be [0 . 9 , 1] , which basically contains all the information in the original image.
Text summarization T S condenses the original caption T , removing some redundant and overly detailed information in the T .
G and T S both capture global information and can be used as pairs of positive samples.
The projected embedding v g and l s of G and T S are pulled closer through contrastive learning.
Fine-grained Local Contrast Since the alignment of the global view G with the text summarization
T S described above is relatively coarse, finer-grained information is largely discarded.
Intuitively, some image sub-regions can be aligned with the original caption T .
To this end, we introduce fine-grained local contrast.
We set the random cropping ratio for generating local view L to be [0 . 5 , 1] , which focuses on the sub-region of the image I .
Then the projected embeddings v l and l t of
L and T are also brought together through contrastive loss (refer to Section 3.4).
Naturally, we have also tried to bring the finer-grained, peer-level ( v r , l a ) closer, but there is no further gain (see Appendix F).
3.3 Cross-level Relation Alignment
To further improve the alignment precision, we introduce the ROI feature of each salient object in the image and the corresponding description with category and attributes, using a pre-trained objectattribute detector, as the most fine-grained semantic level to provide more accurate supervisions.
Specifically, given an image I with M salient objects, the extracted visual semantics of each object region is formulated as [ o ′ m , z m ] , where m denotes the m th object, o ′ m is a 2048-dimensional feature vector and z m is a 4-dimensional normalized position vector indicating the coordinates of top-left and bottom-right corners.
By concatenating o ′ m and z m , we have the 2052-dimensional position-sensitive
ROI feature vector o m , forming the ROI feature sequence R = {
o 1 , o 2 , ..., o M } with the order organized from high confidence to low.
Then R ∈ R M × 2052 is transformed into R M × d using the projector in the embedding module φ , where d indicates the latent dimension of the MHSA layers in the image encoder.
A randomly initialized d -dimensional class token is additionally appended at the front, resulting F ∈ R (1+ M ) × d , which is further feed into the rear part f 2 of the image encoder to compute the normalized ROI relation embedding v r , i.e. , F = φ ( R ) and v r = f 2 ( F ) .
Note that feature sequence F is position-sensitive following R .
Thus, positional embedding is not applied before it enters into the MHSA layer.
Meanwhile, the detected names of category and attributes for each salient object form a phrase with one or more adjectives (attributes) modifying a noun (category), like “ gold round brown table ” in Figure 2.
Then all the M phrases from M salient objects are joint into a text description T OA with the same order as the ROI feature sequence, and the phrases are separated by commas.
Next, T OA is input to the text encoder to obtain the embedding l a = h ( T OA ) .
To enhance the relation modelling capacity of the text encoder, while avoiding weakening the reasoning ability of the image encoder, ( v g , l a ), ( v r , l s ), ( v l , l a ) and ( v r , l t ) are used as another four positive pairs, and the distance between each pair is minimized simultaneously.
Since the object-level inputs R and T OA are direct concatenation of feature vectors and phrases respectively, hence very fine-grained, while other inputs G , L , T S and T are complete images or sentences, we term this contrasting process cross-level relation alignment .
In the case that the visual model is a convolution neural network (CNN), the traditional pooling layer is replaced by attention pooling, which actually is a MHSA layer.
So the embedded ROI feature sequence F is input to the attention pooling layer, i.e. , f 2 , which indicates the final attention pooling layer, as shown in Figure 3(a).
For the transformer-based visual model (ViT), the sequence F can be directly input to a transformer layer.
Considering that F already encodes high-level visual semantics, we feed it into the rear part f 2 of the ViT encoder, see Figure 3(b).
Besides, as pointed out in (20; 21), the standard ViT may not fully leverage the local context information, which limits the visual representation capacity of ViT-based image encoder.
Following (21), we incorporate a depth-wise convolution into the Feed-Forward module of ViT, termed Locally-enhanced
Feed-Forward (LeFF), improving the patch-level local perception and interaction.
The structure of
LeFF is shown in Figure 3(c).
First, the patch tokens are projected into a higher dimension through a linear projection layer and reshaped.
Next, a 3 × 3 depth-wise convolution is utilized to capture the local information.
Then the feature maps are flattened to a token sequence and re-projected into the initial dimension.
While the class token is identical during the process and is concatenated with locally-enhanced patch tokens, generating the final output.
As depicted in Figure 3(b), LeFF is only applied in the front part
f 1 of the ViT-based image encoder, since it is clearly not suitable for the embedded ROI feature sequence F .
In Figure 3(b), L denotes the total number of transformer layers in ViT and L s is the number of transformer layers with LeFF in f 1 .
And L = 12 and L s = 9 in the experiments.
The influence of different settings of L s can be seen in Appendix F.
3.4 Softened Objective Function
For a batch of N image-text pairs { ( I i , T i ) } Ni =1 , where i indicates the i th pair, the normalized embedded vectors { v gi , v li , v ri , l si , l ti , l ai } Ni =1 of the same dimension are obtained by the dual-stream encoders.
In this formulation, v gi , v li and v ri are generated by the image encoder from global-crop image G , local-crop image L and ROI feature sequence R respectively, while l si , l ti and l ai are generated by the text encoder from text summarization T S , original text T and object-attribute description T OA respectively.
Then we use this vector group to construct six supervision signals L
GS , L LT , L GA , L RS , L LA and L RT for in-batch contrastive learning, which can be calculated with { ( v gi , l si ) } Ni =1 , { ( v li , l ti ) } Ni =1 , { ( v gi , l ai ) } Ni =1 , { ( v ri , l si ) } Ni =1 , { ( v li , l ai ) } Ni =1 and { ( v ri , l ti ) } Ni =1 respectively.
Our six contrastive losses, with the formulation of InfoNCE (22), are designed to achieve the alignment between visual representation and linguistic representation from disparate semantic levels.
Take the first loss term L GS from { ( v gi , l si ) } Ni =1 as an example.
For the i th pair, the normalized vision-to-language similarity p vi ( G ) =
{ p vij ( G ) } Nj =1 and language-to-vision similarity p li ( T S ) = { p lij ( T S ) } Nj =1 can be calculated through:
[Equation]
where τ is a learnable temperature parameter initialized with 0 .
07 and the function sim( · ) conducts dot product to measure the similarity scores.
In standard practice, for the i th pair, the corresponding one-hot label vectors of the ground-truth y vi ( G ) =
{ y vij ( G ) } Nj =
and y li ( T S )
= {
y lij ( T S ) } Nj =1 , with positive pair denoted by 1 and negatives by 0 , are used as the targets to calculate cross-entropy.
This kind of hard targets assumes there is absolutely no similarity between unpaired image and text.
However, within a large-size batch, unpaired image and text may have more or less local similarities, i.e. , some local regions in an image may be matched with some words or phrases in other unpaired texts.
To address this problem for better generalization, we use label smoothing to soften the hard targets.
The corresponding softened targets (cid:101) y vi ( G ) and (cid:101)
y li ( T S ) for the i th pair can be formulated as:
[Equation]
where α is the smoothing hyper-parameter set to 0 .
2 in our experiments.
Then L GS can be written as:
[Equation]
The other loss terms L LT , L GA , L RS , L LA and L RT can be calculated similarly.
We then divide them into three groups that are respectively the peer-level alignment L peer =
( L GS + L LT ) / 2 , the global-relation cross-level alignment L globalcross =
( L GA + L RS ) / 2 and the local-relation cross-level alignment L localcross =
( L LA + L RT ) / 2 .
Therefore, the overall objective function of PyramidCLIP is:
[Equation]
where the loss weights λ and µ are both set to 1 / 3 in our experiments.
Experiments 4.1 Implementation Details and Datasets
Pre-training Stage We experiment on three different architectures, PyramidCLIP-ResNet50/
ViTB32/ViT-B16, according to the choice of image encoder.
Their detailed architecture designs follow that of CLIP (6).
LAION99M contains 99M image-text pairs with the highest similarity selected from
LAION400M (23) according to the similarity scores provided by the producer.
We use the publicly
[Table]
released T5 model (19) to extract text summarization for all texts and utilize an object-attribute detector pre-trained by VinVL (16) to extract salient object features together with category and attribute information in the image.
Please refer to Appendix A for details.
Downstream Tasks We validate the effectiveness of our proposed method on five downstream tasks: zero-shot image classification, zero-shot image-text retrieval, linear probe, object detection and instance segmentation.
For classification, experiments are carried out on 11 datasets, such as ImageNet (13), CIFAR-100 (28).
For image-text retrieval, experiments are conducted on Flickr30K (29) and MS-COCO (30).
For object detection and instance segmentation, the proposed method is verified on PASCAL VOC (31) and MS-COCO.
More details can be found in Appendix B.
4.2 Fair Comparison against SoTA
[Table]
We first compare our method against other SoTA approaches on ImageNet zero-shot classification task using the same amount of pre-training data YFCC15M-V2 and the results are shown in Table 2.
It can be seen that compared to CLIP, PyramidCLIP improves the top-1 accuracy by 10.6% / 13.2% / 10.0% when the visual model is
ResNet50 / ViT-B32 / ViT-B16 respectively.
Furthermore,
PyramidCLIP outperforms all other SoTA approaches by a large margin.
In addition, since the distribution of different datasets can vary vastly, we also conduct experiments on YFCC15M-V1 and LAION15M, which is obtained by sampling 15 million image-text pairs from
LAION99M for fair comparison.
The results can be seen in Appendix C and our PyramidCLIP still shows great superiority.
4.3 Comparison on Large-scale Datasets
We further validate the effectiveness of our method on a large-scale dataset, i.e. , 143M image-text pairs, and downstream zero-shot image-text retrieval and image classification results are shown in
Table 3.
It can be seen that on image-text retrieval task, PyramidCLIP exceeds CLIP trained on 400M data and DECLIP by a large margin.
And on ImageNet classification task, with the same amount of pre-training data, PyramidCLIP significantly exceeds the results of CLIP by 6.1%/3.8%/3.5% using ResNet50/ViT-B32/ViT-B16 as image encoder.
Furthermore, it is worth noting that, when the vision model is ResNet50, PyramidCLIP trained on 143M data surpass CLIP using 400M data, which greatly improves data efficiency.
4.4 Transferability to Small-scale Classification Datasets
In this section, we validate the transferability of our method on 10 relatively small downstream classification datasets, both on zero-shot and linear probe tasks.
The results are shown in Table 4.
It can be seen that the average accuracy of PyramidCLIP on 10 datasets all exceed CLIP trained on 400M data on two kinds of tasks.
It is worth noting that our pre-training data is less than 36% of
CLIP, but the average accuracy is better, indicating higher data utilization.
4.5 Transferability to Object Detection and Instance Segmentation
In order to verify that our model can better exploit the relationship between objects in the image, we further validate our models on object detection and instance segmentation tasks.
Specifically, we take the visual model ResNet50 to initialize the backbone of Faster R-CNN (34) and Mask R-CNN (35) and then all the parameters are fine-tuned.
The results are shown in the Table 6.
It can be seen that our model significantly outperforms random initialization and surpasses CLIP and DECLIP model.
[Table] [Table]
4.6 Ablation Study
[Table]
In this section, we verify the effectiveness of each component in PyramidCLIP on downstream zeroshot ImageNet classification task, and all the experiments are pre-trained for 8 epochs on YFCC15M-V1.
The results are listed in Table 5, which indicate that on the basis of the peer-level alignment, all the other components including cross-level global-relation and local-relation alignment, LeFF in ViT and softened objectives can bring significant gains individually.
More ablation results can be seen in Appendix F, including the detailed ablation on L s , the choice of peer-level alignment and the effectiveness each component on downstream object detection task.
4.7 Visualization
Semantic Features We utilize t -SNE (36) to visualize the learned semantic features of CIFAR-10 (28).
Each text feature of 10 categories is obtained using the ensemble of 80 prompt templates.
And image features are extracted with ResNet50 visual encoder.
As depicted in Figure 4, CLIP pre-trained on 143M data has a poor clustering performance, with the visual features of most categories overlapping heavily and the textual features of some two categories being very close, such as ( automobile , truck ) and ( frog , deer ).
Although CLIP pre-trained on 400M data (6) performs better than CLIP on 143M data, the visual features of some categories like dog and cat still have a large overlap.
In comparison, the semantic features extracted by PyramidCLIP on 143M data, are well separated with each textual feature surrounded by most visual features of the same category.
[Table]
[Figure]
Grad-CAM Heatmaps Grad-CAM (37) is also utilized to help understand why PyramidCLIP outperforms CLIP.
Specifically, we conduct this through text-to-image retrieval on MS-COCO, using
CLIP/PyramidCLIP-ResNet50 pre-trained on 143M data.
For each query text, the model retrieves top5 images with highest similarities.
Then for each of the five retrieved images, we use Grad-CAM to find which areas have the highest activation to the query text.
As shown in Figure 5, PyramidCLIP has better retrieval performance than CLIP.
Moreover, PyramidCLIP can accurately capture the complete object regions that are highly matched with the query texts, while CLIP captures regions with either components missing or additional noises, or even obtains completely irrelevant matches.
More visualization results can be found in Appendix G.
Conclusion
In this paper, we have proposed a hierarchical pre-training method, termed PyramidCLIP, to achieve improved alignment between visual and linguistic modalities.
It resolves the issue that the weblycrawled data is not in perfect one-to-one correspondence by explicitly constructing pyramidal semantic inputs at the both sides of dual-stream network.
We also show that softened peer-level semantics alignment and cross-level relation alignment can interact between two modalities and are beneficial.
PyramidCLIP achieves the state-of-the-art results on five downstream tasks, which demonstrates the superiority.
[Table]
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[No] (c) Did you discuss any potential negative societal impacts of your work?
[No] Our work does not have any potential negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] Our codes may be released in the future.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Appendix A. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] We did not run the same experiment for multiple times.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix A. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[No] (c) Did you include any new assets either in the supplemental material or as a URL?
[No] Our codes may be released in the future.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[No] Existing assets used in our work are all publicly released.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[No] Datasets used in our work are all publicly released without this problem.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]