H I CLIP: C ONTRASTIVE L ANGUAGE -I MAGE P RETRAINING WITH H IERARCHY - AWARE A TTENTION
A BSTRACT
The success of large-scale contrastive vision-language pretraining (CLIP) has benefited both visual recognition and multimodal content understanding.
The concise design brings CLIP the advantage in inference efficiency against other visionlanguage models with heavier cross-attention fusion layers, making it a popular choice for a wide spectrum of downstream tasks.
However, CLIP does not explicitly capture the hierarchical nature of high-level and fine-grained semantics conveyed in images and texts, which is arguably critical to vision-language understanding and reasoning.
To this end, we equip both the visual and language branches in
CLIP with hierarchy-aware attentions, namely Hierarchy-aware CLIP (HiCLIP), to progressively discover semantic hierarchies layer-by-layer from both images and texts in an unsupervised manner.
As a result, such hierarchical aggregation significantly improves the cross-modal alignment.
To demonstrate the advantages of HiCLIP, we conduct qualitative analysis on its unsupervised hierarchy induction during inference, as well as extensive quantitative experiments on both visual recognition and vision-language downstream tasks 1 .
I NTRODUCTION
In recent years, vision-language pretraining has achieved significant progress pairing with large-scale multimodal data.
Contrastive vision-language pretraining (CLIP) features its generalization ability for zero-shot tasks and robustness to domain shift (Radford et al., 2021).
Moreover, the spectrum of problems that CLIP can solve range from visual recognition, image-text retrieval, and vision-language reasoning tasks via providing appropriate prompt engineering (Zhou et al., 2022; Gao et al., 2021;
Xu et al., 2021; Shridhar et al., 2021;
Rao et al., 2022; Zhong et al., 2022).
Since CLIP is built upon simple cross-modal interactions, it has superior inference efficiency over cross-attention based vision-language models (Li et al., 2021;
Chen et al., 2020; Li et al., 2020; Tan & Bansal, 2019;
Dou et al., 2022).
Recent studies including DeCLIP (Li et al., 2022), SLIP (Mu et al., 2021), and
FILIP (Yao et al., 2022) extend CLIP by either leveraging extra self-supervision training objectives or performing contrastive loss on dense token features.
As humans perceive the world in a hierarchical manner Hubel & Wiesel (1968)
; Fukushima & Miyake (1982); Kuzovkin et al.
(2018), such hierarchical nature in vision and language contents has been explored to assist the design of various model architectures.
However, contrastive vision-language learning methods like CLIP often cannot capture visual and linguistic hierarchies in an explicit way.
In the example of right figure in Fig.
1 (a), the pixels first form local patches as image encoder’s inputs, and are further merged into semantic groups denoting objects (“traffic lights”, “sky”), attributes (“cloudy”), etc.
Similarly, syntax hierarchies can also be observed in natural languages where the caption can be decomposed into constituents as shown in Fig.
1 (b).
Therefore, we argue that the hierarchical nature (i.e., merging from local to global) in vision and language is critical and can be explicitly utilized for improving CLIP’s capability on multimodal tasks, especially those requiring high-level understanding and reasoning.
A couple of traffic lights sitting under a cloudy sky.
LanguageHierarchy traffic lights sitting a under cloudy sky
VisionHierarchy Visual Hierarchy 0.
≈
Language Hierarchy trafﬁc
A couple of traffic lights sitting under a cloudy sky.
LanguageHierarchy traffic lights sitting a under sky cloudy
VisionHierarchy “traffic lights sitting under a cloudy sky” lights sitting under a cloudy sky (a) (b) cloudy ≈0.
To this end, we introduce hierarchy-aware attention into CLIP, denoted as HiCLIP. Hierarchy-aware attention applies an attention mask to the conventional attention mechanism to indicate the tendency to merge certain vision patches and language tokens into groups because they are spatially and semantically or visually similar. We generalize hierarchy-aware attention to both images and texts, where its mask is obtained by first calculating the neighbouring affinity score among adjacent patches or tokens, and then propagating the scores across any given patch or token pairs.
In addition, we formulate the affinity score with an increasing trend as the layer gets deeper to ensure the merged groups remains the same.
In this way, we progressively aggregate hierarchies in a layer-by-layer manner for both images and texts.
To be specific, for modeling hierarchies in natural language, we share similar intuitions with the previous studies on unsupervised grammar induction, which aim at unsupervised hierarchical mining (Shi et al., 2019; Drozdov et al., 2019).
Tree Transformer (Wang et al., 2019) proposes a similar modified attention mechanism which is essentially a special case of hierarchy-aware attention, where the attention mask is instantiated as constituent prior to encourage the merging of semantically similar tokens.
Capturing hierarchies in visual contents is more challenging, because spatial correlation should also be considered in addition to visual similarities.
Therefore, we extend the hierarchy-aware attention to Vision Transformers (Dosovitskiy et al., 2021) by creating a Group Transformer to progressively aggregate image patches into semantic groups until all patches are merged in one common group which is the original image.
Different from the 1D scenario in Tree Transformer, the neighboring affinity score is computed among the four adjacent neighbors of each image patch (Fig.
1 (a)).
Afterwards, we propagate neighboring affinity scores by comparing two special paths connecting image patches on the 2D grid graph.
When we apply such hierarchy-aware attention to both image and text branches in CLIP, we obtain the proposed hierarchy-aware CLIP (HiCLIP) which features the following advantages: (1) it is able to automatically discover hierarchies in vision and language that matches human intuitions in an unsupervised manner; (2) it generates better multimodal representations especially for visionlanguage downstream tasks; and (3) it features a comprehensive hierarchy visualization to help parse visual and textual hierarchies.
To prove the aforementioned advantages, we pretrain HiCLIP and other CLIP-style approaches on large-scale image-text pairs, and conduct extensive experiments on downstream tasks including visual recognition, image-text retrieval, visual question answering, and visual entailment reasoning.
To sum up, our contributions are summarized as follows:
• We incorporate hierarchy-aware attention into CLIP (HiCLIP) for both image and text contents, which achieves better performances on vision and vision-language downstream tasks.
• To model images in a hierarchical manner, we propagate neighboring affinity scores through two special paths on 2D grid graphs and generalize the hierarchy-aware attention to Vision Transformer.
• We visualize the evolution of hierarchies in images and texts to demonstrate the ability of unsupervised hierarchy induction of HiCLIP, which contributes to a better interpretability.
R ELATED W ORK
Vision-Language Models.
With the proliferation of multimodal information, exploring the interaction of vision and language information becomes an important topic.
As a result, many vision-language models have flourished recently.
Based on how the training objective is designed, they can be divided into three categories.
The first category includes early bilinear pooling and attention based multimodal models such as MUTAN (Ben-Younes et al., 2017),
BAN (Kim et al., 2018), bottom-up top-down attention (Anderson et al., 2018), and intra-inter modality attention (Gao et al., 2019).
The second category is built upon the masked language modeling (MLM) pretraining objective and consists of approaches such as ViLBERT (Lu et al., 2019), LXMERT (Tan & Bansal, 2019), UNITER (Chen et al., 2020) and ALBEF (Li et al., 2021).
Several recent approaches such as SOHO (Huang et al., 2021) and BEiT (Wang et al., 2022) futhur extends MLM to masked visual modeling (MVM) to push the boundary of multimodal learning.
In addition, CLIP family models (Radford et al., 2021; Li et al., 2022; Mu et al., 2021; Yao et al., 2022; Chen et al., 2023) that rely on vision-language contrastive learning and large-scale image-text pairs constitutes the last category.
Unsupervised Grammar Induction.
Unsupervised grammar induction is a classic topic in NLP domain aiming at automatically inducing phrase-structure grammars from free-text without parse tree annotations.
In the earlier age, probabilistic context free grammars (PCFGs) built upon contextfree grammar is widely applied and are solved by inside-outside algorithm (Baker, 1979) or CYK algorithm (Sakai, 1961).
More recently, many deep learning based approaches have been proposed by extending the conventional methods into neural networks such as C-PCFG (Kim et al., 2019) and
DIORA (Drozdov et al., 2019), designing special modules to induce tree structures such as PRPN (Shen et al., 2018), ON-LSTM (Shen et al., 2019), Tree Transformer (Wang et al., 2019), or assisting unsupervised grammar induction with the help of cross-modality alignment such as VG-NSL (Shi et al., 2019), VC-PCFG (Zhao & Titov, 2020), and CLIORA (Wan et al., 2022).
Hierarchical Discovery in Vision.
Discovering the hierarchy in visual contents is a well-established area of vision research.
For example, Lin et al.
(2017) constructs the hierarchy of feature pyramids in object detection to help the model capture semantics at all scales.
More recent work on transformers (Liu et al., 2021; Zhang et al., 2020) also adopts similar intuition to generate hierarchical feature maps with special local-global attention designs.
Meanwhile, another line of research on designing new fine-grained parsing tasks aims to understand hierarchy within a scene, such as scene graph parsing (Krishna et al., 2017; Zhang et al., 2019) and action graph parsing (Ji et al., 2020).
Recently, more efforts are devoted to automatic hierarchy learning with self-supervised or weakly-supervised objectives (Xie et al., 2021; Dai et al., 2021), designing special inductive bias for the self-attention mechanism (Yu et al., 2022; Zheng et al., 2021), and automatically merging semantically similar embeddings (Xu et al., 2022; Locatello et al., 2020).
Our work has the scope of developing special attention constraint and utilizing contrastive learning objectives for unsupervised hierarchy discovery.
H IERARCHY - AWARE A TTENTION IN CLIP
As discussed in Section 1, both vision and language share a hierarchical nature in information parsing.
The lower level of the hierarchy contains more localized and finer-grained information while the higher levels capture more holistic semantics.
These properties are in line with how we humans understand vision (Hubel & Wiesel, 1968; Fukushima & Miyake, 1982; Kuzovkin et al., 2018) and language information (Chomsky, 1956; Manning, 2022).
3. A F RAMEWORK OF H IERARCHICAL I NFORMATION A GGREGATION
Hierarchy-aware attention is based on the attention mechanism in conventional Transformers.
Given query Q , key K , and value V , and the scaling factor √ d h that maintains the order of magnitude in features where d h denotes the feature dimension, the general Attention function is defined as:
[Equation]
As illustrated in Fig.
2, we propose to enhance CLIP’s vision and language branch with a hierarchyaware attention.
Following the common transformer architecture, given the modality inputs being split into low-level image patches and text tokens, we recursively merge patches and tokens that are semantically and spatially similar, and gradually form more semantic-concentrated clusters such as image objects and text phrases.
First, we define the hierarchy aggregation priors as follows: [Figure] Group [Figure]
Layer
Layer Layer CLIP Loss A couple of traffic lights sitting under a cloudy sky.
LanguageHierarchy traffic lights sitting a under sky cloudy
VisionHierarchy “traffic lights sitting under a cloudy sky”
Layer
Layertrafﬁc lights sitting under a cloudy sky
Layerexample
[Figure]
Tendency to merge.
We recursively merge patches and tokens into higher-level clusters that are spatially and semantically similar.
Intuitively, if two nearby image patches have similar appearances, it is natural to merge them as one to convey the same semantic information.
Non-splittable.
Once the patches or tokens are merged, they will never be split at later layers.
With this constraint, we aim to enforce that the hierarchical information aggregation will never get degraded, and as a result, preserve the complete process of hierarchy evolving layer-by-layer.
We then incorporate these hierarchy aggregation priors into an attention mask C which serves as an extra inductive bias to help the conventional attention mechanism in Transformers to better explore the hierarchical structures adaptive to each modality format, i.e., 2D grid on images and 1D sequence on texts.
Therefore, the proposed hierarchy-aware attention can be defined as:
[Equation]
Note that C is shared among all heads and progressively updated bottom-up across Transformer layers.
We elaborate on the formulations of the hierarchy-aware mask C for each modality as follows.
3.1. H IERARCHY I NDUCTION FOR L ANGUAGE B RANCH
In this section, we revisit the tree-transformer method from the proposed hierarchy-aware attention point of view and show how to impose hierarchy aggregation priors on C with three steps.
Generate neighboring attention score.
Neighboring attention score describes the merging tendency of adjacent word tokens.
Two learnable key, query matrices W ′ Q , W ′ K are adopted to transfer any adjacent word tokens ( t i , t i +1 ) , so that the neighboring attention score s i,i +1 is defined as their inner-product:
[Equation]
Here σ t is a hyper-parameter to control the scale of the generated scores.
Then, for each token t i , a softmax function is employed to normalize its merging tendency of two neighbors:
[Equation]
For neighbor pairs ( t i , t i +1 ) , the neighboring affinity score ˆ a i,i +1 is defined as the geometric mean of p i,i +1 and p
i +1 ,
i : ˆ a i,i +1 = √ p
i,i +1 · p i +1 ,i .
From a graph perspective, it describes the strength of edge e i,i +1 by comparing it with edges e i − 1 ,i ( p i,i +1 v.s. p
i,i − 1 ) and e
i +1 ,
i +
( p i +1 ,
i v.s. p
i +1 ,i +2 ) .
Enforcing Non-splittable property.
Intuitively, a higher neighboring affinity score indicates that two neighbor tokens are more closely bonded.
To assure merged tokens will not be splitted, layer-wise affinity scores a li,i +1 should increase as the network goes deeper, i.e., a li,i +1 ≥ a l − 1 i,i +1 for all l , to help gradually generate a hierarchy structure as desired:
[Equation]
Modeling the tendency to merge.
To measure the tendency to merge, namely C i,j , for any word token pair ( t i , t j ) , we propagate the affinity scores of neighboring tokens between ( t i , t j ) .
Specifically, C i,j is derived through the multiplication operation as C i,j = (cid:81)
j − 1 k
= i a k,k +1 .
Note that C is a symmetric matrix, so we have C i,j = C j,i .
3.1. H IERARCHY I NDUCTION FOR V ISUAL B RANCH
From a graph perspective, it is easier to generalize the hierarchy-aware mask C from the 1D sequence in language to the 2D grid in vision domain.
First, we also employ query and key matrices W ′′ Q , W ′′ K to calculate the neighboring attention scores among the four-adjacency neighbors of each patch t i,j :
[Equation]
(6) where σ v is used to control the scale of the generated scores, and the neighborhood t i ′ ,j ′ is limited to the four-adjacency patches of t i,j as ( i ′ ,
j ′ ) ∈
( i + δ, j + η ); δ, η ∈ {− 1 , +1 }} ≡ A .
Next, for each patch p i,j , the softmax normalizing function is employed to get the merging tendency of t i,j to its four neighbors:
[Equation]
Similar to the formulation in the language branch, the neighboring affinity score with non-splittable property can be obtained by:
[Equation]
where ˆ a
[Equation]
Lastly, the neighboring affinity scores needs to be propagated to the whole image to acquire
C ( i 1 ,j 1 ) , ( i 2 ,j 2 ) between any two patches ( t i 1 ,j 1 , t i 2 ,j 2 ) .
As the image can be seen as a 2D-grid graph, a natural solution is considering C ( i 1 ,j 1 ) , ( i 2 ,j 2 ) as the length of the shortest path with − log( a ( i,j ) , ( i ′ ,j ′ ) ) as the edge weights.
To achieve better computational efficiency, we consider two special paths: connecting ( t i 1 ,j 1 , t i 2 ,j 2 ) along the grid with only one turn.
The length of these two paths can be calculated by horizontal and vertical propagation as follows:
[Equation]
2 − 1 (cid:89) n = ia ( n,j 2 ) , ( n +1 ,j 2 ) , (9) and C ( i 1 ,j 1 ) , ( i 2 ,j 2 ) =
max( C 1 , C 2 ) .
Intuitively, C ( i 1 ,j 1 ) , ( i 2 ,j 2 ) finds the maximum merging tendency along two possible paths, either horizontal-first or vertical-first.
In this way, both spatial and visual similarities have contributed to the attention mask C for 2D images.
Since our approach tries to organize vision patches sharing high similarities into groups, we thus dub it “Group Transformer”.
Relation to Recursive Bilateral Filtering.
Recursive bilateral filtering (Yang, 2012) shares similar spirit with our Group Transformer.
Given two pixels in an image, recursive bilateral filtering decomposes the calculation of range filtering kernel R into two 1D operations – horizontal and vertical.
For each 1D operation, let x k , x i denote two pixels on a scanline of the 2D image, the 1D range filtering kernel
R k,i =
R k,k +1 R k
+1 ,k +2 · · · R i − 2 ,i − 1 R i − 1 ,i = (cid:81)
i − 1 j = k R j,j +1 , where R j,j +1 is computed with a radial basis function kernel: R j,j +1 = exp (cid:16) − | x j − x j +1 | 2 2 σ 2 R (cid:17) .
We can observe the similar property that
Tree Transformer possesses.
As bilateral filters tend to preserve sharp edges while smoothing other parts of an image, this is in accordance with the goal of our Group Transformer – aggregating similar patches into a group.
The differences between recursive bilateral filtering and our framework are mainly two perspectives: 1) the basic operation unit is pixel in recursive bilateral filtering while our approach uses either patch or word token; 2) radial basis function kernel is employed to measure the range similarity in recursive bilateral filtering while we use neighboring affinity score instead.
3. H IERARCHY - AWARE CLIP
Pretraining with HiCLIP.
To equip both CLIP branches with the ability of dynamic hierarchy discovery, our Hierarchy-aware CLIP adopts Group Transformer as the image encoder and employs
Tree Transformer as the text encoder.
Let v and u denote the image and text feature vectors, the contrastive pretraining objective L can be written as:
[Equation]
where τ is a learnable temperature parameter, and N is the total number of image-text pairs.
Unsupervised Hierarchy Induction.
During inference, we follow Eq.
(5) and Eq.
(8) to generate all neighboring affinity scores { a li,i +1 } Ll =1 and { a l ( i,j ) , ( i ′ ,j ′ ) }
Ll =1 from bottom to top layer L for the texts and images, respectively.
These neighboring affinity scores are then used for hierarchy induction.
Intuitively, a low affinity score at a certain layer indicates the two corresponding neighbours remain split within this layer.
When repeating such process in a top-down greedy manner, we are able to generate tree hierarchies for texts and similar group hierarchies for images in an unsupervised fashion.
E XPERIMENTS 4. E XPERIMENTAL S ETTINGS
Pretraining Datasets To make a fair comparison with the state-of-the-art contrastive vision-language pretraining approaches, we adopt the YFCC15M benchmark proposed in (Cui et al., 2022) which builds on a subset from YFCC100M
(Thomee et al., 2016)
consisting of 15M image-text pairs.
In addition, we construct a 30M version of pretraining data by including Conceptual Caption 3M (CC3M) (Sharma et al., 2018) and 12M (CC12M) (Changpinyo et al., 2021).
We thus validate our model on the two different scales of pretraining data.
Downstream Datasets Following CLIP and DeCLIP, we select 11 visual recognition datasets under the zero-shot setting, namely ImageNet (Deng et al., 2009), CIFAR 10 & CIFAR 100 (Krizhevsky et al., 2009), StanfordCars (Krause et al., 2013), Caltech101 (Fei-Fei et al., 2004), Flowers102 (Nilsback & Zisserman, 2008), SUN397 (Xiao et al., 2010), DTD (Cimpoi et al., 2014), FGVCAircraft (Maji et al., 2013), OxfordPets (Parkhi et al., 2012), and Food101 (Bossard et al., 2014).
Same zero-shot classification protocol is applied following Radford et al.
(2021) which uses predefined prompts as text inputs.
The full list of used prompts is provided in the Appendix.
Although CLIP and DeCLIP only evaluates on visual recognition, we also provide comprehensive comparisons on vision-language tasks which are more desired in evaluating multimodal models, including: imagetext retrieval on MSCOCO Caption (Chen et al., 2015), as well as vision-language reasoning on
VQAv2 (Antol et al., 2015) and SNLI-VE (Xie et al., 2019).
Implementation Details Two variants of Vision Transformer (Dosovitskiy et al., 2021) are used as the image encoder in our experiments – ViT-B/32 and ViT-B/16, while the text encoder is a vanilla Transformer (Vaswani et al., 2017) following CLIP as a fair comparison.
The embedding size of both image and text features are 512 throughout our paper.
To make a fair comparison with CLIP family baselines, we train all models for 32 epochs under the same set of pretraining hyperparameters including learning rate, warmup steps, weight decay, etc.
The input image size is set to 224 × 224 , and the input text sequence length is truncated or padded to 77 .
The scaling factor σ t and σ v of Hierarchy-aware attention are both set to 256 for Group Transformer and Tree Transformer.
Following CLIP and DeCLIP, the learnable temperature parameter τ is initialized as 0 .
4. V ISUAL R ECOGNITION
We first compare HiCLIP with state-of-the-art CLIP family approaches on YFCC15M benchmark (Cui et al., 2022) containing only ImageNet zero-shot setting.
Then we present zero-shot classification results on the other common visual recognition datasets.
Both results are presented in Table 1.
YFCC15M Benchmark.
CLIP Radford et al.
(2021), SLIP Mu et al.
(2021), and FILIP Yao et al.
(2022) all leverage a contrastive learning and can be directly compared with HiCLIP.
When limiting [Table] the comparisons within this scope, HiCLIP achieves the largest performance gain (7.7%) over the other CLIP-style models.
Since DeCLIP Li et al.
(2022) and DeFILIP
Cui et al.
(2022) apply multiple single-modal self-supervised tasks in addition to CLIP, we incorporated the same objectives into our hierarchy-aware model for a fair comparison (denoted as HiDeCLIP).
By combining the contrastive learning and self-supervised learning loss functions, our HiDeCLIP further improves the zero-shot
ImageNet classification performance by 2.7% over DeCLIP, and overall 13.1% higher than CLIP.
11 Visual Recognition Benchmarks.
Note that we included both versions of training data, YFCC15M (short as 15M) and 30M, in this experiments as discussed in Section 4.1.
We observed that the zero-shot performance on Cars and Aircraft datasets are very low for all models, because in the
YFCC benchmark there are 0.04% and 0% of descriptions contains aircraft and car labels used in these datasets, such as “Audi V8 Sedan 1994” .
HiCLIP achieves significant improvement in average over CLIP on both pretraining datasets, indicating that HiCLIP maintains substantial advantage over
CLIP when scaling up the size of training data.
Despite the fact that the absolute improvements by incorporating hierarchy-aware attentions into CLIP is relatively less significant than adding multiple self-supervised tasks, it is interesting that hierarchy-aware attention is compatible with self-supervised learning (DeHiCLIP) and further achieves performance improvement over DeCLIP.
4. P ERFORMANCE C OMPARISON ON V ISION -L ANGUAGE T ASKS
In Table 2, we compare different CLIP-style methods on downstream vision-language tasks, including image-text retrieval which emphasizes on cross-modal alignment and two vision-language reasoning tasks (VQA and SNLI-VE) which focus more on collaborative multimodal reasoning.
Zero-shot Image-Text Retrieval on MSCOCO.
On all different algorithms and training datasets,
HiCLIP and HiDeCLIP improve the retrieval performance by a large margin.
It is worth noting that without complicated self-supervised learning objectives, HiCLIP constantly outperforms DeCLIP when merely relying on CLIP’s contrastive loss which is different from visual recognition tasks.
This finding suggests that the benefits brought by adding self-supervised learning is effective within the scope of visual recognition, while our approach fully explores the hierarchical nature of multimodal contents which contributes to a significantly performance boost in vision-language tasks.
Fine-tuning on Vision-Language Reasoning Tasks.
Similar to the results on zero-shot retrieval, we observe consistent performance gains for all visual reasoning tasks and pretraining data, indicating that hierarchy-aware attentions are more efficient multimodal learners and is capable of tackling tasks that require content understanding and reasoning capabilities.
4. A BLATION S TUDY
In this section, we provide additional ablation studies on influence factors including the patch granularity of visual encoder and the training data volumes in Table 3.
We report the following experimental results including zero-shot accuracy on ImageNet and averaged accuracy on all [Table] [Table] visual recognition dataset, Rsum over recall@1, 5, 10 on zero-shot image-text retrieval, as well as accuracy on VQA and SNLI with fine-tuning.
In addition, we conduct component analysis in Tableto show that Group Transformer and Tree Transformer both play important roles in HiCLIP.
On Patch Granularity.
We compare all downstream tasks using ViT-B/32 and ViT-B/16 as visual encoders.
Since the Group Transformer is based on visual patches and benefits from finer-grained patch segments, we expect HiCLIP and HiDeCLIP achieves consistent performance improvements when directly comparing the same method across different visual encoder variants.
When we fix the visual encoder and compare HiCLIP and HiDeCLIP with their corresponding baselines (i.e., CLIP and
HiCLIP),
HiCLIP and HiDeCLIP constantly outperform CLIP and DeCLIP on all tasks with the help of hierarchical-aware attention.
It is worth noting that HiCLIP alone without complex self-supervised losses outperforms DeCLIP on three out of five tasks, with the exceptions on ImageNet and VQA by a small margin.
This shows that hierarchical information captured by HiCLIP potentially benefits more to the vision-language contrastive learning paradigm.
On Pretraining Data Scale.
As shown in Table 3, for most vision recognition tasks, we observe that the benefits contributed by a better modeling strategy saturates when more data is used during pretraining, which is in line with the findings reported by many other works including CLIP Radford et al.
One possible explanation is that, in order to achieve further improvements on visual recognition tasks, a more vision-specific training scheme such as self-supervised learning potentially benefits more, because the ability of multimodal high-level reasoning are not as critical in vision-only tasks.
In contrast, by scaling up the pretraining data, the performance improvements achieved on vision-language tasks are more significant and consistent across all methods.
Similarly, HiCLIP and
HiDeCLIP still enjoys large improvements against CLIP and DeCLIP when the pretraining dataset scales up.
In addition, HiCLIP pretrained on 30M data achieves better vision-language performances on all three tasks over DeCLIP suggesting a potential better scalability of HiCLIP on vision-language reasoning tasks, while DeCLIP features better vision recognition performances.
On Component Analysis.
In Table 4, we demonstrate that using Group Transformer alone (HiCLIPGroup) for vision modeling yields comparable improvements on visual recognition task (zeroshot ImageNet classification) with using Tree Transformer alone (HiCLIP-Tree).
In addition, the
[Table]
improvements on image-text retrieval are more significant when applying Tree Transformer alone than applying Group Transformer, indicating that language modeling may have more potential impact than visual modeling with regard to such vision-language tasks.
Moreover, when we activate both Group Transformer and Tree Transformer, substantial performance boosts are obtained against
HiCLIP-Group and HiCLIP-Tree, showcasing the synergy between dual hierarchy-aware attentions even under naive cross-modal interactions.
4. U NSUPERVISED H IERARCHY I NDUCTION WITH P RETRAINED H I CLIP MODEL
It is natural to adopt Tree Transformer because texts are essentially discrete tokens among which certain semantic dependencies are shared.
Following the same analogy, since each image is prepatchified in Vision Transformers, we expect the image patches to join semantic groups gradually from bottom layers to top layers for a better representation, although it seems to be more challenging than the language counterpart.
Therefore, in addition to the performance gains achieved over various downstream tasks, we also visualize the hierarchies captured in our Group and Tree Transformers.
As shown in Figure 3, by virtue of explicitly modeling the inputs with hierarchy-aware attentions during pretraining, our model is able to gradually group semantically similar neighbors, showing the ability of performing hierarchical visual and language inductions in an unsupervised manner.
[Figure]
C ONCLUSION AND F UTURE W ORK
In this work, we equip both the visual and language branches of CLIP with hierarchy-aware attention to automatically capture the hierarchies from image-caption pairs.
Following the discovered hierarchical structures, the proposed HiCLIP creates compact image and text embeddings via gradually aggregating spatially and semantically similar patches or tokens into common groups.
Supported by extensive experiments on multiple downstream tasks, we show that hierarchy-aware attention greatly improves the alignment of image and text modalities against several recent CLIP-style approaches.
Moreover, after pretraining, both branches of HiCLIP can be adopted for unsupervised hierarchy induction by analyzing the generated constituent attention weights.
With limited computational resources, we conduct experiments up to 30 million image-text pairs without extensive parameter tuning.
As a future direction, we plan to scale up the pretraining dataset as well as the scale of the visual encoder to fully validate the scalability of our approach.
In addition, we also plan to explore the full potential of hierarchy-aware attentions with better multimodal information fusion operations compared with the simple dot product used in CLIP-style models.
A P ROMPTS E NGINEERING FOR Z ERO - SHOT V ISUAL R ECOGNITION
In this work, we follow the 80 prompts proposed in Radford et al.
(2019) to evaluate zero-shot image classification on ImageNet dataset.
The full list of prompts for ImageNet are presented in Table 5.
For other downstream visual recognition datasets, we also use domain-specific prompts according to
Radford et al.
The whole prompts for the 10 downstream datasets can be found in Table 6.
In addition to the zero-shot image classification tasks presented in Table 1, we also perform linear probe on frozen image features to estimate the quality of pretrained image encoders.
We follow the settings of CLIP and DeCLIP to train a linear classifier with the L-BFGS optimizer from scikitlearn machine learning library.
The linear probe results on downstream visual recognition tasks are presented in Table 7. From the table, we can observe that HiCLIP / HiDeCLIP still outperform CLIP / DeCLIP across all varied visual encoder types and pretraining data sizes. C A DDITIONAL P RETRAINING I MPLEMENTATION D ETAILS
Our implementation is based on the open-source PyTorch implementation 2 .
Following Cui et al. (2022), we use AdamW optimizer (Loshchilov & Hutter, 2019) with a weight decay rate of 0.1 during pretraining.
The learning rate is first linearly increased to 0.001 within 2500 warmup steps, and then decayed to 0 following the cosine strategy.
For the 15M version pretraining data, we set the batch size to 4096 and run all experiments on 32 A100 GPUs. For 30M version pretraining data, we set the batch size to 8192 and run all experiments on 64 A100 GPUs.
[Table] of a { label } .
[Table]
photo [Table] many { label } .
[Table]
sculpture of a { label } .
[Table]
see { label } . [Table]
photo of [Table] { label } .
[Table]
rendering of a { label } . graffiti of a { label } .
[Table]
of the { label } . [Table] [Table]
[Table] of a { label } . [Table] embroidered
[Table]
[Table]
of
[Table]
see { label } .
[Table]
of a { label } .
[Table]
clean { label } .
[Table]
of
[Table]
[Table]
[Table]
[Table]
of a {
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
{ label } .
[Table]
[Table]
[Table]
[Table] [Table] [Table] a { label } .
[Table]
[Table] [Table] { label } . [Table]
[Table] [Table] [Table] photo of the { label } .
[Table]
{ label } . [Table]
rendering [Table] [Table] [Table]
[Table] [Table] game.
[Table] photo of one { label } . a doodle of a { label } . [Table]
of the { label } . [Table] of a { label } .
the origami { label } . [Table] [Table] [Table]
game.
[Table] a { label } . [Table] { label } .
[Table]
{ label } . [Table] a { label } . [Table]
[Table]
rendition of the { label } .
a photo of the clean { label } .
[Table]
photo [Table] large { label } .
a rendition of a { label } .
a photo of a nice { label } .
a photo of a weird { label } . a blurry photo of a { label } .
a cartoon { label } .
art of a { label } .
a sketch of the { label } .
a embroidered { label } .
a pixelated photo of a { label } .
itap of the { label } .
a jpeg corrupted photo of the { label } .
a good photo of a { label } .
a plushie { label } .
a photo of the nice { label } .
a photo of the small { label } .
a photo of the weird { label } .
the cartoon { label } .
art of the { label } .
a drawing of the { label } .
a photo of the large { label } .
a black and white photo of a { label } .
the plushie { label } .
a dark photo of a { label } .
itap of a { label } .
graffiti of the { label } .
a toy { label } .
itap of my { label } .
a photo of a cool { label } .
a photo of a small { label } .
a tattoo of the { label } .
D M ORE V ISUALIZATION R ESULTS & V ISUALIZATION P ROCESS
Besides the visualization results illustrated in Figure 3, we also provide eight more cases on unsupervised hierarchy induction from Figure 4 to Figure 11.
Moreover, we provide the detailed descriptions of the visualization process for input images in Algorithm 1, where we set the list of “break” threshold values { θ 1 , . . .
, θ 12 } to { 0 . 35 , 0 . 5 , 0 . 5 , 0 . 6 , 0 . 8 , 0 . 85 , 0 . 9 , 0 . 9 , 0 . 9 , 0 . 9 , 0 . 9 , 0 . 9 } .
For unsupervised grammar induction, we adopt the same parsing algorithm as in Tree Transformer (Wang et al., 2019).
Based on the visualization results, we can conclude that by integrating hierarchy-aware attention into the conventional attention mechanism, our HiCLIP can discover and aggregate spatially and semantically similar visual patches and language tokens in a layer-by-layer manner. However, current unsupervised hierarchy induction of HiCLIP (visualization of vision encoder especially) follows a top-down style and relies on the threshold values to decide whether to split two adjacent visual patches and language tokens. For the visual hierarchy, we trivially specify thresholds for different layers (the higher layer also has a higher threshold value).
Thus, the threshold list may not be suitable for every image.
In addition, changing the threshold values may influence the visual and language induction results. It would be better if the thresholds are adaptive to every input image and sentence.
Our future work is to find a better way (e.g., a data-dependent algorithm) to parse the
C matrix for each layer.
E V ISUALIZATION OF L EARNED F EATURE S PACE
In Figure 12 and Figure 13, we provide the t-SNE visualization of the learned feature space for CLIP,
HiCLIP and DeCLIP pretrained on YFCC-15M and 30M data, respectively.
We use the 10 classes of
CIFAR-10 dataset to conduct all the visualization experiments.
[Table]
F D ETAILED I LLUSTRATION OF THE C OMPUTATION OF C
In Figure 14, we illustrate the detailed computation steps of the attention mask C .
For the toy example sentence “a blue cat sitting on bench”, we show how the C li,j matrix in each Tree-Transformer layer is calculated from neighbourhood affinity scores a li,i +1 through the multiplication operation (i.e.,
C li,j = (cid:81) j − 1 k
=
i
a lk,k +1 ), where i ∈ { 0 , . . .
, N − 1 } , N is the input sequence length.
[Table]
71.7 HiDeCLIP-ViT-B/ 15M 88.
74.2 ( ↑ +2.5)
CLIP-ViT-B/ 30M 92.
73.8 HiCLIP-ViT-B/ 30M 92.
75.4 ( ↑ +1.6) DeCLIP-ViT-B/ 30M 93. 76.5 HiDeCLIP-ViT-B/ 30M 92. [Figure]
[Figure] [Figure] [Figure]
[Figure] [Figure] [Figure] [Figure]
[Figure]
Require: All neighboring affinity scores a l ( i j ) ( i ′ j ′ ) for l
N } layers, A list “break” threshold values { θ 1 , . . .
, θ N } every layer.
1: l ← N ▷
2: Initialize
▷ Store layer 3: while l > 0 do 4: the patch graph do 5: if a l ( i ( i j ) < then 6: l then 7: Append the edge
B l
N 8: else 9:
Append the edge ( i B layer l 11: end
12: end
13: end 14: end 15: l : end
[Figure]
[Figure]
[Table]
[Figure]