Fine-Grained Semantically Aligned Vision-Language Pre-Training
Abstract
Large-scale vision-language pre-training has shown impressive advances in a wide range of downstream tasks.
Existing methods mainly model the cross-modal alignment by the similarity of the global representations of images and texts, or advanced cross-modal attention upon image and text features.
However, they fail to explicitly learn the fine-grained semantic alignment between visual regions and textual phrases, as only global image-text alignment information is available.
In this paper, we introduce LOUPE , a fine-grained semantically a L igned visi O nlang U age P r E -training framework, which learns fine-grained semantic alignment from the novel perspective of game-theoretic interactions.
To efficiently compute the game-theoretic interactions, we further propose an uncertainty-aware neural
Shapley interaction learning module.
Experiments show that LOUPE achieves stateof-the-art performance on a variety of vision-language tasks.
Furthermore, without any object-level human annotations and fine-tuning, LOUPE achieves competitive performance on object detection and visual grounding.
More importantly, LOUPE opens a new promising direction of learning fine-grained semantics from largescale raw image-text pairs.
The repository of this work is at https://github.
com/YYJMJC/LOUPE .
Introduction
Learning transferable cross-modal representations from large-scale vision-language pre-training has exhibited remarkable performance on a wide variety of downstream tasks.
Most existing works can be classified into two categories: dual-encoder and fusion-encoder .
The dual-encoder methods [17, 26, 35, 49] adopt two separate encoders to embed images and texts, and model the cross-modal alignment by the cosine similarity between the global features of images and texts.
While such architecture is efficient for large-scale image-text retrieval by pre-computing image and text representations offline, they fail to model fine-grained semantic alignment between visual regions and textual phrases.
On the other hand, the fusion-encoder methods [7, 19, 20, 25, 30, 34, 43, 42] attempt to use a single multi-modal encoder to jointly model the concatenated sequence of images and texts.
These methods simulate soft alignment via advanced cross-modal attention [45].
However, they can only learn implicit alignment by end-to-end training, lacking explicit supervision to encourage semantic
[Table]
alignment between visual regions and textual phrases.
And the learned cross-modal attention matrices are often scattering and uninterpretable.
Further, they are inefficient for retrieval since it requires jointly encoding every image-text pair during inference.
Learning fine-grained semantic alignment from image-text pre-training is crucial to many crossmodal reasoning tasks ( e.g., visual grounding [51], image captioning [48]), but it is particularly challenging as the alignment information between visual regions and textual phrases is not available, posing fine-grained semantic alignment learning a weakly-supervised learning problem.
In this paper, we address this problem while simultaneously maintaining high retrieval efficiency by proposing
LOUPE , a fine-grained semantically a L igned visi O n-lang U age P r E -training framework, from the novel perspective of game theory.
We formulate input patch and word tokens as multiple players into a cooperative game and quantify game-theoretic interactions ( i.e., Shapley Interaction [12, 40]) among them to investigate the semantic alignment information.
LOUPE learns fine-grained semantic alignment from two stages: token-level Shapley interaction modeling and semantics-level Shapley interaction modeling , where we first learn to identify semantic regions of images that correspond to some semantically meaningful entities, and then align these regions with phrases in the paired text.
Specifically, token-level Shapley interaction modeling aims to group patch tokens of images into semantic regions that semantically correspond to some visual instances.
From the game-theoretic view, we take patch tokens as players and the similarity score between images and texts as the game function.
Intuitively, supposing a set of patch tokens correspond to a visual instance in the image, then they tend to have strong interaction to form the complete semantics of the corresponding instance, which contributes to the better similarity judgment with the paired text.
Based on this insight, we take the token-level Shapley interaction as soft supervision labels to encourage the model to capture semantic regions from images.
Then, semantics-level Shapley interaction modeling infers the fine-grained semantic alignment between semantic regions and phrases.
We consider every region and phrase as players and define a fine-grained similarity score as the game function.
If a region and a phrase have strong correspondence, they tend to interact with each other and contribute to the fine-grained similarity score.
By measuring the Shapley interaction between each region-phrase pair, we obtain the alignment information to guide the pre-training model.
As computing the exact Shapley interaction is an NP-hard problem [32], existing methods mainly employ sampling-based method [6] to obtain unbiased estimation.
However, as the number of players grows, they require thousands of model evaluations.
To reduce the computational cost, we further propose an efficient hybrid Shapley interaction learning strategy, where an uncertainty-aware neural
Shapley interaction learning module cooperates with the sampling-based method.
Experimental results show that our hybrid strategy significantly saves the computational cost while maintaining the estimation accuracy.
More analysis is shown in Section 4.5.
Our framework serves as a proxy training objective that explicitly establishes the fine-grained semantic alignment between local region and phrase representations.
This proxy objective can be directly removed for downstream tasks, rendering an efficient and semantics-sensitive dual-encoder model.
Experiments show that LOUPE achieves new state-of-the-art on image-text retrieval benchmarks.
For text-to-image retrieval on MSCOCO, LOUPE surpasses its strongest competitor by 4.2% on recall@1.
Further, without any fine-tuning, LOUPE successfully transfers to object detection and visual grounding tasks in a zero-shot manner.
For object detection, it achieves 12.1% mAP on COCO and 19.5% mAP on PASCAL VOC.
For visual grounding, it achieves 26.8% accuracy on RefCOCO and 23.6% accuracy on RefCOCO+.
Our contributions are summarized as follows:
• We propose LOUPE that explicitly learns fine-grained semantic alignment between visual regions and textual phrases while preserving the high retrieval efficiency of dual-encoder.
• We introduce an efficient and effective hybrid Shapley interaction learning strategy, based on an uncertainty-aware neural Shapley interaction learning module and a sampling-based method.
Pre-trained on image-text data, LOUPE achieves new state-of-the-art on image-text retrieval and successfully transfers to the tasks that require more fine-grained object-level visual understanding ( i.e., object detection and visual grounding ) without any fine-tuning.
• As manual annotations for masses of object categories is time-consuming and unscalable, our work demonstrates a promising alternative, that is, learning fine-grained semantics from raw texts about images, which are easily available and contain a broader set of visual concepts.
Related Work
Vision-Language Pre
-Training.
The great success of pre-train-and-fine-tune paradigm in natural language processing [5, 9] and computer vision [10, 14, 47] has been expanded to the joint domain of vision and language [2, 3, 22].
Dominant vision-language pre-training models can be categorized into two groups: dual-encoder and fusion-encoder .
The dual-encoder methods [17, 26, 35, 49] adopt two individual encoders to embed images and texts separately, and model the cross-modal interaction by cosine similarity.
Such architecture is efficient for large-scale image-text retrieval as image and text representations can be pre-computed offline.
However, simply measuring the cosine similarity between global representations is shallow to capture fine-grained semantic relationships between regions and phrases.
The fusion-encoder methods [7, 15, 16, 19, 20, 24, 25, 30, 34, 42, 43, 50, 55] adopt a single multi-modal encoder to jointly model the concatenated sequence of images and texts, which achieves deeper cross-modality interaction.
However, these methods are less efficient as images and texts are intertwined to compute the cross-modal attention and can not be pre-computed offline.
Further, there are no explicit supervision signals to encourage the alignment between regions and phrases.
Some works [7, 24, 25, 30, 43, 50, 55, 56] attempt to leverage an off-the-shelf object detector to extract object features for pre-training.
However, the detector is usually pre-trained on limited object categories.
Furthermore, considering the excessive demand on memory and computation, existing methods usually fix the parameters of detection models and regard region detection as a pre-processing step, disconnected with vision-language pre-training.
Thus, the performance is also restricted by the quality of detection models.
FILIP [49] uses a token-wise maximum similarity to enhance the cross-modal interaction of dual-encoder methods.
To learn explicit fine-grained semantic alignment, GLIP [23] and X-VLM [53] utilize human-annotated datasets, where regions with bounding-box annotations are aligned with text descriptions.
Such a manner is time-consuming and hard to scale to larger raw image-text data from the Internet.
In contrast, our proposed framework explicitly learns the fine-grained semantic alignment from raw image-text data and at the same time maintains the high efficiency of dual-encoder.
Detailed discussions can be found in Appendix K.
Shapley Values.
The Shapley value [40] was initially introduced in game theory.
It has been theoretically proven to be the unique metric to fairly estimate the contribution of each player in a cooperative game such that certain desirable axioms are satisfied [46].
With solid theoretic foundations, Shapley value has recently been studied as post-hoc explanation methods for Deep
Neural Networks (DNN) [8, 31, 54].
Lundberg et al.
[31] propose a unified attribution method based on Shapley value to interpret the predictions of DNN.
Ren et al.
[38] propose to explain adversarial attacks by Shapley value.
In this paper, we propose to model fine-grained semantic alignment by game-theoretic interactions, along with an efficient Shapley interaction learning strategy.
Method
In this section, we first introduce the problem formulation of fine-grained semantically aligned vision-language pre-training in Section 3.1.
Then, we propose the corresponding LOUPE framework for fine-grained semantic alignment learning in Section 3.2 and an efficient approach for Shapley interaction learning in Section 3.3.
3.1 Problem Formulation and Model Overview
Generally, vision-language pre-training aims to learn an image encoder f
I and a text encoder f T by cross-modal contrastive learning, where the matched image-text pairs are optimized to get closer and the mismatched pairs are optimized to get further.
Let f I
( I i ) and f T ( T i ) denote the global representations of the image and text.
Then the cross-modal contrastive loss can be formulated as:
[Equation]
where B is the batch size and τ is the temperature hyper-parameter.
While intuitive, such a manner can only learn coarse alignment between images and texts but fails to explicitly capture the fine-grained semantic alignment between visual regions and textual phrases.
To learn fine-grained semantic alignment while simultaneously maintaining high retrieval efficiency,
we
[Figure]
propose LOUPE, a fine-grained semantically aligned vision-language pre-training framework that germinates from cooperative game theory.
As illustrated in Figure 1, LOUPE learns fine-grained semantic alignment from two stages: tokenlevel Shapley interaction modeling and semantics-level Shapley interaction modeling .
For token-level Shapley interaction modeling, we learn to aggregate patch tokens of images into semantic regions that semantically correspond to some visual concepts, under the guidance of token-based semantic aggregation loss L TSA .
As for semantics-level Shapley interaction modeling, the semantic alignment between the aggregated regions and textual phrases is learned, supervised by the finegrained semantic alignment loss L FSA .
Combined with the two newly proposed losses, the full objective of fine-grained semantically aligned vision-language pre-training can be formulated as:
[Equation] [Equation]
Such a new pre-training objective enforces the image encoder to capture semantic regions and establishes fine-grained semantic alignment between visual regions and textual phrases.
During inference, it can be directly removed, rendering an efficient and semantics-sensitive dual-encoder .
3.2 Interpreting Fine-Grained Semantic Alignment as Game-Theoretic Interaction 3.2.1 Preliminaries
Shapley Values.
The Shapley value [40] is a classic game theory solution for the unbiased estimation of the importance or contribution of each player in a cooperative game.
Considering a game with
N = { 1 , ..., n } players, S ⊆
N denotes a potential subset of players.
A game v ( · ) is implemented as a function that maps each subset S of players to a score, modeling the outcome of a game when players in S participate in.
Specifically, v ( N ) − v ( ∅ ) denotes the contribution obtained by all players in the game.
The Shapley value ϕ
( i |N ) for player
i is defined as the average marginal contribution of player i to all possible coalitions S that are formed without i :
[Equation]
where p ( S ) is the likelihood of S being sampled.
The Shapley value has been proved to be the unique metric that satisfies the following axioms: Linearity , Symmetry , Dummy , and Efficiency [46].
We summarize these axioms in Appendix B.
Shapley Interaction.
In the game theory, some players tend to form a coalition and always participate in the game together.
The players in the coalition might interact or cooperate with each other, which brings additional contributions to the game.
The Shapley interaction [12] measures this additional contributions brought by the coalition compared with the case when the players work individually.
For a coalition S , we consider [ S ] as a single hypothetical player, which is the union of the players in
S .
Then, the reduced game is formed by removing the individual players in S from the game and adding [ S ] to the game.
The Shapley value ϕ
([ S ] |N \ S ∪ { [ S ] } ) for player [ S ] can be computed using Equation 3 over the reduced game.
Similarly, we can obtain ϕ
( i |N \ S ∪ { i } ) , where i is the individual player in S .
Finally, the Shapley interaction for coalition S is formulated as:
[Equation]
In this way, I ([ S ]) reflects the interactions inside S .
The higher value of I ([ S ]) indicates that players in S cooperate closely with each other.
3.2.2 Token-Level Shapley Interaction Modeling
Due to inherent semantic unit mismatch between texts and images, it is ineffective to directly compute the alignment between words and pixels (patches).
A textual phrase usually refers to a specific image region, which is composed of multiple patches and represents a visual instance.
Thus, we first introduce token-level Shapley interaction modeling to aggregate patches into semantic regions.
Input Representations.
Given an image-text pair, the input image I is sliced into patches and flattened.
Followed by linear projection layer and position embeddings, we obtain patch token
sequence
X I = { x Ii } L
i =1 with an additional [CLS_I] token embedding.
The input text T is tokenized and embedded into word token sequence X T
= { x Ti } L 2 i =1 , added with position embeddings.
We also prepend a learnable special token [CLS_T] to the word token sequence.
Then, we adopt a dual-encoder structure to encode the patch token sequence and word token sequence, separately.
On top of the image and text encoders, we obtain the representations of patch token sequence ˜ X I =
{ ˜ x Ii } ˜ L 1 i =1 and word token sequence ˜ X T
= { ˜ x Ti } ˜ L 2 i =1 .
We take the learned representations of [CLS_I] and [CLS_T] tokens as the global representations for images and texts.
And the global similarity of image-text pairs is measured by the cosine similarity between them.
Understanding Semantic Region via Shapley Interaction.
Supposing a set of patches represent a complete visual instance in an image, then they tend to have a strong Shapley interaction because they work jointly to form a visual instance, which contributes to the better similarity judgment with the text.
From the game-theoretic view, we take patch tokens and word tokens as players X =
X I ∪ X T , and the global similarity between images and texts as the game score v 1 ( · ) .
To compute v 1 ( S ) , we keep tokens in S and mask input tokens in X \ S to zeros.
Thus, the global similarity only considers the tokens in S , which reflects the contribution of the tokens in S to the global similarity judgment.
Semantic Region Generation.
Inspired by YOLOv3 [37], we design a lightweight region generation module.
It takes each patch token representation ˜ x Ii as input and generates a bounding box prediction centered on ˜ x Ii , which corresponds to a visual region R i = { x Ii,k } K i k
=1 with K i patch tokens.
The region generation module also predicts a confidence score
s
( R i ) for each region.
We select the top- M predictions as the semantic regions.
Then, the Shapley interaction of R i can be defined as:
[Equation]
According to the Equation 3, we can reformulate Shapley value into the form of expectation:
[Equation]
where c represents the coalition size.
ϕ ( x Ii,k |
X \ R i ∪ { x Ii,k } ) can be defined in a similar manner, and the Shapley interaction of R i can be reformulated as (we provide the proof in Appendix C):
[Equation]
Taking normalized I ′
([ R i ]) as the soft supervision label, the token-based semantic aggregation loss is defined as cross-entropy loss:
[Equation]
which propagates gradients to the region generation module and image encoder to adjust bounding box predictions such that more accurate semantic regions can be captured.
3.2.3 Semantics-Level Shapley Interaction Modeling
After obtaining the inferred semantic regions, we propose semantics-level Shapley interaction modeling to explicitly model the fine-grained semantic alignment between regions and phrases.
We first define the fine-grained similarity score and then explain semantic alignment based on game theory.
We adopt Avg-Pooling over learned patch representations in each R i to obtain region representation h Ii ∈ R d .
We employ an off-the-shelf constituency parser to extract phrases from text and obtain phrase representation h Ti ∈ R d by Avg-Pooling .
Totally, we obtain M regions H
I = { h Ii } Mi = and N phrases H T = { h Tj } Nj =1 .
And the alignment matrix can be defined as: A = [ a ij ] M × N , where a ij = h Ii ⊤ h
Tj represents the alignment score between i -th region and j -th phrase.
Next, we apply softmax-normalization over each row of A , obtaining ˜ A .
For the i -th region, we calculate its maximum alignment score as max j ˜ a ij .
Then, we use the average maximum alignment score over all regions as the fine-grained image-to-text similarity p 1 .
Similarly, we can obtain the fine-grained text-to-image similarity p 2 , and the total fine-grained similarity score can be defined: p
= ( p 1 + p 2 ) / 2 .
Understanding Semantic Alignment via Shapley Interaction.
If a region and a phrase have strong semantic correspondence, then they tend to cooperate with each other and contribute to the fine-grained similarity score.
Thus, we can consider H
= H
I ∪H T as the players and the fine-grained similarity score p as the game score v 2 ( · ) .
The Shapley interaction of them can be formulated as:
[Equation]
where [ H ij ] represents the single player formed by the coalition of i -th region and j -th phrase.
Taking normalized I ′
([ H ij ]) as soft labels, the fine-grained semantic alignment loss can be defined as:
[Equation]
3.3 Uncertainty-Aware Neural Shapley Interaction Learning
According to Equation 3 and Equation 4, computing exact the Shapley value is an NP-hard problem [32].
Previous methods mainly apply sampling-based method [6] to approximate it.
While sampling-based approximation is unbiased, an accurate approximation requires thousands of model evaluations.
To reduce the computational cost, we propose an uncertainty-aware neural Shapley interaction learning (UNSIL) module to cooperate with the sampling-based method, rendering an efficient and effective hybrid strategy.
Specifically, the sampling-based method [6] estimates the expectation terms in Equation 7 and
Equation 10 by sampling to compute the Shapley interaction.
Inspired by noisy label learning [18], the UNSIL module learns to predict the Shapley interaction and the corresponding uncertainty σ ∈ (0 , 1) .
Intuitively, if the UNSIL module makes a prediction with low uncertainty, we can directly apply its prediction to L TSA and L FSA , avoiding thousands of model evaluations.
If the uncertainty is high, we then resort to the sampling-based method for a more accurate estimation.
During training, the UNSIL module first predicts the target interaction with uncertainty σ .
Then, we sample a value ϵ from a uniform distribution on (0 , 1) .
If ϵ > σ , we directly use its prediction.
If
[Table] [Table]
n
[Table] [Table]
C oun
[Table]
r y
F
[Table]
R
A i r c r a f t s
O x f o r d P e t s
C a lt ec h
I m a g e N e t
A v e r a g e
CLIP 96.
[Table] [Table]
LOUPE 95.
[Table]
ϵ ≤ σ , we then use the sampling-based method to compute the Shapley Interaction and update the
UNSIL module based on the sampling-based results.
Note that, for the first few iterations, we employ the sampling-based method directly, and use its results to train the UNSIL module.
Let I ∗ and ˆ I denote the results from the sampling-based method and UNSIL module, respectively.
Taking I ∗ as the ground-truth, the UNSIL module is trained by:
[Equation]
where the first term is mean
squared error
L MSE weighted by the uncertainty, the second term serves as a regularization term for the prediction uncertainty, and β is the weight hyper-parameter.
The
UNSIL module implicitly learns the uncertainty from the regression loss function.
We discuss the implementation details of the UNSIL module in Section 4.5 and Appendix D.
Experiments 4.1 Pre-training Details
As sufficient data is a prerequisite for vision-language pre-training, we construct a dataset with 240M image-text pairs from the Internet.
We implement the image encoder by Swin-L [28] and the text encoder by BERT-Small [9].
The input images are resized to 224 × 224 and the input texts are tokenized by WordPiece with a maximum length of 60.
We pre-train the model for 20 epochs using a batch size of 512 on 128 NVIDIA V100 GPUs.
We utilize AdamW [29] optimizer with a learning rate of 2 × 10 − 4 and a weight decay of 0.01.
More pre-training and evaluation details are provided in
Appendix D, E. We also analyze the image encoder and training efficiency in Appendix G, J.
4.2 Zero-Shot Image-Text Retrieval
We compare LOUPE on the widely used MSCOCO [27] and Flickr30K [33] datasets.
First, the results in Table 1 show that LOUPE achieves new state-of-the-art zero-shot performance on most metrics of the two datasets, demonstrating the stronger generalizability of our pre-training framework.
Second, while previous works mainly pre-train on larger datasets (CLIP 400M, ALIGN 1800M,
FILIP 340M), LOUPE still achieves superior performance using less training data (240M).
Third, compared with FILIP which directly computes token-wise similarity, our model captures semantic alignment between visual regions and textual phrases, which is more semantically meaningful.
For text-to-image retrieval on MSCOCO, LOUPE significantly outperforms FILIP by 4.2% on recall@1.
4.3 Zero-Shot Image Classification
In this section, we evaluate LOUPE on the zero-shot image classification task.
We compare LOUPE with CLIP on 11 downstream classification datasets, following the same evaluation setting as
[Table]
[Figure]
CLIP [35].
Table 2 summarizes the results.
As shown in Table 2, our LOUPE outperforms CLIP with average improvement of 3.1%.
Notably, on ImageNet, the largest dataset among 11 datasets, our LOUPE surpasses CLIP by 0.8%.
Also, we observe that LOUPE achieves substantial performance gains on several fine-grained image classification datasets (i.e., Flowers102 and Aircrafts).
It demonstrates the superiority of our LOUPE on fine-grained semantics understanding.
We also evaluate the linear probing performance of our LOUPE on image classification.
The detailed results can be found in Appendix I.
4.4 Zero-Shot Transfer to Object Detection and Visual Grounding
To answer whether our model has learned fine-grained semantics, we further evaluate LOUPE on object detection [39] and visual grounding [51], which require more fine-grained semantic understanding ability to identify specific visual regions in images according to the object labels or referring expressions.
Visual grounding can be seen as generalized object detection, where the pre-defined class labels are replaced by language referring expression sentences.
As LOUPE can generate a set of semantic regions that are aligned with textual phrases, it can be easily applied to object detection and visual grounding without structure modification.
For visual grounding, we take referring expressions as input text.
For object detection, as illustrated in Figure 2, we use prompt to expand detection labels to input text.
Then, we encode input text by the learned text encoder, and these tasks can be completed by measuring the similarity between candidate region representations and text representations.
For comparison, we zero-shot transfer CLIP (ViT-L/14) to object detection and visual grounding by applying several non-parametric approaches on the spatial feature maps of CLIP.
We also compare with AdaptCLIP [21], which is a concurrently unpublished method that leverages classic superpixel (SLIC [1]) and bounding box proposal (selective search [44]) methods to zero-shot transfer
CLIP to phrase localization.
We use its public official implementations to get the experiment results.
For object detection, we evaluate their mean Average Precision (mAP) at IoU thresholds of { 0 . 3 , 0 . 5 } on COCO [27] (65 classes) and PASCAL VOC [11] (20 classes).
For visual grounding, we evaluate their top-1 accuracy at IoU thresholds of 0.5 on RefCOCO [51] and RefCOCO+ [51].
The experiment details of CLIP variants and LOUPE are provided in Appendix E.
Table 3 summarizes the results.
1) Overall, LOUPE outperforms all CLIP variants by a large margin.
The significantly higher performance illustrates the stronger zero-shot transfer ability of our fine-grained semantically aligned pre-training paradigm.
Second, all CLIP variants rely on pre-processing steps on CLIP’s feature map ( e.g., AdaptCLIP first uses SLIC to group pixels and then uses selective search to generate a large number of proposals), which is time-consuming.
In contrast, our method directly predicts the semantic regions based on the patch token representations.
Third, the consistently competitive performance across four benchmarks validates that LOUPE can learn fine-grained semantics from raw text supervision.
LOUPE demonstrates a promising alternative, that is, learning fine-grained semantics from large-scale raw image-text pairs, which are easily available and contain a broader set of visual concepts.
[Table]
As time-consuming human annotations are unscalable for massive object classes in the real world, some recent works [4, 36] target at training object detectors with annotations on base object classes to generalize to the remaining object classes of the same dataset.
The latest works [13, 52] leverage the generalizability of vision-language pre-training models to further improve the zero-shot performance on novel classes.
However, these zero-shot approaches still require bounding box annotations on base classes for task-specific supervised learning.
In contrast, our LOUPE is trained on large-scale raw image-text pairs, which are already accessible on the Internet and contain more diverse semantics.
4.5 Ablation Study
Effectiveness of Individual Components.
In this section, we investigate the effectiveness of each component in Table 4.
Given the costly training time, all ablation studies are based on a relatively small dataset (Conceptual Captions 3M [41]).
We start with the backbone model that consists of a dual-encoder trained by cross-modal contrastive loss.
We then gradually add tokenlevel Shapley interaction modeling supervision L TSA (Row 2), semantics-level Shapley interaction modeling supervision L FSA (Row 3), and UNSIL module (Row 4).
For Row 2 and Row 3, the
Shapley interaction is only computed by the sampling-based method.
The results in Table 4 show that both L TSA and L FSA bring significant improvement for all tasks.
We observe that L TSA boosts a 3.8% improvement on object detection.
And the improved fine-grained visual semantic understanding further facilitates the cross-modal retrieval performance (+1.4%).
The semantics-level
Shapley interaction modeling further improves the performance on all tasks by modeling the semantic alignment between visual regions and textual phrases.
Comparing Row 3 and Row 4, we observe that the UNSIL module maintains the estimation accuracy while avoiding intensive computations.
The averaged training time is reduced from 9.90 seconds per iteration to 1.93 seconds per iteration.
Accuracy of the Shapley Interaction Learning.
Since we use the sampling-based method [6] to compute the Shapley Interaction and train the UNSIL module, we conduct a study to evaluate the accuracy of the sampling-based method and the error of the UNSIL module.
As [54], we compute the interaction multiple times and measure the instability of them.
A lower instability means that we obtain similar interactions from different sampling processes.
It indicates a high accuracy.
Specifically, the instability is defined as E u,v : u ̸
= v |
I u − I v | E w |
I w | , where I w denotes the interaction computed in the w -th time.
We average the instability values over Shapley interaction of 100 image-text pairs.
We report the average instability values with respect to different sampling numbers.
As shown in Figure(a), the instability decreases along with the increase of the sampling number.
When the sampling number is larger than 500, the approximated Shapley interaction is stable enough with instability less than 0.06.
Further, we attempt different models ( i.e., Conv1D, 3-Layer MLP + Attention, 3-Layer
Transformer ) to implement the UNSIL module (please see Appendix D for more details).
We test them on 1000 samples and report their mean uncertainty and relative error in Figure 3 (b) and (c).
We observe that MLP + Attention is good enough to predict the interaction with lower complexity.
Thus, we implement the UNSIL module by MLP + Attention .
[Figure] [Figure]
4.6 Qualitative Analysis
Qualitative Examples.
As shown in Figure 4, LOUPE successfully captures the regions that correspond to the detected objects, and grounds the referring expressions onto the referred regions.
Visualization of Learned Fine-Grained Semantic Alignment.
In Figure 5, we visualize some key semantic regions and corresponding alignment matrices inferred by LOUPE.
We present the regions with top-3 confidence (Region 1 – 3) and two randomly sampled regions (white boxes).
The red boxes at the bottom of bounding boxes indicate their normalized token-level Shapley interaction values.
Comparing their Shapley interaction values, we observe that the token-level Shapley interaction successfully distinguishes semantic regions from randomly sampled regions.
The semantically meaningful regions tend to have stronger interaction.
It indicates that token-level Shapley interaction can provide correct supervision for semantic region generation.
Further, we show the alignment matrices inferred by semantics-level Shapley interaction and LOUPE, respectively.
As shown in the right case of Figure 5, LOUPE successfully recognizes the leash region and aligns it with the “a leash” phrase.
Note that existing object detection datasets do not contain the “leash” category.
Conclusion
This paper introduces a novel vision- language pre-training framework, LOUPE , which models the fine-grained semantic alignment between visual regions and textual phrases by game-theoretic interactions.
To efficiently compute the interactions, we further propose an uncertainty-aware neural
Shapley interaction learning module.
Comprehensive experiments show that LOUPE achieves new state-of-the-art on image-text retrieval datasets and can transfer to object detection and visual grounding in a zero-shot manner.
This work demonstrates a new promising direction of learning fine-grained semantics from large-scale raw image-text data.
Limitations.
The phrases are extracted by off-the-shelf constituency parsers, whose predictions might not be completely accurate.
The web data might inevitably contain mismatched image-text pairs, leading to noisy supervision.
Social Impacts.
Our model is trained on noisy data from the Internet, which may contain unsuitable images, violent text, or private information.
Thus, additional analysis of the data is necessary.
Further, the use of our model for privacy surveillance or other nefarious purposes should be prohibited.
Acknowledgment.
This work has been supported in part by the National Key Research and Development Program of China (2018AAA0101900), Zhejiang NSF (LR21F020004), Key Research and
Development Program of Zhejiang Province, China (No. 2021C01013), Chinese Knowledge Center of Engineering Science and Technology (CKCEST).
We thank all the reviewers for their valuable comments.
[-X I = { x Ii } L 
-]