U NIVERSAL V ISION -L ANGUAGE D ENSE R ETRIEVAL : L EARNING A U NIFIED R EPRESENTATION S PACE FOR M ULTI -M ODAL R ETRIEVAL
A BSTRACT
This paper presents Universal Vision-Language Dense Retrieval (UniVL-DR), which builds a unified model for multi-modal retrieval.
UniVL-DR encodes queries and multi-modality resources in an embedding space for searching candidates from different modalities.
To learn a unified embedding space for multi-modal retrieval,
UniVL-DR proposes two techniques: 1)
Universal embedding optimization strategy, which contrastively optimizes the embedding space using the modality-balanced hard negatives; 2) Image verbalization method, which bridges the modality gap between images and texts in the raw data space.
UniVL-DR achieves the state-ofthe-art on the multi-modal open-domain question answering benchmark, WebQA, and outperforms all retrieval models on the two subtasks, text-text retrieval and text-image retrieval.
It demonstrates that universal multi-modal search is feasible to replace the divide-and-conquer pipeline with a united model and also benefits single/cross modality tasks.
All source codes of this work are available at https: //github.com/OpenMatch/UniVL-DR .
I NTRODUCTION
Although search engines primarily focus on textual data (Singhal et al., 2001) multi-media is necessary to satisfy user needs during retrieval.
A user query can be answered by the information in variant formats, such as a text document or a picture.
The growth of multi-media content has been one of the most notable trends on the internet (Mei et al., 2014), and various studies have proved that users prefer more vivid multi-media content in search results (Datta et al., 2008).
Current multi-media search systems often employ a divide-and-conquer approach.
As shown in
Figure 1(a), they first conduct search in individual modalities, including text, image, video, etc.
(Bajaj et al., 2016; Grubinger et al., 2008; Kwiatkowski et al., 2019; Awad et al., 2021), and then fuse the retrieval results from various verticals together, e.g., building another ranking layer on top of these single/cross modality retrievers (Escalante et al., 2008; Grubinger et al., 2008).
Both relevance modeling and retrieval result fusion are usually entwined to achieve more accurate multi-modal retrieval results.
However, due to the modality gap, they can be only pipeline-modeled in divide-andconquer, making it challenging to fuse retrieval results from different modalities.
In this paper, we explore the potential of universal multi-modal retrieval to build an end-to-end model and retrieve multi-modality documents for user queries.
Illustrated in Figure 1(b), universal multi-modal retrieval maps queries and multi-modality resources to one universal embedding space and retrieves multi-modality candidates via KNN search.
As a result, the relevance modeling, cross-modality matching, and retrieval result fusion are done by one model.
More specifically, we propose a Universal Vision-Language Dense Retrieval (UniVL-DR) model to get the representations of queries, texts, and images and learn a tailored vision-language embedding space for multi-modal retrieval.
UniVL-DR optimizes the vision-language embedding space using hard negatives (Xiong et al., 2021a) and balances the modalities of these negatives to alleviate
… Images
Texts
Text IR Text IR
ImageRetrieval
ImageRetrieval … … … …
Ranking List
Sub-Ranking Lists
Query (
[Figure]
Construction of the Xanadu house in Kissimmee, Florida, began with the pouring of a concrete …
In 1946, he was honored on the first coin to feature an African American, the Booker T. Washington Memorial Half Dollar …
The National Air and Space Museum of the Smithsonian Institution, also called the Air and Space Museum … …
KNN Search
Ranking List
Query
[Figure]
Search.
the modality preference of multi-modal retrievers.
Furthermore, UniVL-DR introduces an image verbalization method, which regards language as a kind of mentalese (Cavanagh, 2021) and mitigates the modality gap between images and texts.
Our image verbalization method first aligns the semantics of image captions and figure pixels (Huang et al., 2021a), and then paraphrases the image facts.
It helps to bridge language and vision understanding modules of UniVL-DR via natural language.
To build a multi-modal retrieval benchmark, we leverage a multi-modal question answering (QA) benchmark
WebQA (Chang et al., 2022) and convert it to a standard open-domain setting: retrieving multi-modality candidates from text and image collections for a user query.
Divide-and-conquer is an intuitive way to build a multi-modal retrieval system and we pre-route queries to oracle modality to show the upper bound performance of such a system.
Compared with the divide-and-conquer system,
UniVL-DR addresses the retrieval result fusion challenge, achieves state-of-the-art multi-modal retrieval performance, and brings more than 5% improvement in single/cross modality retrieval.
Our experiments show that UniVL-DR learns an effective embedding space for multi-modal retrieval by separating texts and images into different areas and guiding queries to return candidates from corresponding modalities.
Our further analyses show that UniVL-DR can alleviate overfit singlemodality signals by balancing hard negatives during training and bridging the modality gap between vision and language by verbalizing images.
All experimental results show that learning one universal representation space is starting to benefit single-modality tasks—pretraining representation models on multi-modality and using our techniques can learn additional signals from multi-modalities, overcome the modality boundary, and provide convincing gains in single/multi-modality tasks.
R ELATED W ORK
Document retrieval is a typical single modality retrieval task, which aims to return related documents for user queries and can be tackled with dense retrievers (Xiong et al., 2021b; Lewis et al., 2020; Zhan et al., 2021;
Li et al., 2021b; Yu et al., 2021).
Dense retrievers encode queries and documents with pretrained language models (Devlin et al., 2019) and map them in an embedding space to conduct an efficient search.
The query and document encoders are usually contrastively trained with in-batch negatives, BM25 retrieved negatives, and hard negatives (Karpukhin et al., 2020; Xiong et al., 2021a).
Recently, lots of work has focused on multi-modal retrieval tasks, which retrieve texts and images to satisfy the multi-modality information needs of users (Hannan et al., 2020; Singh et al., 2021;
Talmor et al., 2021; Chang et al., 2022).
WebQA (Chang et al., 2022), an open-domain multi-modal question answering benchmark, is built to encourage the following work to represent multi-modal knowledge in a unified space and answer user queries with the information from attribute modalities.
It is a more realistic setting, which avoids synthesizing queries with templates (Talmor et al., 2021) and downplays the role of modality disambiguation (Hannan et al., 2020) in the multi-modality modeling.
To search information from large-scale multi-modality sources, WebQA (Chang et al., 2022) employs a divide-and-conquer pipeline to search text and image candidates with BM25 and CLIP (Radford et al., 2021) and then fuse these retrieval results using a vision-language model.
However, single-
Text1: William Bradford was an English Separatist leader in Leiden and Holland.
He served as Plymouth Colony Governor five times covering about thirty years between 1621 and 1657.
Text2:
William Bradford was a passenger on the Mayflower in 1620.
He travelled to the New World to live.
Text3: William Bradford was the governor of Plymouth Colony for 30 years.
The colony was founded by people called Puritans.
Query: how many years did william bradford serve as governor of plymouth colony?
Retrieval Candidates:
[Figure]
modality retrievers, such as BM25 and CLIP, usually show distinct retrieval effectiveness (Chang et al., 2022), leading to modality discrimination during fusing retrieval results from different modalities. When building a unified multi-modal retriever, vision-language pretraining (VLP) is crucial to learn universal representations for texts and images, which has also shown success on lots of visionlanguage benchmarks (Uppal et al., 2022; Han et al., 2020; Khan et al., 2021; Du et al., 2022). Most VLP approaches encode texts and images and pretrain encoders with two tasks: masked token prediction and text-image matching (Zhang et al., 2021).
These VLP methods teach vision-language models to learn the semantic alignments between texts and images, as well as encode images with the regional features of detected objects (Chen et al., 2019;
Lu et al., 2019; Tan and Bansal, 2019;
Su et al., 2020; Li et al., 2019; 2021a; Cho et al., 2021;
Hu et al., 2020; Gan et al., 2020) or the whole image features (Xu et al., 2021;
Kim et al., 2021; Huang et al., 2021b; Wang et al., 2021).
M ULTI -M ODAL R ETRIEVAL T ASK
As shown in Figure 3, we compare different retrieval tasks and tell apart the differences between multi-modal retrieval and other two tasks, single modality retrieval and cross modality retrieval.
Single Modality Retrieval.
Single modality retrieval focuses on conducting relevance searching in one modality space, which includes text-text retrieval and image-image retrieval.
Text-text retrieval (Bajaj et al., 2016) aims to search relevant candidates from the text collection T
= { T 1 , ..., T n } to answer a query q .
And image-image retrieval (Yoon et al., 2021) focuses more on returning similar images from the image collection
I =
{ I 1 , ..., I m } for the given image
I j .
Cross Modality Retrieval.
The cross modality retrieval, e.g. MSCOCO (Chen et al., 2015) and
Flickr30K (Young et al., 2014), contains two subtasks: text-image retrieval and image-text retrieval.
Given an image caption T i or an image I j , these tasks require retrieval models to conduct crossmodality matching between images and captions, aiming to search candidates from images
I =
{ I 1 , ..., I m } or image captions T
= { T 1 , ..., T n } , respectively.
Such cross-modality interactions are built to align semantics between captions and images, which is distinct from the search relevance.
Multi-Modal Retrieval.
Given a query q , the multi-modal retrieval task (Chang et al., 2022) helps users uncover the information from multi-modality sources D
= { T 1 , ..., T n , I 1 , ..., I m } .
Different from single/cross modality retrieval, multi-modal retrieval aims at returning relevant candidates from the multi-modality documents D .
The retrieval results may consist of texts, images, or a mixture of them according to user query q .
Different from existing text and sketch base image retrieval (Sangkloy et al., 2022; Dutta and Akata, 2020; Dey et al., 2018; Mai et al., 2017), the multimodal retrieval focuses more on relevance modeling between queries and documents, single/cross modality matching, and modality routing, making this task more challenging.
Moreover, we can pre-route queries to a single modality and convert the multi-modal retrieval to two subtasks, text-text retrieval and text-image retrieval, which are single and cross modality retrieval tasks.
Text Document
Query
Text Encoder
Image Encoder
Query
Image Document
ImageCaption
PictureFeature
Verbalize
Text
Image Caption
Image Feature
Sum
Image
A li g n
Training with Modality-
Balanced Hard Negatives
GroundTruth
P u s h
A w a y
U NIV S EARCH BY L EARNING A U NIFIED E MBEDDING S PACE
This section describes our Universal Vision-Language Dense Retrieval (UniVL-DR).
As shown in
Figure 3, given a query q and multi-modality documents D = { d 1 Text , ..., d n Text , d 1 Image , ..., d m Image } , it directly encodes query q , text document d i Text and image document d j Image in one embedding space, which conducts relevance modeling, modality routing, and result fusion in such a space (Sec. 4.1). Texts and images usually have different understanding mechanisms, making it difficult to tackle multi-modality tasks.
Nevertheless, language and vision can be commonly translated as a type of mentalese to better communicate between different modules in our brains (Cavanagh, 2021), thus a unified representation method has the ability to break the boundary of different modalities and benefit vision-language learning.
To build a unified multi-modal retrieval system, UniVL-DR learns a universal embedding space by contrastively optimizing vision-language representations using hard negatives with balanced-modality sampling (Sec. 4.2) and bridging the modality gap via verbalizing the picture to paraphrase pixel semantics in the raw text space (Sec. 4.3).
4. M ULTI -M ODALITY D ENSE R ETRIEVAL
UniVL-DR gets representations of queries, image documents and text documents with two encoders:
TextEnocder and ImgEncoder .
Specifically, the image document d j Image consists of a picture I j and an image caption C j , thus we utilize ImgEncoder and TextEnocder to encode I j and C j .
Query Encoding.
UniVL-DR directly encodes the query q to get its representation ⃗q :
[Equation]
Text Document Encoding.
To represent text documents, UniVL-DR also leverages the TextEnocder to encode the i -th text document d i Text as ⃗d i
Text :
[Equation]
Image Document Encoding.
Different from text documents, image documents can be represented by picture features and image captions and the textual captions can help better understand the semantics of image documents (Baldrati et al., 2022).
Thus, UniVL-DR encodes picture I j and image caption
C j
and then sums these embeddings to get the representation ⃗d j Image of j -th image document:
[Equation]
The representations ⃗d j Image and ⃗d i
Text of image document and text document use the same
TextEnocder to encode their textual information, which bridges different modalities in the text space and helps to build a universal embedding space for multi-modality retrieval.
Multi-modality Document Retrieval.
The cosine similarity score f ( q, d ) of query q and document candidate d ∈ D can be calculated to estimate the relevance between q and d :
[Equation]
where ⃗q and ⃗d are the representations of q and d .
The efficient similarity calculation between queries and the multi-modality documents can be provided by FAISS (Johnson et al., 2019).
4. U NIVERSAL R EPRESENTATION L EARNING
UniVL-DR employs a vision-language model, CLIP (Radford et al., 2021), to learn universal representations for queries and multi-modality documents, which is knowledgeable about crossmodality retrieval.
UniVL-DR optimizes the universal embedding space through training with modality-balanced hard negatives, which avoids overfitting to the signals of single-modality during multi-modal co-training.
Given the query q and its relevant candidate d + ∈ D , the embedding space can be optimized by sampling hard negatives D − and minimizing the following contrastive training loss L :
[Equation]
where τ is the temperature to scale the similarity score.
During training, we in fact maximize L Align and minimize L Image and L Text , which make queries closer to related documents and away from unrelated documents.
If k 1 > k 2 or k 2 > k 1 , we can achieve a smaller loss L Image + L Text by simply making queries far away from the image collection or the text collection.
Such a behavior can win a lower loss L but overfits the ranking features from single/cross modality matching, leading to a modality discrimination during retrieval.
Our modality-balanced negative training strategy keeps k 1 = k
=
k to better train the modality selection ability of retrievers.
4. I MAGE V ERBALIZATION FOR E XPANSION
UniVL-DR provides another way to bridge the modality gap between texts and images by verbalizing picture pixel features, including image caption and query generation methods.
Following Li et al.
(2020), we can represent a picture
I j using detected objects O = { O 1 , ..., O l } .
For each image object O i , we can get its pixel feature
⃗O
i and the predicted class
ˆ O i .
Then UniVL-DR uses a vision-language model, such as VinVL (Zhang et al., 2021), to verbalize image documents.
Specifically, we generate potentially matched captions or related queries as the image verbalization results V ( I j ) , according to the picture I j or the image document d j Image
= { I j , C j } .
We can first feed the predicted classes { ˆ O 1 ; ... ; ˆ O l } and regional features { ⃗O 1 ; ... ; ⃗O l } of detected objects into image verbalization models.
Then we train the model to generate image caption C j :
[Equation]
or replace the detected object classes { ⃗O 1 ; ... ; ⃗O l } in the input sequence X cj with the image caption
C j to generate related query q of the image document d j Image :
[Equation]
where ; is the concatenation operation, and [CLS] and [SEP] are special tokens.
During training or inference, we utilize Masked Language Modeling (MLM) (Devlin et al., 2019) to mask and predict some or all of the tokens of image caption C j and query q in the inputs X cj and X qj , aiming to train image verbalization models or generate verbalized captions and queries.
Finally, we enhanced the representations of image documents by expending their text representations
C ∗ j by expanding the raw caption C j with image verbalization results V ( I j ) :
[Equation]
where the enhanced text representation C ∗ j is used to replace the raw caption C j in E.q. 3 during encoding the image document d j Image .
E XPERIMENTAL M ETHODOLOGY
This section describes the dataset, baselines, some vision language models used in our experiments, and implementation details.
Dataset.
A multi-hop and multi-modal open domain question answering dataset WebQA (Chang et al., 2022) is used in our experiments.
We process the WebQA dataset in an open domain retrieval setting and show the details in Appendix A.1.
Evaluation Metrics.
We use NDCG@ K , MRR@ K , Recall@20, and Recall@100 as the evaluation metrics.
K can be 10 and 20.
And we regard MRR@10 as our main evaluation (Bajaj et al., 2016).
Vision-Language Models.
In our experiments, we employ two state-of-the-art vision-language models, VinVL (Zhang et al., 2021) and CLIP (Radford et al., 2021) to implement different retrieval models in our experiments.
VinVL (Zhang et al., 2021) inherits Oscar (Li et al., 2020) architecture, which extracts object tags and region features to represent images, and learns cross-modal representations by aligning semantics between images and texts.
Different from VinVL, CLIP (Radford et al., 2021) utilizes a dual encoder to project images and texts in the same semantic space for computing their similarity scores and is trained on a large-scale dataset WebImageText that contains 400 million image-text pairs.
It has shown strong effectiveness in cross-modality retrieval.
Baselines.
Our baselines contain several models in the settings of single modality retrieval, divideand-conquer, and universal multi-modal retrieval.
Single modality retrieval .
In this setting, we represent image documents with captions and employ text retrievers, BM25 and DPR (Karpukhin et al., 2020) as baselines.
DPR is trained with
NQ (Kwiatkowski et al., 2019), which is similar to the textual source of WebQA.
Then we continuously train DPR with in-batch and hard negatives to implement NQ-DPR and NQ-ANCE models.
Divide-and-conquer.
We first employ three widely used retrievers, BM25, VinVL-DPR, and CLIPDPR, to conduct text-text retrieval and text-image retrieval.
Then the multi-modality retrieval results are fused according to their uni-modal rank reciprocals or oracle modality routing.
The latter one shows the upper bound of the retrieval performance of our divide-and-conquer models.
Multi-modal retrieval.
In our experiments, we also build two multi-modal retrieval baselines: VinVLDPR and CLIP-DPR.
VinVL-DPR and CLIP-DPR represent image documents with caption and picture features.
And then they optimize VLP models, VinVL (Zhang et al., 2021) and CLIP (Radford et al., 2021), with in-batch negatives to learn universal representations for multi-modal retrieval.
Implementation Details.
During training UniVL-DR, we employ the text and image encoders from
CLIP, truncate the text with the max length of 77 1 and set the batch size to 64, learning rate= 5 e − 6 , max training epoch to 20, and the temperature hyperparameter τ = 0 .
In our experiments, we retrieve Top 100 documents using CLIP-DPR and sample two hard negatives of different modalities ( k = 1 ) from these candidates.
All models are tuned with AdamW optimizer, are evaluated persteps, and set early stop step as 5.
More experimental details are shown in Appendix A.2.
E VALUATION R ESULTS
In this section, we study the performance of UniVL-DR, its advantages in multi-modal retrieval, the effectiveness of our modality-balanced hard negative training strategies, and how our image verbalization methods bridge the modality gap between texts and images.
6. O VERALL P ERFORMANCE
The multi-modal retrieval performance of different models is shown in Table 1.
Our UniVL-DR outperforms all baselines with more than 7% improvement on ranking evaluation, recalls more than 6% relevant multi-modality documents, and even outperforms the divide-andconquer model guided by oracle modality routing.
Such significant improvements illustrate the effectiveness of UniVL-DR in building a multi-modal retrieval system.
[Table]
Similar to UniVL-DR, BM25 learns universal textual representations for image/text documents and shows strong ranking effectiveness.
To build a divide-and-conquer system, we use BM25 and
CLIP-DPR to implement text-text and text-image retrievers and then fuse the results from different retrievers.
With the help of oracle modality routing, the divide-and-conquer system shows better ranking results and recalls more relevant documents than BM25.
Nevertheless, this system shows a distinct performance when using the uni-modal rank reciprocals to route queries, showing the challenge of fusing retrieval results in divide-and-conquer.
CLIP-DPR and UniVL-DR can deal with this problem by learning universal representations for queries and multi-modality documents, which unifies the multi-modality relevance modeling and retrieval result fusion.
Thanks to our multi-modality training strategies, UniVL-DR achieves more than 10% improvement on multi-modal retrieval than CLIP-DPR.
The following experiments further explore how UniVL-DR learns universal representations for multi-modal retrieval and bridges the gap between images and texts.
6. A BLATION S TUDIES
The ablation studies are conducted to study model performance on multi-modal retrieval.
And we also evaluate the effectiveness of UniVL-DR on both text-text and text-image retrieval tasks, which aims at showing the influence of multi-modal learning on these single/cross modality retrieval tasks.
As shown in Table 2, we evaluate the retrieval effectiveness of different vision-language models,
VinVL-DPR and CLIP-DPR.
They are trained with in-batch negatives on text-text/image and multimodal retrieval tasks.
In the single/cross modality setting, we fine-tune vision-language models with a group of queries that only contain related documents in text modality or image modality.
Our multi-modality training setting uses all queries to train these vision-language models and equally samples in-batch negatives from the documents of different modalities.
For both CLIP-DPR and VinVL-DPR, image captions are usually more effective to represent image documents than figure features, which demonstrates the difficulty in understanding figure semantics with only figure pixels.
Thus, UniVL-DR tries to verbalize the figure features by extracting the objects that appear in the figure and describing the figure facts among detected objects (Zhang et al., 2021).
The image verbalization results paraphrase picture pixel facts in natural language and help to enhance the textual representations of images by expanding image verbalization results to image captions.
As a result, UniVL-DR uses such an enhanced text representation for image documents and then employs the same module to encode text information of image documents and text documents.
It helps to build universal representations for multi-modality documents by breaking the modality boundary and fully using additional training signals from different modalities, making UniVL-DR achieve the best retrieval performance on multi-modal retrieval among all baseline models.
UniVL-DR also shows its advantages by outperforming all baseline models on both text-text and text-image retrieval tasks, demonstrating that multi-modality modeling indeed benefits single/cross modality retrieval.
In the multi-modal retrieval setting, CLIP-DPR is converted from a text-text retriever to a multi-modal retriever after adding figure features.
CLIP-DPR achieves better performance on the text-image retrieval task than CLIP-DPR w/o figure feature, which illustrates that image features provide additional signals to help multi-modality models distinguish related image documents. On the contrary, the multi-modal retrieval performance of CLIP-DPR is decreased, [Table] [Table]
of the Top 10 retrieved candidates are shown to evaluate the modality preference [Figure] showing that CLIP-DPR fails to fuse retrieval results from different modalities.
UniVL-DR uses a modality-balanced hard negative training strategy to learn universal representations for queries and documents, which deals with the challenge of fusing retrieval results, helps to achieve more gains on the multi-modal retrieval task, and enhances the modality disambiguation ability.
6. E FFECTIVENESS OF B ALANCED H ARD N EGATIVE S AMPLING
In this experiment, we study the training strategies of UniVL-DR that are used in learning universal multi-modality representations and show the effectiveness of different negative sampling methods.
As shown in Table 3, we start from the multi-modal retriever CLIP-DPR, continuously fine-tune it with different hard negative sampling methods, and show their performance on different retrieval tasks.
Our experimental results show that the in-batch trained models prefer to return text documents than image documents as the ranking results, even the image-answerable queries take a larger portion (about 51.6%) in the training data.
It illustrates that training multi-modality retrievers with modality-unbalanced negatives usually leads to undesired modality bias during retrieval.
Then we continuously train CLIP-DPR with hard negatives sampled from top-retrieved multi-modality results from CLIP-DPR and significantly improve its retrieval performance in all testing scenarios.
Our modality-balanced hard negative sampling strategy achieves the best retrieval performance among all negative sampling methods, showing its important role in building a universal multi-modal retrieval model.
Compared with ANCE (Random), our modality-balanced sampling strategy mitigates the modality variance during contrastive training and provides more useful signals to train the modality disambiguation ability of universal multi-modal retrievers.
Finally, we visualize the embedding space of different retrieval models in Figure 4.
After training with modality-balanced hard negatives, UniVL-DR learns a more uniform and effective embedding space for multi-modal retrieval.
In this embedding space, both text and image documents are [Table] assigned in different areas of the embedding space, and queries are routed to different areas for returning documents from corresponding modalities.
As shown in Figure 4(b) and Figure 4(c), when the retrieval models are only trained with hard negatives of text and image documents, the query embeddings are concentrated and respectively assigned closer to the areas of image and text documents.
It demonstrates that multi-modality retrieval model usually overfits the training signals of in-batch majority modality to win a lower contrastive loss during training.
UniVL-DR alleviates this problem by balancing the modalities of hard negatives in contrastive training.
6. B RIDGING C ROSS -M ODALITY M ATCHING WITH I MAGE V ERBALIZATION
UniVL-DR uses image verbalization methods to generate matched captions or related queries to bridge the modality gap between texts and images.
In this experiment, we show the effectiveness of different image verbalization strategies on text-text, text-image, and multi-modal retrieval tasks.
As shown in Table 4, our image verbalization methods demonstrate their ability on enhancing the text representations of image documents by achieving better text-image retrieval results.
These image verbalization methods aim to generate informative text clues to help retrievers distinguish the queryrelated image documents in the text space.
Then the text-text and multi-modal retrieval performance is also improved with the help of verbalized captions or queries, showing the effectiveness of our image verbalization methods in bridging the modality gap between images and texts and benefiting the single modality tasks using additional training signals from different modalities.
Compared with verbalized captions, our query verbalization method aligns the necessary semantics in captions, e.g. mentioned entities, with image objects and verbalizes the figure pixels with the help of caption semantics.
Enhancing image representations using verbalized queries usually achieves better retrieval effectiveness than using verbalized captions.
It showcases that our query verbalization method can provide more meaningful text clues for relevance modeling and multi-modality learning.
Moreover, some additional experiments are provided to study the effectiveness of different image verbalization methods.
We first show the relationship between the effectiveness of verbalized queries and manual caption lengths in Appendix {+A.
+} and then conduct some case studies in Appendix A.5 to explore the characteristics of different image verbalization methods.
C ONCLUSION
This paper proposes UniVL-DR, which models singe/cross modality matching and retrieval result fusion in one universal embedding space.
UniVL-DR proposes an effective multi-modality training strategy to learn universal representations for queries and documents, which breaks the modality boundary between vision and language, and helps to achieve state-of-the-art multi-modal retrieval performance.
Our experiments show that UniVL-DR can bridge the modality gap with image verbalization technologies and avoid overfitting the training signals of one modality by optimizing retrievers with modality-balanced hard negatives. A CKNOWLEDGMENTS
This work is supported by Beijing Academy of Artificial Intelligence (BAAI), the Natural Science
Foundation of China under Grant No. 62206042, No. U1811261 and No. 62006129, the Fundamental
Research Funds for the Central Universities under Grant No. N2216013, China Postdoctoral Science Foundation under Grant No. 2022M710022, and National Science and Technology Major Project (J2019-IV-0002-0069). A A PPENDIX A. D ATA S TATISTICS A multi-hop and multi-modal open domain question answering dataset WebQA (Chang et al., 2022) is used in our experiments.
The dataset contains images and passages that are crawled from the general Web and Wikipedia.
In our experiments, we randomly sample 5,000 queries from the original training set of WebQA as the development set for evaluation.
All data statistics are shown in Table 5.
To build an open-domain benchmark, we collect 389,750 images and 787,697 texts as multi-modal retrieval sources.
The image collection contains all images collected by the WebQA dataset, while the text collection contains all relevant passages of all 41,732 queries, which are Wikipedia snippets selected by matching noun chunks in the queries (Chang et al., 2022).
A. A DDITIONAL E XPERIMENT D ETAILS
This subsection describes additional implementation details.
In our experiments, we employ two pretrained vision-language models, VinVL (Zhang et al., 2021) and CLIP (Radford et al., 2021), and the pretrained language model, BERT (Devlin et al., 2019) to implement different retrieval models.
VinVL-DPR.
For VinVL variant models, we first detect the image objects and extract corresponding region features following VinVL 2 .
Then we concatenate image captions and image region features as inputs to feed into VinVL models and get the image representations.
We initialize VinVL with the checkpoint trained on the MSCOCO image retrieval task and continuously train the model on the WebQA dataset with in-batch negatives.
During training, we set the batch size to 32, learning rate= 2 e − 5 , accumulate step as 1, and max training epoch to 30.
We truncate the queries, image captions, text documents, and image region features with max lengths of 70, 70, 200, and 50.
CLIP-DPR.
For training CLIP-DPR, we start from the ViT-B/32 version of CLIP and continuously train CLIP on the WebQA dataset with in-batch negatives.
We truncate texts with the max length of 77 and set accumulate step as 1, batch size to 64, learning rate= 5 e − 6 , max training epoch to 20, and the temperature hyperparameter τ = 0 .
The cosine annealing strategy is used to schedule the learning rate during training.
BERT-DPR.
We initialize our retriever with the bert-base-uncased checkpoint, which is provided by Hugginface Transformers 3 .
During training, we set the batch size to 32, learning rate= 5 e − 5 , accumulate step as 1, and max training epoch to 30.
We truncate the queries, text documents, and image captions with max lengths of 70, 200, and 70.
NQ-DPR/NQ-ANCE.
NQ-DPR and NQ-ANCE start from the NQ-trained DPR model (Karpukhin et al., 2020), which uses a dual encoder architecture to encode queries and documents.
All experimental settings keep the same with BERT-DPR.
Besides, NQ-ANCE is tuned with the hard negatives sampled from the Top 100 retrieved candidates of NQ-DPR Xiong et al. (2021a).
A. E XPERIMENTAL D ETAILS OF I MAGE V ERBALIZATION
The image verbalization models are used to generate potentially matched captions or related questions for an image.
Our experiments start from the image caption generation model, which is trained on the
MSCOCO image caption task (Zhang et al., 2021) to generate related captions or queries to verbalize images.
2 https://github
com/microsoft/scene_graph_benchmarkhttps://github
Short
Medium
Long
M
RR @
Raw Caption Lengths
UniVL-DR
UniVL-DR w. Verbalized Query
During generating image-related queries, as shown in E.q. 7, we concatenate the image-related query, image caption, and image regional features as the input.
We continuously train the VinVL-based image caption model by randomly masking the tokens in queries, and optimizing vision-language models to fill in the masked positions. Different from image caption models, our query generation method tries to align the semantics in image captions and image pixel features instead of mapping the predicted classes and regional image features of detected objects, which can help vision-language models better understand the image semantics (Huang et al., 2021a). During training and inference, we set the generated tokens up to 20 and the beam size to 5.
We truncate the queries, image captions, and image region features with the max lengths of 40, 30, and 50, respectively. The mask probability is set to 0.15. More experimental details can be referred to Zhang et al.
A. I MAGE V ERBALIZATION P ERFORMANCE WITH D IFFERENT C APTION L ENGTHS
In this subsection, we evaluate the multi-modal retrieval performance of UniVL-DR with different verbalized queries.
Specifically, we evaluate the effectiveness of image-verbalized queries in the multi-modal retrieval task. These image-verbalized queries are generated with the image documents with different manual caption lengths.
We group the testing examples into three categories according to the manual caption lengths of the image documents and calculate the average MRR@10 score for each group.
The ratios are 42.33%, 36.84%, and 20.83% of the short, medium, and long caption length groups.
As shown in Figure 5, the experimental results show that our query generation method mainly helps to improve the retrieval effectiveness on the queries of short length and medium length, illustrating that these generated queries can provide some crucial textual clues in image representations of shorter captions.
These expanded text clues help retrieval models better understand image semantics, more effectively represent images via enhanced textual information, and conduct cross-modality matching more easily.
Moreover, the queries in the medium caption length group achieve the best performance, because the image captions of medium lengths can cover more necessary text clues for generating more informative verbalization results. [Table]
A. C ASE S TUDIES ON D IFFERENT I MAGE V ERBALIZATION M ETHODS
This experiment shows some image verbalization cases in Table 6.
We randomly sample queries that can be answered by image documents and show the manual captions, verbalized captions, and verbalized queries of the image documents.
Overall, these cases can be categorized into two groups according to the lengths of manual captions.
The first three cases are longer and more informative to describe the image facts among mentioned objects, which can be directly used in text-image relevance modeling.
On the contrary, the manual captions in the last three cases are written by the most representative entities that appeared in the images, making it difficult to distinguish the related images only according to these manual captions.
UniVL-DR employs two image verbalization methods to enhance the textual semantics of images.
Generating image captions is the most intuitive way to paraphrase images with some pre-defined classes of image objects.
Nevertheless, these object classes are too general and may be uninformative to provide semantics for matching, because the specific entities are critical to retrieving related documents in a question-answering system (Sciavolino et al., 2021).
Different from these verbalized captions, the verbalized queries are usually more informative and meaningful and specify the image objects by copying entity names from the manual captions, such as the names of persons, places, and buildings.
These entities can be directly matched with the given queries, which benefits cross-modality matching and helps to mitigate the modality gap between images and texts.
A. M ULTI -M ODAL R ETRIEVAL WITH D IFFERENT I MAGE R EPRESENTATION C OMBINATION M ETHODS
In this subsection, we conduct experiments to show the effectiveness of different methods in combining the representations of image captions and image features. [Table] [Table]
As shown in Table [Table] we evaluate the effectiveness of different combination models using CLIP-DPR.
We concatenate, dot product (outer product) [Table] and sum the representations of image captions and image features to conduct three models: CLIP-DPR (Concatenation), CLIP-DPR (Outer Product), and CLIP-DPR (Sum).
CLIP-DPR (Sum) shows its effectiveness by achieving the best performance among all baselines.
The sum operation is a commonly used semantic combination method, which is also used in BERT (Devlin et al., 2019) to combine token embedding and position embedding.
On the contrary, the concatenation operation usually regards the representations of image captions and image features as subvectors and separates them into subspaces, making it hard to learn the semantics of image documents.
On the other hand, the outer product operation conducts orthogonal representations during combining representations, which is not a typical combination method.
A. A DDITIONAL E VALUATIONS ON M ULTI -M ODEL R ETRIEVAL
In our experiments, we follow previous widely used retrieval benchmarks, MS MARCO (Bajaj et al., 2016) and BEIR (Thakur et al., 2021), and use NDCG@10/20 and MRR@10/20 to show the retrieval effectiveness of different retrieval models.
MRR scores and NDCG scores are calculated by the MS MARCO official scripts 4 and TREC’s evaluation tool 5 .
As shown in Table 8, we also conduct some evaluations to show the retrieval performance of higherranked candidates using MRR@1 and NDCG@5.
UniVL-DR also shows strong effectiveness by outperforming BM25 and CLIP-DPR with more than 6% improvements.
Notably, UniVL-DR even shows better retrieval effectiveness than the BM25 & CLIP-DPR (Oracle Modality) model, which is in an ideal setting.
It supports our claim that multi-modality modeling can also benefit single/cross-modality tasks.
A. E XAMPLES OF H ARD N EGATIVES
In this subsection, we randomly sample two queries and show some hard negatives in Figure 6, which are top-ranked documents using our CLIP-DPR model.
In the first case, when we ask “Is there greenery at Centennial Olympic Park?”, the CLIP-DPR model can provide some image and text documents, which are regarded as hard negatives to continuously train dense retrievers.
The negative images are about buildings, lawns, and trees, but these objects are not located at Centennial Olympic Park.
Evidently, these negative images are on-topic with “greenery” but are not related to the given query.
Training dense retrievers with these hard negatives can better teach retrievers to distinguish the subtle difference among these confusing images. [Figure] Query2:
The genus name Columba is Latin word means dove or what, which are particularly fond of roof spaces?
Text Doc1:
The city's permanent memorial to the 1996 Olympics is Centennial Olympic Park, which was built as a focal point for the Games. Text Doc2: Built as a legacy of the 1996 Olympic Games, Centennial Olympic Park, located on 21-acre (85, m2) area of Downtown, is the largest downtown park in the United States developed in the last 25 years. Text Doc3: Centennial Olympic Park was designed as the "town square" of the Olympics, and thousands of spectators had gathered for a late concert and merrymaking.
[Figure]
Golden Doc: Pigeons are particularly fond of roof spaces.
These often contain water tanks.
Any water tank or cistern on a roof must, therefore, be secured and sealed off to keep the pigeons out of them.
The popularity of a nesting area does not seem to be affected by the pigeons' population density.
[Figure]
: The common
[Figure]
"columbine" comes from the Latin for "dove", due to the resemblance of the inverted flower to five doves clustered together.
[Figure]
[Figure]
[Figure]
[Figure]
of
[Figure]
[Figure]
For both cases, the hard negatives from different modalities showcase some necessary semantics, which are needed by the retrieval model to find relevant information.
For example, text documents in Case 1 can provide background knowledge of the Olympic Games and Centennial Olympic Park; image documents in Case 2 supply the “doves” semantics from the visual modality.
These informative documents from different modalities can provide sufficient clues to guild dense retrievers to learn necessary semantics during contrastive training.