Multi-Lingual Acquisition on Multimodal Pre-training for Cross-modal Retrieval
Abstract
Vision and diverse languages are important information sources in our living world.
A model that understands multi-modalities and multi-languages can be applied to a wider range of real-life scenarios.
To build such a multimodal and multilingual model, existing works try to ensemble vision-language data from multiple languages in pre-training.
However, due to the large number of languages, these works often require huge computing resources and cannot be flexibly extended to new languages.
In this work, we propose a M ulti- L ingual A cquisition (MLA) framework that can easily empower a monolingual Vision-Language Pre-training (VLP) model with multilingual capability.
Specifically, we design a lightweight language acquisition encoder based on state-of-the-art monolingual VLP models.
We further propose a two-stage training strategy to optimize the language acquisition encoder, namely the Native Language Transfer stage and the Language Exposure stage.
With much less multilingual training data and computing resources, our model achieves state-of-the-art performance on multilingual image-text and video-text retrieval benchmarks.
Introduction
[Figure]
We live in a multimodal and multilingual world.
The information we receive in our daily lives may come from different modalities and languages.
Therefore, building multimodal and multilingual models to effectively understand such information has attracted much research attention [13, 44, 24, 4].
Recently, Multilingual Vision-Language Pre-training (M-VLP) achieves convincing performance in various cross-lingual cross-modal tasks such as multilingual image-text retrieval [32, 50, 12, 19, 21] and multimodal machine translation [40].
As shown in Figure 1(a), M-VLP models handle multiple languages and modalities simultaneously during pre-training.
Despite their successes, M-VLP models suffer from two problems.
First, pre-training on vision and multilingual data consumes huge computing resources.
For example, the state-of-the-art M-VLP model MURAL [21] is pre-trained on 128 Cloud TPUv3 for four days.
It could support multimodal tasks on 100+ languages.
However, considering there are 6,900+ languages worldwide [50], building such a single model to handle all languages will be highly expensive.
Second, M-VLP models cannot be flexibly extended to new languages.
Additional training is required for M-VLP models to achieve satisfactory performance on a new language.
However, this training process will cause performance degeneration of M-VLP models on the original languages due to the limited model capacity.
For example, the limited model capacity even results in M-VLP models performing worse than their monolingual counterparts on
English [32, 50].
To build multimodal and multilingual models with low-cost and high-flexibility, we refer to our human learning habits when acquiring new languages.
We humans normally learn our native language during childhood and practice it through interactions with the multimodal living environments.
When learning a new language, we humans initially tend to align it with the native language, as we can easily map words in the native language to real-world objects and concepts.
After having a certain language foundation, we could further master it by interacting with the environment directly using the new language.
This is known as the language exposure [6].
The whole learning process rarely degrades our native language capability.
Inspired by this, we propose a new framework, M ulti- L ingual A cquisition (MLA), which constructs multimodal and multilingual models based on monolingual VLPs.
The topology of the MLA-based multimodal and multilingual model is illustrated in Figure 1(b).
Unlike M-VLPs, which handle data from multiple languages and modalities in a single model, MLA empowers monolingual VLPs with multilingual capability using much less training data through a language acquisition encoder.
The language acquisition encoder is realized by inserting our proposed lightweight language acquirers into the pre-trained monolingual encoder of the VLP model.
During training, original parameters in the pre-trained monolingual encoder are fixed, only multi-lingual embeddings and language acquirers for each new language are optimized.
Following the human learning habits, we propose a two-stage training strategy to train the language acquisition encoder.
In the Native Language Transfer (NLT) stage, the model is trained to establish the correspondence between the new languages with the native language.
In the Language Exposure (LE) stage, the model is optimized to build cross-modal alignment between new languages and images.
We apply our proposed MLA to the monolingual VLP model CLIP [35] and achieve state-of-the-art results on both multilingual imagetext and video-text retrieval benchmarks with much less training data and computing resources.
Ablation studies demonstrate the effectiveness of our training strategy.
Owing to the independence merit of the language acquirers, the MLA-based models can be easily extended to new languages without compromising the performance of their original languages.
Since cross-modal retrieval is a fundamental task for multilingual and multimodal learning, and most M-VLP [32, 50, 21] works are evaluated using this task, we focus on cross-modal retrieval in the paper.
The application of MLA to other tasks will be explored in our future work.
The main contributions of our work are as follows: 1) We propose a lightweight Multi-Lingual Acquisition (MLA) framework that can easily empower monolingual VLPs with multilingual capability. 2)
We propose a two-stage training strategy to optimize the MLA-based models inspired by the language learning habits of humans.
Ablation studies prove the effectiveness of the strategy.
We apply MLA to the monolingual VLP model CLIP and achieve the new state-of-the-art results on both multilingual image-text and video-text retrieval benchmarks with much less training data and parameters.
Related Work
Vision-Language Pre
-training: There are increasing interest in building Vision-Language Pretraining (VLP) models.
From the perspective of how to interact between vision and language modalities, existing models can be divided into two categories: single-stream and dual-stream models.
The single-stream models perform interaction on image and text directly with a crossmodal transformer [8, 30, 25].
In contrast, the dual-stream models encode image and text with two independent encoders and optimize via simple objectives like image-text contrastive learning [35, 22, 49].
Compared with the single-stream models, the dual-stream models are more efficient to utilize noisy image-text data harvested from the web [20], and thus achieve better performance on downstream tasks.
Meanwhile, the dual-stream models are more flexible for extension.
Since the dual-stream models process images and text through independent encoders, we can fix the vision encoders and focus on extending the text encoders to support new languages.
Therefore, we focus on empowering dual-stream VLPs with multilingual capability in this work.
Multilingual Vision-Language Pre-training: To achieve both multilingual and multimodal capability, many works try to learn the relationship between multiple languages and modalities simultaneously through pre-training.
M 3 P [32] introduces the multimodal code-switched training method to enhance multilingual transferability.
UC 2 [50] augments the English image-text data to other languages through machine translation and proposes fine-grained pre-training objectives to encourage alignment between image regions and multilingual tokens.
More recently, MURAL [21] adopts the dual-stream structure.
It is pre-trained with image-text and text-text contrastive objectives on multilingual image-text pairs and translation pairs.
M-VLP models significantly outperform previous non-pretraining models [13, 44, 24, 4] on multilingual image-text retrieval.
Despite their success, these models typically consume huge computing resources and large-scale multilingual training data.
Moreover, they fail to take full advantage of the cross-modal knowledge learnt in monolingual VLP, and building cross-modal cross-lingual representations from scratch can be very hard.
In contrast, our
MLA framework aims to empower VLP models with multilingual capability and it builds multimodal and multilingual models with much less data and computing cost.
Multilingual Extension: Some works explore making pre-trained monolingual language models multilingual.
Reimers et al.
[37] extend sentence embeddings from monolingual to multilingual by
Multilingual Knowledge Distillation (MKD).
Artetxe et al.
[2] extend monolingual models by training additional word embeddings .
MAD-X [34] extends multilingual pre-training models to support low-resource languages through adapters [18].
By extending state-of-the-art pre-trained language models, these works have achieved impressive results in NLP tasks such as bitext retrieval [37], cross-lingual QA and NER [34, 37].
However, few works focus on making VLP models multilingual.
OSCAR+ Ada [33] extends single-stream model OSCAR [30] to multilingual for VQA tasks by adopting a similar strategy with MAD-X [34]. It trains language adapters with Masked Language
Modeling (MLM) for each language, and replaces the English adapter with the target language one during inference. This strategy [34, 33] shares the same idea with MLA that handles different languages through independent structures but with two core differences: i) It is designed for MLMbased VLPs and cannot be applied to dual-stream models such as CLIP [35]. Consequently, it is not suitable for cross-modal retrieval considering the gap between retrieval and MLM.
ii)
It generalizes poorly on other languages since MLM can only implicitly establish the correspondence between these languages and English, let alone vision correspondences.
In contrast, MLA directly builds the connection between the other languages with English and then with vision through the twostage training strategy.
It thus achieves comparable results on these languages as on English in the downstream retrieval tasks.
Method
The Multi-Lingual Acquisition (MLA) framework is proposed to empower a dual-stream monolingual
VLP model with multilingual capability.
We define the native language of a VLP as its pre-training language.
In this paper, we choose CLIP-ViT-B [35] as the VLP model.
It is pre-trained with 400M image-text pairs in English [35].
Note that MLA can also be applied to VLP models with different native languages.
Since the state-of-the-art VLP models can project vision and native language into a shared multimodal space, we design a language acquisition encoder to process non-native languages.
We then simulate the learning habits of human beings and propose a two-stage training strategy to optimize the language acquisition encoder.
We first introduce the architecture of the MLA framework in Sec.3.1.
Then, we describe our training strategy in Sec.3.2.
3.1 Architecture
Figure 2(a) illustrates the overview of the MLA framework, which consists of three modules: the pre-trained text encoder, the pre-trained vision encoder, and the language acquisition encoder.
Pre-trained Text Encoder.
Given a sentence S in the native language, the corresponding sentence representation s =
Φ( S ; θ Φ ) is generated through the pre-trained text encoder Φ .
To preserve the cross-model knowledge of VLP, θ Φ is keep fixed during training.
As shown in the top part of Figure 2(a), the pre-trained text encoder contains a native embedding block and l transformer layers [43].
The native embedding block first tokenizes S with byte pair encoding (BPE)
Then, it converts words into embeddings E S =
[ e 0= [SOS] , e 1 , . .
, e M = [EOS] ] .
[SOS] and [EOS] are special 一个男人与他的狗 … Transformer layerTransformerlayer- 𝒍 … a man and his dogs … Language Acquirer Shared
Stage One Stage two Transformer layerTransformerlayer- 𝒍 Pre-trained Text Encoder 𝚽 … Native embedding
Non-native embedding
Transformer layerTransformer layer- 𝒍
Pre-trained Vision Encoder 𝚿 …
Image embedding
Language Acquisition Encoder 𝚽′
Language Acquirer
Attention
Feed-forward
Add & Norm
Add & Norm
ReLU
Down Projection
Upper Projection (a) (b) 𝑆 𝑇 𝑉 𝒔 𝒕 𝒗 tokens denoting the boundary of S .
The word embeddings are then passed through the transformer layers:
[Equation]
where H i =
[ h i 0 , . . .
, h iM ] is the hidden state of the layer i .
θ i Φ denotes the parameters of the layer i .
E pos is the positional encoding.
Note that the causal self-attention mask is used in the transformer layers [35].
The last hidden state of the [EOS] token is chosen to generate the sentence representation:
[Equation]
where s is the sentence representation of S , and W a denotes a linear projection.
Pre-trained Vision Encoder.
We extract the representation v =
Ψ
( V ; θ Ψ ) of an image V with the pre-trained vision encoder Ψ .
Similar with the pre-trained text encoder, θ Ψ is also frozen.
The pre-trained vision encoder is implemented as a Vision Transformer [10].
As shown in the bottom part of Figure 2(a), it consists of a image embedding block and l transformer layers.
Given an image V , the image embedding block first divides V into patches V ′ =
[ v ′ 1 , . . .
, v ′ N ] following [10].
Then, they are linearly projected into patch embeddings E p =
[ e [CLASS] , W p v ′ 1 , . .
, W p v ′ N ] , where e [CLASS] is a special embedding for the whole image and W p is the linear projection.
The patch embeddings are then fed into transformer layers:
[Equation]
where Z i =
[ z i 0 , . . .
, z iN ] is the hidden state of the layer i .
The last hidden state of the [CLASS] embedding z l 0 is selected to produce the representation of image V :
[Equation]
where v is the image representation of V , and W b denotes a linear projection.
Language Acquisition Encoder.
As shown in the middle part of Figure 2(a), the language acquisition encoder is built upon the pre-trained text encoder.
Suppose T is a sentence written in a non-native language L , we get the representation of T through language acquisition encoder t =
Φ ′ ( T ; θ Φ , θ emb , θ L ) , where θ Φ are fixed parameters of the pre-trained text encoder, θ emb refers to a shared non-native embedding block and θ L represents specialized language acquirers for language L .
Non-native sentence T is first tokenized and processed into word embeddings
E T =
[ u 0=
[SOS] , . .
u M =
[EOS] ] through the non-native embedding block.
The word embeddings are then encoded through the pre-trained transformer layers and language acquirers:
[Equation]
iL ) (9) where X i =
[ x i 0 , . .
, x im ] is the hidden state of the layer i .
W e is a linear projection to keep dimension consistency.
θ iL denotes the parameters of the i -th language acquirer for language L .
As shown in Figure 2(b), the language acquirer is implemented as a bottleneck MLP with residual connection [15]:
[Equation]
Similar with the pre-trained text encoder, the last hidden state of the [EOS] token is projected into the sentence representation t :
[Equation]
Eq.11 shares the same linear projection W a with Eq.3.
The main advantage of the language acquisition encoder is that it can extend the VLP models to support new languages without influencing the existing languages, as it handles different languages with independent language acquirers.
3.2 Training Strategy
To simulate the language learning habits of humans, we optimize the model in two stages: the Native
Language Transfer (NLT) stage and the Language Exposure (LE) stage.
Native Language Transfer.
When learning a new language, we humans initially tend to align it with the native language.
To simulate this learning phase, we align the non-native representations to the native representations during the Native Language Transfer (NLT) stage.
Specifically, suppose { ( S 1 , T 1 ) , ..., ( S n , T n ) } are translation pairs, where S i is in the native language, and T i is in a non-native language L .
The objective in the NLT stage is minimizing the Mean Square Error (MSE) between the native representation s i =
Φ( S i ; θ Φ ) and the non-native representation t i =
Φ ′
( T i ; θ Φ , θ L , θ emb ) :
[Equation]
(12) where B is the batch size.
Note that θ Φ is loaded from the VLP model and is kept frozen.
θ L is trained for non-native language L .
θ emb is shared among non-native languages.
During the NLT stage, the non-native language correspondence with vision can be built by pivoting on the native language, since the correspondence between the native language and vision is well established through VLP.
Language Exposure.
After the NLT stage, the model has built an implicit connection between non-native languages and vision.
However, due to the existence of synonyms, two same words in the native language may correspond to different images.
Thus, ambiguity may arise when learning non-native languages solely by relying on the native language.
Actually, we can regard the language acquisition encoder after the NLT stage as a person with a certain language foundation.
He/
She has learned the basic usage of a language through native language teaching.
To master it, he/she may practice the non-native language by interacting with the multimodal living environments.
Inspired by this learning phase, we directly establish the cross-modal alignment between non-native languages and vision during the Language Exposure (LE) stage.
Given image-text pairs { ( V 1 , T 1 ) , ..., ( V n , T n ) } where T i is in a non-native language L , the sentence representation t i =
Φ ′
( T i ; θ Φ , θ L , θ emb ) should be closer to the aligned image representation v i =
Ψ( V i ; θ Ψ ) , and away from the misaligned one v j = Ψ
( V j ; θ Ψ ) , j ̸
= i .
This can be achieved by performing contrastive learning between non-native languages and images.
For a non-native sentence T i , we treat the corresponding image
V i as a positive sample, and other images in the same batch V j , j ̸
= i as negative samples.
Vice versa for images.
The objective in the LE stage is minimizing the NCE loss [14] defined as follows:
[Equation]
where B is the batch size.
sim( x , y ) =
x ⊤ y ∥ x ∥∥ y ∥ is the cosine similarity between two vectors.
τ is a temperature hyper-parameter to scale the logits.
Note that though the image-to-text loss L v 2 t is optimized, the pre-trained vision encoder is kept frozen during training.
Similar to NLT, the trainable parameters in LE come from the language acquirers and the non-native embedding block.
Experiments 4.1 Dataset Description
We train our model with the Conceptual Captions (CC) dataset [39] and two translation enhanced versions of the CC [50, 5].
We use Multi30K [11], MSCOCO [7, 29, 46] and XTD [1] for multilingual image-text retrieval evaluation, and MSRVTT [45, 19] for multilingual video-text retrieval evaluation.
Conceptual Captions (CC) [39] contains 3.3 million image-text pairs in English crawled from the Web.
2 We also randomly select 300K image-text pairs denoted as CC300K for training our model to show the low-cost merit of MLA.
For multilingual sentences, we leverage two translation augmented CC datasets: (1) CC6L [50] that translates all English captions of the CC into 5 languages (German(de), French(fr), Czech(cs), Chinese(zh)); 3 and (2) CC69L [5] that contains 27K captions in each of the 68 languages translated from English.
4 Considering the languages of the downstream datasets, we train the model with CC6L for multilingual image-text retrieval, and with CC69L for multilingual video-text retrieval.
Multi30K [11] is built upon Flickr30K [47].
The English(en) captions are manually translated into
German(de), French(fr) and Czech(cs).
It contains 31K images paired with 5 captions per image in
English and German, and 1 caption in French and Czech.
We use the standard train, dev and test splits defined by Young et al.
MSCOCO [7] contains 123K images with 5 English captions per image. Yoshikawa et al.
[46] annotate 5 Japanese captions per image, and Li et al. [29] extends MSCOCO with Chinese captions for 20K images.
We follow the standard train, dev and test splits for English and Japanese as in [23].
For Chinese, we can only perform zero-shot evaluation on the test split defined by Li et al. [29], as the full splits have overlaps with English and Japanese splits.
XTD [1] provides captions in 11 languages (English(en), German(de), French(fr), Chinese(zh),
Japanese(ja), Italian(it), Spanish(es), Russian(ru), Polish(pl), Turkish(tr), Korean(ko)) for 1K
MSCOCO images.
Except for Japanese, all non-English captions are translated from the English caption directly.
We use this dataset for zero-shot image-text retrieval evaluation only.
MSRVTT [45] is a video caption dataset with 10K videos, where each video is annotated with[-
-]
English captions.
Huang et al.[19] translates the English captions into 8 languages (German(de),
French(fr), Russian(ru), Spanish(es), Czech(cz), Swahili(sw), Chinese(zh) and Vietnamese(vi))
via machine translation service.
We follow the standard train/dev splits in [45], and evaluate on the 1K test split as described in [48].
4.2 Implementation Details
We apply MLA on two VLP models: CLIP-ViT-B-32 and CLIP-ViT-B-16 [35], denoted as MLA CLIP and MLA CLIP16 respectively.
The hidden dimension of the language acquirers is set to 256, and all language acquirers for each non-native language cost only 3.14 MB parameters.
The non-native embedding matrix is initialized with M-BERT [9].
It costs 92.2 MB and shared with all non-native languages.
We train two separate models for multilingual image-text retrieval and video-text retrieval.
For the image model, we train with CC6L [50].
For the video model, we use multilingual captions from CC69L [5].
For both models, we optimize multiple language acquirers iteratively with a batch size of 128.
The NLT stage performs 117,150 steps with a learning rate of 1e-4, and the LE stage performs 11,715 steps with a learning rate of 3e-6.
The temperature τ is set to 0.01.
For both stages, we use the Adam optimizer [26] with a linear warm-up for the first 10% of steps.
The whole training process takes about 12 hours to converge on 1 Nvidia V100 GPU. [Table]
4.3 Evaluation on Multilingual Image-Text Retrieval
In multilingual image-text retrieval, models are given a sentence in a certain language to find the most semantically relevant image from an image database and vice versa.
We compare our model with state-of-the-art multilingual vision-language pre-training methods under three settings: (1) Zero-shot: we directly evaluate the model without fine-tuning on downstream datasets.
(2) Fine-tune on English: we first fine-tune the VLP model on downstream English data.
We then insert the language acquirers and non-native embedding block into the fine-tuned model and evaluate on other languages directly. (3) Fine-tune on All: after (2), we fine-tune the language acquirers and non-native embedding block on the downstream dataset and freeze other parts of the model.
Following previous works [32, 50, 21], we report Average Recall (AR), which is the average score over Recall@1, Recall@5, and Recall@ on two retrieval directions (image → text, text → image).
The results are shown in Table 1.
For a fair comparison, we re-implement MURAL [21] on the same training data with MLA and marked them with ⋄ .
More details and experiments about the implementation can be found in Appendix B.5.
Also, the comparison of supported languages, computing costs and parameters can be found in Table 2.
[Table]
Under the Zero-shot setting, we observe that
MLA CLIP performs significantly better than state-of-the-art M-VLP models on English.
This is because MLA CLIP could completely maintain the strong English performance of CLIP.
In contrast, M-VLP models typically perform worse than their monolingual counterparts on English (M 3 P 57.9 vs. Unicoder-VL [28] 72.0, MURAL 80.9 vs. ALIGN [22] 84.3).
MLA CLIP also outperforms M-VLP models on other languages.
For example, MLA CLIP achieves 78.7 average recall on German by using 300K image-text pairs, outperforming MURAL pre-trained with the same data by 16.0% and CC12M+EOBT by 2.7%.
It demonstrates that MLA is a high-data-efficient method to empower monolingual VLP models with multilingual capability.
Meanwhile, M-VLP models like MURAL require pre-training over a large amount of data to achieve convincing performance.
Under the Fine-tune on English setting, MLA shows strong cross-lingual transfer capability.
Under the Fine-tune on All setting,
MLA CLIP performs slightly worse than MURAL which was pre-trained on publicly unavailable dataset AT+MBT [21].
We consider the reason is that MURAL has a larger transformer-based text encoder (768 hidden dimensions, 12 heads) comparing with MLA CLIP (512 hidden dimensions, 8 heads). This makes MURAL easier to fit the downstream datasets with a certain scale such as Multi30K and MSCOCO.
MLA CLIP16 achieves state-of-the-art results on all languages under three settings.
These results suggest that MLA could probably reach better performance on multilingual image-text retrieval if stronger VLP models such as ALIGN-L2 [22] or Florence [49] are provided. Results of applying MLA to CLIP with different sizes can be shown in Appendix B.2.
4.4 Evaluation on Multilingual Video-Text Retrieval
In multilingual video-text retrieval, the model searches for the most semantically relevant videos given a text query in a certain language.
Following [31], we first uniformly sample 12 frames from each video, and use the pre-trained vision encoder to extract representations for each frame.
We then perform mean pooling over frame representations to get the video representation.
We also evaluate the models under three settings as in Sec.4.3.
We report the text → video Recall@1 score in Table 3.
Under Zero-shot setting, MLA CLIP , which is trained on CC69L without using any video data, achieves comparable or even better results than the fine-tuning results of the state-of-the-art M-VLP model XLM-R-MMP [19] on several languages (de: 20.1 vs. 21.1; fr: 22.0 vs. 21.8; es: 20.2 vs. 21.9).
Under the Fine-tune on English and Fine-tune on All settings, MLA CLIP also outperforms XLM-R-MMP significantly.
We consider the convincing performance comes from two reasons: 1) CLIP is a strong VLP model that can generalize well on video data.
The proposed MLA framework can well transfer the open-domain knowledge learned by CLIP to other languages.
These results suggest that MLA could maintain the open-domain capability of the VLP model which generalizes well on different downstream data. [Table]
4.5 Ablation Studies A. Training Strategy
[Table]
We conduct an ablation study in Table 4 to validate the effectiveness of the proposed MLA training strategy.
For those settings with NLT and LE at the same stage, we add the loss of the two objectives together during training.
By comparing row 1 to row 2&3, we observe that
LE at stage one leads to poor performance.
This indicates that aligning with the native language is more important for the VLP model to acquire new languages at an early stage.
It is consistent with the learning habits of humans.
By comparing rowand row 4, we see that LE at stage two could bring improvements on the new languages.
Additionally, comparing row 4 and row 5 suggests that optimizing the model with NLT and LE together at stage two does not bring improvements.
B. Language Acquirers and Embedding Initialization
[Table]
In order to validate the effectiveness of the proposed Language Acquirers, we remove the language acquirers and the M-BERT embedding initialization from the model respectively and evaluate on zero-shot multilingual image-text retrieval.
As shown in Table 5, the performance on all languages drops significantly without language acquirers.
Meanwhile, initializing the embedding with M-BERT [9] only brings incremental improvements.
It indicates that the language acquirers contribute most to the performance, and MLA does not depend much on the initialization of non-native embedding. C. Comparing with Machine Translation based method
[Table]
We compare MLA with a translate-based method CLIP+MT as follows.
For nonEnglish text queries in Multi30K and MSCOCO,
CLIP+MT first translates them into English with a strong commercial machine translation service Iflytek 5 , and then performs image-text retrieval. As shown in Table 6, the CLIP+MT performs significantly worse than MLA in all languages. This is because the translate-based method can introduce extra translation errors or ambiguity although the commercial translation service has been optimized with a large volume of multilingual corpus.
By directly optimizing the alignment between vision and each language, MLA does not suffer from this issue.
D. Low-resource Languages
[Table]
Image-text pairs may be rare for low-resource languages.
To explore the performance of MLA under this situation, we further simulate a lowresource scenario using XTD dataset.
We finetune MLA CLIP , UC 2 and MURAL with small amount of data from XTD in an unseen language. Both UC 2 and MURAL are pre-trained on CC6L with 3M images.
We randomly sample 600 pairs for finetuning, and the remained 400 samples are evenly divided for validation and testing.
Korean is chosen to perform simulation as its script and language family are not covered by CC6L. Experimental results in Table 7 show that MLA can achieve competitive results with very small amount of text-text pairs only (row 3), and adding image-text pairs brings further improvement (row 4).
It demonstrates that MLA is still an attractive method for low-resource languages even without any image-text pairs.
E. Amount of Training Data
[Figure] We conduct experiments to control the numbers of image-text pairs used for each language.
We train the models with CC6L and evaluate on
MSCOCO 1K and Multi30K under the zero-shot setting.
The corresponding mean AR over nonEnglish languages (de, fr, cs, ja, zh) are drawn in Figure 3.
We observe that MLA performs significantly better than MKD [37] in all cases.
Note that when the amount of training data is small, the advantage of MLA is more obvious, which could outperform MKD even without the
LE training stage.
Additionally, when training with only 30K image-text pairs per language,
MLA outperforms UC 2 , which is pre-trained with 3M pairs per language.
MLA is thus a data-efficient method to build multilingual and multimodal models.
F. Language Extensibility
Multilingual models often encounter the need to support new languages that do not occur in the training stage.
We conduct language extension experiments to compare MLA CLIP with M-VLP model UC 2 [50] on the XTD dataset [1].
XTD supports 11 languages, and 5 of them (en, de, fr, cs, zh, ja) are seen in the pre-training stage of UC 2 , while other 6 languages (it, es, ru, pl, tr, ko) are unseen.
To make a fair comparison, we first train MLA CLIP with the same data as UC 2 and then train both of them on unseen languages with CC69L.
The zero-shot image-text retrieval results on XTD are shown in Table 8.
We observe a significant performance degeneration on the seen languages for
UC 2 when training solely with unseen languages (row 1 vs. row 2).
Even keep training with the seen languages, the performance is still significantly reduced due to the limited model capacity (row 1 vs. row 3).
In contrast, as MLA decoupled multiple languages through acquirers, the performance of the seen languages is rarely affected (row 4 vs. row 5) .
This suggests that MLA framework can build multimodal multilingual models that are suitable for supporting increasing numbers of languages.
[Table]
Conclusion and Limitations
In this paper [Table] we propose the Multi-Lingual Acquisition (MLA) framework that can empower multilingual capability on monolingual Vision-Language Pre-training models with low-cost and highflexibility.
MLA injects language acquirers and a non-native embedding block into VLPs to support new languages.
Inspired by the language learning habits of humans, we propose a two-stage training strategy to optimize the language acquirers and non-native embedding block.
MLA applied on CLIP achieves state-of-the-art performances on multilingual image-text and video-text retrieval benchmarks with much less computing costs and training data.
Extensive ablation studies demonstrate that MLA is a flexible, effective, and efficient method to empower multilingual capability on multimodal models.
Though MLA shows high data efficiency that achieves high performance with a small amount of multilingual training data in our experiments, it has the following limitations, which also indicate potential research avenues: i) MLA relies on bilingual corpora, which limits its application on low-resource languages. Meanwhile, state-of-the-art models, such as UC 2 and MURAL, also rely on bilingual information. How to achieve comparable performance with monolingual corpora only is still an open problem. ii) MLA is currently limited to coarse-grained retrieval tasks.
We will investigate the further application of MLA to fine-grain tasks like VQA in our future work.
Furthermore, the majority of our training data is automatically constructed through machine translation, so the ethical prejudice from the machine translation service may potentially affect the behavior of multilingual models produced by MLA.
One way to mitigate such concern is to use human annotated or reviewed data for training.
Acknowledgement
This work was partially supported by the National Key R&D Program of China (No.
2020AAA0108600), the National Natural Science Foundation of China (No. 62072462) and
the
Large-Scale Pre-Training Program 468 of Beijing Academy of Artificial Intelligence (BAAI)
In Figure 4, we visualize the top-1 retrieved images for given text queries in 11 languages on XTD dataset [1].
Compared with the multilingual vision-language pre-training model UC 2 [50], MLA can better capture entities, attributes, and actions to retrieve the correct image. Specifically, given simple queries that contain few entities such as Query #1 or Query #2, the images retrieved by MLA show high consistency across languages, since the representations of non-English queries are aligned to English in the NLT stage. For the more complex queries such as Query #3 or Query #4, MLA also shows better fidelity to all entities in most cases.
[Figure]
To visualize the multimodal and multilingual representation space, we translate the English class labels of CIFAR10 [27] into 5 languages including German (de), French (fr), Czech (cs), Chinese (zh), and Japanese (ja).
The images and labels in 6 languages are encoded into representations through MLA CLIP . Figure 5 shows the t-SNE [42] visualization of these representations. We can see that the representations from different languages and modalities are clustered according to the semantics. It suggests that MLA CLIP indeed can project images and multilingual sentences into a shared multimodal and multilingual space.
[Figure] label
We conduct additional ablation studies to verify the effectiveness of MLA.
All experiments in this section are conducted on zero-shot image-text retrieval.
In our proposed MLA, we implement the language acquirer as a bottleneck MLP.
In Table.9, we compare the different structure of the language acquirer, the bottleneck MLP and a linear projection layer with the same amount of parameters. MLP works slightly better than the linear projection. Thus, we choose MLP to conduct our major experiments.
[Table]
We additionally apply MLA to CLIP in different sizes with two kinds of structures: ResNet and ViT. The results in Table 10 indicate that MLA can perform better on all languages when stronger monolingual VLPs are provided.
[Table]
In the default setting, we use the MSE objective during the NLT stage and the NCE objective during the LE stage. The MSE objective requires paired representations to be completely consistent, while the NCE objective only requires positive pairs to be closer than negative ones.
We conduct experiments to use different objectives in the two stages. As shown in Table 11, we observe that the MSE objective is more suitable for the NLT (row 1 vs. row 2, row 7 vs. row 8) stage
[Table] and the
NCE objective performs better for the LE stage (row 3 vs. row 4, row 5 vs. row 6). We consider the reason is that in the NLT stage, we leverage translation pairs to build alignment between languages. Since the two sentences of a translation pair are highly semantically related, their representations can be very similar. Thus, optimizing a strong objective like MSE during the NLT stage is feasible.
However, during the LE stage, the optimization is conducted with image-text pairs. Although the image and text are semantically related, one sentence can hardly describe all the information in the image. Therefore, a weak objective like NCE is suitable for the LE stage. [Table]
MLA adopts the "Multimodal → Multilingual" strategy that empowers VLP models with multilingual capability. However, there is another option of "Multilingual → Multimodal" that empowers multilingual pre-training models with multimodal capability. To make a comparison between these two strategies, we implement the Cross-Modal Acquisition (CMA) that inserts cross-modal acquirers in each layer of the multilingual pre-training model M-BERT [9]. We keep the pre-trained M-BERT fixed and train the cross-modal acquirers with the same two-stage strategy as MLA.
From Table 12, we find that CMA performs worse than MLA in all languages. It suggests that generalizing multilingual models to multimodal is harder than generalizing multimodal models to multilingual through lightweight acquirers. [Table] B.5 Details of implementing MURAL
We implement MURAL [21] on the 6 languages considering our computing budgets. The dualencoders of MURAL are implemented with CLIP-ViT-32 and M-BERT-base respectively, since we find that initializing the dual-encoders with both pre-trained models can boost the performance. We train MURAL on CC300K (same as MLA) using 1 V100 GPU with a batch size of 128, and on
CC3M (the largest dataset we can access) using 8 V100 GPUs with a batch size of 1024.
The learning rate is set to 1e-4.
Both models converge in about 1 day and 4 days respectively.
The results are shown in Table 13.
It indicates that even initializing the dual-encoders, MURAL performs worse than
MLA.
Note that under the fair comparison, MLA also shows its low-cost merit, since the data and computing resources of MURAL pre-trained on CC3M are much larger.
[Table]
We conduct an evaluation on WIT [41] dataset to further examine MLA on real low-resource languages.
WIT [41] contains Wikipedia-based image-text pairs in 108 languages.
We follow the test set proposed in the IGLUE benchmark [3] that contains 500-1000 image-text pairs in 10 languages.
We train both MLA and MURAL-CC3M in the 10 languages with CC69L and perform the evaluation on WIT directly.
As shown in Table 14, MLA still outperforms MURAL-CC3M in most languages on this benchmark, which validates the effectiveness of MLA.
[Table]
C Open-domain Image Classification
In order to test the open-domain capability of models, we conduct zero-shot open-domain image classification experiments on CIFAR100 [27], ImageNet-V2 [36], ImageNet-R [16] and ImageNet-A
[17] datasets.
As shown in Table 15, MKD [37] performs badly on open-domain image classification.
We consider the reason is that MKD abandons the original text encoder which contains open-domain multimodal knowledge from large-scale pre-training.
In contrast, MLA keeps the original text encoder fixed and thus could maintain the open-domain capability of the pre-training model.
[Table]