Parameter-Efficient Cross-lingual Transfer of Vision and Language Models via Translation-based Alignment (2023)

Abstract: Pre-trained vision and language models such as CLIP have witnessed remarkable success in connecting images and texts with a primary focus on English texts. Despite recent efforts to extend CLIP to support other languages, disparities in performance among different languages have been observed due to uneven resource availability. Additionally, current cross-lingual transfer methods of those pre-trained models would consume excessive resources for a large number of languages. Therefore, we propose a new parameter-efficient cross-lingual transfer learning framework that utilizes a translation-based alignment method to mitigate multilingual disparities and explores parameter-efficient fine-tuning methods for parameter-efficient cross-lingual transfer. Extensive experiments on XTD and Multi30K datasets, covering 11 languages under zero-shot, few-shot, and full-dataset learning scenarios, show that our framework significantly reduces the multilingual disparities among languages and improves cross-lingual transfer results, especially in low-resource scenarios, while only keeping and fine-tuning an extremely small number of parameters compared to the full model (e.g., Our framework only requires 0.16% additional parameters of a full-model for each language in the few-shot learning scenario).

## introduction cross-lingual transfer of models facilitates the transfer of learned representations or knowledge from one language to another. it plays a vital role in enhancing the performance of target languages, where the availability of labeled data and linguistic resources are particularly limited. cross-lingual transfer has found applications in various nlp tasks, including sentiment classification @xcite , dependency parsing @xcite , named entity recognition @xcite , question answering @xcite , and dialogue @xcite , among many others. recent advancements, such as xlm-r @xcite , mbart @xcite , and mt5 @xcite , have extended the capabilities of large language models in a multilingual manner, enabling them to comprehend and process multiple languages concurrently. two-stream vision-language pre-trained model clip @xcite has demonstrated remarkable performance in image-text retrieval @xcite by encoding images and text into a shared representation space. however, it primarily focuses on english and cannot comprehend other languages. to address this limitation, multilingual-clip @xcite has been proposed to enhance the clip's ability to support multiple languages through cross-lingual transfer. nevertheless, multilingual-clip treats english as a pivot language, leading to performance disparities across languages, especially low-resource languages. while previous work @xcite has accessed and highlighted this multilingual disparity, there is currently a lack of proposed solutions to address it. multilingual models often encounter a performance trade-off across different languages @xcite in the sense that overfitting the model in one language may degrade its performance in another. this can be a significant issue as the need to train and maintain separate models for each language can become resource-intensive when dealing with a large number of languages. the goal of this paper is to address the multilingual disparity in a parameter-efficient manner. to achieve this, we introduce a framework that extends the capabilities of the multilingual-clip model. more specifically, within this framework, we propose a translation-based alignment method that effectively minimizes the distribution gap between translated and natural language distributions. this alignment method plays a crucial role in significantly reducing the multilingual disparity exhibited by multilingual-clip. @xcite also point out the importance of machine translation in classification tasks. additionally, we adopt parameter-efficient fine-tuning (peft) methods @xcite @xcite @xcite @xcite @xcite as a solution to achieve parameter efficiency. furthermore, we find that, in the zero-shot scenario, hard prompt can also reduce the multilingual disparity and improve multilingual ability in addition to parameter efficiency. compared with full-model fine-tuning on each language, our framework mitigates the multilingual disparity and obtains higher average performance across all languages, using much fewer additional parameters than a single model. we conduct our experiments on xtd and multi30k datasets covering 11 languages in zeroshot, few-shot, and full-dataset learning scenarios. through extensive analytical experiments, we verify the effectiveness of our framework and provide answers to our research questions. based on the results of our experiments, we conclude the following: multilingual-clip clip @xcite , proposed by openai, is a two-stream visionlanguage pre-trained model with textual and visual encoder. it is trained on a large scale imagetext pair dataset using a contrastive loss to encode the image and text into a shared embedding space. clip calculate the cosine similarity between image and text features to measure their semantic similarity. recently, multilingual-clip @xcite extend clip to a multilingual version. this work replaces original english text encoder with a pre-trained multilingual language model such as m-bert @xcite and trains it using teacher learning @xcite . although multilingual-clip endowed clip with multilingual capabilities, the performance of multilingual-clip in other languages is worse than in english due to the limited amount of data available in lowresource languages, leading to insufficient training in these languages. furthermore, training data for other languages are translated from english text, which can result in a distribution gap during training and practical application. noticing this problem, we aim to reduce this multilingual disparity in this paper. parameter-efficient fine-tuning as the size of foundation models @xcite increases, fine-tuning and saving the entire model becomes very resource-intensive. many parameterefficient fine-tuning (peft) methods have been proposed to solve this issue. these approaches add additional parameters inside the model @xcite @xcite @xcite , optimize a small portion of the parameters or their low-rank decomposition matrix @xcite @xcite , or add trainable token embedding into the input @xcite . moreover, @xcite and @xcite analyze and combine these approaches from a unified perspective. furthermore, @xcite and @xcite propose automatic methods to search for an optimal combination of these peft methods for language models and visual models, respectively. many works @xcite @xcite also apply peft methods to clip models. nevertheless, those peft methods have not been thor-oughly explored for the multilingual-clip model in the cross-lingual transfer setting. it is important to note that peft methods often result in a decline in performance to varying degrees compared to full-model fine-tuning. therefore, it is essential to conduct experiments to verify the effectiveness of these methods and determine the most appropriate approach for specific tasks and models. ## framework our main contribution is a cross-lingual transfer framework for multilingual-clip (figure 1 ). in this framework, we propose a novel translationbased cross-lingual alignment method that reduces the multilingual disparity and exploits parameterefficient tuning methods to solve the resource consumption problem in cross-lingual transfer. ## experimental setup dataset we work with two datasets: (1) flickr30k @xcite is an image captioning datasets in english, split into train/dev/test datasets with the number of 29000/1024/1000. the multi30k @xcite is a cross-lingual dataset for the image-text retrieval task covering 11 languages. it only has a test split with 1000 samples per language with the same images. for few-shot setting, we randomly split the original test split into train/dev/test sets with 50/50/900 image-text pairs. base model and translation tool we use xlm-r large -vit l/14 @xcite as our base model. the model fixes the original visual encoder of openai vit l/14 @xcite and replaces the text encoder by xlm-roberta large @xcite trained by teacher learning @xcite . we use google translation 1 , a strong neural machine translation (nmt) system, to translate between all the different languages. to reduce the computational overhead, we translate the dataset in advance, rather than when it is used. we give the results of the translation in the code part. analysis of different machine translation tools can be seen in appendix b. ## experiments and analysis in this section, we first present the overall results of our framework with the optimal combination and then conduct analytical experiments to demonstrate the effectiveness of machine translation (section 5.2), routine 3 and mse loss is the best choice for alignment (section 5.3) and hard prompt, lora and adapter is respectively outperforms in zero-shot, few-shot and full-dataset scenario (secion 5.4). the details of our experimental configurations are in appendix a. ## conclusion in this paper, we propose a framework that significantly mitigates the multilingual disparity of multilingual-clip in a parameter-efficient manner in zero-shot, few-shot and full-dataset scenarios. our framework uses a translation-based alignment method and adopts parameter-efficient tuning methods. analytical experiments indicate that machine translation is effective for cross-lingual transfer; exploiting pivot language can help reduce the disparity; parameter-efficient tuning methods are beneficial for reducing resource consumption without too much performance degradation.