ICU : Conquering Language Barriers in Vision-and-Language Modeling by Dividing the Tasks into Image Captioning and Language Understanding (2023)

Abstract: Most multilingual vision-and-language (V&L) research aims to accomplish multilingual and multimodal capabilities within one model. However, the scarcity of multilingual captions for images has hindered the development. To overcome this obstacle, we propose ICU, Image Caption Understanding, which divides a V&L task into two stages: a V&L model performs image captioning in English, and a multilingual language model (mLM), in turn, takes the caption as the alt text and performs cross-lingual language understanding. The burden of multilingual processing is lifted off V&L model and placed on mLM. Since the multilingual text data is relatively of higher abundance and quality, ICU can facilitate the conquering of language barriers for V&L models. In experiments on two tasks across 9 languages in the IGLUE benchmark, we show that ICU can achieve new state-of-the-art results for five languages, and comparable results for the rest.

## introduction in recent times, there has been a growing interest in extending the success of vision-and-language (v&l) models beyond english to encompass non-english languages. however, the scarcity of training data has posed challenges in the development of multilingual models. to address this issue, various code-switch strategies @xcite have been proposed to encourage models to learn the relationships between corresponding words in different languages. additionally, machine translation (mt) techniques have been employed to augment existing english-only datasets @xcite . although some improvements have been achieved using mt-enhanced translated data, the quality of translations varies across languages. furthermore, fine-tuning strategies (liu 1 code to reproduce our results is available at https:// github.com/gjwubyron/icu figure 1 : we employ xvnli as a case study to exemplify the partitioning of the original task into two stages. the original task comprises an image premise and text hypothesis, which we display at the top. below it, we present the two stages: image captioning (ic) and crosslingual natural language inference (xnli). the english translations of the text are provided within the brackets. @xcite have been explored to enhance cross-lingual generalization. however, there still exists a significant performance gap between english and other languages, highlighting the challenge of scarcity. to address these challenges, this paper introduces icu (image caption understanding), which approaches v&l tasks by dividing them into two stages: image captioning (ic) and cross-lingual language understanding (xlu). as depicted in figure 1 , we use the cross-lingual visual natural language inference (xvnli) task as an example. initially, we employ a v&l model to perform ic and generate an english caption for the image. this caption is then treated as the alt text for the image, enabling cross-lingual natural language inferthe left image shows {lef t_caption} while the right image shows {right_caption}. in this study, we assess our approach using two tasks from iglue: xvnli and marvl @xcite , a multicultural reasoning over vision and language dataset. our findings indicate that icu, even in zero-shot scenarios, achieves remarkable performance on both tasks. additionally, we observe that employing few-shot learning techniques for xvnli further enhances the model's performance. moreover, we explore frame engineering techniques, wherein we assign captions to different frames (refer to our contributions are summarized as follows: ## icu: image caption understanding in this section, we will discuss the challenges posed by the implementation of icu. firstly, a crucial task is to adapt the second stage (xlu) to a suitable nlp task. for xvnli, this can be easily addressed since nli has already been extensively studied. however, for marvl, the model needs to determine whether a textual description is true or false about a pair of images. in this case, the adaptation is achieved by assigning the two captions to different frames, as illustrated in ## experiments in this section, we will provide a comprehensive description of the models employed in icu, along with the experimental settings and evaluations conducted. ## results and analysis in this section, we present the results of icu in comparison to existing works within the iglue benchmark. additionally, we analyze the impact of few-shot learning and frame engineering techniques on the performance of icu. ## conclusion in this paper, we introduce icu, a divide-andconquer approach designed to address the challenges of multilingual vision-and-language (v&l) tasks. icu leverages the strengths of both v&l models and multilingual language models (mlm) to tackle the inherent difficulties in these tasks. by dividing the original tasks into two stages, we transfer the burden of multilingual processing from the v&l model to the mlm, making it a more feasible objective. this approach not only helps alleviate the scarcity problem to some extent but also proves to be more efficient. we provide valuable insights into adapting v&l tasks to be compatible with mlms. furthermore, we explore the benefits of few-shot learning and frame engineering techniques in enhancing performance. our experimental results demonstrate the efficacy of recycling existing models, achieving state-of-the-art performance. overall, icu presents a promising solution for multilingual v&l tasks and opens up avenues for future research.