Towards Developing a Multilingual and Code-Mixed Visual Question Answering System by Knowledge Distillation (2021)

Abstract: Pre-trained language-vision models have shown remarkable performance on the visual question answering (VQA) task. However, most pre-trained models are trained by only considering monolingual learning, especially the resource-rich language like English. Training such models for multilingual setups demand high computing resources and multilingual language-vision dataset which hinders their application in practice. To alleviate these challenges, we propose a knowledge distillation approach to extend an English language-vision model (teacher) into an equally effective multilingual and code-mixed model (student). Unlike the existing knowledge distillation methods, which only use the output from the last layer of the teacher network for distillation, our student model learns and imitates the teacher from multiple intermediate layers (language and vision encoders) with appropriately designed distillation objectives for incremental knowledge extraction. We also create the large-scale multilingual and code-mixed VQA dataset in eleven different language setups considering the multiple Indian and European languages. Experimental results and in-depth analysis show the effectiveness of the proposed VQA model over the pre-trained language-vision models on eleven diverse language setups.

## introduction visual question answering (vqa) is a challenging problem in computer vision (cv) and natural language processing (nlp) that have gained popularity due to its many-fold benefits ranging from assisting visually impaired users to establishing effective communication with robots via intuitive interfaces. the existing works @xcite on vqa are mainly limited to the english questions, making it challenging to acknowledge progress in foreign languages. moreover, the current language-vision models do not serve the purpose in the code-mixed setting, where the morphemes, words, phrases of one language are embedded into the other language. since code-mixing has been a mean of communication in a multi-cultural and multi-lingual society, the next generation of artificial intelligence (ai) agents should be capable to understand the multilingual and code-mixed (mcm) questions about the image. in the recent past, the pre-trained language-andvision models @xcite @xcite have become the state-of-the-arts for solving a variety of cv and nlp problems. however, the majority of these models are predominantly built for resource-rich languages like english. therefore, their abilities to process and answer the mcm questions are limited (c.f. fig. 1 ). to address this, we propose a highly effective and unified vqa method that allows us to extend the existing monolingual language-and-vision models to multilingual (in 6 different languages) and code-mixed (in 5 different code-mixed languages) scenarios. specifically, we develop a novel knowledge distillation @xcite approach to distill the knowledge from the monolingual language-and-vision transformer network (teacher model) to multilingual and code-mixed language-and-vision transformer network (student model). this enables the student model to adapt to any language and code-mixed scenarios. to effectively transfer the knowledge from the teacher network to the student network, we introduce multiple distillation objectives which ensure the incremental knowledge extraction from multiple intermediate layers of language-and-vision transformer model. these objectives are formulated to guide the student model to learn two key figure 1 : example multilingual and code-mixed questions (same question), where pre-trained language-and-vision models fails to correctly predict the answer except for english question. furthermore, to combat the data scarcity, we also create a large-scale (3.7m image-question pairs) multilingual and code-mixed vqa dataset. towards this, we utilize the english vqa1.0 @xcite dataset and extend it to multiple languages. we also create their code-mixing counterparts by designing the linguistically informed strategy to formulate the code-mixed question by mixing the words/phrases from the english question and the foreign language question. we evaluate our proposed approach on the created dataset and achieve 11.74% average improvement across all the languages over the pre-trained language-vision model. ## multilingual and code-mixed vqa dataset dataset creation: the large-scale vqa dataset (vqav1.0) released by @xcite contains the triplet information in the form of the question, image, and answers. @xcite introduced a vqa dataset named mcvqa, which comprises questions in hindi and hinglish (i.e., code-mixed english and hindi). however, the approach to create mcvqa dataset has two major shortcomings: (1) algorithm is not scalable to other languages, and (2) it requires the nlp components (part-of-speech tagger, named entity recognizer, transliteration, etc.) for the resource-scarce languages, which, themselves are an active research area for the resource-scare languages. to address these shortcomings, in this work, we create the large-scale "multilingual and codemixed visual question answering" (muco-vqa) dataset which supports the five (5) languages (hi, bn, es, de, and fr) and five (5) different code-mixed settings (en-hi, en-bn, en-es, en-de, and en-fr). to generate the code-mixed questions, we follow the matrix language frame (mlf) theory (myers-scotton, 1997) of code-mixed text. according to mlf, a code-mixed sentence will have a dominant language (matrix-language) and inserted language (embedded-language). we utilize the google machine translation to translate the english questions from vqav1.0 dataset to the foreign language @xmath0, bn, es, de, fr}. from the parallel questions (en-xx), we learn the alignment of english words in the foreign language question. given a pair of questions from the two languages (en-xx), we identify the words following @xcite from the english question and substitute their aligned counterparts (in foreign language question) with the identified english words to synthesize the english embedded code-mixed questions. please see appendix for the implementation details and samples of the muco-vqa dataset. analysis: similar to the vqav1.0 dataset, our created muco-vqa dataset consists of 248, 349 training and 121, 512 test questions for each of the five different languages and five code-mixed settings. we perform a qualitative analysis of this dataset by randomly selecting 5, 00 questions, each from en, hi and corresponding en-hi. we seek annotation help from two bilingual (en, hi) experts to manually translate and create the code-mixed questions. towards this, we compute the bleu @xcite , rouge @xcite and translation error rate (ter) @xcite considering the manually created code-mixed questions as the gold standard; and the generated code-mixed questions from muco-vqa as the candidates. we compute the mean values of the individual scores obtained from both experts. we found the bleu: 78.34, rouge-l: 91.13, and ter: 8.23, which show the generated code-mixed questions are close to the human formulated code-mixed questions. the detailed analysis and statistics in terms of codemixed complexity can be found in the appendix. ## methodology our proposed knowledge distillation framework for the vqa model is tailored to predict the answer for multilingual and code-mixed questions. we utilize lxmert @xcite , a pre-trained english vision-language model, as the teacher network to train our student network. our student network is inspired by the teacher network and has three components, viz. (1) mcm question encoder that processes and effectively encode the multilingual and code-mixed questions, (2) image encoder which learns the representation of the objects detected in the image, (3) cross-modality encoder, that learns the joint feature representation by applying the cross-attention on the language and image features, and (4) answer prediction, which predicts the answer for mcm questions. ## conclusion this paper proposes a unified framework for multilingual and code-mixed vqa by distilling the knowledge from the monolingual language-vision pre-trained lxmert model. to fully utilize the rich information from the question, image, and cross-modal encoders, we devise effective distillation objectives to encourages the student model to learn from the teacher through a multi-layer distillation process. to train and evaluate the proposed approach, we have created a large-scale muco-vqa dataset supporting eleven different mcm settings. extensive experiments over the muco-vqa and mcvqa datasets demonstrate the effectiveness of our proposed approach.