A Unified Framework for Multilingual and Code-Mixed Visual Question Answering (2020)

Abstract: In this paper, we propose an effective deep learning framework for multilingual and code- mixed visual question answering. The pro- posed model is capable of predicting answers from the questions in Hindi, English or Code- mixed (Hinglish: Hindi-English) languages. The majority of the existing techniques on Vi- sual Question Answering (VQA) focus on En- glish questions only. However, many applica- tions such as medical imaging, tourism, visual assistants require a multilinguality-enabled module for their widespread usages. As there is no available dataset in English-Hindi VQA, we firstly create Hindi and Code-mixed VQA datasets by exploiting the linguistic properties of these languages. We propose a robust tech- nique capable of handling the multilingual and code-mixed question to provide the answer against the visual information (image). To better encode the multilingual and code-mixed questions, we introduce a hierarchy of shared layers. We control the behaviour of these shared layers by an attention-based soft layer sharing mechanism, which learns how shared layers are applied in different ways for the dif- ferent languages of the question. Further, our model uses bi-linear attention with a residual connection to fuse the language and image fea- tures. We perform extensive evaluation and ablation studies for English, Hindi and Code- mixed VQA. The evaluation shows that the proposed multilingual model achieves state-of- the-art performance in all these settings.

## introduction visual question answering (vqa) is a challenging problem that requires complex reasoning over visual elements to provide an accurate answer to a natural language question. an efficient vqa system can be used to build an artificial intelligence (ai) agent which takes a natural language question * work carried out during the internship at iit patna and predicts the decision by analyzing the complex scene(s). vqa requires language understanding, fine-grained visual processing and multiple steps of reasoning to produce the correct answer. as the existing research on vqa are mainly focused on natural language questions written in english @xcite @xcite @xcite @xcite , their applications are often limited. multilingual speakers often switch back and forth between their native and foreign (popular) languages to express themselves. this phenomenon of embedding the morphemes, words, phrases, etc., of one language into another is popularly known as code-mixing @xcite . codemixing phenomena is common in chats, conversations, and messages posted over social media, especially in bilingual / multilingual countries like india, china, singapore, and most of the other european countries. sectors like tourism, food, education, marketing, etc. have recently started using code-mixed languages in their advertisements to attract their consumer base. in order to build an ai agent which can serve multilingual end users, a vqa system should be put in place that would be language agnostic and tailored to deal with the code-mixed and multilingual environment. it is worth studying the vqa system in these settings which would be immensely useful to a very large number of population who speak/write in more than one language. a recent study @xcite also shows the popularity of code-mixed english-hindi language and the dynamics of language shift in india. our current work focuses on developing a language agnostic vqa system for hindi, english and code-mixed english-hindi languages. let us consider the examples shown in fig 1 . the majority of the vqa models @xcite @xcite are capable enough to provide correct answers for english questions q e , but our evaluation shows that the same model could not predict correct answers for hindi q h and code-mixed question q cm . the questions q h and q cm correspond to the same question q e , but are formulated in two different languages. in this paper, we investigate the issue of multilingual and code-mixed vqa. we assume that there are several techniques available for monolingual (especially, english) vqa such that a strong vqa model can be built. however, we are interested in building a system that can answer the questions from different languages (multilingual) and the language formed by mixing up of multiple languages (code-mixed). we show that in a cross-lingual scenario due to language mismatch, applying directly a learned system from one language to another language results in poor performance. thus, we propose a technique for multilingual and code-mixed vqa. our proposed method mainly consists of three components. the first component is the multilingual question encoding which transforms a given question to its feature representation. this component handles the multilinguality and code-mixing in questions. we use multilingual embedding coupled with a hierarchy of shared layers to encode the questions. to do so, we employ an attention mechanism on the shared layers to learn language specific question representation. furthermore, we utilize the self-attention to obtain an improved question representation by considering the other words in the question. the second component (image features) obtains the effective image representation from object level and pixel level features. the last component is multimodal fusion which is accountable to encode the question-image pair representation by ensuing that the learned representation is tightly coupled with both the question (language) and image (vision) feature. it is to be noted that designing a vqa system for each language separately is computationally very expensive (both time and cost), especially when multiple languages are involved. hence, an endto-end model that integrates multilinguality and code-mixing in its components is extremely useful. we summarize our contribution as follows: to the best of our knowledge, this is the very first attempt towards this direction. 2. we propose a unified neural model for multilingual and code-mixed vqa, which can predict answer of a multilingual or code-mixed question. 3. to effectively answer a question, we enhance the vision understanding by combining local image grid and object-level visual features. we propose a simple, yet powerful mechanism based on soft-sharing of shared layers to better encode the multilingual and code-mixed questions. this bridges the gap between vqa and multilinguality. 4. we perform extensive evaluation and ablation studies for english, hindi and code-mixed vqa. the evaluation shows that our proposed multilingual model achieves state-of-the-art performance in all these settings. ## related work multilingual and code-mixing: recently, researchers have started investigating methods for creating tools and resources for various natural language processing (nlp) applications involving multilingual @xcite @xcite and code-mixed languages @xcite @xcite @xcite . developing a vqa system in a code-mixed scenario is, itself, very novel in the sense that there has not been any prior research towards this direction. vqa datasets: quite a few vqa datasets @xcite @xcite @xcite @xcite have been created to encourage multi-disciplinary research involving natural language processing (nlp) and computer vision. in majority of these datasets, the images are taken from the large-scale image database mscoco @xcite or artificially constructed @xcite @xcite . there are a few datasets @xcite for multilingual vqa, but these are limited only to some chosen languages, and unlike our dataset they do not offer any code-mixed challenges. ## mcvqa dataset dataset creation: the popular vqa dataset released by @xcite contains images, with their corresponding questions (in english) and answers (in english). this is a challenging large scale dataset for the vqa task. to create a comparable version of this english vqa dataset in hindi and code-mixed hinglish, we introduce a new vqa dataset named "multilingual and codemixed visual question answering" (mcvqa) which comprises of questions in hindi and hinglish. our dataset foot_0 , in addition to the original english questions, also presents the questions in hindi and hinglish languages. this makes our mcvqa dataset suitable for multilingual and code-mixed vqa tasks. a sample of question-answer pairs and images from our dataset are shown in fig 2 . we do not construct the answer in code-mixed language because a recent study @xcite has shown that code-mixed sentences and their corresponding english sentences share the same nouns (common nouns, proper nouns, spatiotemporal nouns), adjectives, etc. for example, given an english and its corresponding code-mixed question: q e : where is the tree in this picture? q cm : is picture me tree kahan hai? it can be observed that both q e and q cm share the same noun { picture, tree}. the majority of answers in the vqa v1.0 dataset are of type 'yes/no', 'numbers', 'nouns', 'verbs' and 'adjectives'. therefore, we keep the same answer in both english and code-mixed vqa dataset. we follow the techniques similar to @xcite for our code-mixed question generation, which takes a hindi sentence as input and generates the corresponding hinglish sentence as the output. we translate original english questions and answers using the google translate foot_1 that has shown remarkable performance in translating short sentences @xcite . we use this service as our original questions and answers in english are very short. for the code-mixed question generation, we first obtain the part-of-speech foot_2 (pos) and named entity foot_3 (ne) tags of each question. thereafter, we replace the hindi words having the pos tags (common noun, proper noun, spatio-temporal noun, adjective) with their best lexical translation. same strategy is also followed for the words having the ne tags as location and organiza-tion. the remaining hindi words are replaced with their roman transliteration. in order to obtain the best lexical translation, we follow the iterative disambiguation algorithm @xcite . we generate the lexical translation by training the statistical machine translation (smt) model on the publicly available english-hindi (en-hi) parallel corpus @xcite . please refer to the appendix for the comparison with other vqa datasets. ## conclusion in this work, we propose a unified end-to-end framework for multilingual and code-mixed question answering and create a dataset for hindi and codemixed vqa. we believe this dataset will enable the research in multilingual and code-mixed vqa. our unified end-to-end model is capable of predicting answers for english, hindi and code-mixed questions. experiments show that we achieve state-ofthe-art performance on multilingual vqa. we believe our work will pave the way towards creation of multilingual and code-mixed ai assistants. in the future, we plan to explore transformer-based architectures for vqa in multilingual and code-mixed setups considering various diverse languages. (a) (b) (c) figure 6: analysis of code-mixed vqa dataset on various code-mixing metrics: (a), (b) and (c) show the distribution of code-mixed questions from training and test set w.r.t the code-mixing index (cmi) score, complexity factor (cf2 and cf3), respectively. metrics manually annotated automatically generated training testing cmi score 0.2946 0.3223 0.3228 cf2 14.765 16.094 16.114 cf3 13.122 14.0708 14.096 @xcite mscoco crowd workers (amt) visual7w @xcite mscoco crowd workers (amt) clevr @xcite synthetic shapes automatically generated kb-vqa @xcite mscoco in-house participants fvqa @xcite mscoco in-house participants japanese vqa @xcite mscoco crowd workers (yahoo) mcvqa (ours)