M u C o T : Multilingual Contrastive Training for Question-Answering in Low-resource Languages (2022)

Abstract: Accuracy of English-language Question Answering (QA) systems has improved significantly in recent years with the advent of Transformer-based models (e.g., BERT). These models are pre-trained in a self-supervised fashion with a large English text corpus and further fine-tuned with a massive English QA dataset (e.g., SQuAD). However, QA datasets on such a scale are not available for most of the other languages. Multi-lingual BERT-based models (mBERT) are often used to transfer knowledge from high-resource languages to low-resource languages. Since these models are pre-trained with huge text corpora containing multiple languages, they typically learn language-agnostic embeddings for tokens from different languages. However, directly training an mBERT-based QA system for low-resource languages is challenging due to the paucity of training data. In this work, we augment the QA samples of the target language using translation and transliteration into other languages and use the augmented data to fine-tune an mBERT-based QA model, which is already pre-trained in English. Experiments on the Google ChAII dataset show that fine-tuning the mBERT model with translations from the same language family boosts the question-answering performance, whereas the performance degrades in the case of cross-language families. We further show that introducing a contrastive loss between the translated question-context feature pairs during the fine-tuning process, prevents such degradation with cross-lingual family translations and leads to marginal improvement. The code for this work is available at https://github.com/gokulkarthik/mucot .

## introduction india has a population of 1.4 billion people speaking 447 languages and over 10,000 dialects, making it the country with the fourth-highest number of languages @xcite . however, indian lan-guages are highly under-represented on the internet and natural language processing (nlp) systems for indian languages are in their nascency @xcite .tamil is a member of the southern branch of the dravidian languages, a group of about 26 languages indigenous to the indian subcontinent. it is also classed as a member of the tamil language family, which contains the languages of around 35 ethnolinguistic groups, including the irula and yerukula languages @xcite @xcite @xcite . malayalam is tamil's closest significant cousin; the two began splitting during the 9th century ad. although several variations between tamil and malayalam indicate a pre-historic break of the western dialect, the process of separating into a different language, malayalam, did not occur until the 13th or 14th century @xcite @xcite . even state-of-the-art multilingual nlp systems perform sub-optimally on dravidian languages @xcite . this can be explained by the fact that multilingual language models are often jointly trained on 100+ languages and indian languages constitute only a small fraction of their vocabulary and training data (as shown in figure 2 ). machine learning models and tools have been proposed for many natural language understanding tasks. in this work, we focus on extractive question-answering (qa), where the goal is to localize the answer to a question within a large context (see figure 1 ). specifically, we aim to develop a common multilingual question answering model for multiple indian languages. a multilingual model has several advantages: (1) learning of cues across different languages, (2) a single model for many languages, and (3) avoiding dependency on english translation during inference. in this work, we start with a pre-trained multilingual bidirectional encoder representations from transformers (mbert) model and further pre-train it with squad @xcite , a large-scale question answering dataset in english. the resulting english-language mbert-qa model is fine-tuned and evaluated for indian languages tamil and hindi using the chaii dataset @xcite . fine-tuning the mbert-qa model using only the training instances in the chaii dataset is less effective because of the small number of training samples (1114 records with approximately twothirds in hindi and the rest in tamil). to overcome this problem, we use translation and transliteration to other languages as a data augmentation strategy. the translation is the process of transforming the source content from one language to another, while the transliteration just involves modifying each word from the source content into another script. both these operations are executed on the training dataset for the contexts, questions, and answers separately; then new locations of transformed answers in the transformed contexts are computed as shown in figure 1 . using translation and transliteration increases the size of the chaii dataset manifold. the choice of languages used for translation and transliteration is critical. @xcite showed that languages under the same family have similar representations in multilingual models. hence, we put together translations and transliterations from related languages within the same language family to achieve better performance. this will also help with better use of the vo-cabulary corpora from the low-resource languages. we also study the impact of translation and transliteration on languages outside the family of the target language. since the cross-family language transfer degraded the qa performance, we introduce a contrastive loss @xcite between the translated pairs to help retain or improve the original performance by encouraging the embeddings from all languages to be similar regardless of the family group. thus, the contributions of the paper are three-fold: ## related work bidirectional encoder representations from transformers (bert) @xcite is a deep learning model for general-purpose language representations. bert is often used as the backbone model for several nlp tasks like semantic analysis, question answering, and named entity recognition. the bidirectional transformer used in bert has a deeper sense of language context and generates intricate semantic feature representations. these representations are learned through a pre-training step using next sentence prediction (nsp) and masked language modelling (mlm) as pretext tasks and transferred to the downstream nlp tasks. the goal of the next sentence prediction task is to identify whether the two input sentences are consecutive or not. in masked language modelling, bert is trained to predict randomly masked words in a sentence. the transformer network receives a sequence of tokens as input and utilizes the attention mechanism to learn the contextual relationships between words in a text. these relationships can then be used to extract high-quality language features, which can be fine-tuned for applications like semantic analysis and question answering. multi-lingual-bert (mbert) is a bert model pre-trained using the wikipedia text corpus @xcite in more than 100 languages around the world. xlm-roberta @xcite scaled this idea with more than 2 terabytes of common crawl data. deep models such as transformers rely heavily on the availability of a large amount of annotated data, which is available only for prominent languages like english, russian, german or spanish @xcite . for a majority of other languages with a minimal number of annotations, cross-lingual transfer learning @xcite @xcite has been proposed as a possible solution. this approach can transfer knowledge from the annotation-rich source language to low-resource or zero-resource target languages. furthermore, multilingual models @xcite can be used to mitigate the data scarcity problem. for example, laser @xcite used a bidirectional lstm @xcite encoder with a byte pair encoding vocabulary shared between languages. this work showed that joint training of multiple languages helped to improve the model performance for low-resource languages. labse @xcite used the mbert @xcite encoder pre-trained with masked language modelling and translation language modelling @xcite tasks. it attempted to optimize the dual encoder translation ranking @xcite loss during pre-training to achieve similar embedding for the same text in different languages. the work of @xcite showed that large pre-trained multilingual models are not enough for question-answering in underrepresented languages and presented several novel strategies to improve the performance of mbert with translations. this work achieved languageindependent embeddings, which improved the cross-lingual transfer performance with additional pre-training on adversarial tasks. it also introduced a language arbitration framework (laf), which consolidated the embedding representations across languages using properties of translation. crosslingual manifold mixup (x-mixup) @xcite achieved better cross-lingual transfer by calibrating the representation discrepancy, which resulted in a compromised representation for target languages. it was shown that the multilingual pretraining process can be improved by implementing x-mixup on parallel data. contrastive language-image pre-training (clip) @xcite introduced an efficient way to learn scalable image representations with natural language supervision. drawing inspiration from convirt @xcite , clip used a contrastive objective that maximizes the cosine similarity of the correct pairs of images and text, while minimizing the same for incorrect pairs. building upon the work of @xcite , we show that translations of a small-scale dataset into cross-family languages could degrade the qa performance. to overcome this problem, we propose multilingual contrastive training to encourage cross-lingual invariance. our approach is relatively simpler compared to adversarial training and laf used in @xcite . though the proposed contrastive loss has a similar objective to the pretraining loss in @xcite , there are subtle differences because we use it in multi-task learning setup along with the original task loss for finetuning. ## conclusion and future work with internet usage expanding every day, there is an increasing need to develop better nlp models for a variety of downstream tasks in vernacular lan-guages. as most of these languages do not have labeled resources that are sufficient to train standalone modern deep learning models, we need to rely on multilingual models and enhance them. our work is a step in this direction and is an attempt to understand and evaluate the impact of cross-lingual knowledge transfer through pre-training and finetuning. we utilize modern open-source deep learning models to translate the chaii dataset into different languages from two language families namely, dravidian, and indo-aryan, and use them to improve the question-answering performance. our analysis shows an effective way to pick languages for translation, which can be used for fine-tuning. we also showed that introducing a contrastive loss with the original task training loss increases the performance even for cross-family languages. despite the inclusion of translations and contrastive loss, we observed that there is only a marginal improvement in the qa performance. this can be attributed to the smaller size of the chaii dataset with 1114 instances @xcite , which is clearly insufficient to fine-tune a 177m parameter model. hence, the proposed techniques have to be evaluated on other larger datasets as well as using other multilingual models like xlm-roberta @xcite , distill-mbert @xcite , muril @xcite and indic-bert @xcite . we hope that the proposed techniques will motivate further research in this field, including exploration of the same phenomenon of cross-lingual transfer in other language families and multilingual tasks.