I ndic XNLI : Evaluating Multilingual Inference for I ndian Languages (2022)

Abstract: While Indic NLP has made rapid advances recently in terms of the availability of corpora and pre-trained models, benchmark datasets on standard NLU tasks are limited. To this end, we introduce INDICXNLI, an NLI dataset for 11 Indic languages. It has been created by high-quality machine translation of the original English XNLI dataset and our analysis attests to the quality of INDICXNLI. By finetuning different pre-trained LMs on this INDICXNLI, we analyze various cross-lingual transfer techniques with respect to the impact of the choice of language models, languages, multi-linguality, mix-language input, etc. These experiments provide us with useful insights into the behaviour of pre-trained models for a diverse set of languages.

## introduction natural language inference (nli) is a well-studied nlp task @xcite that assesses if a premise entails, negates, or is neutral towards the hypothesis statement. the task is well suited for evaluating semantic representations of state-ofthe-art transformers @xcite models such as bert @xcite . two large scale datasets, such as snli @xcite and multinli @xcite , has recently been developed to enhanced the relevance of the nli task. with the availability of multi-lingual pre-trained language models such as mbert @xcite , and xlm-roberta @xcite promising cross-lingual transfer and universal models, multi-lingual nlp has recently gained a lot of attention. however, most languages have a scarcity of datasets resources. some multi-lingual datasets have attempted to fill this gap, including xnli @xcite for nli, xquad ## the indicxnli dataset we created indicxnli, a nli data set for indic languages. indicxnli is similar to existing xnli dataset in shape/form, but focusses on indic language family. indicxnli include nli data for eleven major indic languages that includes assamese ('as'), gujarat ('gu'), kannada ('kn'), malayalam ('ml'), marathi ('mr'), odia ('or'), punjabi ('pa'), tamil ('ta'), telugu ('te'), hindi ('hi'), and bengali ('bn'). next we describe the in-dicxnli construction and its validation in details. indicxnli construction. to create in-dicxnli, we follow the approach of the xnli dataset and translate the english xnli dataset (premises and hypothesis) to eleven indic-languages. we use the indictrans @xcite , a state-of-the-art, publicly available translation model for indic languages, for translating from english to indic languages. the train (392,702), validation (2,490), and evaluation sets (5,010) of english xnli were translated from english into each of the eleven indic languages. indictrans is a large transformer-based sequence to sequence model. it is trained on samanantar dataset @xcite , which is the largest parallel multi-lingual corpus over eleven indic languages. indictrans outperforms other open-source models based on mbart @xcite and mt5 @xcite for indic language translations and is competitive with paid translation models such as google-translate or microsoft-translate on several benchmarks @xcite . our choice of indictrans was motivated by cost, language coverage and speed, refer appendix §a. ## error analysis in this section, we investigate the correlation between the language similarity and the model performance. we see that the model performs similarly on similar languages. we evaluate our results on muril on the english+indic finetuning strategy. in figure 1 (appendix), we observe that the overall correct and incorrect predictions, bengali vs assamese pair has the total of 81% overlap,tamil vs kannada has 83% overlap, hindi vs maratha has 82% overlap. all the language pairs have the largest overlap for entailment label for correct labels and largest overlap in contradiction label for incorrect overlaps. in figure 2 (appendix), interestingly bengali vs assamese pair and hindi vs marathi has the highest percentage of overlap in predictions where the most overlap is in entailment and minimum overlap is in contradiction. while for tamil vs kannada pair has the highest overlap for neutral and minimum for contradiction. we have also done error analysis of model performance on original hindi test data already present in xnli and data obtained through translations from indictrans in figure 3 (appendix). we observe a total of 82% overlap in error consistency, 3 further analysis in appendix §f and we observe that the greatest number of correct overlaps is for the entailment label, whereas the greatest number of incorrect predictions is for the contradiction label. we see the maximum overlap in neutral prediction and the least overlap in contradiction prediction in terms of consistency. this demonstrates that the model performs identically on both the original hindi data and the machinetranslated hindi data, bolstering the legitimacy of our dataset. ## related work recently many indic-specific resources are developed such as indicnlpsuite @xcite , which include (a.) word embeddings: indicft, (b.) transformer models: indicbert, (c.) monolingual corpora: indiccorp, (d.) and, evaluation benchmark: indicglue . furthermore, indic-specific pre-processing libraries such as inltk @xcite and indic-nlp-library @xcite , other indic monolingual corpora: common crawl oscar corpus @xcite , multilingual parallel corpora: pmindia @xcite and samantar @xcite , transformer model muril @xcite and language specific indic-transformers @xcite exists. ## conclusion with indicxnli we extend the xnli dataset for indic languages family. we benchmark indicxnli with several multi-lingual models using various train-test strategies. we also study the use of english xnli as pre-finetuning dataset. furthermore, we also evaluate models on mixedlanguage inference input and cross-lingual transfer ability. we aim to integrate indicxnli and benchmark models in indicglue @xcite . we also intend to enhance indicxnli with advanced translation techniques. another direction is accessing model performance on indic-indicxnli task, where both premises and hypothesis are in two distinct indic languages. 10998 one of our work's key limitations is that the dataset indicxnli was created by machine translation of the original english xnli dataset. although in-dicxnli is not human translated, it has been carefully evaluated for translation accuracy by a number of natural bilingual indic speakers (2 for each language). furthermore, as shown in our research ( second, adapting an existing dataset risks transferring biases and shortcomings from the original xnli dataset into ours. however, it has been established that xnli is a typical benchmark for evaluating multilingual and cross-lingual sentence representation, and it has been used to evaluate several multilingual models @xcite . morphological analysis of related languages, as well as insights into their performance behavior, may be useful. the authors, however, are not experts in that area, and such an assessment would have been outside the scope of the current work. this study might be expanded to include language groups other than indian languages such as indo-european. third, because of limited resources, the current study did not include large versions of well-known models such as xlm-roberta-large and muril-large. however, for indicbert, mbert, xlm-roberta, and muril, we assessed model performance in relation to model size (#parameters) in for english on these rare indic languages, which are not explore by prior studies. furthermore, we also investigate the cross-lingual transfer effect across indic languages, also not explored before. we explore the impact of multilingual training, english-data augmentation, unified indic model performance, cross-lingual transfer of closely related indic family and english-indic nli through our work. all the above mention topics are not explore for indic language before. we aim to integrate in-dicxnli and benchmark models in indicglue @xcite . such a benchmark dataset is required for investigating the linguistic properties of indian languages and how accurately they are interpreted by various multilingual models. another direction is accessing model performance on indic-indicxnli task, where both premises and hypothesis are in two distinct indic languages. why indictrans for translation? we use the indictrans as a translation model for converting english xnli to indicxnli because of the following reasons: (a.) open-source: indictrans is open-source to public for non-commercial usage without additional fees, while google-translate and microsoft-translate require a paid subscription. (b.) light weight: indictrans is the fastest and the lightest amongst mbart and mt5 on single gpu machines. google-translate and microsoft-translate are also relatively slower due to repeated network-intensive api calls. (c.) indic coverage: seq2seq models like mbart and mt5 are not designed for all languages in the indic family. mbart supports seven (excludes kn,or,pa,as) while mt5 supports nine languages (excludes or,as) out of eleven indic languages. google-translate supports ten out of eleven indic languages (excludes assamese). microsoft translate supports all the eleven indic languages. in future, we plan to enhance indicxnli with better translation.