XNLI eu: a dataset for cross-lingual NLI in B asque (2024)

Abstract: XNLI is a popular Natural Language Inference (NLI) benchmark widely used to evaluate cross-lingual Natural Language Understanding (NLU) capabilities across languages. In this paper, we expand XNLI to include Basque, a low-resource language that can greatly benefit from transfer-learning approaches. The new dataset, dubbed XNLIeu, has been developed by first machine-translating the English XNLI corpus into Basque, followed by a manual post-edition step. We have conducted a series of experiments using mono- and multilingual LLMs to assess a) the effect of professional post-edition on the MT system; b) the best cross-lingual strategy for NLI in Basque; and c) whether the choice of the best cross-lingual strategy is influenced by the fact that the dataset is built by translation. The results show that post-edition is necessary and that the translate-train cross-lingual strategy obtains better results overall, although the gain is lower when tested in a dataset that has been built natively from scratch. Our code and datasets are publicly available under open licenses.

## introduction the natural language inference (nli) task consists in classifying pairs of sentences -a premise and a hypothesis-according to their semantic relation: entailment, when the meaning of the premise entails that of the hypothesis; contradiction, when both sentences have opposing truth conditions and can not co-occur at the same time; and neutral, when both sentences are not semantically related (see nli is an important task towards natural language understanding (nlu), and is often used to test the semantic understanding of language models. it provides a general framework where different nlp tasks can be reframed, including information previous work has emphasized the importance of the origin of the train and test data in cross-lingual settings, i.e., whether they are original or created through translation. in particular, @xcite show that a mismatch in the origin between training and test data may have a serious impact on the results, particularly when comparing different cross-lingual strategies. moreover, nli datasets are known to be biased and contain artifacts that lead models to rely on superficial clues @xcite @xcite . to analyze the impact of these factors in xnlieu, we have created a native test set completely from scratch with original premises extracted from sources with content in basque and hypotheses provided by basque speakers, which were specifically told to avoid such biases. using these datasets, we have conducted a series of experiments using mono-and multilingual language models for basque, both discriminative and generative, and have tested different training variants for cross-lingual nli in basque. the experiments set a new baseline for nli in basque, and have served us to analyze the effect of professional post-edition compared to the automatic machine-translation system. we have also identified the most effective cross-lingual strategy for nli in basque, considering both translated and native sets. this paper makes the following contributions: this paper is structured as follows: section 2 covers some relevant research and resources related to the topic in hand, our dataset is further explained in section 3, the description of the experiments and experimental settings in section 4, the results are covered in section 5, section 6 includes the analysis of the errors in the outputs of our models, section 7 a summary of the research and its conclusions; and there is a final section that expands on the limitations of our research. ## related work cross-lingual nli. the best results on nli benchmarks to date are based on supervised learning, which requires large amounts of training data that are only available for resource-rich languages such as english. examples of english nli datasets are the stanford nli corpus @xcite , the multi-genre nli corpus @xcite and the adversarial nli corpus @xcite . the nli task is also included among the tasks of the popular nlu benchmarks glue @xcite and superglue @xcite . crosslingual nli is an alternative approach that leverages pre-trained multilingual models which are fine-tuned in resource-rich languages, then tested in the desired target language. this transfer approach, called zero-shot, is often compared to strategies that involve machine translation: translate-train, where the training set is translated to each target language and used to train the models on their respective language and translate-test, where the test set is translated to the high-resource language, usually english. alternatively, large multilingual autoregressive models are also known to perform well in cross-lingual settings, by providing them with a set of correct input/label pairs as prompts for new inputs @xcite . ## experimental design we have conducted a series of experiments on cross-lingual nli for basque, using different discriminative and generative language models, both mono-and multilingual. all models have been tested using the three datasets described in section 3. we aim to determine if post-edition introduces significant changes to the dataset that enhance its reliability. we also want to compare the results on the xnli-derived datasets with the native humandevised dataset, and analyze the effect of biases and artifacts introduced by translation. since there is no training set in basque for nli, we consider different cross-lingual alternatives foot_1 : discriminative name language # of parameters ixambert multilingual 177m multilingual bert multilingual 179m xlm-roberta (base) multilingual 279m xlm-roberta (large) multilingual 561m berteus basque 124m roberta-eus euscrawl basque 355m generative latxa multilingual 7b bloom multilingual 7.1b xglm multilingual 7.5b at least in english and basque. these models are then fine-tuned on the english mnli corpus. in a further experiment, we explore fine-tuning with source languages beyond english. regarding the models, we have experimented with the following discriminative models: ixam-bert @xcite , multilingual bert @xcite , xlm-roberta large @xcite , berteus @xcite and roberta-eus-large @xcite . further details about these models can be found in we have also tested three multilingual generative models that include basque among their pretraining languages: bloom (bigscience workshop et al., 2023), xglm @xcite and latxa @xcite , a model based on llama 2 tuned for basque with continual pretraining on basque corpora. the prompts used in our experiments can be seen in appendix b. as for evaluation, we select the label whose log-likelihood is highest, according to the model. the code used for testing the generative models is based on that included in the language model evaluation harness project @xcite . following usual practice, we use accuracy as our evaluation metric: the ratio of correctly classified instances divided by the total number of instances. ## results in this section, we show the main results of our experiments and discuss the main findings. we start by analyzing the results on the datasets derived from xnli (xnlieu and xnlieu mt ), followed by a comparison with those obtained using the native dataset. finally, we detail the results of experiments that involved fine-tuning with source languages other than english. ## analysis this section provides additional analyses of the results. we begin by considering the performance of the best model on a per-label basis, followed by a manual comparison of the model outputs on the xnlieu and xnlieu tm datasets to analyze the effects of post-edition. ## conclusions in this work, we introduce xnlieu, a new dataset for cross-lingual nli in basque. xnlieu is developed by machine-translating the english part of xnli followed by a post-edition step with the assistance of professional translators. along with xnlieu we release the full machine-translated version, as well as a basque native version carefully built to avoid known biases in nli datasets. we have conducted a series of cross-lingual basque nli experiments using a set of language models and different cross-lingual strategies. the experiments show that translate-train is the best strategy, particularly when there is no mismatch between the origin of the train and test data. in the native dataset, translate-train still yields the best results, but the difference is comparatively smaller. this finding aligns with prior research examining the effects of translation-based datasets. we also manually analyze the results of the models and find that machine-translation often introduces artifacts that change the meaning of the premises or hypotheses, and that professional translators correct those errors when post-editing. we conclude that postedition is a crucial step towards reliable evaluation of cross-lingual nli. all of the datasets developed in this paper are publicly available under the same licenses as xnli. we believe that they are an important resource that will contribute to filling the gaps in resources that exist in basque, which can hinder the development of research and applications with a focus on semantics in this language.