C - XNLI : C roatian Extension of XNLI Dataset (2023)

Abstract: Comprehensive multilingual evaluations have been encouraged by emerging cross-lingual benchmarks and constrained by existing parallel datasets. To partially mitigate this limitation, we extended the Cross-lingual Natural Language Inference (XNLI) corpus with Croatian. The development and test sets were translated by a professional translator, and we show that Croatian is consistent with other XNLI dubs. The train set is translated using Facebook’s 1.2B parameter m2m_100 model. We thoroughly analyze the Croatian train set and compare its quality with the existing machine-translated German set. The comparison is based on 2000 manually scored sentences per language using a variant of the Direct Assessment (DA) score commonly used at the Conference on Machine Translation (WMT). Our findings reveal that a less-resourced language like Croatian is still lacking in translation quality of longer sentences compared to German. However, both sets have a substantial amount of poor quality translations, which should be considered in translation-based training or evaluation setups.

## introduction natural language processing has developed rapidly in recent years. models are starting to achieve human-like performance, but most of these achievements are concentrated on only a small fraction of the world's 7000+ languages. this is to be expected due to the nature of linguistic annotation, which is not only tedious, subjective, and costly, but also requires domain experts, which are in decline @xcite . there are two main approaches commonly used to handle that problem from the models' perspective. the first approach relies on cross-lingual transfer, where the model is pretrained to learn multilingual representations @xcite , while the other approach relies heavily on machine translation (mt) systems to translate the text from a low-resource language to a high-resource language (or vice versa). both approaches can be easily evaluated on cross-lingual benchmarks such as xtreme @xcite or xglue @xcite . they consist of crosslingual datasets grouped by task to allow comprehensive evaluation. unfortunately, xtreme covers 40 languages and xglue only 19. since none of these benchmarks include croatian language in any of their datasets, and crosslingual natural language inference (xnli; @xcite corpus is included in both, we decided to extend xnli with croatian (c-xnli). the task is to classify whether a premise contradicts, entails, or is neutral to the hypothesis. xnli's development and test sets are crowdsourced in english and human-translated into 14 languages, while multinli's @xcite training set is used for training. it also consists of machine-translated sets required for the translatetrain and translate-test paradigms. our croatian extension is created in the same manner as its xnli parent. the development and test sets are translated by a professional translator. since xnli provides translate-train, translate-dev and translate-test sets, we opted for facebook's 1.2b parameter m2m_100 mt model @xcite to create our own translations. it has been shown that mt models still suffer from errors like mistranslations, non-translations and hallucinations @xcite , which motivated us to analyze the quality of our dataset. for this purpose, we sampled 2000 sentences per language in both croatian and german, and evaluated the translations using a variant of the direct assessment (da) score proposed in the multilingual quality estimation dataset (mlqe; @xcite . to summarize, our contributions are the following: (1) we create and analyze the croatian exten-sion of xnli and provide baseline models, (2) we create quality estimation (qe) datasets for croatian and german to evaluate the quality of machinetranslated sentences from the translate-train sets, and (3) we quantify the textual overlap between hypothesis and premise and analyze its impact on baseline models. ## conclusion in this work, we extended xnli to include the croatian language. the development and test sets were translated by a professional translator. we have successfully demonstrated that the quality of the development and test sets is comparable to that of the other languages. to validate the machine-translated training set, we compare our croatian translations with those available for a high-resourced language -german. the comparison is based on 2000 manually scored sentences from german and croatian train sets using a variant of da scores normalized by z-score. our results show that the croatian mt model performs slightly better because it's more up-to-date, even though it's a lower-resourced language. we also found that the croatian translation model performs poorly on longer sentences -premises. finally, we present an overlap metric to measure the textual overlap between the premise and hypothesis. we find that the training set has larger overlaps than the development and test sets. these overlaps resulted in a high correlation between the models' scores, indicating that a model uses cues from the data that also correlate with overlaps. we provide our datasets under the same license foot_3 as the xnli dataset, and also make the accompanying code available on github foot_4 . we hope that by sharing our datasets, researchers will have the opportunity to gain further insights and expand their knowledge in the field of cross-lingual transfer.