A Novel Cartography-Based Curriculum Learning Method Applied on R o NLI : The First R omanian Natural Language Inference Corpus (2024)

Abstract: Natural language inference (NLI), the task of recognizing the entailment relationship in sentence pairs, is an actively studied topic serving as a proxy for natural language understanding. Despite the relevance of the task in building conversational agents and improving text classification, machine translation and other NLP tasks, to the best of our knowledge, there is no publicly available NLI corpus for the Romanian language. To this end, we introduce the first Romanian NLI corpus (RoNLI) comprising 58K training sentence pairs, which are obtained via distant supervision, and 6K validation and test sentence pairs, which are manually annotated with the correct labels. We conduct experiments with multiple machine learning methods based on distant learning, ranging from shallow models based on word embeddings to transformer-based neural networks, to establish a set of competitive baselines. Furthermore, we improve on the best model by employing a new curriculum learning strategy based on data cartography. Our dataset and code to reproduce the baselines are available at https://github.com/Eduard6421/RONLI.

## introduction given a sentence pair composed of a premise and a hypothesis, natural language inference (nli) @xcite , a.k.a. textual entailment recognition, is the task of determining if the premise entails, contradicts, or is neutral to the hypothesis. nli is an actively studied problem @xcite @xcite @xcite @xcite @xcite , being an essential task to be solved before addressing natural language understanding (nlu). its complexity stems from the fact that nlu is generally considered an ai-hard problem @xcite . notably, nli forms the foundation of numerous advanced natural language processing systems @xcite , providing a backbone for multiple study areas such as language modeling @xcite , conversational agents @xcite , zero-shot text classification @xcite , image captioning @xcite ), text summarization @xcite , discourse parsing @xcite , and many others @xcite . the importance of nli is well recognized, being included as a reference task in benchmarks such as glue @xcite and superglue @xcite . to date, the nli task has been intensively studied in the english language @xcite @xcite @xcite and other languages, such as chinese @xcite , turkish @xcite , portugese @xcite and indonesian @xcite , as well as in multi-lingual scenarios @xcite . however, the sparsity of resources led researchers to overlook the development of nli models for low-resource languages, such as romanian @xcite . in this context, we introduce a novel romanian natural language inference corpus (ronli) composed of 64k sentence pairs. the samples are divided into 58k for training, 3k for validation, and 3k for testing. we collected the sentence pairs from the romanian version of wikipedia, while searching for certain linking phrases between contiguous sentences to automatically label the sentence pairs. the training pairs are obtained via an automatic rule-based annotation process known as distant supervision @xcite , while the validation and test sentence pairs are manually annotated with the correct labels. to the best of our knowledge, ronli is the first publicly available corpus for romanian natural lan-guage inference. ronli is meant to attract the attention of researchers in studying and developing nli models for under-studied languages, where nlu systems often produce subpar results, due to data scarcity and cross-lingual settings. to this end, we believe that ronli is a valuable resource. we publish our data and source code under the open-source cc by-nc-sa 4.0 license at: https://github.com/eduard6421/ronli . we carry out experiments with multiple machine learning methods, ranging from shallow models based on word embeddings to transformer-based neural networks, to establish a set of competitive baselines. moreover, we employ data cartography @xcite to characterize our training samples as easy-to-learn (e2l), ambiguous (a), and hard-to-learn (h2l), from the perspective of the best baseline model. further, we study several approaches that harness the resulting data characterization to mitigate the inherent labeling noise caused by the automatic labeling process. we manage to improve the top model via a novel curriculum learning method based on data cartography. ## related work to date, there are multiple nli datasets for the english language that are derived from different data sources and capture a different label space. for example, the rte dataset @xcite , which was crucial for the emergence of the nli task, consists of premises extracted from news articles, with @xmath0, @xmath1. sick @xcite contains sentence pairs derived from image captions and video descriptions classified into three classes: entailment, contradiction, or neutral. similar to sick, snli @xcite consists of premises extracted from image captions (with hypotheses being written by human annotators), covering the same label space as sick. despite being very large in size, one limitation of snli is that it lacks data source diversity. @xcite introduced mnli to specifically increase the diversity of the data. that is, in mnli, the premises are extracted from diverse sources such as face-to-face conversations, travel guides, and the 9/11 event, while the hypotheses are derived exactly as in snli. while both snli and mnli are widely used to track progress in natural language understanding, they have been shown to contain spurious correlations @xcite ., category romanian english translation contrastive în contrast in contrast în contradicţie in contradiction în opoziţie in opposition entailment cu alte cuvinte in other words în alţi termeni in other terms pe larg in broader terms reasoning astfel thus prin urmare therefore în concluzie in conclusion neutral n/a n/a there are many other relevant nli datasets for english, which are briefly mentioned next. qnli @xcite is derived from the squad question answering dataset @xcite . scitail @xcite is derived from a school level science question-answer corpus with the sentence pairs being classified into two classes: entailment or not-entailment. mednli (romanov and shivade, 2018) is obtained from medical records of patients with @xmath2, @xmath3, contradiction, or neutral. contractnli @xcite is derived from document-level contracts and annotated with a set of hypotheses (or claims) per contract with @xmath4, @xmath5, contradiction, or neutral. scinli @xcite is obtained from research articles published in the acl anthology, but unlike the above nli datasets where the hypotheses are written by humans specifically with the nli task in mind, in scinli, both sentences in a pair are extracted from research articles. these sentences are written by researchers without the nli task in mind, and thus, are expected to contain fewer spurious correlations. the labels in scinli are entailment, contrasting, reasoning, and neutral (extending the usual set with the reasoning relation that often occurs in formal writing). owing to the importance of the nli task, several efforts have been directed to the development of datasets in other languages. for example, @xcite introduced xnli, a cross-lingual evaluation dataset obtained by translating examples from mnli. @xcite created ocnli, the first large chinese dataset for nli, whereas @xcite explored nli in the spanish language. other works focused on the creation of nli datasets from low-resource languages such as creole @xcite , indonesian @xcite , and turkish @xcite , either by human annotations or automatic translations from english. similar to these latter works, we focus on a low-resource language, romanian, to create a large nli dataset, the first of its kind. in contrast to these studies and inspired by the creation of scinli @xcite , instead of resorting to english translations, we use the linking phrases between contiguous sentences in text to automatically label a large training set for the romanian language, while the validation and test sets are manually annotated. the resulting text samples are further split using the sentence tokenizer available in nltk @xcite . however, not all sections of wikipedia articles are equally valid sources to extract sentence pairs for natural language inference. to ensure the quality and relevance of our dataset, we exclude segments such as the list of references, the image xml markers, the external links to other articles, and the metadata articles, which usually contain unstructured or less meaningful text. due to the collaborative nature of wikipedia's content, there is a high variety in the quality of sentences. we observed that shorter sentences tend to introduce content that is often deemed unrepresentative for the nli task, lowering the overall quality of our dataset. to extract high-quality sentences from wikipedia, we thus set a minimum length of 50 characters for each sentence. we select this limit empirically, based on visually inspecting the sentences extracted with our parsing methodology. building upon the methodology presented by sadat and caragea (2022b), we divide the sentence pairs into four separate relationship types: contrastive, entailment, reasoning and neutral. to automatically label the sentence pairs using distant supervision, we establish a set of language-specific linking phrases and transition words that are very likely to predetermine the relationship between two sentences. some examples of linking phrases for the related sentence pairs are shown in while recognizing that this approach is not infallible, it is undoubtedly an effective strategy for automatically collecting representative data for nli in an unsupervised manner. the method allows us to harness the inherent relational information embedded in the romanian vocabulary without requiring specific knowledge about the subject matter. by employing this methodology, we successfully extract a total of 64k sentence pairs. in all the extracted examples, the linking phrases are removed to make sure that models do not learn superficial patterns (based on linking phrases) from the data. to create our training, validation, and test sets, we employ stratified sampling to maintain the inherent distribution of the original data. in as illustrated in figure 1 , data cartography provides a characterization of the training examples as easy-to-learn (e2l), ambiguous (a), and hard-tolearn (h2l), for the chosen ro-bert model. easyto-learn examples reside in the top-left quadrant of the data map, describing items which exhibit low variability and high confidence. the model's strong certainty in its decisions indicates that it faces minimal difficulty in comprehending these examples. in contrast, hard-to-learn examples, occupying the bottom-left quadrant, demonstrate low variability but differentiate themselves from the previous category in terms of confidence, which stays low in this case. the low confidence of the model in the ground-truth label suggests that the examples from this region are likely to be mislabeled. this is because most examples from the underrepresented classes (contrastive, entailment) are labeled as h2l (see for most training examples, our data map shows that ro-bert presents predominantly high confidence (over 30k training samples have the average confidence above 0.9) and high correctness (over 40k samples are correctly classified), suggesting that the model is particularly effective in recognizing and learning consistent patterns. a secondary cluster can be observed pertaining to the hard-tolearn scenarios reflecting that the model starts and remains strongly biased across training epochs for a relatively small portion of samples. in summary, the data map suggests that most samples are consistently labeled. corroborating this observation with the substantial cohen's kappa agreement between the automatic and manual labels suggests that the automatic labels are mostly correct. nonetheless, we further investigate the implications of the observed data distribution in section 5 by considering training scenarios where we deliberately choose the type of data to incorporate, encompassing varying combinations of e2l, a, and h2l instances. ## models we employ grid-search to tune the hyperparameters of all models (see details in appendix a). classifiers based on word embeddings. our first two baselines are based on word embeddings, leveraging the 300-dimensional word vectors returned by the fasttext framework @xcite , which includes support for the romanian language. we select fasttext primarily due to its inherent capability to manage out-of-vocabulary words, a crucial aspect considering the diverse nature of wikipedia content. we represent each sentence as a continuous bag-of-words (cbow), i.e., we compute the average of the word embeddings in each sentence. next, the cbow representations generated for a sentence pair are concatenated, resulting in a joint representation of the premise and the hypothesis. we consider two alternative classifiers to learn from the extracted cbow embeddings, namely a linear support vector machines (svm) model and a multinomial logistic regression (softmax) model. rogpt2. the next baselines draw upon the capabilities of transformer models @xcite , which have shown impressive performance on various nlp tasks, including nli. our third baseline is based on the generative capabilities of the gpt family of models. by harnessing the inherent generative knowledge embedded within gpt, we can establish meaningful correlations between two sentences, a premise and a hypothesis, and accurately infer their relationship. for this approach, we employ a gpt2 model pre-trained on romanian corpora @xcite . we extract the last hidden states of the end-of-sentence (eos) token and feed it to a softmax classification head. chatgpt 3.5 turbo. another way to utilize transformer-based large language models (llms) for various tasks is via in-context (prompt-based) learning. this approach leverages the metalearning capabilities of llms. to this end, we employ a powerful multilingual llm, namely chat-gpt 3.5 turbo @xcite , with incontext learning (two examples per class), to directly predict the test labels. this model does not require any fine-tuning. multilingual bert. we employ a multilingual bert model which is pre-trained on the xnli corpus @xcite . we then consider two options, one based on zero-shot learning, and one based on fine-tuning. the zero-shot multilingual bert follows the definition-restrictive approach described by @xcite . this is because our set of classes (defined in appendix a) do not coincide exactly with those of xnli. ro-bert. our sixth baseline is a variant of bert @xcite pre-trained on romanian corpora, namely ro-bert @xcite . we hypothesize that ro-bert discerns inherent characteristics within and between sentences, allowing it to accurately predict the relationship type. we extract the [cls] token from the final hidden state of each sentence in order to obtain an internal representation. the extracted embeddings are then concatenated and fed into a softmax classification head, enabling the system to predict the relationship between pairs. ro-bert (spurious). natural language inference datasets commonly face the challenge of spurious correlations @xcite , which are statistical associations that machine learning models learn to interpret and rely upon when making predictions. such correlations do not necessarily reflect a robust understanding of nli, but rather illustrate common language patterns and expressions. in order to assess how much ronli is affected by this phenomenon, we fine-tune ro-bert only on the hypotheses, with the same hyperparameters as above. the performance of this model is proportional to the amount of spurious correlations. ro-bert + cartography subsampling. we explore additional baselines based on the categorization produced by data cartography. following the work of @xcite , we fine-tune the ro-bert model on subsets of the training set, determined by the difficulty categories: easy-tolearn, ambiguous and hard-to-learn. we train one ro-bert model for each of the three categories of samples. in addition, we train a ro-bert model on easy-to-learn and ambiguous (e2l+a) samples, based on the intuition that hard-to-learn samples suffer from high rates of labeling noise, and should thus be discarded. ro-bert + length-cl. we employ a ro-bert model based on curriculum learning, a training paradigm proposed by @xcite . the technique emulates the human learning process, which typically progresses from simpler to more complex concepts, and applies it to the training of machine learning models. the idea behind the technique is to expose the model to easier examples first, thus building a foundational understanding of the task, which will in turn allow it to accommodate more complex samples faster. curriculum learning has received significant attention from researchers due to its vast applicability, as shown by @xcite . following nagatsuka et al. ( 2023 ), the considered baseline applies the curriculum with respect to text length. for a fair comparison with the conventional training paradigm, we ensure that the total number of training iterations is equal with those of the base model. we perform the curriculum training for half the number of total iterations. subsequently, we continue training under the standard paradigm, using all available data for the remaining iterations. as for the baseline ro-bert, we introduce early stopping to prevent overfitting. we apply the same training procedure to all curriculum models described below. ro-bert + sts-cl. an intuitive way to introduce curriculum learning with respect to the nli task is to use the semantic text similarity (sts) between sentences to measure difficulty. we thus add a baseline ro-bert model that uses curriculum learning based on sts. to calculate sts, we utilize the cosine similarity based on the fine-tuned ro-bert embeddings of the sentences. the training starts with the most similar sentence pairs, and progressively adds less similar sentence pairs. ro-bert + cart-cl. we propose a ro-bert model based on curriculum learning with respect to the difficulty groups established via data cartography. we start the training on the e2l samples for a number of iterations. we gradually incorporate more challenging batches, adding ambiguous data instances, then h2l samples. if the total number of iterations is n , we perform n/4 iterations with e2l samples, n/4 iterations with a samples, and n/2 iterations with h2l samples. ro-bert + cart-cl++. our next model confronts certain caveats introduced by data cartography @xcite in terms of data difficulty characterization. given that the grouping of data instances relies on one metric (either confidence or variability) instead of both, we observed that the e2l, a, and h2l groups defined by @xcite do not form a partition of the training set, leaving samples outside these groups, while assigning other samples to two different groups. leaving data outside or duplicating some of the examples can bias the model and lead to suboptimal performance. we conjecture that a better curriculum can be created by designing a difficulty scoring function that jointly takes confidence and variability into account. let c i and v i be the confidence and variability of the i-th training sample. we introduce a new difficulty scoring function s : [0, 1] × [0, 1] → [0, 3] defined as follows: ## results and observations experimental setup. we perform nli experiments with the set of baselines described above. we also carry out experiments with the data cartography groups, considering scenarios when our top performing model is trained with subsets of the training set, namely using only e2l examples, a examples, h2l examples, and e2l+a examples. we add the models based on curriculum learning to our line-up of models, specifically length-cl, sts-cl, cart-cl, cart-cl++, and cart-stra-cl++. as evaluation metrics, we report the precision, recall and f 1 score for each of the four classes, on the manually labeled test set. we also report the micro and macro f 1 scores to determine how models are affected by the imbalanced nature of ronli. note that the micro f 1 is equivalent to the classification accuracy, while the macro f 1 is also known as the group-averaged f 1 . our experimental methodology is based on conducting multiple trials of each experiment, specifically five runs, to ensure the robustness and reliability of our findings. the model that demonstrates the best performance on the validation set across the five trials is the one selected for evaluation on the test set. this procedure is fairly performed for all models, including the baselines. results. the results obtained by the baselines on the individual classes are presented in the overall results in terms of the micro and macro f 1 scores obtained by the baselines are shown in bert lead to some interesting observations. when ro-bert is trained on hypotheses, its performance drops by 3% in terms of micro f 1 , and 2% in terms of macro f 1 , indicating that spurious correlations are not enough to reach high performance levels. the ro-bert trained on e2l examples performs worse than the vanilla ro-bert, suggesting that the former model does not benefit much from the homogeneity of the easy examples. the ro-bert trained on h2l examples reaches an even lower performance, suggesting that focusing too much on complex examples can hurt overall performance, possibly due to overfitting on these hard cases (and their potentially erroneous labels). in contrast, the ro-bert trained on ambiguous examples and the one trained on easy and ambiguous examples (e2l + a) are fairly close to vanilla ro-bert, indicating that the ambiguous samples represent the most useful group to train the model on. the state-of-the-art curriculum method, length-cl @xcite , as well as our first curriculum learning approach, cart-cl, exhibit lower performance than the vanilla ro-bert. cart-cl++ outperforms cart-cl, confirming that our difficulty scoring function s is useful. furthermore, our function boosts the macro f 1 score of ro-bert + cart-cl++ over the vanilla ro-bert. however, it is likely that cart-cl++ is still suboptimal because, in the early stages of the curriculum, when using only the e2l group, there is a complete lack of contrastive and entailment pairs (see ## discussion a unique characteristic of ronli, by contrasting it with other existing nli datasets, is that the hypotheses in ronli are sentences from wikipedia which are not written by crowd-workers with the nli task in mind, and these hypotheses are more diverse. in contrast, hypotheses in snli @xcite and mnli @xcite are written by paid crowd-workers and lack diversity. for example, in snli, we observed many pairs such as those presented in furthermore, we emphasize that romanian is a unique eastern romance language, being part of a linguistic group that evolved from several dialects of vulgar latin which separated from the western romance languages between the 5th and the 8th centuries ad. being surrounded by slavic neighbors, it has a strong slavic influence, which makes romanian a unique language (among the latin languages). therefore, the results obtained by multilingual bert in the zero-shot and fine-tuned settings (presented in tables 4 and 5 ) suggest that there are linguistic differences between our dataset for nli in romanian and other nli-supported languages. this confirms the need for a languagespecific dataset such as ronli, despite the presence of other latin languages, such as french and spanish, and even slavic languages, such as bulgarian and russian, in xnli. ## conclusion in this work, we introduced ronli, the first public corpus for romanian natural language inference. we trained and evaluated several models to establish a set of competitive baselines for future research on our corpus. we also performed experiments to analyze the effect of spurious correlations, as well as the effect of harnessing data cartography to establish groups of useful samples or to develop curriculum learning strategies. notably, we introduced a novel curriculum learning strategy based on data cartography and stratified sampling, boosting the overall micro and macro f 1 scores of ro-bert by 2% and 3%, respectively. these improvements are statistically significant. nevertheless, our empirical results indicate that there is sufficient room for future research on the romanian nli task. we make our dataset and code available to the community to lay the grounds for future research on romanian nli. our work is slightly limited by the number of samples with manual labels included in our corpus. this limitation is caused by the laborious annotation process involving human effort. however, we underline that ronli is not the only nli corpus with training data based on automatic labels. for example, scinli @xcite is created using a very similar process, based on automatically labeling english sentence pairs via linking phrases.