Assessing Monotonicity Reasoning in D utch through Natural Language Inference (2023)

Abstract: In this paper we investigate monotonicity reasoning in Dutch, through a novel Natural Language Inference dataset. Monotonicity reasoning shows to be highly challenging for Transformer-based language models in English and here, we corroborate those findings using a parallel Dutch dataset, obtained by translating the Monotonicity Entailment Dataset of Yanaka et al. (2019). After fine-tuning two Dutch language models BERTje and RobBERT on the Dutch NLI dataset SICK-NL, we find that performance severely drops on the monotonicity reasoning dataset, indicating poor generalization capacity of the models. We provide a detailed analysis of the test results by means of the linguistic annotations in the dataset. We find that models struggle with downward entailing contexts, and argue that this is due to a poor understanding of negation. Additionally, we find that the choice of monotonicity context affects model performance on conjunction and disjunction. We hope that this new resource paves the way for further research in generalization of neural reasoning models in Dutch, and contributes to the development of better language technology for Natural Language Inference, specifically for Dutch.

## introduction natural language inference (nli) is one of the standard benchmark tasks for current-day nlp architectures. in this task a model takes two sentences as input, and has to classify the relationship between the former (premise) sentence and the latter (hypothesis) sentence, typically between entailment, contradiction, and neutral. nli makes for a interesting task as drawing the correct inference may require subtle aspects of syntax, lexical semantics, and even pragmatics. while many nli datasets exist like sick @xcite , snli @xcite and its extensions (mnli @xcite , @xcite , e- @xcite ), much is still unknown about how and why neural language models (lms) like bert @xcite perform on the task. evidence shows that fine-tuned lms don't generalize well across nli benchmarks @xcite , and other investigation shows that lms may be exploiting dataset heuristics to solve the task @xcite . more generally, lms do seem to encode a certain amount of syntactic structure @xcite , but the relation to nli remains unclear. in order to shed light on the performance of largescale lms, specific datasets have been developed to understand what models do and don't understand. specifically in the context of nli, @xcite introduce the monotonicity entailment dataset (med), which targets models' capacity for understanding monotonicity reasoning @xcite . monotonicity reasoning is a staple test of human reasoning which requires lexical knowledge, as well as syntactic knowledge, making it suitable for an nli benchmark. in cases of monotonicity reasoning, a particular lexical item in the sentence licenses inferences by means of substituting specific syntactic constituents by either more general (upward context) or more specific (downward context). in figure 1 , the quantifier every is downward entailing in its first argument, and upward entailing in its second arguments, meaning that either man may be replaced by a more specific instance to obtain an inference pair -as in 1(b) -while sung and danced ought to be substituted for a more general constituent to preserve inference, as in 1(c). violating the entailment context leads to a hypothesis for which there is no entailment (but not necessarily a contradiction), as in 1(d). while the field of research into nli is lively, it is largely focused on english. in this article, we work with dutch, a language that has a relatively high digital prevalence, while at the same time being underrepresented in terms of typical sentence-level nlp benchmarks. for dutch there is the lassy corpus, which contains a smaller gold standard, and a larger silver standard syntactically annotated corpus of written @xcite , and the sonar corpus of written dutch @xcite . given the availability of these corpora combined with a rich wikipedia dump, two transformer-based language models have been developed for dutch, based on the bert architecture @xcite ) and the roberta architecture (robbert, @xcite ), both available through huggingface's transformers library. 1 in terms of investigations into these dutch language models, de @xcite argues that bertje encodes a typical 'nlp pipeline', which had been argued for bert before @xcite , whereas @xcite show through probing that long-distance dependencies are hard to recognize for both dutch language models. in order to extend the research done on nli and on dutch nlp, we add a benchmark for monotonicity reasoning in dutch by translating the med dataset of @xcite . we perform an evaluation of large-scale language models for dutch on this novel benchmark, that we dub med-nl. we corroborate the findings of @xcite , observing that the dutch lms similarly have difficulty with inferences coming from downward entailing contexts. further inspection suggests that the main problem comes from inference pairs containing negation. in what follows, we first detail the creation of the dataset and the experimental setup for the evaluation, after which we report results and inspect the model predictions. ## dataset creation & evaluation the dataset is obtained by translation from the english med dataset of @xcite . first, all 5241 unique sentences are collected and lexicographically sorted to ensure consistency among sentence translations. these sentences are given to a native dutch speaker for translation who could ensure quality and naturality of the translated examples. using the translated sentences, we populate the original dataset with its dutch incarnation. since the original entailment/neutral labels derive from monotonicity properties, the entailment labels are preserved in dutch. it is important to note that the labelling is binary, since med only considers entailment and non-entailment (or neutral). to evaluate, we then performed a standard language model fine-tuning routine. we use two state of the art dutch neural language models; bertje (de vries et al., 2019), a bert-based model pretrained for dutch, and robbert @xcite , a roberta-based model for dutch. for multilingual comparison, we furthermore train multilingual bert @xcite . each model was trained on the sick-nl dataset @xcite , which is the only existing nli benchmark for dutch. we binarize the labels in sick-nl by conflating all neutral and contradiction labels into one class, as to make the training data compatible with the binary format of med-nl. training proceeds for 20 epochs, and the model is saved for the epoch for which highest development accuracy is obtained. 2 we test on the sick-nl for validation purposes, after which testing is performed on med-nl. to reduce any potential influences of performance perturbation due to model seed initialization, we train each model thrice and report seed-averaged accuracy. ## results & analysis performance on the development and test set of sick-nl are slightly higher than reported in related work @xcite , which may be due to the fact that the classification labels have been binarized. the high drop in accuracy on med-nl is however on par with reported results on its english counterpart @xcite , despite the models and training dataset being different. in terms of difference between the models, overall accuracy barely distinguishes bertje and robbert in terms of pure performance. interestingly, multilingual bert has a performance decline of ca 15% compared to rob-bert, yet reached highest performance on med-nl. the multilingual model has more trouble with the dutch training data, although all three runs reached peak validation accuracy after one epoch of training. monotonicity contexts a breakdown of accuracy results by the type of monotonicity context is given in specifically, these results contrast the performance of the monolingual dutch lms with multilingual bert, the latter doing the worst on downward entailing contexts while trumping the former models on non-monotone contexts. ## conclusion in this paper we provided med-nl, a novel nli dataset for dutch, which specifically targets monotonicity reasoning. the evaluation of two dutch language models on this test set shows that the models specifically struggle with cases in downward entailing contexts, which had earlier been established for english as well @xcite . however, we indicate specifically that the role of negation words may play a large role in the poor model performance on such cases, giving way for future research into language models and negation. on the other hand, the evaluation also shows that disjunction is much easier to handle by the models than conjunction, for which no explanation was found. in future investigations, we hope to provide more analysis of these language models, specifically regarding negation.