SICK - NL : A Dataset for D utch Natural Language Inference (2021)

Abstract: We present SICK-NL (read: signal), a dataset targeting Natural Language Inference in Dutch. SICK-NL is obtained by translating the SICK dataset of (Marelli et al., 2014) from English into Dutch. Having a parallel inference dataset allows us to compare both monolingual and multilingual NLP models for English and Dutch on the two tasks. In the paper, we motivate and detail the translation process, perform a baseline evaluation on both the original SICK dataset and its Dutch incarnation SICK-NL, taking inspiration from Dutch skipgram embeddings and contextualised embedding models. In addition, we encapsulate two phenomena encountered in the translation to formulate stress tests and verify how well the Dutch models capture syntactic restructurings that do not affect semantics. Our main finding is all models perform worse on SICK-NL than on SICK, indicating that the Dutch dataset is more challenging than the English original. Results on the stress tests show that models don’t fully capture word order freedom in Dutch, warranting future systematic studies.

## introduction one of the primary tasks for natural language processing (nlp) systems is natural language inference (nli), where the goal is to determine, for a given premise sentence whether it contradicts, entails, or is neutral with respect to a given hypothesis sentence. for english, several standard nli datasets exist, such as sick @xcite , snli @xcite and mnli @xcite . having such inference datasets available only for english may introduces a bias in nlp research. @xcite introduce xnli, a multilingual version of a fragment of the snli dataset, that contains pairs for natural language inference in 15 languages and is explicitly intended to serve as a resource for evaluating crosslingual representations. however, dutch is not represented in any current nli dataset, a lack that we wish to complement. dutch counts as a high-resource language, with the sixth largest wikipedia (2m+ articles), despite having ca. 25m native speakers. moreover, the syntactically parsed lassy corpus of written dutch @xcite , and the sonar corpus of written dutch @xcite provide rich resources on which nlp systems may be developed. indeed, dutch is in the scope of the multilingual bert models published by google @xcite , and two monolingual dutch bert models have been published as part of hugging-face's transformers library @xcite . compared to english, however, the number of evaluation tasks for dutch is limited. there is a named entity recognition task coming from the conll-2003 shared task @xcite ; from a one million word hand annotated subcorpus of sonar @xcite one derives part-of-speech tagging, named entity recognition and semantic role labelling tasks. more recently a sentiment analysis dataset was introduced, based on dutch book reviews (van der burgh and verberne, 2019). moreover, allein et al. (2020) introduce a classification task where a model needs to distinguish between the pronouns die and dat. given the focus on word/token-level tasks in dutch, we aim to complement existing resources with an nli task for dutch. we do so by deriving it from the english sick dataset, for the following reasons: first, this dataset requires a small amount of world knowledge as it was derived mainly from image captions that are typically concrete descriptions of a scene. therefore, no world knowledge requirements will be imposed on an nlp model for the task, but rather its ability for reasoning will be assessed. secondly, due to the structure of the sentences in sick, the types of inferences can be attributed to particular constructs, such as hypernymy/hyponymy, negation, or choice of quantification. thirdly, sick contains 6076 unique sentences and almost 10k inference pairs, making it a sizeable dataset for nlp standards, while deriving a dutch version is more manageable than with other datasets. we make the dataset, code and derived resources (see section 5) available online foot_0 . ## dataset creation we follow a semi-automatic translation procedure to create sick-nl, similar to the portuguese version of sick @xcite . first, we use a machine translator to translate all of the (6076) unique sentences of sick foot_1 . we review each sentence and its translation in parallel, correcting any mistakes made by the machine translator, and maintaining consistency of individual words' translation, in the process guaranteeing that the meaning of each sentence is preserved as much as possible. finally, we perform a postprocessing step in which we ensure unique translations for unique sentences (alignment) with as few exceptions as possible. in this way we obtain 6059 unique dutch sentences, which means that the dataset is almost fully aligned on the sentence level. it should be noted, however, that we can not fully guarantee the same choice of words in each sentence pair in the original dataset, as we translate sentence by sentence. in the whole process, we adapted 1833 of the 6076 automatic translations, either because of translation errors or alignment constraints. as we are interested in collecting a comparable and aligned dataset, we maintain the relatedness and entailment scores from the original dataset. ## baseline evaluation and results we evaluate two types of models as a baseline to compare sick-nl with its english original. first, we evaluate embeddings that were not specifically trained on sick. the relatedness results show that (a) using the [cls] token embedding as a sentence encoding performs worse than taking the average of word embeddings, and that (b) the dutch incarnation of sick is harder than the original english dataset. it may be noted that relatedness scores are less robust then entailment labels, and so our first result may not be enough support for the claim that sick-nl poses a more challenging task. for example, it could be that relatedness scores will differ slightly if we were to ask a number of annotators to reevaluate the dutch dataset. for each model, we report the best score out of 20 epochs of fine-tuning. ## error analysis in order to understand the differences between the dutch and english language models on the respective tasks, we dive deeper into the classification results. we plot confusion matrices for each model in in the case of english, performance on classifying contradictions is worse for multilingual bert and roberta, and roberta also gives highest recall values for the neutral and entailment labels. this is all not surprising given that roberta has the overall highest test set accuracy. the surprising results come mainly from the comparison between english and dutch models. where bertje is rather indecisive when it comes to neutral sentence pairs (it classifies roughly equal numbers as neutral and entailment), it classifies 74% of entailment pairs as neutral. for multilingual bert the situation is reversed, with 47% of neutral entailments classified as entailment, although for cases of entailment, the classifier did not clearly distinguish neutral from entailment. the most surprising pattern was observed in robbert: where roberta still has high recall for neutral and entailment, its dutch counterpart robbert mistakes most neutral cases as entailment and even more so vice versa. for all models, in these four cases of misclassification, in the case of the english task the correct inference was made in at least 99% of the cases. following @xcite , we inspect these prominent cases of misclassification in dutch by looking at the number of cases of high overlap (at most four words not in common), and at the number of length mismatches (the difference between sentence length exceeds 4), and set off these distributions against that of the test set, in the main finding here is that word overlap does provide a strong cue in the english dataset, especially given that sick has more cases (1970) overall than sick-nl (1385), and that in sick they are more concentrated in cases of entailment. length mismatches occur more often in sick-nl but seem to provide less of a cue to the models to make strong inference decisions. ## stress testing one of the potential sources of error could have been the passive form translation of a verb. such constructions, combined with a prepositional phrase, form an interesting testbed for dutch as they allow the prepositional phrase to be moved in front of the verb in a sentence without changing the meaning. for example, "een vrouw is aan het wakeboarden op een meer" ("a woman is wakeboarding on a lake"), may in dutch be used interchangeably with "een vrouw is op een meer aan het wakeboarden"). we select all (87) sentences in sick-nl that contain both the 'aan het' construction and a prepositional phrase, and generate their permutations. then, we replace all (225) inference pairs with these sentences such that they now contain a sentence with different word order but the exact same meaning and therefore the inference label is preserved. we then verify how the model's predictions do on those inference pairs that were in the test set (116). additionally, we check whether the models are able to interchange sentences and their rewritten equivalent (i.e. classify as entailment). as a second test, we investigate the role of the simple present versus the present continuous. we take all the (383) cases of present continuous in the dutch dataset and replace them by a simple present equivalent, leading to 1137 pairs, out of which 576 occur in the test data. for example, we turn the sentence "de man is aan het zwemmen" into the simple form "de man zwemt". we then repeat the same procedure as above, asking how many inference predictions change as a result of this form change, and whether the forms can be used interchangeably for the models. the results in ## conclusion in this paper we introduced an nli dataset for dutch by semi-automatically translating the sick dataset. to our knowledge this is the first available inference task for dutch. despite the common perception that dutch is very similar to english, sick-nl was significantly more difficult to tackle, even for language models that had access to the training data for fine-tuning. we hypothesised that the difference in result may be due to a larger vocabulary in sick-nl, and a decline in word overlap between inference pairs. in addition we performed two stress tests and found that pretrained models that were exposed to the training data had difficulty detecting semantically equivalent sentences that differ only in word order. further work will therefore more systematically assess such phenomena.