Can current NLI systems handle G erman word order? Investigating language model performance on a new G erman challenge set of minimal pairs (2023)

Abstract: Compared to English, German word order is freer and therefore poses additional challenges for natural language inference (NLI). We create WOGLI (Word Order in German Language Inference), the first adversarial NLI dataset for German word order that has the following properties: (i) each premise has an entailed and a non-entailed hypothesis; (ii) premise and hypotheses differ only in word order and necessary morphological changes to mark case and number. In particular, each premise and its two hypotheses contain exactly the same lemmata. Our adversarial examples require the model to use morphological markers in order to recognise or reject entailment. We show that current German autoencoding models fine-tuned on translated NLI data can struggle on this challenge set, reflecting the fact that translated NLI datasets will not mirror all necessary language phenomena in the target language. We also examine performance after data augmentation as well as on related word order phenomena derived from WOGLI. Our datasets are publically available at https://github.com/ireinig/wogli.

## introduction german is endowed with a rather free word order @xcite , especially when it comes to ordering nominal arguments in a sentence. currently, large german nli datasets are only available as translations from other languages. for example, the training portion (392k pairs) of the german xnli dataset @xcite is a machine translation of the english multinli training set @xcite . the testing portion of german xnli is a manual translation of 5k english premise-hypothesis pairs that were newly created by the authors of xnli. such translated sets do not necessarily mirror all german-specific linguistic phenomena, such as the freer german word order. we construct a new german challenge set named wogli (word order in german language inference). this dataset is handcrafted and does not stem from translation. it contains 16k premises where each premise is accompanied by one entailed (e) and one non-entailed (ne) hypothesis that both contain the same lemmata as the premise but change argument order. morphological markers are indicative of subject and (direct) object, thus informing about the hypothesis' entailment relationship to the premise. in other words, wogli serves as a test bed for current language models' capabilities to distinguish subject from object in the context of german word order. our contributions are as follows: all our datasets are publically available foot_0 . the topological model. the topological model @xcite describes regularities in german clause order prefield l brack. middlefield r brack. count (% of accus.) main so peter sieht den mann 231 (86%) peter sees the man acc peter sees the man os den mann sieht peter 38 (14%) the man acc sees peter peter sees the man emb. so dass peter den mann sieht 546 (99%) that peter the man acc sees that peter sees the man os dass den mann peter sieht 6 (1%) that the man acc peter sees that peter sees the man declarative main clauses, such as peter sieht den mann at the top of while subject followed by object (so) is viewed as the canonical word order, it is possible to place the object before the subject (os) in both embedded and main clauses ( os acceptability and minimal pairs. the marked os order is more frequent in main clauses involving the prefield @xcite (around 14% of main clauses with accusative direct object) and in the active voice @xcite (see data and examples in ## wogli construction verb collection. we collected 50 frequent german transitive verb types including agentive (such as warnen/warn), object-experiencer (such as erschrecken/startle) and subject-experiencer (such as lieben/love) verbs. all verbs can take animate (human) subjects as well as animate (human) direct objects, and all objects take the accusative case. all verbs are not symmetric, meaning that they do not lead to bidirectional entailments. 2 in addition, none of the verbs need to split prefixes when used in main clauses so that the resulting premises have a very simple svo structure. all verbs occur at least 70 times in gxnli-train. consequently, any difficulties that a language model will experience are unlikely to be due to verb rarity. noun collection. we collected 144 noun types describing humans that function as direct object or subject in our premises/hypotheses. these include 38 masculine common nouns such as gast/guest, each of which was seen at least 10 times in gxnlitrain and 24 feminine common nouns such as lehrerin/(female) teacher. we collected feminine common nouns by searching for the suffix in in gxnli-train, which often indicates female persons in german. the unbalanced masculine-feminine split is due to the automatic translation of gxnlitrain as gender-neutral english job descriptions, for example doctor, are most frequently translated via the german male form, e.g. arzt instead of the female form ärztin foot_2 . we also collected 41 female and 41 male first names that occur at least 10 times in gxnli-train. the 144 noun types yield 181 different noun surface forms (nominative/accusative, plural/singular). premise and hypothesis generation. we automatically generated german premises as declarative, present tense, main clauses in the active voice with svo structure (see lines 1 and 5) in we have 17 patterns due to combinations of different argument nps, including masculine and feminine proper names and common nouns as well as singular and plural arguments. subjects/objects are either a simple proper name (such as maria) or consist of an article foot_3 and a common noun, e.g. der arzt/the doctor. consequently, each sentence always has a length of four or five words. a list of all 17 patterns is provided in premise der/dieser/ein n om -sg-m arzt warnt sg den/diesen/einen acc-sg-m kunden † the/this/a n om -sg-m doctor warns sg the/this/a acc-sg-m client the/this/a doctor warns the/this/a client h1-so (ne) der/dieser/ein n om -sg-m kunde † warnt sg den/diesen/einen acc-sg-m arzt the/this/a n om -sg-m client warns sg the/this/a acc-sg-m doctor the/this/a client warns the/this/a doctor h2-os (e)* den/diesen/einen acc-sg-m kunden † warnt sg der/dieser/ein n om -sg-m arzt the/this/a acc-sg-m client warns sg the/this/a n om -sg-m doctor the/this/a doctor warns the/this/a client h3-os (ne)* den/diesen/einen acc-sg-m arzt warnt sg der/dieser/ein n om -sg-m kunde † the/this/a acc-sg-m doctor warns sg the/this/a n om -sg-m client the/this/a client warns the/this/a doctor premise der/dieser/ein n om -sg-m minister empfiehlt sg die/diese acc-p l-f autorinnen the/this/a n om -sg-m all word lists with gxnli-train frequencies and translations can be found in our github repository. each of the 50 verb types appears between 308 and 383 times (mean: 339.4 times) in the 16,971 premises. they also appear 20 times on average per pattern in the premises. since models were fine-tuned on gxnli-train in a three-class setting, we merge contradiction and neutral into non-entailed predictions for evaluations on wogli. fine-tuning details are provided in section c of the appendix. results (see ## error analysis all analyses in this section are carried out on ensemble predictions (majority vote of the 5 runs) of the strongest model in ## data augmentation we sample 1,037 premises and their corresponding e/ne hypotheses from wogli, resulting in 2,074 training instances. each of the 17 patterns occurs 61 times. all 50 verb lemmas are represented, each appearing between 18 and 25 times. all 181 noun forms appear at least once. 12 we concatenate these wogli instances with gxnli-train, name the resulting augmented training set gxnli+1037 and shuffle it before finetuning gbert-large 10 times on this augmented training set. we evaluate on the remaining 31,868 wogli instances, named wogli-test-1037. this augmented training set allows gbert-large to classify wogli almost perfectly, while maintaining its performance on gxnli-test ( ## related work many english adversarial nli datasets have been proposed. some of these @xcite @xcite , like us, include minimal pairs with a high word overlap between premise and hypotheses. @xcite , for example, change argument order to generate non-entailments so that "understanding" word order is necessary to solve these. however, in wogli, changes in argument order regarding adversarial nli datasets for german, @xcite investigate negation but do not work on word order. @xcite propose nli diagnostic datasets for french, german and swedish. sentence pairs are manually translated from the russian terra dataset @xcite as well as from the diagnostic dataset of glue @xcite . we inspected a random 100 hypotheses of the german terra dataset, none of which were in marked word order. the translated glue benchmark is annotated with linguistic features relevant for entailment such as lexical semantics, logic, and predicate-argument structure. only the predicate-argument structure examples include a handful where word order of arguments has been inverted between premise and hypothesis. however, resulting hypotheses were often ambiguous and -in our opinion -wrongly annotated as not-entailed. consider the premise john zerbrach das fenster/john broke the window and the hypothesis das fenster hat john eingeschlagen, which is ambiguous between the window n om broke john acc (so order, ne) or the window acc broke john n om (os order, e). this is annotated as non-entailment in the dataset, assuming so order with an implausible semantic reading, whereas the marked word order with a plausible semantic reading leads to entailment. unlike us, both datasets do not emphasise word order. they are also based on translations and therefore rarely contain os hypotheses. ## conclusion we created wogli, a new nli challenge set, in order to examine the challenges brought by the freer german word order. premises, entailed and not-entailed hypotheses contain exactly the same lemmata; the two hypotheses differ only in word order and morphological changes but change label. three current bert-based models fine-tuned on gxnli-train struggle on wogli pairs. this poor performance mirrors the fact that translated nli training sets such as gxnli do not incorporate all required linguistic phenomena that are specific to the target language, german. we find that the number of wogli pairs for augmentation during fine-tuning must be sufficiently high in order to (i) learn wogli and (ii) generalize to other woglilike pairs. even with a larger augmentation set and a large pretrained model, a generalization set that differs more from wogli, such as wogli-oshard (ne) , remains difficult. in future experiments, we will expand wogli datasets to contain additional variation, such as tense variation, more complex sentence structure (additional arguments and adjuncts, active/passive), more complex constituent structure and other sentence types (non-declarative, embedded). this will also allow us to conduct more fine-grained error analyses regarding the hierarchies that influence the linearization of arguments and thus word order.