When Beards Start Shaving Men: A Subject-object Resolution Test Suite for Morpho-syntactic and Semantic Model Introspection (2020)

Abstract: In this paper, we introduce the SORTS Subject-Object Resolution Test Suite of German minimal sentence pairs for model introspection. The full test suite consists of 18,502 transitive clauses with manual annotations of 8 word order patterns, 5 morphological and syntactic and 11 semantic property classes. The test suite has been constructed such that sentences are minimal pairs with respect to a property class. Each property has been selected with a particular focus on its effect on subject-object resolution, the second-most error-prone task within syntactic parsing of German after prepositional phrase attachment (Fischer et al., 2019). The size and detail of annotations make the test suite a valuable resource for natural language processing applications with syntactic and semantic tasks. We use dependency parsing to demonstrate how the test suite allows insights into the process of subject-object resolution. Based on the test suite annotations, word order and case syncretism can be identified as most important factors that affect subject-object resolution.

## introduction subject-object resolution remains a difficult task for syntactic parsing of languages with relatively free word order and case syncretism. for such languages, subject and object cannot always be disambiguated based on morpho-syntactic surface structures @xcite . parser performance on german and dutch, for example, has been shown to suffer significantly from incorrect identification of subject and object @xcite . while there have been task-specific test suites for difficult syntactic phenomena in german such as prepositional phrase attachment, coordination, and verb phrase complementation @xcite @xcite , subjectobject resolution has widely been neglected in existing test suites. in this paper, we introduce the sorts subject-object resolution test suite of german minimal sentence pairs. the test suite has been created manually from hand-selected subject-verb-object triples and contains 18,502 transitive clauses with universal dependency @xcite ) annotations, template-based annotations of 8 word order patterns and manual annotations of 16 morphological, syntactic and semantic property classes. for instance, the sentence sie trifft eine entscheidung. 'she makes a decision.' has been annotated with word order vf[s]lk[v]mf @xcite , the syntactic property subject pronoun and the semantic property light verb in addition to dependency relations relevant for subject-object resolution, as illustrated in figure 1 . one domain of application for this test suite is syntactic parsing. we will use six neural dependency parsers with different architectures to show that the test suite is able to expose the linguistic properties that make subject-object resolution easier or more difficult for parsing. experimental results imply that parsers are not able to resolve certain syntactic structures well, e.g. subject-object pairs with objectsubject order and case syncretism. the paper is structured as follows: section 2 gives an overview of subject-object resolution in german as background to the test suite description in section 3. section 4 shows an application of the test suite in german dependency parsing. related work is presented in section 5 before concluding with some guiding remarks in section 6. sie trifft eine entscheidung . she makes a decision . ## conclusion and future work we presented the sorts test suite for model introspection into subject-object resolution via german minimal sentence pairs. with a total size of 18,502 transitive clauses and 24 syntactic-semantic property classes, the test suite is a valuable resource for inspecting syntactic nlp systems of german. its application in syntactic parsing revealed weaknesses of all parsers when syntactic and morphological cues are insufficient to resolve subjects and objects. particularly difficult are sentences with object-subject order and case syncretism of subject and object. how to best resolve such sentences will remain an interesting case for future research on parsing. another direction for future work focuses on the extension of the test suite to cover more languages and linguistic phenomena. german still provides several cues to subject-object resolution: case marking and subject-verb agreement. in dutch, only some remainders of case marking persist in the pronoun paradigm whereas nominal subjects and objects can only be disambiguated morpho-syntactically through subject-verb agreement. for this reason, a second test suite is currently being constructed for dutch. for ease of comparison, it will to a large extent be a translation of the german test suite. subject-object resolution is only the second largest error class behind pp attachment. a second test suite subset will be dealing with different aspects of pp attachment. in contrast to subject-object resolution, pp attachment has the advantage that it applies to a wider range of languages.