Do UD Trees Match Mention Spans in Coreference Annotations? (2021)

Abstract: One can find dozens of data resources for various languages in which coreference - a relation between two or more expressions that refer to the same real-world entity - is manually annotated. One could also assume that such expressions usually constitute syntactically meaningful units; however, mention spans have been annotated simply by delimiting token intervals in most coreference projects, i.e., independently of any syntactic representation. We argue that it could be advantageous to make syntactic and coreference annotations convergent in the long term. We present a pilot empirical study focused on matches and mismatches between hand-annotated linear mention spans and automatically parsed syntactic trees that follow Universal Dependencies conventions. The study covers 9 datasets for 8 different languages.

## introduction the relation of coreference holds between expressions in a text which (co-)refer to the same realword entity or event. the dominating practice in coreference-related language resources is that the individual referring expressions (often called mentions) are represented simply as linearly delimited sequences of tokens (often called mention spans) within the original sentences, without using syntactic representation of the sentences. we argue that integrating coreference annotation with syntactic annotation, more specifically with dependency trees, would be beneficial in the long term from various linguistic and computational perspectives, especially if we hypothesize that: in this paper, we aim to bring empirical evidence for supporting the first hypothesis. we show that mention spans usually correspond to subtrees of dependency trees quite straightforwardly; and if they do not, it can be explained by errors in the dependency trees or by mention span errors in most cases. this also shows a useful side effect of our line of thought: a joint view on coreference and syntactic annotations could be used for detection of annotation inconsistencies in both. there are dozens of coreference-related resources annotated for various languages under highly diverse schemes. in our study, we use 9 coreference datasets for 8 different languages selected from corefud 0.1 @xcite , which is a collection of coreference datasets unified under a common scheme. mention spans in all 9 datasets result from manual annotation. dependency trees available in the collection follow the universal dependencies (ud) scheme @xcite and result from manual annotation in one case and from automatic parsing with udpipe @xcite in the 8 remaining cases. in all cases, the dependency trees came into existence independently of the coreference annotation. ## related work the idea of considering coreference and syntactic information together was quite popular in the last two decades of the 20 th century, generally accepted in the meaning-text theory @xcite or in the functional-generative description @xcite . coreference is also one of the main concepts underlying binding phenomena in generative syntax @xcite . however, with the advent of large-scale annotated corpora, coreference and syntax have somewhat diverged. the syntax-aware annotation of coreference demands for manual syntactic annotation, which is very expensive and not always feasible. as a result, coreference relations in most existing large-scale annotated resources are marked on raw texts, textual spans being defined as coreferring mentions, see, e.g. hinrichs et al. (2005); uryupina et al. (2020); hendrickx et al. (2008); désoyer et al. (2016); landragin (2016 ); bourgonje and stede (2020); guillou et al. (2014); lapshinova-koltunski et al. (2018); žitkus and butkienė (2018); @xcite . some of these datasets @xcite label syntactic heads of the mentions. for some other datasets, syntactic annotation exists but it was created independently of coreference annotation. this is the case of gum for english @xcite or szegedkoref, a coreferentially annotated corpus of hungarian, which is a part of the szeged treebank @xcite . coreference in ontonotes @xcite is a special case, as potentially coreferential mentions have been first excerpted from constituency trees and then annotated independently. to the best of our knowledge, there are only two large-scale coreference-annotated datasets where syntax is closely linked to coreference relations. in ancora-co @xcite , coreferring mentions are nodes in constituency trees, and in the prague dependency corpora @xcite @xcite , coreference relations are annotated directly between syntactic heads in dependency trees and mention spans are implicitly defined as subtrees of the heads. finkel and @xcite deal with issues similar to our work and have developed a model that performs named entity recognition and constituency parsing consistently, guaranteeing that named entity spans do not conflict with the phrasal spans of the parse tree. as for coreference resolution systems, some earlier algorithms took syntactic information into account, see e.g. hobb's naive approaches to pronoun resolution @xcite , carter's shallow processing approach @xcite or fully symbolic lappin and leass' algorithms for resolving third person pronouns and traversing syntactic trees @xcite . morpho-syntactic features were later largely used in statistical approaches (e.g., @xcite @xcite , especially for morphologically rich languages (e.g, @xcite . with the advent of neural networks and contextual embeddings for coreference resolution (e.g., @xcite @xcite , the explicit treatment of morpho-syntax has practically vanished, even for the related task of mention detection. such models are able to encode syntactic aspects implicitly, as shown by e.g., @xcite and @xcite . ## data selection we draw our empirical observations about correspondences between manually annotated mention spans and manually or automatically produced dependency trees from corefud 0.1 @xcite , the biggest collection of coreference datasets converted to a harmonized scheme. corefud 0.1 contains 17 datasets for 11 languages. we excluded corefud datasets that are much smaller than the rest (english and german data originated in the parcorfull project @xcite ), and also datasets in which mentions and syntactic structures correspond by design. in the latter case, coreference annotations made use either of constituency trees -an english dataset from ontonotes @xcite , and spanish and catalan datasets from the ancora project @xcite ), or of dependency trees -a czech dataset from the prague dependency treebank (hajič et al., 2020), and english and czech datasets from the prague czech-english dependency treebank @xcite . the selection resulted in 9 datasets, for which we use their corefud labels: (1) english-gum: georgetown multilayer corpus @xcite (the only resource with hand-annotated dependency trees); (2) french-democrat: democrat (landragin, 2016); (3) german-potsdamcc: potsdam com-mentary corpus (bourgonje and stede, 2020); (4) hungarian-szegedkoref: szegedkoref: hungarian coreference corpus (vincze et al., 2018); (5) lithuanian-lcc: lithuanian coreference corpus (žitkus and butkienė, 2018); (6) polish-pcc: polish coreference corpus (ogrodniczuk et al., 2013); (7) russian-rucor: rucor: russian coreference corpus (toldova et al., 2014); (8) dutch-corea: coreference corpus and resolution system for dutch @xcite ; (9) english-arrau: the arrau corpus of anaphoric information @xcite ). foot_0 ## matches and mismatches between mention spans and dependency trees we took a random sample of sentences from the data described in section 3 and manually examined instances where syntactic and coreference annotations do not match. the annotation work took about 5-8 hours per 100 examples, depending on the language. in cases where the language was understood by none of this work's authors (hungarian, lithuanian, dutch), we took advantage of public machine-translation services during the annotation process. in this section, we first describe how we understand matching coreference and syntax; then we discuss individual types of mismatches and their possible causes. ## evaluation with the exception of english-gum, in all other datasets considered in this study the syntactic annotation has been assigned by an automatic parser. certain amount of parsing errors is thus to be expected and these errors may or may not cause the syntactic structure to diverge from the annotated mention spans. sometimes the mismatches revealed errors in mention span annotation. such cases are less frequent than parsing errors, yet they are still considerable in some datasets (lithuanian-lcc, german-potsdamcc, english-gum), and the syntactic annotation is helpful in identifying them. both missing and extra words have been observed. note that this is not about by-design inclusion/omission of certain modifiers such as prepositions -these would affect the subtree condition but not the catena condition. our annotation focused only on non-catena mentions, thus the percentage of wrongspan and wrongtree reported in we have investigated the correspondence of syntactic and coreferential annotation in a subset of corefud, a harmonized collection of coreference datasets with ud-style syntax. based on a manually annotated sample, we have shown that most mention spans (86% to 98%, depending on dataset) correspond to catenae. more than a half of the mismatches can be attributed to errors in the syntactic structure, in some datasets there is also a significant number of errors in the mention spans. an automatic confrontation of independently annotated syntax and coreference can thus be used to find and correct annotation errors on either side. a relatively small proportion of the mismatches is justified by the ud guidelines for specific syntactic constructions (coordinations and flat structures). even if our findings are intrinsic with respect to corefud, the observation that certain elements of syntactic and coreference annotations tend to be strongly correlated even if created independently, suggest that the syntactic viewpoint should be taken into account for annotation of coreference in order to increase its consistency. the decision whether a particular mismatch has been incurred by the parse tree or by coreference annotation must be based on the parser's quality and assumed coreference quality for individual resources. further inspection of the resources may help to design heuristics to make the decision automatically. otherwise, the corrections would have to be made manually. another natural future extension of this work is to look at mentions that are catenae but not com-6 it is possible that a catena mention contains a parsing error and in the correct tree, the span would not be catena anymore. similarly, there may be a catena mention with wrongly annotated span and the correct span would not form a catena in the tree. furthermore, both cases may coincide and after correcting both the tree and span, the mention may be catena again (as pointed out by a reviewer of this paper). we have not noticed any cases described in this footnote in the data, but for a proper investigation of these phenomena, we would need to annotate all mentions (including subtree mentions).