ez C oref: Towards Unifying Annotation Guidelines for Coreference Resolution (2023)

Abstract: Large-scale, high-quality corpora are critical for advancing research in coreference resolution. However, existing datasets vary in their definition of coreferences and have been collected via complex and lengthy guidelines that are curated for linguistic experts. These concerns have sparked a growing interest among researchers to curate a unified set of guidelines suitable for annotators with various backgrounds. In this work, we develop a crowdsourcing-friendly coreference annotation methodology, ezCoref, consisting of an annotation tool and an interactive tutorial. We use ezCoref to re-annotate 240 passages from seven existing English coreference datasets (spanning fiction, news, and multiple other domains) while teaching annotators only cases that are treated similarly across these datasets. Surprisingly, we find that reasonable quality annotations were already achievable (90% agreement between the crowd and expert annotations) even without extensive training. On carefully analyzing the remaining disagreements, we identify the presence of linguistic cases that our annotators unanimously agree upon but lack unified treatments (e.g., generic pronouns, appositives) in existing datasets. We propose the research community should revisit these phenomena when curating future unified annotation guidelines.

## introduction coreference resolution is the task of identifying and clustering together all textual expressions (mentions) that refer to the same discourse entity in a given document. impressive progress has been made in developing coreference systems @xcite @xcite , enabled by datasets annotated by experts @xcite @xcite and crowdsourcing @xcite . however, these datasets vary widely in ontonotes: maybe we need a [cia] version of the miranda warning: you have the right to conceal your coup intentions, because we may rat on you. arrau: maybe [we]e1 need [a [cia] version of [the miranda warning]]: [you]e4 have [the right to conceal [[your]e5 [coup] intentions]], because [we]e6 may rat on [you]e7. crowd (this work): maybe [we]e1 need [a [cia] version of [the [miranda] warning]]: [you]e3 have [the right] to conceal [[your]e3 coup intentions], because [we]e1may rat on [you]e3. figure 1: we visualize a common sentence from news domain annotated by two expert-curated datasets, ontonotes @xcite and arrau @xcite , along with the crowd annotations collected via our ezcoref platform. ontonotes does not mark generic pronouns. arrau does not consider them as coreferent and annotates them using a special relation "undef-reference" (markables with vague interpretations). on the contrary, our crowdworkers assign all mentions of the generic pronoun "you" to the same coreference chain. the situation is also similar for the generic "we." it is thus unclear which guidelines one should employ when collecting coreference annotations in a new domain or language. traditionally, existing guidelines have leaned towards lengthy explanations of complex linguistic concepts, such as those in the ontonotes guidelines @xcite , which detail what should and should not be coreferent (e.g., how to deal with headsharing noun phrases, premodifiers, and generic mentions). as a result, coreference datasets have traditionally been annotated by linguists (experts) already familiar with such concepts, which makes the process expensive and time-consuming. crowd-sourced coreference data collection has the potential to be significantly cheaper and faster; however, teaching an exhaustive set of linguistic guidelines to non-expert crowd workers remains a formidable challenge. as a result, there has been a growing interest among researchers in curating a unified set of guidelines @xcite suitable for annotators with various backgrounds. more recently, games-with-a-purpose (gwaps) @xcite were proposed to aid crowdsourcing of large coreference datasets (e.g., @xcite . while gwaps make it enjoyable for crowdworkers to learn complex guidelines and perform annotations using them @xcite , they also require significant effort to attract and maintain workers. for instance, phrase detectives corpus 1.0 was collected over a span of six years @xcite @xcite , which motivates us to instead study coreference collection on more efficient payment-based platforms. specifically, our work investigates the quality of crowdsourced coreference annotations when annotators are taught only simple coreference cases that are treated uniformly across existing datasets (e.g., pronouns). by providing only these simple cases, we are able to teach the annotators the concept of coreference, while allowing them to freely interpret cases treated differently across the existing datasets. this setup allows us to identify cases where our annotators unanimously agree with each other but disagree with the expert, thus suggesting cases that should be revisited by the research community when curating future guidelines. our main contributions are: coreference annotation tools: several coreference annotation tools have been developed (see table a3 in appendix for more details). however, these are difficult to port to a crowdsourced workflow, as they require users to install software on their local machine @xcite @xcite @xcite , or have complicated ui design with multiple drag and drop actions and/or multiple windows @xcite @xcite @xcite @xcite . closest to ezcoref is corefi @xcite , a web-based coreference annotation tool that can be embedded into crowdsourcing websites. subjectively, we found its user interface difficult to use (e.g., users have to memorize multiple key combinations). it also does not allow for nested spans, reducing its usability. crowdsourcing linguistic annotations: several efforts have been made to crowdsource linguistic annotations @xcite @xcite , including on payment-based microtasks via platforms like @xcite . many gwaps @xcite @xcite have been used in nlp to collect linguistic annotations including coreferences; with some broader platforms @xcite aiming to gamify the entire text annotation pipeline. one solution to teaching crowd workers complex guidelines is to incorporate learning by progression @xcite @xcite , where annotators start with simpler tasks and gradually move towards more complex problems, but this requires subjective judgments of task difficulty. in contrast to the payment-based microtask setting studied in this work, gwaps are not open-sourced, need significant development, take longer to collect data, and require continuous efforts to maintain visibility @xcite . ## ezcoref: a crowdsourced coreference annotation platform the ezcoref user experience consists of (1) a stepby-step interactive tutorial and (2) an annotation interface, which are part of a pipeline including automatic mention detection and amt integration. annotation structure: two annotation approaches are prominent in the literature: (1) a local pairwise approach, annotators are shown a pair of mentions and asked whether they refer to the same entity @xcite @xcite , which is time-consuming; or (2) a cluster-based approach @xcite @xcite , in which annotators group all mentions of the same entity into a single cluster. in ezcoref we use the latter approach, which can be faster but requires the ui to support more complex actions for creating and editing cluster structures. example phenomena taught [john] doesn't like [fred], but [he] still invited [him]to [the party]. (1) personal pronouns (2) singletons [this dog] likes to play [catch].[it]'s better than other [dogs] at [this game]. [[its] owner] is really proud. (1) possessive pronouns (2) semantically similar expression which are not coreferring (3) non-person entities (animals) [director [mackenzie]] spent [last two years] working on a ["young adam"]. during [this time] [he] often had to make [compromises] but [the movie] turned out to exceed expectations. (1) nested spans (2) non-person entities (time, item) [the office] wasn't exactly small either. [i]'m sure that 50, or maybe even 60, [people] could easily fit [there]. (1) non-person entities (place) user interface: we spent two years iteratively designing, implementing, and user testing the interface to make it as simple and crowdsourcingfriendly as possible (figure 2 ). 4 marked mentions are surrounded by color-coded frames with entity ids. the currently selected mention ("the book"), is highlighted with a flashing yellow cursor-like box. the core annotation action is to select other mentions that corefer with the current mention, and then advance to a later unassigned mention; annotators can also re-assign a previously annotated mention to another cluster. advanced users can exclusively use keyboard shortcuts, undo and redo actions were added to allow error correction. finally, ezcoref provides a side panel showing mentions of the entity currently being annotated to spot mentions assigned to the wrong cluster. coreference tutorial: to teach crowdworkers the basic definition of coreference and familiarize them with the interface, we develop a tutorial (aimed to take ∼ 20 minutes) that introduces them to the mechanics of the annotation tool, and then trains them on simple cases of coreferences. these cases (e.g., personal/possessive pronouns or determinative phrases which corefer with their antecedents as shown in annotation workflow: the annotators are presented with one passage (or "document") at a time (figure 2 ), and all mentions have to be annotated before proceeding to the next passage. there is no limitation to the length or language of the passage. in this work, we divide an initial document into a sequence of shorter passages of complete sentences, on average 175 tokens, as shorter passages minimize the need to scroll, reducing annotator effort. while this obviously cannot capture longer distance coreference, foot_4 a large portion of important coreference phenomena is local: within the ontonotes written genres, for pronominal mentions, the closest antecedent is contained within the current or previous two sentences more than 95% of the time. ## using ezcoref to re-annotate existing coreference datasets we deploy ezcoref on the amt crowdsourcing platform to re-annotate 240 passages from seven existing datasets, covering seven unique domains. in total, we collect annotations for 12,200 mentions and 42,108 tokens. we compare our workers' an-notations both quantitatively and qualitatively to each other and to existing expert annotations. datasets: we collect coreference annotations for the seven existing datasets described in from each domain in each dataset, we then select documents and divide them into shorter passages (on average 175 tokens each), creating 20 such passages per dataset. for datasets with multiple domains, we choose 20 such passages per domain (see appendix a.1 for detail). overall, we collect annotations for 240 passages with 5 annotations per passage to measure inter-annotator agreement. procedure: we first launch an annotation tutorial and recruit the annotators on the amt platform. 9 at the end of the tutorial, each annotator is asked to annotate a short passage (around 150 words). only annotators with a b3 score @xcite ) of 0.90 or higher are then invited to participate in the annotation task. training annotators with simplified guidelines using ezcoref: as the goal of our study is to understand what crowdworkers perceive as coreference, we train our annotators with simple guidelines. we carefully draft our training examples to include only cases which are considered as coreference by all the existing datasets. the objective is to 8 the preco dataset is interestingly large but seems difficult to access. in november 2018 and october 2021 we filled out the data request form at the url provided by the paper, and attempted to contact the preco official email directly, but did not receive a response. to enable a precise research comparison, we scraped all documents from preco's public demo in november 2018 (no longer available as of 2021); its statistics match their paper and our experiments use this version of the data. preco further suffers from data curation issues @xcite ; it uses text from english reading comprehension tests collected from several websites, but the original document sources and copyright statuses are undocumented. when reading through preco documents, we found many domains including opinion, fiction, biographies, and news (table a1 in appendix); we use our manual categories for domain analysis. 9 we allow only workers with @xmath2,000 approved tasks who are from the us, canada, australia, new zealand, or the uk. teach crowdworkers the broad definition of coreference while leaving space for different interpretations of ambiguous cases or those resolved differently across the existing datasets. note that a comparable experiment with more complex guidelines is infeasible since it is unclear which guidelines to choose, and also providing complex linguistic guidelines to crowdworkers remains an open challenge. overall, ezcoref is aimed to minimize both researcher and annotator effort for new coreference data collection, compared to prior work (figure 3 ). worker details: overall, 73 annotators (including 44 males, 20 females, and one non-binary person) foot_6 completed the tutorial task, which took 19.4 minutes on average (@xmath3.2 minutes). they were aged between 21 and 69 years (@xmath4.9, @xmath5.3) and identified themselves as native english speakers. most of the annotators had at least a college degree (47 vs 18). 89.0% of annotators, who did the tutorial, received a b3 score of 0.90 or higher for the final screening example, and were invited to the annotation task. 50.7% of the invited annotators returned to participate in the main annotation task, and 29.2% of them annotated five or more passages. annotation of one passage took, on average, 4.15 minutes, a rate of 2530 tokens per hour. the total cost of the tutorial was @xmath04.50 per tutorial). we paid @xmath11440. 11 ## analysis in this section, we perform quantitative and qualitative analyses of our crowdsourced coreference annotations. first, we evaluate the performance of our mention detection algorithm, comparing it to gold mentions across seven datasets. next, we measure the quality of our annotations and their agreement with other datasets. finally, we discuss interesting qualitative results. ## conclusion existing coreference datasets vary in their definition of coreferences and have been collected via complex guidelines. in this work, we investigate the quality of annotations when crowdworkers are taught only few coreference cases that are treated similarly across existing datasets. we develop a crowdsourcing-friendly coreference annotation methodology, ezcoref and use it to re-annotate 240 passages from seven existing english coreference datasets. we observe reasonable quality annotations were already achievable even without extensive training. on analyzing the remaining disagreements, we identify linguistic cases that crowd unanimously agree upon but lack unified treatments in existing datasets, suggesting cases the researchers should revisit when curating future unified annotation guidelines.