C o R efi: A Crowd Sourcing Suite for Coreference Annotation (2020)

Abstract: Coreference annotation is an important, yet expensive and time consuming, task, which often involved expert annotators trained on complex decision guidelines. To enable cheaper and more efficient annotation, we present CoRefi, a web-based coreference annotation suite, oriented for crowdsourcing. Beyond the core coreference annotation tool, CoRefi provides guided onboarding for the task as well as a novel algorithm for a reviewing phase. CoRefi is open source and directly embeds into any website, including popular crowdsourcing platforms. CoRefi Demo: aka.ms/corefi Video Tour: aka.ms/corefivideo Github Repo: https://github.com/aribornstein/corefi

## introduction coreference resolution is the task of clustering textual expressions (mentions) that refer to the same concept in a described scenario. this challenging task has been mostly investigated within a single document scope, seeing great research progress in recent years. the rather under-explored crossdocument coreference setting is even more challenging. for example, consider the following sentences originating in two different documents in the standard cross-document coreference dataset ecb+ @xcite : ## the corefi annotation tool corefi provides a suite for annotating single and cross-document coreference, designed to embed into crowdsourcing environments. since coreference annotation is an involved and complex task, we target a controlled crowdsourcing setup, as proposed by @xcite . this setup consists of selecting designated promising crowd workers, identified in preliminary trap-tasks, and then quickly training them for the target task and testing their performance. this yields a pool of reliable lightly trained annotators, who perform the actual annotation of the dataset. corefi supports both the annotator training (onboarding) and annotation production phases, as illustrated in figure 1 . the training phase (section 2.4) consists of two crowdsourcing tasks, first teaching the tool's functionality and then practicing guided annotation, interactively learning basics of the annotation guidelines. the annotation production phase also consists of two crowdsourcing tasks: first-round coreference annotation, providing a user-friendly interface designed to reduce annotation time (section 2.2), and a novel reviewing task, in which an additional annotator reviews and improves the initial annotation (section 2.3). ## pilot study to further assess corefi's effectiveness in a crowdsourcing environment, we performed a smallscale trial on amazon mechanical turk, employing 5 annotators, focusing on the coreference annotation functionality (rather than mention validation). to allow objective assessment of annotation quality, we experimented with replicating coreference annotations from the ecb+ dataset @xcite , the commonly used dataset for cross-document coreference over english news articles @xcite @xcite @xcite . accordingly, we considered the ecb+ gold mentions as input, requesting crowdworkers to assign them to coreference clusters. focusing on the controlled crowdsourcing setting, we hired five annotators that were previously selected for annotation by @xcite . on-boarding annotators were given corefi's walk-through tutorial and guided annotation tasks, 2 https://vuejs.org/ adapted to the ecb+ guidelines and applied to a part of an ecb+ subtopic (cluster of documents). these two tasks took altogether 11 minutes on average to complete, at a rate of @xmath08 to annotate two additional subtopics in full (of about 150 and 200 mentions; in ecb+ only a few sentences are annotated per document, and these were presented for annotation). each subtopic took 27 minutes on average to annotate, corresponding to an annotation rate of ~400 mentions per hour. reviewing for the reviewing trial, the two best annotators, a2 and a3, were selected as r1 and r2. two additional annotators (a1 and a5) were each assigned a new unique subtopic, which was then reviewed by both r1 and r2. ## related work as mentioned in section 2, prior tools for coreference annotation are based on two prominent workflows: pair-based, treating coreference as a pairwise annotation decision, and cluster-based, in which mentions are assigned to clusters. while targeting simplicity, only two pair-based tools supported crowdsourcing annotation, yet they were not applied for producing exhaustively annotated daasets: phrase detective @xcite , which was employed in a web-based game setting, and @xcite , which was applied in an active learning environment. pair-based tools differ in their annotation approaches. in certain tools, such as brat @xcite , glozz @xcite , analec @xcite , and mmax2 @xcite , the annotator first determines mention span boundaries and then links a pair of men-tions. other pair-based tools @xcite either provide annotators a single (pre-determined) mention, asking to find a coreferring antecedent, or provide a pair of mentions, asking to judge whether the two corefer. notably, pair-based tools are less effective for exhaustive coreference annotation, for two reasons. first, they require comparing each mention to all other mentions, rather than to already constructed clusters. second, local pairwise decisions lack awareness of previous cluster assignments, which might hurt annotation quality. cluster-based tools, including cromer @xcite , model based annotation tool (aralikatte and søgaard, 2020), corefannotator @xcite , and sacr @xcite , ask annotators to first detect mention spans and then cluster them, thus complicating the overall task without allowing the delegation of mention detection to a preprocessing phase. such a method does not guarantee exhaustive annotation, since annotators may miss some mentions. with respect to operation efficiency, mentions are often linked to clusters via somewhat slow operations, such as drag-and-drop or selection from a drop-down list, in comparison to the fast keyboard operations in corefi. notably, to the best of our knowledge, corefi is the first cluster-based crowdsourcing tool that provides an end-to-end annotation suite, including automated onboarding tasks and exhaustive reviewing, the latter enabled by our novel reviewing algorithm. furthermore, it is the first tool that was developed using the webcomponent standard, embeddable in any website. ## conclusion in this paper, we aim to facilitate crowdsourced creation of needed large-scale coreference datasets, in both the within-and the cross-document setting. our comprehensive end-to-end tool suite, corefi, enables high quality and fairly cheap crowdsourcing of exhaustive coreference annotation in various domains and languages. our experiments demonstrate that corefi's automatic onboarding is effective at augmenting @xcite 's controlled crowdsourcing. corefi provides the first reviewing algorithm and implementation for cluster-based coreference annotation. overall, we demonstrated that non-expert annotators can be trained to effectively perform and review coreference annotations, allowing for cost-effective annotation efforts.