CoReﬁ: A Crowd Sourcing Suite for Coreference Annotation
Abstract
Coreference annotation is an important, yet expensive and time consuming, task, which often involved expert annotators trained on complex decision guidelines.
To enable cheaper and more efﬁcient annotation, we present C O R EFI , a webbased coreference annotation suite, oriented for crowdsourcing.
Beyond the core coreference annotation tool, C O R EFI provides guided onboarding for the task as well as a novel algorithm for a reviewing phase.
C O R EFI is open source and directly embeds into any website, including popular crowdsourcing platforms
Introduction
Coreference resolution is the task of clustering textual expressions ( mentions ) that refer to the same concept in a described scenario.
This challenging task has been mostly investigated within a single document scope, seeing great research progress in recent years.
The rather under-explored cross-document coreference setting is even more challenging.
For example, consider the following sentences originating in two different documents in the standard cross-document coreference dataset
ECB+ (1):
A man suspected of shooting three people at an accounting ﬁrm where he had worked ... 2.
A gunman shot three people at a suburban Detroit ofﬁce building Monday morning. Recognizing
that both sentences refer to
the same event (“shooting”,“shot” at the same location (“accounting ﬁrm”, “Detroit ofﬁce”) can be very useful for downstream tasks, particularly across documents, such as multi-
document
summarization (2; 3) or multi-hop question answering (4; 5).
High-quality annotated datasets are valuable to develop efﬁcient models.
While Ontonotes (6) provides a useful dataset for generic single-document coreference resolution, large-scale datasets are lacking for cross-document coreference (1; 7; 8) or for targeted domains, such as medical (9).
Due to the complexity of the coreference task, existing datasets have been annotated mostly by linguistic experts, incurring high costs and limiting annotation scale.
Aiming to address the cost and scalability issues in coreference annotation, we present C O R EFI , an embeddable web-component tool suite that supports an end-to-end crowdsourcing process (Figure 1), while providing several contributions over earlier annotation tools (Section 4).
C O R EFI includes an automated onboarding training phase, familiarizing annotators with the tool functionality and the task decision guidelines.
Then, actual annotation is performed through a simple-to-use and efﬁcient interface, providing quick keyboard operations.
Notably, C O R EFI provides a reviewing mode, by which an additional annotator reviews and improves the output of an earlier annotation.
This mode is enabled by a non-trivial algorithm, that seamlessly integrates reviewing of an earlier annotation into the progressive construction of the reviewer’s annotation.
By open sourcing C O R EFI , we hope to facilitate the creation of large-scale coreference datasets, especially for the cross-document setting, at modest cost while maintaining quality.
The C O R EFI Annotation Tool
C O R EFI provides a suite for annotating single and cross-document coreference, designed to embed into crowdsourcing environments.
Since coreference annotation is an involved and complex task, we target a controlled crowdsourcing setup, as proposed by Roit et al.
This setup consists of selecting designated promising crowd workers, identiﬁed in preliminary trap-tasks, and then quickly training them for the target task and testing their performance.
This yields a pool of reliable lightly trained annotators, who perform the actual annotation of the dataset.
C O R EFI supports both the annotator training (onboarding) and annotation production phases, as illustrated in Figure 1.
The training phase (Section 2.4) consists of two crowdsourcing tasks, ﬁrst teaching the tool’s functionality and then practicing guided annotation, interactively learning basics of the annotation guidelines.
The annotation production phase also consists of two crowdsourcing tasks: ﬁrst-round coreference annotation, providing a user-friendly interface designed to reduce annotation time (Section 2.2), and a reviewing task, in which an additional annotator reviews and improves the initial annotation (Section 2.3).
2.1 Design Choices
Our ﬁrst major design choice regards the annotation ﬂow.
As elaborated in Section 4, two different coreference annotation ﬂows were prominent in prior work.
The local pair-based approach aims at annotation simplicity, often motivated by a crowdsourcing setting.
Here, an annotator has to decide for a pair of mentions whether they corefer or not, or to proactively ﬁnd such pairs of corefering mentions.
Since coreference is annotated at the level mention pairs, it might require, in the worst case, comparing a mention to all other mentions in the text.
In the cluster-based ﬂow, annotators assign mentions to coreference clusters .
Here, a mention needs to be compared only against the clusters accumulated so far, or otherwise be deﬁned as starting a new cluster.
Indeed, the number of coreference clusters is often substantially lower than the number of mentions, particularly in the cross-document setting, where the same content gets repeated across the multiple texts.
For example, in the most popular dataset for cross-document coreference, ECB+ (1), the number of clusters is about one third of the number of mentions (15122 mentions split intocoreference clusters, including singletons).
In C O R EFI , we adopt the cluster-based approach since we aim at exhaustive coreference annotation across documents, whose complexity would become too high under the pairwise approach.
At the same time, we simplify the annotation process and functionality, making it crowdsourceable.
Our second design choice regards detecting referring mentions in text.
As elaborated in Section 4, coreference annotation tools, particularly cluster-based (e.g. (11; 12)), often require annotators to ﬁrst detect the target mentions before annotating them for coreference.
Conversely, recent local pair-based decision tools (13; 14) delegate mention extraction to a preprocessing phase, presenting coreference annotators with pre-determined mentions.
This simpliﬁes the task and allows annotators to focus their attention on the coreference decisions.
As we target exhaustive crowdsourced coreference annotation, we chose to follow this recent facilitating approach.
In addition to the input texts, C O R EFI takes as input an annotation of the targeted mentions, while optionally allowing annotators to ﬁx this mention annotation.
In our tool suite, we followed the approach of Prodigy, 1 where corpus developers may implement their own automated (non-overlapping) mention extraction recipes, or use a separate manual annotation tool for mention annotation, according to their desired mention detection guidelines (which often vary across projects).
The resulting mentions can then be fed into C O R EFI for coreference annotation.
We provide an example mention extraction recipe that detects as mentions common nouns, proper nouns, pronouns, and verbs (for event coreference).
Such mention detection is consistent with approaches that consider reduced mention spans, mostly pertaining to syntactic heads or named entity spans (15).
2.2 Annotation
Figure 2 shows the annotation interface of C O R EFI .
As initialization, the ﬁrst candidate mention is automatically assigned to the ﬁrst coreference cluster, which is placed in the “cluster bank”, appearing at the bottom of the screen ((3) in Figure 2).
In this bank, each cluster is labeled by the text of its ﬁrst mention.
The annotator is then shown the subsequent mentions, one at a time, with the current mention to assign underlined in purple (2).
For each mention, the annotator decides whether to accept it as a valid mention (doing nothing) or to modify its span (easily highlighting the correct span and pressing the ‘F’ key, for “Fix”).
Similarly, the annotator may introduce a new span, missing from the input.
To simplify annotation, the tool allows only non-overlapping spans.
The annotator then makes a coreference decision, by assigning the current mention to a new or existing cluster.
An existing cluster can be rapidly selected either by selecting it in the cluster bank or by selecting one of its previously-assigned mentions in the text.
Once a cluster is selected, it is highlighted in blue along with all its previous text mentions ((3) and (1) in the ﬁgure).
Rather than assigning mentions to clusters through a slower drag and drop interface (11; 12) or buttons (16; 17), annotation is driven primarily by faster keyboard operations, such as S PACE (assign to an existing cluster) and C TRL +S PACE (new cluster), with quick navigation through arrow keys and mouse clicks.
At any point, the annotator can re-assign a previously assigned mention to another cluster or view any cluster mentions.
C O R EFI supports an unlimited number of documents to be annotated, presented sequentially in a conﬁgurable order.
Finally, C O R EFI guarantees exhaustive annotation by allowing task submission only once all candidate mentions are processed.
2.3 Reviewing
To promote annotation quality, annotation projects typically rely on multiple annotations per item.
One approach for doing that involves collecting such annotations in parallel and then merging them in some way, such as simple or sophisticated voting (18).
Another approach is sequential, where one or more annotations are collected initially, and are then manually consolidated by an additional, possibly more reliable, annotator (a “consolidator” or “reviewer”) (10).
In our case, coreference annotation is addressed as a global clustering task, where an annotator generates a complete clustering conﬁguration for the input text(s).
Automatically merging such multiple clustering conﬁgurations, where cluster assignments are mutually dependent, might become unreliable.
Therefore, in C O R EFI we follow the sequential manual reviewing approach.
To that end, we introduce a reviewing task, which receives as input a previously annotated clustering conﬁguration and allows an additional annotator to review and improve it.
The reviewing task follows the same ﬂow of the annotation task, making it trivial to learn for annotators that already experienced with C O R EFI annotation.
At each step, the reviewer is presented with the next mention in the reviewed conﬁguration, and may ﬁrst decide to modify its span.
Next, the reviewer has to decide on cluster assignment for the current mention.
The only difference at this point is that the reviewer is presented with candidate cluster assignments which reﬂect the original annotator assignment (as explained below), displayed just above the cluster bank (Figure 3).
In fact, it is not trivial to reﬂect the cluster assignment by the original annotator to the reviewer, since that assignment has to be mapped to the current clustering conﬁguration of the reviewer.
Ambiguity may arise, resulting in multiple candidate clusters, since an early cluster modiﬁcation by the reviewer can impact the interpretation of downstream cluster assignments in the original annotation.
To illustrate this issue, consider reviewing a cluster assigned by the original annotator, consisting of three mentions, { A, B, C } .
When presented with the mention A , the reviewer agrees that it starts a new cluster.
Then, when reaching B , the reviewer is presented with { A } as B ’s original cluster assignment.
Suppose the reviewer disagrees with the annotator that A and B corefer and decides to assign B to a new cluster.
Now, when reviewing the mention C , it is no longer clear whether to attribute C ’s original assignment to { A } or { B } .
Hence, the reviewing tool presents both { A } and { B } as candidate clusters that reﬂect the original annotator’s assignment.
The reviewer may then choose either of them, or override the original annotation altogether and make a different assignment.
Similar ambiguities arise when the reviewer splits an original mention span and assigns its parts to separate clusters.
To address this challenge, we formulate an algorithm that maps an original cluster assignment to a set of candidate clusters in the current clustering conﬁguration of the reviewer.
Generally speaking, the algorithm considers the cluster to which the current mention was assigned in the original annotation, and tracks all earlier token positions in that cluster.
These token positions are then mapped back to clusters in the current reviewer’s clustering conﬁguration, which become the candidate clusters presented to the reviewer.
The algorithm pseudocode is presented in Appendix A, along with a comprehensive example of its application.
2.4 Onboarding
In the proposed controlled crowdsourcing scheme of (10), annotators were trained in an “ofﬂine” manner, reading slides and receiving individual feedback.
We propose augmenting this phase with automated training, delivered through two crowdsourcing tasks, described below.
2.4.1 Walk-through Tutorial
During this task, a trainee is walked through the core concepts and functionality of C O R EFI , such as the “current mention” and the “cluster bank” (Figure 2), and through the annotation operations.
These functionalities are presented through a series of intuitive dialogues.
To ensure that each feature is correctly understood, the user is instructed to actively perform each operation before continuing to the next (see Appendix B).
2.4.2 Guided Annotation
After acclimating with C O R EFI ’s features, users are familiarized with the coreference decision guidelines through a guided annotation task, practicing annotation while receiving automated guiding feedback.
If an annotation error is made, the trainee is notiﬁed with a pre-prepared custom response, which guides to the correct decision before allowing to proceed.
Additionally, following certain decisions, speciﬁc important guidelines can be communicated (see Appendix C for examples).
The content of the guided annotation task and the automated responses are easily conﬁgurable using a simple JSON conﬁguration schema.
This allows tailoring them when applying C O R EFI for different datasets and annotation guidelines.
Augmenting the controlled crowdsourcing scheme (10) with automated training provides key beneﬁts.
First, since feedback is automated, the amount of required personalized manual feedback is reduced.
Second, annotators beneﬁt from an immediate response for each decision, allowing them to understand their mistakes earlier and improve in real time.
We suggest that, for optimal learning of annotation guidelines, these beneﬁts should be coupled with the additional training means of controlled crowdsourcing.
These include the provision of guideline slides, for learning and later for reference, and some personalized manual feedback during the training phase.
2.5 Implementation Beneﬁts
C O R EFI was developed using the web component standard and the V UE .
JS framework.
2 allowing, it to easily embed into any website, including crowdsourcing platforms.
Additionally, C O R EFI provides output in the standard CoNLL coreference annotation format, enabling training state of the art models and scoring with the ofﬁcial coreference scorer (19).
All C O R EFI features are easily conﬁgurable with HTML encoded JSON and support any UTF-8 encoded language.
Pilot Study
To further assess C O R EFI ’s effectiveness in a crowdsourcing environment, we performed a small-scale trial on Amazon Mechanical Turk, employing 5 annotators, focusing on the coreference annotation functionality (rather than mention validation).
To allow objective assessment of annotation quality, we experimented with replicating coreference annotations from the ECB+ dataset (1), the commonly used dataset for cross-document coreference over English news articles (20; 21; 22; 23; 24; 25).
Accordingly, we considered the ECB+ gold mentions as input, requesting crowdworkers to assign them to coreference clusters.
Focusing on the controlled crowdsourcing setting, we hired ﬁve annotators that were previously selected for annotation by (10).
On-boarding Annotators were given C O R
EFI ’s walk-through tutorial and guided annotation tasks, adapted to the ECB+ guidelines and applied to a part of an ECB+ subtopic (cluster of documents).
These two tasks took altogether 11 minutes on average to complete, at a rate of $1.5 a task.
Next, workers were asked to annotate an entire ECB+ subtopic (through the actual annotation task). We
[Table] [Table] provided them manual feedback for their mistakes, which consumed 30-40 minutes of researcher time per trained annotator.
Annotation After training, we paid workers $8 to annotate two additional subtopics in full (of about 150 and 200 mentions; in ECB+ only a few sentences are annotated per document, and these were presented for annotation).
Each subtopic took 27 minutes on average to annotate, corresponding to an annotation rate of ~400 mentions per hour.
Table 1 presents the performance (F1) of each of the annotators, compared to the ECB+ gold annotations, averaged over the two subtopics.
The results are reported using the common evaluation metrics for coreference resolution: MUC (26), B 3 (27), CEAFe (28), and CoNLL — the average of the three metrics.
Considering the decision volume and complexity, as well as the limited training (not providing guideline slides and a single practice round), we ﬁnd that these results support C O R EFI ’s effectiveness for crowdsourcing.
As previously mentioned, we expect that annotation quality may be further improved in an actual dataset creation project, by providing additional guided tasks, annotation guidelines slides, and additional manual feedback.
Reviewing For the reviewing trial, the two best annotators, A2 and A3, were selected as R1 and
R2.
Two additional annotators (A1 and A5) were each assigned a new unique subtopic, which was then reviewed by both R1 and R2.
Table 2 presents the reviewing results, showing consistent improvements after reviewing and assessing the ease of using the reviewing functionality.
Related work
As mentioned in Section 2, prior tools for coreference annotation are based on two prominent workﬂows: pair-based, treating coreference as a pairwise annotation decision, and cluster-based, in which mentions are assigned to clusters.
While targeting simplicity, only two pair-based tools supported crowdsourcing annotation, yet they were not applied for producing exhaustively annotated daasets: Phrase Detective (13), which was employed in a web-based game setting, and (14), which was applied in an active learning environment.
Pair-based tools differ in their annotation approaches.
In certain tools, such as BRAT (29), Glozz (30), Analec (31), and MMAX2 (32), the annotator ﬁrst determines mention span boundaries and then links a pair of mentions.
Other Pair-based tools (13; 14) either provide annotators a single (pre-determined) mention, asking to ﬁnd a coreferring antecedent, or provide a pair of mentions, asking to judge whether the two corefer.
Notably, pair-based tools are less effective for exhaustive coreference annotation, for two reasons.
First, they require comparing each mention to all other mentions , rather than to already constructed clusters.
Second, local pairwise decisions lack awareness of previous cluster assignments, which might hurt annotation quality.
Cluster-based tools, including Cromer (16), Model based annotation tool (17), CorefAnnotator (11), and SACR (12), ask annotators to ﬁrst detect mention spans and then cluster them, thus complicating the overall task without allowing the delegation of mention detection to a preprocessing phase.
Such a method does not guarantee exhaustive annotation, since annotators may miss some mentions.
With respect to operation efﬁciency, mentions are often linked to clusters via somewhat slow operations, such as drag-and-drop or selection from a drop-down list, in comparison to the fast keyboard operations in C O R EFI .
Notably, to the best of our knowledge, C O R EFI is the ﬁrst cluster-based crowdsourcing tool that provides an end-to-end annotation suite, including automated onboarding tasks and exhaustive reviewing, the latter enabled by our novel reviewing algorithm.
Furthermore, it is the ﬁrst tool that was developed using the WebComponent standard, embeddable in any website.
Conclusion
In this paper, we aim to facilitate crowdsourced creation of needed large-scale coreference datasets, in both the within- and the cross-document setting.
Our comprehensive end-to-end tool suite,
C O R EFI , enables high quality and fairly cheap crowdsourcing of exhaustive coreference annotation in various domains and languages.
Our experiments demonstrate that C O R EFI ’s automatic onboarding is effective at augmenting Roit et al.(10) ’s controlled crowdsourcing.
C O R EFI provides the ﬁrst reviewing algorithm and implementation for cluster-based coreference annotation.
Overall, we demonstrated that non-expert annotators can be trained to effectively perform and review coreference annotations, allowing for cost-effective annotation efforts.
Acknowledgments
The work described herein was supported in part by grants from Intel Labs, Facebook, the Israel Science Foundation grant 1951/17, the Israeli Ministry of Science and Technology and the German Research Foundation through the German-Israeli Project Cooperation (DIP, grant DA 1600/1-1).
In addition to the support above, we would like to thank Uri Fried, Ayal Klein, Paul Roit, Amir
Cohen, Sharon Oren, Chris Noring, Asaf Amrami, Ori Shapira, Daniela Stepanov, Ori Ernst, Yehudit
Meged, Valentina Pyatkin, Moshe Uzan and Ofer Sabo for their support with architecture, design and crowdsourcing.
A Reviewing Algorithm
Algorithm 1 implements the mechanism to ﬁnd potential clusters given the initial annotation and previous reviewing modiﬁcations.
To support span modiﬁcation, we build two main data structures at the token level (lines 1 and 2).
Given the original annotation M , we build a static mapping Ant (line 1), where each single token is associated with all tokens from previous mentions that belong to the same coreference cluster.
T 2 C is a growing mapping that will keep track of the reviewer decisions.
After the initialization phase, the reviewer is shown all the annotator mentions in a sequential order.
For each presented mention, the reviewer ﬁrst decides whether to agree or to modify the mention span boundaries (line 5).
Future mentions in the stack M that are fully covered by the reviewed span need to be removed (lines 6-8).
The reviewer may also split the current mention or partially cover next mentions (line 9-11).
In order to ﬁnd the potential coreference clusters (line 13), we ﬁrst use Ant to retrieve the antecedent tokens in the original annotation, for each single token in the reviewed span Sp (cid:48) .
Then, we use the reviewer mapping T 2 C for each antecedent tokens to identify the possible cluster(s) that will be displayed to the reviewer (Figure 3).
Given the coreference decision (assigning to an existing cluster or to a new one) of the reviewer (line 14), we update the reviewer mapping T 2 C (line 15) and coreference assignments (line 16).
Table 3 illustrates the reviewing decision step by step, given an initial annotation that incorrectly assigned the following gold clustered mentions {{ Bank of America, bank, BoA } { American }} into one coreference cluster { Bank of America, American bank, BoA } .
Input
M of initial clustering assignment Output: R : Reviewed Assignment 1 Ant ← CreateAntecedentMapping M C
ID 3 while do // Set reviewer span Sp (cid:48) ← ReviewSpan( M top () ) while M.top () .end ≤ Sp (cid:48) .end
do M.pop () end if M.top () .start ≤ Sp (cid:48) .end then popSplitPush( M, Sp (cid:48) ) end //
Set reviewer cluster C ← getCandidates ( Sp (cid:48) , Ant, T 2 C ) cluster ← selectCluster ( C ) T 2 C.update ( Sp (cid:48) , cluster ) R.push ( Sp (cid:48) , cluster ) 17 end
MentionStack
AnnotatorCandidates
ReviewerDecision
Explanation
[Table]
[Table] [Table] [Table] } (cid:51)
TherevieweragreesthatBankofAmericaisthestartofanewcluster [-
-]2 [
[Table] [Table] [Table]
Split [Table]
Twomentionsarecreated, [Table] and [Table]
.The reviewerwillnextdeterminetheclusterassignmentofAmerican.
3 [ American bank, [Table] [Table]
Assign American toanewcluster
Thereviewerisshown {
[Table]
[Table] - didateclustersincethe [Table] wasassigned withBankofAmericabytheannotator.
4 [ bank , [Table] ] { BankofAmerica } , { [Table]
Assign [Table] [Table] Thereviewerisshownboth
[Table] } and [Table] American } ascandidateclustersforbank.
Now,the reviewerdecidestoassignbanktothe { [Table] - ica [Table]
[Table] B Tutorial Figure 4 and 5 demonstrate notiﬁcations that explains conceptual aspect of C O R EFI . Figure 4 explains what the current mention to assign is where as Figure 5 explains what clusters are and how to manage them in the cluster bank.
Figure 6 demonstrates a more interactive tutorial prompt. It shows how to make an active coreference decision with the keyboard and encourages the trainee to experiment with in the conﬁnes of the tutorial environment to familiarize themselves with the feature.
C Guided Annotation
Figure 7 demonstrates the guided experience of the on-boarding ﬂow of C O R EFI .
In Figure 7, the trainee is learning the nuances of coreference and makes the mistake of attempting to assign the mention name to the same cluster as another mention with the exact same expression.
However, in context the name event mention expressed by the current mention does not refer to the selected cluster.
The current mention refers to the naming of the Dr. Regina Benjamin as U.S Surgeon General where as the selected cluster refers to the event of naming Dr. Sanjay Gupta to Surgeon General.
Since, the correct decision is subtle the user receives a toast informing them that Words can have the same meaning but not corefer.
This toast helps to guide the annotator to the correct decision and reinforces the coreference guidelines.
As the trainee is familiarized with the subtleties of coreference they are less likely to make similar mistakes during annotation of the real dataset.