How Good Is the Model in Model-in-the-loop Event Coreference Resolution Annotation? (2023)

Abstract: Annotating cross-document event coreference links is a time-consuming and cognitively demanding task that can compromise annotation quality and efficiency. To address this, we propose a model-in-the-loop annotation approach for event coreference resolution, where a machine learning model suggests likely corefering event pairs only. We evaluate the effectiveness of this approach by first simulating the annotation process and then, using a novel annotator-centric Recall-Annotation effort trade-off metric, we compare the results of various underlying models and datasets. We finally present a method for obtaining 97% recall while substantially reducing the workload required by a fully manual annotation process.

## introduction event coreference resolution (ecr) is the task of identifying mentions of the same event either within or across documents. consider the following excerpts from three related documents: e 1 : 55 year old star will replace m 1 matt smith, who announced in june that he was leaving the sci-fi show. annotating ecr links can be challenging due to the large volume of mention pairs that must be compared. the annotating task becomes increasingly time-consuming as the number of events in the corpus increases. as a result, this task requires a lot of mental effort from the annotator and can lead to poor quality annotations @xcite . indeed, an annotator has to examine multiple documents simultaneously often relying on memory to identify all the links which can be an error-prone process. to reduce the cognitive burden of annotating ecr links, annotation tools can provide integrated model-in-the-loop for sampling likely coreferent mention pairs @xcite @xcite . these systems typically store a knowledge base (kb) of annotated documents and then use this kb to suggest relevant candidates. the annotator can then inspect the candidates and choose a coreferent event if present. the model's querying and ranking operations are typically driven by machine learning (ml) systems that are trained either actively @xcite @xcite or by using batches of annotations @xcite . while there have been advances in suggestion-based annotations, there is little to no work in evaluating the effectiveness of these systems, particularly in the use case of ecr. specifically, both the overall coverage, or recall, of the annotation process as well as the degree of annotator effort needed depend on the performance of the model. in order to address this shortcoming, we offer the following contributions: ## related work previous work for ecr is largely based on modeling the probability of coreference between mention pairs. these models are built on supervised classifiers trained using features extracted from the pairs. most recent work uses a transformer-based language model (lm) like bert @xcite to generate joint representations of mention pairs, a method known as cross-encoding. the cross-encoder is fine-tuned using a coreference scoring objective @xcite @xcite @xcite . these methods use scores generated from the scorer to then agglomeratively cluster coreferent events. over the years, a number of metrics have been proposed to evaluate ecr @xcite @xcite @xcite ). an ecr system is evaluated using these metrics to determine how effectively it can find event clusters (recall) and how cleanly separated the clusters are (precision). from the perspective of annotation, it may only be necessary to focus on the system's recall or its effectiveness in finding ecr links. however, an annotator might still want to know how much effort is required to identify these links in a corpus to estimate their budget. in the remainder of the paper, we attempt to answer this question by first quantifying annotation effort and analyzing its relation with recall of the system. we use the event coreference bank plus (ecb+; @xcite ) and the gun violence corpus (gvc; vossen et al. ( 2018 )) for our experiments. the ecb+ is a common choice for assessing ecr, as well as the experimental setup of @xcite and gold topic clustering of documents and gold mention annotations for both training and testing foot_1 . on the other hand, the gvc offers a more challenging set of exclusively event-specific coreference decisions that require resolving gun violence-related events. ## annotation methodology we implement an iterative model-in-the-loop methodology 3 for annotating ecr links in a corpus containing annotated event triggers. this approach has two main components -(1) the storage and retrieval of annotated event clusters, which are then compared with each new target event, and (2), an ml model that ranks and prunes the sampled candidate clusters by evaluating their semantic similarity to the target mention. as illustrated in figure 1 , our annotation workflow queries the annotated event store for the target event (m 1 ), retrieving three potential coreferring candidates (m 2 , m * , and m 4 ). the ranking module then evaluates these candidates based on their lexical and semantic similarities to m 1 . the annotator then compares each candidate to the target and determines if they are coreferent. upon finding a coreferent candidate, the target is merged into the coreferent cluster, and any remaining option(s) (m * ) are skipped. ## evaluation methodology we evaluate the performance of the model-in-theloop annotation with the ranking methods through simulation on two aspects: (1) how well it finds the coreferent links, and (2) how much effort it would take to annotate the links using the ranking method. ## conclusion we introduced a model-in-the-loop annotation method for annotating ecr links. we compared three ranking models through a novel evaluation methodology that answers key questions regarding the quality of the model in the annotation loop (namely, recall and effort). overall, our analysis demonstrates the viability of the models, with cdlm exhibiting the best performance on the ecb+ dataset, followed by bert and lemma. the choice of ranking method depends on the specific use case, dataset, and resource constraints, but all three methods offer valuable solutions for different scenarios.