Cross-Document Event Coreference Resolution: Instruct Humans or Instruct GPT ? (2023)

Abstract: This paper explores utilizing Large Language Models (LLMs) to perform Cross-Document Event Coreference Resolution (CDEC) annotations and evaluates how they fare against human annotators with different levels of training. Specifically, we formulate CDEC as a multi-category classification problem on pairs of events that are represented as decontextualized sentences, and compare the predictions of GPT-4 with the judgment of fully trained annotators and crowdworkers on the same data set. Our study indicates that GPT-4 with zero-shot learning outperformed crowd-workers by a large margin and exhibits a level of performance comparable to trained annotators. Upon closer analysis, GPT-4 also exhibits tendencies of being overly confident, and force annotation decisions even when such decisions are not warranted due to insufficient information. Our results have implications on how to perform complicated annotations such as CDEC in the age of LLMs, and show that the best way to acquire such annotations might be to combine the strengths of LLMs and trained human annotators in the annotation process, and using untrained or undertrained crowdworkers is no longer a viable option to acquire high-quality data to advance the state of the art for such problems.

## introduction cross-document event coreference resolution (cdec) is the task of identifying coreferent events from different documents. traditionally, cdec has been approached as a mention-pair classification problem, in which the goal is to determine if two event mentions refer to the same event based on their contextual information in their containing articles @xcite , followed by a clustering step where coreferent events are grouped into clusters. cdec is a challenging problem for both data annotation and computational modeling because, in order to determine whether two event mentions are coreferent with each other, their participants, time, and location would have to be the same or at least compatible, and such information would have to be found in the global context of the document or even outside of the document. another challenge for cdec annotation is that in a random collection of documents, coreferent event pairs are likely to be very sparse. in order to make a cdec annotation task feasible, it is necessary to pre-select the documents by their topics to increase the density of coreferring events, and to limit the length of the document to reduce the size of the context that needs to be examined. as a result, existing datasets such as ecb+ @xcite , a widely used benchmark for cdec, consists of relatively short articles and has limited coverage and diversity of event types. in this paper, we explore cdec data creation with regular sized news articles. we frame cdec as a multi-class classification problem on pairs of events represented as sentences containing event trigger words. to make the task feasible, we perform decontextualizaton on these sentences to make them self-contained with the necessary contextual information. the decontextualized event sentence pair are illustrated in figure 1 . we also pre-select candidate event pairs using a state-of-theart cdec model cdlm @xcite ) by maximizing the recall so that we don't unintentionally rule out coreferring event pairs. performing these steps allows us to present candidate event pairs to annotators who can determine if they are coreferent based on just the event pairs. given the impressive performance of gpt-based llms, several recent studies explored using gpts to create annotated datasets for text generation @xcite and social computing @xcite . to investigate how well @xcite doc 1: [...] despite @xmath0, it is not expected that the construction will resume in the near future. [...] decontextualized event sentence: despite @xmath1, it is not expected that the construction of the thirty meter telescope project will resume in the near future. doc 2: [...] in march, protesters who had been camped out on mauna kea to block construction of thirty meter telescope dismantled their large tents @xmath2. [...] decontextualized event sentence: in march, protesters, who who had been camped out on mauna kea mountain to block construction of thirty meter telescope dismantled their large tents @xmath3. performs against human annotators with different levels of training, we gave the same data set of event mention pairs to fully trained annotators, crowd-workers, and gpt-4, requiring them to provide nine-way decisions. to establish our ground truth, we conducted adjudication meetings with our trained annotators to resolve disagreements. subsequently, we calculated the accuracy of both human annotations and gpt-4 annotations against the ground truth. our findings reveal that gpt-4 outperformed crowd workers by a large margin and achieved a level of performance comparable to trained human annotators. upon closer examination, our results show that for both human annotators and gpt-4, performance varies across topics and coreference categories. they also show that gpt-4 exhibits a strong tendency to make inferences even when there is not sufficient contextual basis, and bridge gaps in understanding by resorting to hallucination. we believe that our work has implications for the creation of a complex and labor-intensive annotations such as cdec. the far superior performance of gpt-4 against untrained crowd workers means that there is little value in performing the cdec annotation in this setting. gpt-4 also has the potential to accelerate the annotation process by reducing the need for trained human annotators, resulting in significant time and cost savings. the most effective data creation process for com-plicated datasets in the age of llms might be one that combines strengths of llms and trained annotators. we leave it to future research exactly how that should be carried out. the rest of the paper is organized as follows. in §2, we discuss related work. in §3, we describe our data preparation process. we present and discuss our experimental results in §4, and conclude in §5. ## data preparation a number of critical steps need to be taken to prepare the data for annotation by human annotators and gpt-4, and they include source article selection and deduplication, event extraction, decontextualization, and pre-selection of candidate event pairs. a flow chart illustration of this pipeline can be found in figure 4 in the appendix a.3. ## https://platform.openai.com/docs what is the relation between the two marked events in the following sentence pair: 1. isabel dos santos has @xmath4. 2. the order said the central bank would ensure that no @xmath5. the relation has to be one of the following: identity, concept-instance, instance-concept, set-member, member-set, subevent-whole, whole-subevent, not-related, cannot-decide. provide an explanation ## conclusion to address scalability challenges in creating cross-document event coreference (cdec) datasets, we explored the feasibility of employing crowdsourcing and gpt-4 using a decontextualized representation of events. our findings indicate that gpt-4 outperforms crowd workers by a large margin and shows comparable performance to trained annotators. we also observe variations in performance across different topics and individual coreference categories and uncovered issues related to reasoning and hallucination in gpt-4's performance in the cdec annotation task. despite its limitations, our work suggests that gpt-4 has the potential to replace human effort in creating complex and labor intensive cdec datasets in at least some settings at scale. given the far superior performance of gpt-4 over crowd workers, it no longer makes sense to resort to untrained annotators in crowdsourcing settings for such complex annotation tasks. the best approach might be one that combines the strengths of llms such as gpt-4 with highly trained annotators. we leave it to future work as to exactly how that combination should work.