Cross-document Event Identity via Dense Annotation (2021)

Abstract: In this paper, we study the identity of textual events from different documents. While the complex nature of event identity is previously studied (Hovy et al., 2013), the case of events across documents is unclear. Prior work on cross-document event coreference has two main drawbacks. First, they restrict the annotations to a limited set of event types. Second, they insufficiently tackle the concept of event identity. Such annotation setup reduces the pool of event mentions and prevents one from considering the possibility of quasi-identity relations. We propose a dense annotation approach for cross-document event coreference, comprising a rich source of event mentions and a dense annotation effort between related document pairs. To this end, we design a new annotation workflow with careful quality control and an easy-to-use annotation interface. In addition to the links, we further collect overlapping event contexts, including time, location, and participants, to shed some light on the relation between identity decisions and context. We present an open-access dataset for cross-document event coreference, CDEC-WN, collected from English Wikinews and open-source our annotation toolkit to encourage further research on cross-document tasks.

## introduction coreference resolution is the task of identifying events (or entities) that refer to the same underlying activity (or objects). accurately resolving coreference is a prerequisite for many nlp tasks, such as question answering, summarization, and dialogue understanding. for instance, to get a holistic view of an ongoing natural disaster, we need to aggregate information from various sources (newswire, social media, public communication, etc.) over an extended period. often this requires resolving 1 data and code are available at https://github.com/ adithya7/cdec-wikinews . @xcite nearly 200 people are confirmed dead and approximately 2600 are ill in a central haitian cholera outbreak. figure 1 : an illustration of the quasi-identity nature of events. the event [haitian cholera] 'outbreak' is expressed by instances with varying counts of infections and deaths. the identity of this event continuously evolves over space and time, attributed to a new type of quasi-identity, spatiotemporal continuity. coreference between mentions across documents. 2 recasens et al. @xcite defines coreference as "identity of reference". therefore, modeling event coreference requires understanding the extent of the shared identity between event mentions. numerous factors determine this identity, including the semantics of the event mention, its arguments, and the document context. resolving coreference across documents is more challenging, as it requires modeling identity over a much longer context. to this end, we identify two major issues with existing cross-document event coreference (cdec) datasets that limit the progress on this task. first, many prior datasets often annotate coreference only on a restricted set of event types, limiting the coverage of mentions in the dataset. second, many datasets and models insufficiently tackle the concept of event identity. as highlighted by @xcite , the decision of whether two mentions refer to the same event is often non-trivial. occasionally, event mentions only share a partial identity (quasiidentity). in this work, we present a new dataset for cdec that attempts to overcome both issues. earlier efforts on cdec dataset collection were limited to specific pre-defined event types, restricting the scope of event mentions that could be studied. in this work, we instead annotate mentions of all types, i.e., open-domain events @xcite , and provide a dense annotation @xcite by checking for coreference relationship between every mention pair in all underlying document pairs. we compile documents from the publicly available english wikinews. 3 to facilitate our goal of dense annotation of mentions and their coreference, we develop and release a new easy-to-use annotation tool that allows linking text spans across documents. we crowdsource coreference annotations on mechanical turk. 4 prior work has attributed the quasi-identity behavior of events to two specific phenomena, membership and subevent @xcite . however, its implications in cross-document settings remain unclear. in this work, we specifically focus on a cross-document setup. as highlighted by @xcite , a direct annotation of quasi-identity relations is hard because annotators might not be familiar with the phenomenon. therefore, we propose a new annotation workflow that allows for easy determination of quasi-identity links. to this end, we collect evidence for time, location, and participant(s) overlap between corefering mentions. we also collect information regarding any potential inclusion relationship between the mention pair. our workflow allowed us to empirically identify a new type of quasi-identity, spatiotemporal continuity, in addition to the existing types defined by @xcite . figure 1 illustrates this phenomenon using the case of [haitian cholera] outbreak. the event gradually evolves over space and time, leading to cases of partial coreference. additionally, traditional coreference annotations cluster mentions together. however, this methodology can be misleading when dealing with cases of quasi-identity (see §5). to overcome this limitation, we frame our annotation task as a (cross-document) mention pair linking. the proposed task simplifies the annotation process by avoiding merging quasi-identical mentions into a single cluster. to serve as a benchmark for future work, we provide two baselines, lemma-match, and a bert-based cross-encoder. in the upcoming sections, we first position our work within the existing cdec literature ( §2). we then describe our methodology for preparing the source corpus ( §3), and our crowdsourcing setup for collecting coreference annotations on this corpus ( §4). in §5, we present a study of quasi-identity of events in our dataset. finally, in §6, we present two baselines models for the proposed dataset. many other datasets have since been curated for the task of cdec. some related works include, meantime @xcite , event hoppers @xcite , gun violence corpus (gvc) @xcite , football coreference corpus (fcc) @xcite , and wikipedia event coreference (wec) @xcite . however, most cdec systems are still evaluated primarily on ecb+. additionally, all of these datasets do not account for the quasi-identity nature of events. though compiled from wikinews, cdec annotations in the meantime corpus were limited to events with participants from a pre-defined list of 44 seed entities. while the fcc corpus was also crowdsourced, the annotation unit was an entire sentence instead of a single event mention. wec corpus uses hyperlinks from wikipedia but primarily handles referential events. in this work, we use open-domain events and treat an event mention as our annotation unit. we collect coreference links across all the mention pairs from all the underlying document pairs. event identity: @xcite postulated entity coreference as a continuum, with identity, non-identity and near-identity relations. in a follow-up work @xcite , they identify near-identity relations using the disagreement between annotators. they say subjects are not fully aware of the near-identity behavior, therefore making direct annotation collection hard. the continuum idea has since extended to events @xcite . determining if two event mentions are identical is not a trivial decision. it depends on the arguments of the mentions (often underspecified in the local context), the semantics of the mention, and the document context. in this work, we are specifically interested in cross-document coreference. @xcite studied the impact of the subevent relationship on quasi-identity, but a more general annotation framework is missing. accurately capturing event identity is critical to cdec dataset construction and the subsequent modeling. therefore, we qualitatively study this phenomenon by collecting supplementary information with each coreference link. ## corpus preparation in our goal of curating a cdec dataset, we first needed to identify documents that exhibit crossdocument coreference. we now describe our document collection process and our methodology for annotating event mentions in these documents. ## annotating coreference via crowdsourcing corefering event mentions share their identity. however, the extent of sharing for them to be considered coreferential is unclear. to empirically study this behavior, we crowdsource annotations on mechanical turk. we use the crowd workers' responses to analyze the influence of quasi-identity on coreference decisions. ## studying quasi-identity of events numerous factors determine the identity of an event mention, including the semantics of the mention, arguments (place, time, and participants), and the overall document context. therefore, overlap in these factors determines the extent of coreference between two given mentions. this overlap leads to cases of partial (quasi-) identity. our annotation workflow allows for empirical investigation of this phenomenon, and we summarize our observations through a taxonomy of event identity in the fire has burned about 4400 acres so far and 15 homes have been lost, however there have been no reported injuries or deaths. 1b reports say that the amount of people fleeing from their homes in california located in the united states due to wildfires has reached the 1,000,000 mark as the fires continue to grow. subevent 3a a freight train in lviv, ukraine derailed, caught fire, and spilled a toxic chemical, releasing dangerous fumes into the air early tuesday morning (local time), and people who live near the site of the crash are still becoming sick. 3b the available information about the phosphorous cloud following the railway accident in the ukraine last monday is becoming more and more cryptic. 4a during the fifteen days of the trial, the prosecutors called 92 witnesses to testify as to the chaotic scenes following the bombing. 4b two explosions within seconds of each other tore through the finish line at the boston marathon, approximately four hours after the start of the men's race. spatiotemporal continuity 5a tropical storm richard is nearing hurricane strength with winds of 70 mph (115 kph) as it lashes honduras with heavy rains 5b hurricane richard made landfall in belize about 20 mi (35 km) south-southeast of belize city with winds of 90 mph (150 kph) at approximately 6:45 local time (0045 utc) according to the national hurricane center (nhc) ## baselines we define the task as a mention pair classification problem. due to the quasi-identity nature of event mentions ( §5), we do not cluster mentions in coreference groups. additionally, we consider both full and partial identity under the coreference label. we present two baseline models, lemma-match, and a cross-encoder model. we split the dataset of 55 subtopics into train and test, with 40 subtopics for training and development, and 15 subtopics for the held-out test set. for our experiments, we assume gold mentions and subtopic information. 19 lemma-match: for our first baseline, we implement the traditional lemma-match baseline. we use spacy's large model 20 to extract the head lemma of the event mentions, and consider two mentions corefering if the lemma's match. following upadhyay et al. (2016) , we also experiment with a lemma-δ baseline. in our experiments, we found the best dev performance with δ=0, resolving to a simple lemma baseline. this could be due to our assumption of access to gold subtopic information. cross-encoder: as a second baseline, we implement bert-based cross-encoder model. the input consists of a pair of sentences with both mentions highlighted using special tokens to indicate the start and end of mention spans (<e>, </@xmath0. we first concatenate the two event-tagged sentences (with [sep] token) and pass it through a bert-baseuncased encoder. we then perform mean pooling on the event start tags (<@xmath1, and pass the pooled embedding through a linear classification layer to predict coreference vs. non-coreference. for training the cross-encoder, in addition to the positive coreference pairs, we generate two types of negative mention pairs. for the first type, we collect noncoreference mention pairs from sentences that have a coreference link between a different mention pair. for the second type, we extract non-coreference mention pairs from random sentence pairs between the documents. during training, we use a dataset ratio of 1:5:5 (positive:negative-i:negative-ii). we use huggingface transformers @xcite , and train the model using adamw @xcite with an initial learning rate of 2e-5. we also use a linear warmup scheduler, with ## conclusion & future work in this work, we present a study of the identity of events through annotation of cross-document event coreference. we use a custom-designed annotation tool to collect coreference annotations on a subset of english wikinews articles. we release our dataset, cdec-wn, under an open license to encourage further research on event coreference. by collecting evidence for the extent of shared identity between events, we identify three types of partialidentity, membership, subevent, and spatiotemporal continuity. to serve as a benchmark for future coreference resolution systems, we provide results on two baseline models, lemma-match and bertbased cross-encoder. we believe that our work will encourage further research on the identity of events in the context of cdec. potential future directions include expanding cdec-wn to include withindocument coreference links, designing coreference resolution systems that account for cases of partial identity between mentions, and expanding the study of the partial identity of event coreference to new domains.