E vent F ull: Complete and Consistent Event Relation Annotation (2025)

Abstract: Event relation detection is a fundamental NLP task, leveraged in many downstream applications, whose modeling requires datasets annotated with event relations of various types. However, systematic and complete annotation of these relations is costly and challenging, due to the quadratic number of event pairs that need to be considered. Consequently, many current event relation datasets lack systematicity and completeness.In response, we introduce EventFull, the first tool that supports consistent, complete and efficient annotation of temporal, causal and coreference relations via a unified and synergetic process.A pilot study demonstrates that EventFull accelerates and simplifies the annotation process while yielding high inter-annotator agreement.

## introduction identifying the semantic relations between events mentioned in a text, notably temporal, causal and coreference relations, has been a fundamental goal in nlp. substantial efforts have been devoted to developing various datasets that capture some or all of these relations @xcite @xcite . these datasets were then leveraged to develop and to evaluate corresponding models for detecting event-event relations. the output of such models has been utilized in a range of downstream applications, with recent examples including event forecasting @xcite , misinformation detection (lei and huang, 2023), and treatment timeline extraction @xcite , among others. models for detecting event relations are expected to produce complete output, that is identifying all event relations that can be inferred from the given text. accordingly, such models should ideally be evaluated, and trained, on datasets in which event relation annotation is in itself complete, in the sense that each pair of targeted events has been classi-fied for its potential relationships. unfortunately, though, exhaustive manual annotation of all eventevent relations is typically considered extremely challenging or impractical, since the number of event pairs to be considered is quadratic in the number of targeted event mentions in the text @xcite . as the number of event mentions grows, this task quickly becomes both too time consuming and cognitively unmanageable. faced with this inherent annotation complexity, many datasets adopted an annotation protocol that restricts the number of events or event pairs considered for annotation through various restrictions and heuristics. notably, tb-dense @xcite , ecb+ @xcite , @xcite , and eventsto-ryline @xcite restrict event pairs to a span of two consecutive sentences. this limitation inherently prevents testing and training models on longer-range relations. other datasets, such as timebank @xcite and @xcite , did not publish a systematic annotation execution protocol that guarantees actual complete annotation, and were subsequently criticized for being incomplete in their relation annotation @xcite . further, some researchers aimed to avoid the cost of manual annotation altogether and employed fully-or partlyautomatic dataset creation methods @xcite @xcite . these approaches inherently incorporate biases introduced by the employed automated method, making them less reliable for testing purposes while being prone to yield biased models when used for training. finally, motivated by similar observations to ours, the recent narrativetime project @xcite does emphasize relation annotation completeness, supported by a corresponding annotation tool. however, their work addresses only temporal figure 1 : the eventfull annotation pipeline begins with a document containing marked targeted events and proceeds through three stages: [1] temporal relations: annotators establish temporal relations between event pairs, supported by three processes, including pair prioritization strategy, consistency checking, and transitive relation detection ( §3.2); [2] coreference: annotators identify coreferring event mentions; [3] causal relations: annotators determine causal relations for pairs of events. relations, while employing a complex annotation scheme that requires expert annotators. indeed, their actual annotation was performed by two of the authors, overall limiting the replicability of this approach. in this paper, we aim to close major gaps in prior annotation protocols. to that end, we introduce a simple-to-use annotation tool that facilitates future creation of event relation datasets (for covering multiple relation types, new genres and languages, etc.), which guarantees complete relation annotation . our web-based tool, eventfull (figure 1 ), fulfills four critical goals, which constitute our primary contributions: (1) supporting joint synergetic annotation of the three prominent event-event relation types: temporal, causal, and coreference; (2) for any given set of targeted events within the text, the gold output relations are guaranteed to be complete, classifying all event pairs for the three relation types; (3) supporting efficient annotation while guaranteeing annotation consistency, via automated transitive completion and consistency checks, while leveraging constraints imposed across the three relation types (see §3.2); (4) ease of use by a non-expert annotator, lowering the bar for future dataset creation. to the best of our knowledge, eventfull is the first available tool that effectively supports and integrates all these goals. in the remainder of the paper, we first survey relevant background and related work ( §2), followed by the presentation of eventfull's input and output structure and its workflow ( §3). finally, we present a pilot study in which we demonstrate the effectiveness of eventfull ( §4). the tool and code are publicly available at https: //github.com/aloneirew/eventfull . ## background and related work this section first provides relevant background regarding event relations, followed by a short survey of prior annotation tools. ## the eventfull annotation tool as described in §1, eventfull guarantees complete and consistent annotation for temporal, causal and coreference relations between events mentioned in an input text, while minimizing the manual annotation effort. we next describe its input and output structure, followed by a detailed description of its functionality and workflow. ## pilot study to evaluate eventfull's effectiveness, we conducted a small-scale study to assess the time, effort and quality of the resulting annotations. this section first describes the annotation procedure ( §4.1) and then reviews the quality and effectiveness of the annotation process ( §4.2). ## conclusion in this paper we introduced eventfull, a novel tool for the end-to-end annotation of complete and con-a b c average time