TIMELINE : Exhaustive Annotation of Temporal Relations Supporting the Automatic Ordering of Events in News Articles (2023)

Abstract: Temporal relation extraction models have thus far been hindered by a number of issues in existing temporal relation-annotated news datasets, including: (1) low inter-annotator agreement due to the lack of specificity of their annotation guidelines in terms of what counts as a temporal relation; (2) the exclusion of long-distance relations within a given document (those spanning across different paragraphs); and (3) the exclusion of events that are not centred on verbs. This paper aims to alleviate these issues by presenting a new annotation scheme that clearly defines the criteria based on which temporal relations should be annotated. Additionally, the scheme includes events even if they are not expressed as verbs (e.g., nominalised events). Furthermore, we propose a method for annotating all temporal relations—including long-distance ones—which automates the process, hence reducing time and manual effort on the part of annotators. The result is a new dataset, the TIMELINE corpus, in which improved inter-annotator agreement was obtained, in comparison with previously reported temporal relation datasets. We report the results of training and evaluating two baseline temporal relation extraction models on the new corpus, and compare them with results obtained on the widely used MATRES corpus.

## introduction understanding the temporal structure of events in text is essential for a wide range of natural language processing tasks, e.g., question answering, information retrieval and inference @xcite @xcite . often, however, there is no explicit temporal information associated with most of the events in news articles. for instance, in the sentence "he pointed to the possibilities of new business models, products and ways of working that could have a dynamic impact on living standards.", there is no temporal expression associated with the event "pointed" that conveys when exactly it occurred. the extraction of temporal relations, i.e., determining whether an event occurred before, after or at the same time as another event, makes it possible to capture the temporal sequence of events, even in cases where the text does not explicitly mention any temporal information with respect to an event @xcite . extracting temporal relations relies heavily on the annotation scheme adopted, which determines the granularity of the types of extracted temporal relations @xcite . in existing temporal information-annotated datasets @xcite @xcite @xcite @xcite @xcite , many types of temporal relations are ignored, ill-defined or focussed only on specific types of events. in most datasets, only relations between events in the same or adjacent sentences are tagged @xcite . such limitation is the main reason for losing more precise temporal information for almost half of the events @xcite . in addition, low agreement between human annotators is a common issue and needs to be improved by making the annotation task more clearly defined @xcite . our work seeks to address these issues by making the following contributions: ## related work timebank is the first temporal information-labelled dataset to provide different types of temporal annotations (i.e., events, time expressions, and temporal relations) in news articles @xcite . however, there are two main issues with timebank: (1) the annotators tagged only temporal relations (referred to as tlinks) which are considered as important @xcite , leading to sparse annotations; and (2) the scheme did not specify when two events should be paired up in a relation; as a result, inter-annotator agreement (iaa) was only around 55%. similar to time-bank is the timeeval3 dataset as it is a cleansed version of the former; it was created mainly for the tempeval shared tasks @xcite . meanwhile, the red corpus considered different relations between events (e.g., temporal, coreference, causal and sub-event relations). it is a rich dataset created mainly to support the development of multi-task systems; the iaa for the relations of interest (e.g., "before" relations) is relatively low, i.e., around 41% (o' @xcite . timebank-dense is a subset of timebank, which was introduced to address the sparsity issue in timebank by annotating all possible event pairs, and all event and time expression pairs in each given sentence and its surrounding sentences @xcite . however, many ill-defined temporal relations were annotated, leading to low iaa. the matres corpus tried to solve this issue by adopting a scheme that takes into consideration multiple timelines, i.e., axes, distinguishing between events that actually happened (which belong to the main axis) and those which are only hypothetical (which belong to an axis parallel to the main one), for example. this multi-axis scheme required that each event relation is annotated while considering the relevant axis, thus improving the iaa significantly (84%) @xcite . however, they focussed only on events centred on verbs and ignored nominalised events. @xcite proposed automatically annotating temporal relations between events in a sentence and its surrounding sentences, using predefined rules based on the events' time anchors. they annotated temporal relations based on an existing dataset where the time anchors for events are already labelled @xcite . @xcite suggested a heuristic algorithm for the automatic inference of relations using the corpus developed by @xcite . moreover, they made the first attempt to capture long-distance relations by asking experts to manually annotate a subset of unlabelled long-distance relations based on textual cues, external knowledge and narrative ordering. however, state-of-the-art models perform worse on their dataset, tddiscourse, compared with other datasets. error analysis shows that the models failed to deal with some of the phenomena in their dataset (e.g., negated/conditional events, event coreference, and the requirement to have access to real-world knowledge). event type category intension, opinion on an orthogonal axis hypothesis, generic on a parallel axis negation not on any axis static, recurrent other ## motivation this work is motivated by earlier relevant studies in the literature; we refer the reader to we reviewed existing temporal relation annotation schemes @xcite to answer the first question. we then decided to discard events that cannot be anchored onto a timeline; these include intended, negated events and events involved in conditional constructions. such events are the source of many ill-defined relations (e.g., vague relations) in existing datasets. a specific temporal relation between two events is labelled vague if there is not enough information about the two events in the text that makes the annotator decide if the first event occurred before or after or at the same time as the second event. consider the following example sentence: "she planned to attend the conference yesterday." the temporal relation between the two events ("planned" and "attend") is vague as we cannot confidently determine the temporal relation between them based on the context alone, i.e., it is possible that the event centred on "attend" did not occur. according to @xcite , events belong to different time axes, hence the distinction between the following axes: (1) the main axis, i.e., a horizontal line where events that actually happened are represented (e.g., the event "planned" in the example sentence); (2) an orthogonal axis (a vertical line that is orthogonal to the main axis) where opinions/intentions are placed (e.g., the event "attend" in the same example); and (3) a parallel axis (a horizontal line parallel to the main axis) where generic and hypothetical events are placed. hence, we focussed only on all events that belong to the main axis (events in the main storyline). we refer the reader to regarding the second question, we concluded that annotating every possible temporal relation in a specific news article is a non-feasible task @xcite . importantly, inconsistent temporal relation annotations are to be expected from a human annotator (e.g., a transitive constraint is not always satisfied) and have been noted in the time-bank corpus @xcite . additionally, employing crowdsourcing for the annotation of these relations is expensive. for instance, @xcite reported that it costs about 400 usd to annotate temporal relations between events in a given sentence and its surrounding sentences in only 36 news articles. also, @xcite highlighted that considering long-distance relations is required to retrieve correct temporal information for 40 % of events in news articles. therefore, to address these issues, we decided to automatically generate temporal relations and to directly infer consistent relations within different windows, i.e., relations between events which are separated by 0...n sentences. further details on how temporal relations are generated will be given later in section 4.2. please refer to figure 1 for an illustration of the relation window. ## dataset construction in this section, we describe the process for collecting the documents included in our corpus. this is followed by a discussion of the details of our proposed annotation scheme. ## corpus reliability three annotators contributed to the annotation of our corpus: the first one (the first author of this paper) annotated all the articles, whilst the second and third annotators annotated 31% of the articles. the second and third annotators are phd computer science students who have received training on the proposed annotation scheme. upon completion of the annotation tasks, they were compensated at an hourly rate of £15. three subsets were defined, containing randomly selected documents: training (70%), development (10%) and test (20%). we refer the reader to we note that prior to training and evaluating each of the said models, all temporal relations labelled as vague in the timeline corpus, were discarded for the following reasons: (1) this type serves as a catch-all category for any relations which are ambiguously expressed in text and yet is overrepresented (accounting for 60.81% of the annotated relations); and (2) more importantly, the performance for the vague relation type was not considered in previously reported work-they treated this label similarly to how they handled events with no temporal relations between them @xcite . in preparation for training the models, we generated bert embeddings @xcite and part-of-speech (pos) embeddings for every token in the sentences containing events that are involved in a temporal relation, as both models take these as input representation. for the training process, we adopted the hyperparameter values used by @xcite . ## evaluation results and ablation study as one can observe in in the subsections below, we discuss the main differences between the two corpora, matres and timeline, and the impact of each key difference on the performance of the baseline models. ## reasoning behind this annotation in the llms era we believe that despite the advent of large language models (llms), this kind of fine-grained annotation is still necessary to support the development of supervised models. we argue that the temporal relation extraction performance of an llm such as chatgpt, for example, is not comparable in relation to that of supervised models. firstly, yuan et al. (2023) investigated chatgpt's capability in zeroshot temporal relation extraction and showed that chatgpt's performance is lower by up to 30% in terms of f1-score compared to supervised methods. furthermore, we investigated the extent to which chatgpt can extract temporal relations by prompting it using the zero-shot prompt proposed by @xcite to identify temporal relations between events in the timeline test set. overall, chat-gpt obtained precision, recall and f1-scores of 31.11%, 35.67% and 33.24%, respectively. these are substantially lower than those of the second baseline method, which obtained 69.05% for precision, 69.05% for recall and 69.05% for f1-score. ## potential applications our annotation scheme and dataset hold promise for various practical uses. extracting temporal relations from news articles can support information extraction applications such as automatic timeline extraction and question answering (qa). moreover, considering that the focus of the dataset is on events on the main axis (i.e., events in the main storyline), this work can potentially support narrative extraction applications such as the analysis of events related to financial markets and event monitoring, e.g., in the context of disaster management @xcite . ## conclusion in this paper, we present a new corpus, timeline, which was annotated following a novel annotation scheme whereby non-verb-centred events are included, as well as long-distance temporal relations between events. the corpus was used in training and evaluating two baseline temporal relation extraction models. based on our evaluation results, we assessed the impact of increasing the relation window and including non-verb-centred events on model performance. in addition, we demonstrated how our annotation scheme can support the development of models that can extract more relations in comparison with earlier datasets. in the future, we aim to increase the size of the dataset and employ it in a timeline generation task.