A nnotating T emporal D ependency G raphs via C rowdsourcing (2020)

Abstract: We present the construction of a corpus of 500 Wikinews articles annotated with temporal dependency graphs (TDGs) that can be used to train systems to understand temporal relations in text. We argue that temporal dependency graphs, built on previous research on narrative times and temporal anaphora, provide a representation scheme that achieves a good trade-off between completeness and practicality in temporal annotation. We also provide a crowdsourcing strategy to annotate TDGs, and demonstrate the feasibility of this approach with an evaluation of the quality of the annotation, and the utility of the resulting data set by training a machine learning model on this data set. The data set is publicly available.

## introduction understanding temporal relations between events in a text is an important part of understanding the "meaning" of text. with the wide adoption of machine learning methods in natural language processing, the ability to achieve a large-scale highquality temporally annotated data set has been the bottleneck in advancing the state of the art in this area. even though the first temporal annotation scheme, timeml @xcite , was proposed over a decade ago, temporally annotated data is still relatively scarce. the largest data set that we are aware of is the data set used in tempeval-3 @xcite , and it consists of 276 articles from the timebank corpus @xcite and the aquaint corpus. this data set was later re-annotated by @xcite to improve its annotation consistency using a crowdsourcing approach. there are many challenges that have contributed to this state of affairs. temporal relations are often confounded with modalities (how do you order an event that actually happened with one that might happen?). some events are ambiguous between an instantaneous and stative reading (does "marriage" refer to the start of the marriage or does it refer to the state when marriage is in effect?) and this complicates its temporal relation with other events. while these have all contributed to the difficulty in temporal annotation, the main challenge is a practical one. the general assumption in temporal annotation has been that the temporal relation between every pair of events in a text has to be specified in order to fully understand the temporal relations in a text. this amounts to constructing a fully connected graph in which every event is connected to another event. with this pair-wise approach, a text of n events has n 2 event pairs that need to be considered. as the value of n increases, the number of event pairs quickly becomes very large. in practice, attempts to achieve complete annotation often fell far short and had to settle with covering all event pairs within a short text window (e.g., within adjacent sentences). even with this restriction, it is still difficult to produce a large data set. for example, there are only 36 articles in timebank-dense @xcite . on the other end of the spectrum, approaches that allow the annotators to select a subset of the event pairs to annotate often end up with sparse and inconsistent annotation, as different annotators often select different event pairs to annotate. for example, while the time-bank corpus has relatively more articles, but only annotates a relatively small number of temporal relations (6,418 in total). there are also efforts that report improved annotation consistency by focusing on specific syntactic constructions @xcite or one aspect of temporal annotation @xcite , but this comes at the cost of incomplete annotation. one promising recent approach to get out of this dilemma is to focus on dependencies between time expressions, between time expressions and events, and between events @xcite , based on the observation that time expressions and events are often expressed in relative terms and their temporal location needs to be understood with a reference time in mind. consider the examples in (1): in (1a), the interpretation of the time expression 9:00am depends on another time expression friday, which in turn depends on when the time this sentence is written, generally known as the document creation time (dct). in (1b), the temporal location of re-examine can only be understood in relation to said (it happens after the saying event). note that it does not depend on today, as it may or may not happen on that day. in contrast, in (1c), the temporal location of re-examine can only be understood with respect to the dct, not said. another example of one event depends on another event for its temporal interpretation is (1d), where the temporal interpretation of demanded depends on entered (it happened after entered), the temporal interpretation of assaulted depends on demanded, and the temporal interpretation of left depends on assaulted. this is a linguistic phenomenon known as temporal anaphora @xcite @xcite @xcite that has been extensively studied in computational linguistics. the working hypothesis of this dependency-based approach is that by annotating the dependencies, additional temporal relations can be inferred, via transitivity, or via common sense reasoning. this hypothesis seems to have been born out in (1d): based on the dependencies, one can additionally infer that assaulted happened after entered, for example. zhang and xue (2018b) made the assumption that there is exactly one reference time for each event or time expression. with this assumption, the temporal relations between time expressions and events in a text will form a temporal dependency tree (tdt). this means that each event or time expression only relates to one other event or time expression, making tdt a much scalable annotation problem in practice. however, there are reasons to believe that this assumption is too stringent, and in some cases, multiple reference times may be needed to properly interpret the temporal location of an event. in this paper, we extend the temporal dependency tree to temporal dependency graph (tdg), allowing each event to have a reference time expression, a reference event, or both. compared with tdt, tdg does not substantially increase the number of temporal relations in a text that need to be annotated while improving its expressiveness. we also investigate the feasibility of annotating tdgs from scratch via crowdsourcing, meaning we start with identifying events and time expressions, and then annotate the temporal relations between them. previous work on crowdsourcing temporal annotations relies on the events and time expressions already identified in the timebank @xcite , and this limits the possibility of expanding temporally annotated datasets beyond what already exist. we show that with a carefully designed annotation strategy, annotating tdgs via crowdsourcing is feasible. we annotated a corpus of 500 wikinews articles with this approach, and created the largest corpus in terms of the number of articles and the number of event or time expression pairs. the remainder of the paper is organized as follows. we provide a brief overview of the tdt representation and propose our extension in section 2. we present our crowdsourcing strategy in section 3. we present a quantitative analysis of our corpus in section 4, and then evaluate the quality of our annotation in section 5. in section 6, we retrain a neural ranking parser @xcite on this data set to demonstrate its utility and establish a baseline for fellow researchers. we discuss related work in section 7, and conclude in section 8. ## crowdsourcing strategy crowdsourcing is generally accepted as a costeffective alternative to the traditional annotation approach where annotators are provided detailed guidelines and carefully trained to meet certain consistency threshold before productive annotation can start. in a crowdsourcing setting, we oftentimes hire a much larger set of annotators that are not professionally trained and may be only working on the task sporadically. therefore, it is infeasible to ask them to follow detailed guidelines. as a result, successful crowdsourcing tasks tend to be simple and intuitive. to make crowdsourcing practical for tdgs, we adopt a divide-and-conquer strategy that decomposes the annotation of a tdg into five steps. a top-level flowchart of our annotation process is in figure 2 . after all annotation steps are completed, we assemble the tdg for each text, and an example that illustrates the step-by-step construction of the tdg of a text is provided in figure 3 . in all steps, each annotation is completed by three crowd workers using the amazon mechanic turk platform. unless otherwise specified, the majority-voted answer is designated as the gold annotation. we explain each annotation step in greater detail in the rest of the section. ## corpus statistics the news articles that we use for our annotation are sampled from english wikinews foot_2 and extracted with the publicly available wikiextractor.py script to remove hypertext markups. 4 a more detailed analysis of temporal relations in our corpus shows that for reference timex identification, 19% of the events have a reference timex that is in the same sentence, while 17% of the events have a reference timex that is in different sentences. around 64% of the events have dct as the reference timex. this indicates that in the majority of cases, the reference timex of an event cannot be found in the same sentence, and our tdg annotation is able to capture these relations as a document-level annotation framework. our analysis also shows that for reference event identification, 27% of the events do not have an reference event, and these are usually the first event of a paragraph. ## annotation evaluation we evaluate the annotation quality of our data set with two evaluation metrics. the first metric measures the agreement between crowd workers and experts, and the second metric, worker agreement with aggregate (wawa) @xcite , measures the agreement among the crowd-workers. both metrics have their advantages and disadvantages but in conjunction, they provide a fuller picture of the annotation consistency of our data set. we also evaluate our annotation with the wawa metric, which measures the average agreement between crowd workers' annotation and the aggregate answer. the wawa score measures the agreement among crowd workers, and as such it is sensitive to the number of crowd workers and whether there are outliers. nevertheless, it is a useful metric, assuming that when an annotation task is well-defined, there should be less variation among the annotators. when computing wawa, we used the majority aggregation instead of the weighted majority aggregation, and we only computed the labeled agreement. the wawa scores for the subtasks are also reported in tables 4 and 5 . ## experiments we test our data with an attention-based neural ranking temporal dependency parser foot_5 that zhang and xue (2018a) developed for tdt, which parses the temporal dependency tree by ranking the candidate parents for each node. to apply the tree parser to the graph data, we first add a meta node as reference event for events that only have a reference timex. then, we rank all the time expressions for events and pick the one with the highest score as its reference timex, and rank all the events and select its reference event. to help the model learn the relations between dct and events, a pos tag feature is added which only distinguishes present tense verb events with other events. this feature is represented as a one hot vector. we use the same hyperparameter values as @xcite . in the 500 documents, 400 are used as training data, 50 as validation data, and 50 as test data. the test data is annotated by experts, and the validation data is generated from crowdsourced annotation as follows: if there is no agreement for one question, i.e. three crowd workers chose three different answers, then experts annotate that question. we also develop a heuristic baseline system as follows. first, each time expression is attached to dct. for each event, if there is a time expression in the same sentence, we attach the event to that time expression, and designate the relation as "include". otherwise that event is attached to dct, and the relation is "before". for the reference events, we attach each event to the immediately previous event in the text, and designate the relation as "overlap", which is the most common relation between events in experts' annotation. as shown in ## conclusion in this paper, we proposed a temporal annotation scheme called temporal dependency graphs which extend previous research on temporal dependency trees. the temporal dependency graphs, like temporal dependency trees, draw inspiration from previous research on narrative times and temporal anaphora, allow a good trade-off between completeness and practicality in temporal annotation. we proposed a crowdsourcing strategy and demonstrated its feasibility with a comparative analysis of the quality of the annotation. we also demonstrated the utility of the data set by training a neural ranking model on this data set, and the data set is publicly available.