D oc T ime: A Document-level Temporal Dependency Graph Parser (2022)

Abstract: We introduce DocTime - a novel temporal dependency graph (TDG) parser that takes as input a text document and produces a temporal dependency graph. It outperforms previous BERT-based solutions by a relative 4-8% on three datasets from modeling the problem as a graph network with path-prediction loss to incorporate longer range dependencies. This work also demonstrates how the TDG graph can be used to improve the downstream tasks of temporal questions answering and NLI by a relative 4-10% with a new framework that incorporates the temporal dependency graph into the self-attention layer of Transformer models (Time-transformer). Finally, we develop and evaluate on a new temporal dependency graph dataset for the domain of contractual documents, which has not been previously explored in this setting.

## introduction understanding the temporal relations between events mentioned in a document is an important natural language task with applications in downstream tasks such as timeline creation @xcite , time-aware summarization @xcite , temporal question-answering @xcite , and temporal information extraction @xcite . this area of research remains important yet challenging due to several limitations such as confounded modalities (eg. events that are certain to happen vs the ones that might happen), event ambiguity (eg. agreeing to terms of a contract vs signing a contract) and need for complete annotation of all event pairs for precise temporal localization @xcite . early work densely annotated all pairs of events to address this problem @xcite , but was limited to short passages or adjacent sentences due to the n 2 complexity of the task, especially for long documents. recently this problem formulation was significantly simplified using temporal dependency trees (tdt) @xcite and temporal dependency graphs (tdg) @xcite by only capturing the reference timex or event to build a dependency graph to capture this information. this enabled the development of temporal dependency parsers @xcite to infer temporal relationships more robustly and efficiently. we introduce doctime -a state-of-the-art temporal dependency parser that parses document-level text to produce temporal dependency graphs. unlike previous approaches using contextual features such as bert @xcite , our model utilizes a graph network and a novel path prediction loss to reason over long-range multi-hop dependencies while maintaining global consistency of temporal ordering of inter-dependent events. to validate the utility of doctime and our generated temporal dependency graph, we go one step further than prior work and explore the question of whether temporal dependency graphs are useful for downstream tasks by introducing time-transformer. it is a framework to incorporate temporal dependency graphs into existing transformer-based architectures without retraining from scratch. we demonstrate the usefulness of our proposed time-transformer on temporal nli @xcite and time-sensitive question answering @xcite tasks. prior work on temporal relationship extraction and temporal dependency parsing have been mostly limited to news @xcite @xcite , narrative stories @xcite or clinical notes @xcite . in addition to experimenting with existing temporal dependency parsing datasets, we introduce a dataset for temporal dependency graphs in a new domain -contractual documents, where temporal reasoning over events has real world legal and monetary implications for users. our main contributions include: ## related work temporal dependency parsing: previous work has been devoted to pairwise classification of relations between events and time expressions, notably timebank @xcite and its extensions like @xcite annotated all relations. pair-wise annotation have multiple problems including polynomial square complexity, global inconsistencies in predictions due to relation transitivity and forced annotation of vague relations @xcite . prior work focuses on extracting temporal relations between event pairs in the same sentence or adjacent sentences @xcite @xcite @xcite . ## time-transformer we would also like to understand our temporal dependency parsing can be useful for downstream tasks requiring temporal reasoning. here we introduce the time-transformer, which allow a tdg generated by doctime to be combined with stateof-the-art transformer models for temporal tasks. the time-transformer augments the flow of information in a transformer network via a temporallyinformed self-attention mechanism. we first for-mulate the time-transformer architecture in §4 and then construct of temporally-informed attention layers in §4. architecture: time-transformer was motivated by recent work incorporating syntax @xcite or co-reference graphs @xcite into the transformer architecture to improve downstream tasks. in each case, these approaches encode additional knowledge from the sparse graphs as a masked self attention layer into the transformer. figure 2 shows the architecture of time-transformer incorporating temporal knowledge into the self-attention layer during finetuning of the transformer model. input text is converted into a temporal dependency graph using doctime parser. the graph is then converted into a set of masks that encodes the temporal relationship between each entity (i.e. after) explained in more detail in the next section: temporally-informed self-attention. the input embedding (token+positional+attention masks) is passed through the time-transformer model which modifies the self-attention layer of the standard transformer architecture with a temporallyinformed self-attention layer to be fine-tuned on downstream tasks. tisa: temporally-informed self-attention : the tdg produced by doctime is sparse and to effectively utilize the graph extracted by the temporal dependency parser for longer range temporal relationships, we utilize k self-attention layers that encode the temporal relationship if traversing k hops in the tdg as shown in 2. more formally starting from node a, the minimum number of hops (k) required to reach another node b can be regarded as k-hop distance between a and b, written as k-hop(a, b). we create k masks to represent the (k)-hop distance between two nodes to allow the model to aggregate information across longer ranges in the tdg. specifically, a mask @xmath0, 1, 2, • • • , r} n×n denotes if there is a relation between entity i and j, and n is the number of tokens in the input text. the value of the mask is the relationship type for i and j. it is found by inferring the relationship using allen's interval algebra @xcite and is set to 0 if there is no relationship or set to "overlap" if there is a conflict. we adopt a soft-mask learning strategy to enable the self-attention layer to re-weight the importance of each mask and avoid the problem of vanishing gradient. a hyperbolic feed-forward layer is used to learn the mask weights as research has shown it can avoid distortion of the feature space in graph representations @xcite . the value of k is a hyperparameter that can be customized according to the nature of input dependency graph. training time-transformer: for each dataset, we optimize the hyper-parameters of time-transformer through grid search on the validation data. in all our experiments, we limit the maximum value of k-hop to 15. detailed settings can be found in the appendix. ## conclusion we present doctime, a new temporal dependency parsing approach that improves upon previous approaches by integrating longer term temporal information through a graph network with a novel path prediction loss. additionally, we are able to show how a tdg can be incorporated into transformer networks with time-transformer to improve on down stream tasks for nli and question answering. finally we introduce a tdg dataset in a new domain (contractual documents) to expand research in this temporal reasoning to a new application domain. future works will aim to explore more ways for integrating temporal dependency graphs into neural architectures across different application domains. in future, we would like to explore temporal event mining to aid various social media applications such as improving hate speech detection @xcite , analyzing temporality in suicidal ideation detection @xcite and abuse detection @xcite . the proposed time-transformer can find applications in augmenting financial tasks @xcite , affective computing @xcite , and ai for social good @xcite with temporal common sense reasoning.