Neural Language Modeling for Contextualized Temporal Graph Generation (2021)

Abstract: This paper presents the first study on using large-scale pre-trained language models for automated generation of an event-level temporal graph for a document. Despite the huge success of neural pre-training methods in NLP tasks, its potential for temporal reasoning over event graphs has not been sufficiently explored. Part of the reason is the difficulty in obtaining large training corpora with human-annotated events and temporal links. We address this challenge by using existing IE/NLP tools to automatically generate a large quantity (89,000) of system-produced document-graph pairs, and propose a novel formulation of the contextualized graph generation problem as a sequence-to-sequence mapping task. These strategies enable us to leverage and fine-tune pre-trained language models on the system-induced training data for the graph generation task. Our experiments show that our approach is highly effective in generating structurally and semantically valid graphs. Further, evaluation on a challenging hand-labeled, out-of-domain corpus shows that our method outperforms the closest existing method by a large margin on several metrics. We also show a downstream application of our approach by adapting it to answer open-ended temporal questions in a reading comprehension setting.

## introduction temporal reasoning is crucial for analyzing the interactions among complex events and producing coherent interpretations of text data @xcite . there is a rich body of research on the use of temporal information in a variety of important application domains, including topic detection and tracking @xcite , information extraction @xcite , parsing of clinical records @xcite , discourse analy-sis @xcite , and question answering @xcite . graphs are a natural choice for representing the temporal ordering among events, where the nodes are the individual events, and the edges capture temporal relationships such as "before", "after" or "simultaneous". representative work on automated extraction of such graphs from textual documents includes the early work by @xcite , where the focus is on the construction of event chains from a collection of documents, and the more recent caevo @xcite and cogcomptime @xcite , which extract a graph for each input document instead. these methods focus on rule-based and statistical submodules to extract verb-centered events and the temporal relations among them. as an emerging area of nlp, large scale pre-trained language models have made strides in addressing challenging tasks like commonsense knowledge graph completion @xcite and task-oriented dialog generation @xcite . these systems typically fine-tune large language models on a corpus of a task-specific dataset. however, these techniques have not been investigated for temporal graph extraction. this paper focuses on the problem of generation of an event-level temporal graph for each document, and we refer to this task as contextualized graph generation. we address this open challenge by proposing a novel reformulation of the task as a sequence-to-sequence mapping problem @xcite , which enables us to leverage large pretrained models for our task. further, different from existing methods, our proposed approach is completely end-to-end and eliminates the need for a pipeline of sub-systems commonly used by traditional methods. we also address a related open challenge, which is a prerequisite to our main goal: the difficulty of obtaining a large quantity of training graphs with human-annotated events and temporal relations. to this end, we automatically produce a large collection of document-graph pairs by using caevo, followed by a few rule-based post-processing steps for pruning and noise reduction. we then encode the graph in each training pair as a string in the graph representation format dot, transforming the text-to-graph mapping into sequence-to-sequence mapping. we fine-tune gpt-2 on this dataset of document-graph pairs, which yields large performance gains over strong baselines on system generated test set and outperforms caevo on timebank-dense @xcite on multiple metrics. figure 1 shows an example of the input document and the generated graph by our system. in summary, our main contributions are: ## related work temporal graph extraction tempeval-3 (uz-zaman et al., 2013) introduced the task of temporal graph extraction as "the ultimate task for evaluating an end-to-end system that goes from raw text to timeml annotation". notable systems developed in response include caevo @xcite , followed by the more recent cogcomptime @xcite . both caevo and cogcomptime use several statistical and rule-based methods like event extractors, dependency parsers, semantic role labelers, and time expression identifiers for the task. our work differs from these systems in both the methodology and desired result in the following ways: i) instead of using specialized sub-systems, we transform the task into a sequence-to-sequence mapping problem and use a single language model to generate such temporal graphs in an end-to-end fashion from text, subsuming all the intermediatesteps. ii) we develop our system using a corpus of 89,000 documents, which is ∼ 300x larger com-pared to datasets used by caevo (36 documents) and cogcomptime on (276 documents); iii) we remove the noisy events included by caevo, but do not limit the extracted events to any specific semantic axis as done by cogcomptime; and finally, iv) our method generates graphs where the nodes are not simple verbs but augmented event phrases, containing the subject and the object of each verb. we use caevo over cogcomptime to generate a largescale corpus for our task and to evaluate our system for the following reasons: i) we found caevo to be much more scaleable, a critical feature for our task of annotating close to 100k documents, ii) caevo over-generates (and not excludes) verbs from its output, giving us the flexibility to filter out noisy events without inadvertently missing out on any critical events. however, our method makes no assumption specific to caevo and is adaptable to any other similar system (including cogcomptime). ## model the training data x for both tasks 1 and 2 comprises of tuples {(x i , y i )} n @xmath0. for task 1 (node generation), x i the concatenation of context, the source, node, and the relation. the target y i consists of the tokens of the target event. for task 2 (graph generation), x i is a document and y i is the corresponding temporal graph represented in dot. we train a (separate) conditional language model to solve both the tasks. specifically, given a training corpus of the form {(x i , y i )}, we aim to estimate the distribution p θ (y i | x i ). given a training example (x i , y i ) we set u @xmath1. p θ (u i ) can then be factorized as a sequence of auto-regressive conditional probabilities using the chain rule: p θ (u i ) = n @xmath2,k |u i,<k ), where u i,k denotes the k th token of the i th sequence, and u i,<k denotes the sequence of tokens {u 1 , u 2 , ..., u k-1 }. language models are typically trained by minimizing a cross-entropy loss -logp θ (u i ) over each sequence u i in x. however, the cross-entropy loss captures the joint distribution p θ (x i , y i ), and is not aligned with our goal of learning conditional distribution p θ (y i | x i ). note that the formulation of masked loss is opaque to the underlying architecture, and can be implemented with a simple change to the loss function. in practice, we use gpt-2 @xcite based on transformer architecture @xcite for our implementation. having trained a p θ for each task, we generate a node (y) given a query (x) (for task 1), or a graph (y) given a document (x) (for task 2) by drawing samples from the appropriate p θ (y | x) using nucleus sampling @xcite . we provide more details of our training procedure and the architecture in the appendix (c.1). ## conclusion and future work current methods for generating event-level temporal graphs are developed with relatively small amounts of hand-labeled data. on the other hand, the possibility of using pre-trained language models for this task has not received sufficient attention. this paper addresses this open challenge by first developing a data generation pipeline that uses existing ie/nlp/clustering techniques for automated acquisition of a large corpus of document-graph pairs, and by proposing a new formulation of the graph generation task as a sequence-to-sequence mapping task, allowing us to leverage and fine-tune pretrained language models. our experiments strongly support the effectiveness of the proposed approach, which significantly outperforms strong baselines. we plan to explore techniques for adapting largescale language models on unseen domains and at multiple granularity levels in the future.