Extracting Temporal Event Relation with Syntax-guided Graph Transformer (2022)

Abstract: Extracting temporal relations (e.g., before, after, and simultaneous) among events is crucial to natural language understanding. One of the key challenges of this problem is that when the events of interest are far away in text, the context in-between often becomes complicated, making it challenging to resolve the temporal relationship between them. This paper thus proposes a new Syntax-guided Graph Transformer network (SGT) to mitigate this issue, by (1) explicitly exploiting the connection between two events based on their dependency parsing trees, and (2) automatically locating temporal cues between two events via a novel syntax-guided attention mechanism. Experiments on two benchmark datasets, MATRES and TB-DENSE, show that our approach significantly outperforms previous state-of-the-art methods on both end-to-end temporal relation extraction and temporal relation classification with up to 7.9% absolute F-score gain; This improvement also proves to be robust on the contrast set of MATRES. We will make all the programs publicly available once the paper is accepted.

## introduction temporal relationship, e.g., before, after, and simultaneous, is important for understanding the process of complex events and reasoning over them. extracting temporal relationship automatically from text is thus an important component in many downstream applications, such as summarization @xcite , dialog understanding and generation @xcite , reading comprehension @xcite @xcite and future event prediction @xcite . while event mentions can often be detected reasonably well @xcite @xcite , extracting event-event relationships, especially temporal relationship, still remains challenging @xcite . temporal relation (e 1 e 2 ): before s2: mr. erdogan' s office (e 1 : said) he had (e 2 : accepted) the apology , " in the name of the turkish people ". s3 : "the desk thing really (e 1 : stuck) with me ", ms. ayotte (e2: said). event mentions are boldfaced, the temporal relations between these events are listed below each sentence, and the temporal cues deciding those temporal relations are highlighted in red. ## approach figure 2 shows the overview of our approach. given an input sentence @xmath0, w 2 , ..., w n ] with n tokens, we aim to detect a set of event mentions {e 1 , e 2 , ...} where each event mention e i may contain one or multiple tokens by leveraging the contextual representations learned from a pre-trained bert @xcite encoder. then, following previous studies @xcite @xcite , we consider each pair of event mentions that are detected from one or two continuous sentences, and predict their temporal relationship. to effectively capture the temporal cues between two event mentions, we build a dependency graph from one or two input sentences and design a new syntax-guided graph transformer network to automatically learn a new contextual representation for each event mention by considering the triples that they are locally involved as well as the triples along the dependency path of the two event mentions within the dependency graph. finally, the two event mention representations are concatenated to predict their temporal relationship. ## related work early studies on temporal relation extraction mainly model it as a pairwise classification problem @xcite @xcite @xcite and rely on hand-crafted features and rules @xcite to extract temporal event relations. recently, deep neural networks @xcite and large-scale pre-trained language models @xcite @xcite are further employed and show state-of-the-art performance. similar to our approach, several studies @xcite @xcite @xcite also explored syntactic path between two events for temporal relation extraction. different from previous work, our approach considers three important sources of temporal cues: local context, denoting the neighbors of each event node within the dependency graph; connection of two event mentions, which is based on the dependency path between two event mentions; and rich semantics of concepts and dependency relations, for example, the dependency relation nmod between two event mentions usually indicates a before relationship. all these indicative features are automatically selected and aggregated with the multi-head self-attention and our new syntax-guided attention mechanism. our work is also related to the variants of graph neural networks (gnn) @xcite @xcite , especially graph transformer @xcite @xcite . different from previous gnns which aim to capture the context from neighbors of each node within the graph, in our task, we aim to select and capture the most meaningful temporal cues for two event mentions from their connections within the graph as well as their surrounding contexts. ## conclusion temporal relationship between events is important for understanding stories described in natural language text, and a main challenge is how to discover and make use of the connection between two event mentions, especially when the event pair is far apart in text. this paper proposes a novel syntax-guided graph transformer (sgt) that represents the connection between an event pair via additional attention heads over dependency parsing trees. experiments on benchmarking datasets, ma-tres, tb-dense, and a contrast set of matres, show that our approach significantly outperforms previous state-of-the-art methods in a variety of settings, including event detection, temporal relation classification (where events are given), and temporal relation extraction (where events are predicted). in the future, we will investigate the potential of this approach to other relation extraction tasks.