LEAF : Linguistically Enhanced Event Temporal Relation Framework (2023)

Abstract: Linguistic structures can implicitly imply diverse types of event relations that have been previously underexplored. For example, the sentence “John was cooking freshly made noodles for the family gathering” contains no explicit temporal indicators between the events, such as before. Despite this, it is easy for humans to conclude, based on syntax, that the noodles were made before John started cooking, and that the family gathering starts after John starts cooking. We introduce Linguistically enhanced Event TemporAl relation Framework (LEAF), a simple and effective approach to acquiring rich temporal knowledge of events from large-scale corpora. This method improves pre-trained language models by automatically extracting temporal relation knowledge from unannotated corpora using diverse temporal knowledge patterns. We begin by manually curating a comprehensive list of atomic patterns that imply temporal relations between events. These patterns involve event pairs in which one event is contained within the argument of the other. Using transitivity, we discover compositional patterns and assign labels to event pairs involving these patterns. Finally, we make language models learn the rich knowledge by pre-training with the acquired temporal relation supervision. Experiments show that our method outperforms or rivals previous models on two event relation datasets: MATRES and TB-Dense. Our approach is also simpler from past works and excels at identifying complex compositional event relations.

## introduction event temporal relation extraction can help us better organize event flow and understand how events develop. for example, in news articles, understanding the causal relationships between events can help us better understand why certain events occurred @xcite . in medical records, understanding the temporal relationships between events can help us better track a patient's medical history @xcite . recently, there have been works focusing on first acquiring temporal relation knowledge automatically and then injecting the acquired knowledge via pre-training. for example, econet @xcite uses explicit keyword search to retrieve the sentences that contain temporal indicators such as before, after, during, and previously as supervision. however, they do not fully exploit knowledge from sentence linguistic structures. while zhou et al. 2020a make an attempt to utilize linguistic structures by extracting patterns from semantic role labeling (srl) parses @xcite , much of the linguistic information available is under-explored and they only utilize keywords found in an event's temporal argument. moreover, previous works which utilize linguistic structure for event relation knowledge do not apply them to neural networks @xcite . we find that there is rich, implicit event knowledge in the linguistic structures that has not been explicitly leveraged in the past. for example, consider the sentence "john was cooking freshly made noodles for the family gathering as adam arrived". notice that in this sentence, there is only one explicit mention of the temporal relationship between any of the events, which is that john was cooking as adam arrived. however, it is possible to extract five times the number of relations according to linguistic structures (figure 1 ): relation 1: the noodles were made before john started cooking, since adjectives of event objects all occur before the event occurs. relation 2: john started cooking before family gathering began, since an event always starts before its purpose event. relation 3: the noodles were made before adam arrived, since we know from the above relations that the noodles were made before john an arrow a -> b indicates that event "a" starts before event "b". we first obtain the srl annotations for all events (shown are the annotations for "cooking" and "told"). then, using a list of atomic patterns, we automatically extract temporal relationships between a target event and other events in its arguments (shown by the single colored arrows). finally, we use transitivity rules to find compositional relations (shown by the arrows with two colors). the table on the right shows the patterns corresponding to each relation. note that this figure displays a subset of the entire set of patterns we use. to this end, we propose leaf, a linguistically enhanced event temporal relation framework. our method aims at capturing a diverse set of linguistic structures implicitly indicating temporal relations (a.k.a, temporal knowledge patterns), and uses them to facilitate language models to learn richer temporal knowledge. we start by manually curating a diverse list of atomic patterns that commonly imply certain temporal relations (appendix a). these patterns involve pairs of events where one event is contained within the argument structure of the other. for example, a target event always starts after events in its prototypical patient (ppt) argument, and we can use this atomic pattern to find that cooking starts after made (figure 1 ). our list encompasses an extension of patterns from pre-vious works @xcite along with novel patterns, including the ppt pattern. as illustrated in figure 1 , if we consider only atomic patterns, many temporal relations are still overlooked. the events made, gathering, and arrived are not within one another's arguments, yet they still hold temporal relations. to capture these relations, we also gather compositional patterns. these are connections between two events that are not directly linked in their argument structures and are derived by utilizing the transitivity of atomic patterns. from applying these temporal knowledge patterns, we are able to extract two more atomic relations (relations 1-2) and three more compositional relations (relations 3-5) from the example sentence than knowledge acquisition methods relying only on temporal indicator word searching. to deploy temporal knowledge patterns for obtaining training supervision at scale, we use al-lennlp's srl parser @xcite on raw text. we then search the collected patterns to determine if any patterns appear in the srl annotations of a given sentence. once a pattern is found, the corresponding temporal relation of the pattern can then be used as supervision to further pre-train language models. with this method, we collect millions of event re-lation supervisions for pre-training from the raw gigaword headline corpus @xcite . this includes around 3.8m atomic relations and 140k compositional relations. our method effectively helps pre-trained language models learn rich temporal knowledge. leaf demonstrates an improvement of up to 9 f1 over vanilla bert base and roberta base on matres @xcite and tb-dense @xcite . it also delivers competitive performance with previous state-of-the-art (sota) methods that use temporal indicators and complex fine-tuning layers. moreover, it greatly exceeds 3-shot chatgpt (openai, 2023) by over 37 f1 points. we also perform ablation studies, which verify that both types of temporal knowledge patterns contribute to high performance. finally, with the aid of acquired atomic and compositional relation supervision, leaf shows an increase of up to 6.8 f1 points over baselines on challenging cases involving compositional relation prediction. ## related work in the early stages of event relation research, experts often used traditional machine learning methods to classify relations @xcite @xcite . these methods required experts to manually identify features and use external resources, which was time-consuming and laborintensive. recently, there have emerged a great number of attempts to incorporate temporal relation knowledge into neural network models @xcite @xcite . one branch of this involves incorporating additional temporal knowledge in the fine-tuning stage on fully labeled datasets, then evaluating on the respective dataset. some add additional parameters to train @xcite @xcite , while others only add objectives during fine-tuning @xcite . another branch called weak supervision is more closely related to our work. weak supervision does not require expensive manually labeled training data, but instead automatically labels unannotated corpora @xcite . this allows for greater transferability of knowledge between tasks, as well as ease of scalability. there are several popular methods for extracting event temporal relations using weakly supervised data. one approach is to perform a keyword search @xcite . another approach is to use a teacher model to label event relations @xcite . the most closely related to our work is @xcite , which uses keywords within a single srl semantic tag to extract relational knowledge. however, this is only a small subset of all available syntactic knowledge. in this work, we expand the range of syntactic structures used to extract relation knowledge, enabling language models to learn more diverse and complex knowledge. ## conclusions in conclusion, our proposed leaf framework demonstrates the effectiveness of using diverse linguistic structures to extract rich temporal knowledge of events from large-scale corpora. the extracted knowledge is able to enhance language models via a simple pre-training procedure. our approach outperforms or rivals previous models on matres and tb-dense, and excels at identifying complex compositional event relations.