C on T empo: A Unified Temporally Contrastive Framework for Temporal Relation Extraction (2024)

Abstract: The task of temporal relation extraction (TRE) involves identifying and extracting temporal relations between events from narratives. We identify two primary issues with TRE systems. First, by formulating TRE as a simple text classification task where every temporal relation is independent, it is hard to enhance the TRE model’s representation of meaning of temporal relations, and its facility with the underlying temporal calculus. We solve the issue by proposing a novel Temporally Contrastive learning model (ConTempo) that increase the model’s awareness of the meaning of temporal relations by leveraging their symmetric or antisymmetric properties. Second, the reusability of innovations has been limited due to incompatibilities in model architectures. Therefore, we propose a unified framework and show that ConTempo is compatible with all three main branches of TRE research. Our results demonstrate that the performance gains of ConTempo are more pronounced, with the total combination achieving state-of-the-art performance on the widely used MATRES and TBD corpora. We furthermore identified and corrected a large number of annotation errors present in the test set of MATRES, after which the performance increase brought by ConTempo becomes more apparent.

## introduction temporal relation extraction (tre) is the task of classifying the temporal relations between pairs of events conveyed in narratives @xcite . this task is important to the natural language processing (nlp) community because the conception of time is a key component of text comprehension and reasoning. additionally, it carries practical importance, as tre is a crucial component of various downstream applications in question answering, information retrieval, and information extraction. tre is difficult. despite its importance and popularity, tre systems still have relatively poor performance compared to other natural language understanding tasks. tre is also salient because recent large language models (llms) and chat systems have substantially worse performance than smaller, fine-tuned tre models @xcite . in this paper, we enhance current tre systems by identifying and addressing two major issue that hinder their performance: tre is typically formulated as a multi-label text classification task. the representation of the relation is determined by a neural classifier, which outputs logits for different label classes. these logits are then converted into label probabilities using a softmax layer. as a result, in a tre model, all relation types are considered independent. however, temporal relations are inherently structured, governed by an underlying temporal calculus @xcite . for instance, if event a occurred before event b, it can be inferred that b happened after a. integrating the meanings of each temporal relation and the principles of the underlying temporal calculus into the neural classifier presents a complex challenge, as there is no straightforward method to do so. we propose a novel temporally contrastive learning method (contempo) 1 to enhance the model's representation of the meaning of temporal relations, and its facility with the underlying temporal calculus. the intuition behind contempo is very simple. contrastive learning aims to pull similar (positive) samples closer and push dissimilar (negative) samples apart. we leverage the symmetric and antisymmetric properties of temporal relations to augment a base graph of positive samples for contrastive learning. for antisymmetric relations (a before b), we want to push the relation's representation away from its inverse (b after a). for a symmetric relation (a equals b), we likewise want to pull its representation closer to its inverse (b equals a). our experimental results indicate that contempo is effective at creating distinctive representations for different types of temporal relation pairs and yields a substantial improvement in performance. then, building upon contempo, we create a unified framework that is compatible with various previous modes of improvement in tre systems. in particular, we have identified two major threads of innovation in the previous literature on this topic: (1) gnn encoding of event and context information, (2) incorporation of common-sense information, and (3) the soft enforcement of logical constraints. our experiment shows that contempo is compatible with techniques from all three major threads of research and can achieve state-of-the-art performance on major evaluation corpora, including matres @xcite and timebank-dense (tbd; @xcite . we nevertheless perceive the need to introduce cleanmatres, a refined version of matres that we created by fixing certain annotation errors. among all of these innovations, an ablation study nevertheless shows that contempo brings the most significant performance increase. this finding underscores our initial hypothesis: the primary challenge for these models has been their difficulty with making sense of relation labels. during our development of the unified comtempo framework, we discovered a large number of annotation errors within matres and tbd, two commonly utilized corpora. tbd's dense annotation scheme, the appropriate level of underspecificity cannot be resolved without further investigation. but we are able to provide a thorough re-examination of the matres corpus. after correcting these errors, performance improves yet again, and the performance increase brought by contempo becomes more apparent. ## contempo we begin our discussion by first formulating the tre task. then, using a comparison of relational representations, we will demonstrate the difficulty that a neural model faces in temporal reasoning and understanding the meaning of temporal labels. in particular, there is a significant correlation between the mistakes a tre model makes and its degree of inference over the available symmetric and antisymmetric properties. next, we will present our contempo method in detail, which enhances a model's awareness of the meaning of each temporal relation and each principles of temporal reasoning. finally, we will empirically demonstrate the effectiveness of contempo and compare it with previous methods in increasing temporal relation awareness. ## the unified framework our survey of previous systems suggests that the reusability of innovations of tre has been limited. for example, papers working on gnn-encoding of document-level information @xcite @xcite rarely incorporate other published innovations, such as additional common-sense information, in their models. we call for more collaboration between different directions of tre research. in this section, we argue that contempo is compatible with various recent innovations across different research directions. in particular, con-tempo is complementary to: (1) gnn encodings of event and context information, (2) incorporat-ing common-sense information, and (3) the soft enforcement of logical constraints. ## conclusion in this paper, we propose a novel contempo method that utilizes contrastive learning on the symmetric and antisymmetric properties to enhance the model's awareness of temporal relations. using contempo, we observe an apparent change in the representation of relations generated by the model.