More than Classification: A Unified Framework for Event Temporal Relation Extraction (2023)

Abstract: Event temporal relation extraction (ETRE) is usually formulated as a multi-label classification task, where each type of relation is simply treated as a one-hot label. This formulation ignores the meaning of relations and wipes out their intrinsic dependency. After examining the relation definitions in various ETRE tasks, we observe that all relations can be interpreted using the start and end time points of events. For example, relation Includes could be interpreted as event 1 starting no later than event 2 and ending no earlier than event 2. In this paper, we propose a unified event temporal relation extraction framework, which transforms temporal relations into logical expressions of time points and completes the ETRE by predicting the relations between certain time point pairs. Experiments on TB-Dense and MATRES show significant improvements over a strong baseline and outperform the state-of-the-art model by 0.3% on both datasets. By representing all relations in a unified framework, we can leverage the relations with sufficient data to assist the learning of other relations, thus achieving stable improvement in low-data scenarios. When the relation definitions are changed, our method can quickly adapt to the new ones by simply modifying the logic expressions that map time points to new event relations. The code is released at https://github.com/AndrewZhe/A-Unified-Framework-for-ETRE

## introduction in order to fully understand natural language utterances, it is important to understand the temporal information conveyed in the text, especially the relations between the events @xcite . such temporal relations play an essential role in downstream applications, such as question answering, event timeline generation, and information retrieval @xcite . the event temporal relation extraction (etre) task is proposed to address the extraction of temporal relations between event pairs from text. researchers have different ideas on how to define the temporal relations between two events. @xcite treats an event as an interval in the timeline and uses 13 relations between two intervals to define the temporal relations between events. the 13 relations, together with a special relation vague, are then adopted by timebank @xcite . however, such a definition is so finegrained that some relations are very hard to distinguish from each other. thus following works make a simplification and aggregate some relations @xcite . for example, tb-dense @xcite aggregates before and before immediately into one coarse re-lation before. other studies, like matres @xcite , think that identifying the duration of events requires very long contexts and even commonsense, making it exceedingly difficult to determine when an event ends. therefore, in ma-tres, only the start time of events is considered for temporal relations. we show some examples of temporal relations and their interval representations in figure 1 . it can be seen that despite the differences across definitions, each relation reflects certain aspects of the full temporal relationship and has rich meanings behind a single label. although the meaning of a relation is important, previous studies did not pay enough attention. they solve etre as a simple text classification task, first using an encoder to get the event pair representation and then feeding it into a multi-layer perceptron to get the prediction. all efforts are focused on generating better event pair representations, such as pre-training a task-specific language model @xcite or applying graph neural networks to incorporate syntactic information @xcite . however, the relations are only used as one-hot labels to provide guidance in cross-entropy loss. such a classification-based method cannot fully use the meaning of relations and could cause the following problems: misunderstanding relations: some relations may correspond to complex scenarios, such as vague in matres. it describes a contradictory situation where event 1 may occur before event 2 and event 2 may also occur before event 1. such complex meaning cannot be conveyed by a simple one-hot vector. ## problem formulation given an input sequence x with two events e 1 and e 2 , the task of event temporal relation extraction is to predict a relation from r ∪ {vague} between the event pair (e 1 and e 2 ), where r is a pre-defined set of temporal relations of interests. label vague means the relation between the two events can not be determined by the given context. ## enhanced baseline model we first introduce our baseline model for etre. it is based on a strong entity relation extraction model @xcite and we integrate other techniques to make it suitable in etre. our baseline can achieve comparable or even better performance with the previous sota in etre, providing a powerful encoder for our unified framework. ## analysis we further explore how our module makes better use of label information in etre tasks. we show the 3 problems of classification-based methods, mentioned in introduction, could be alleviated by our unified framework. ## event re in llms large language models (llms), such as chat-gpt 4 , have shown impressive performance in vari1. which starts first? e1 or e2? 2. which starts first? e2 or e1? 3. which ends first? e1 or e2? 4. which ends first? e2 or e1? classification unified framework ous tasks. in this section, we investigate the performance of llms in event temporal relation extraction and assess the value of our proposed unified framework in the era of llms. we conduct experiments using gpt-3.5-turbo-0301 foot_1 , and figure 5 shows the prompts we used. the classification-based prompt lists all the candidate relations and requires the model to select one from them. if the list of relation candidates changes, the prompt should be modified accordingly and the old results will be useless. unlike the classification-based method, our unified framework is irrelevant to the candidate list. we ask the model to answer four questions and deduce the temporal relation based on the answers, just following what we do on bert-based models. we explore variations of prompts and the detailed experimental settings can be found in appendix e. the order of candidates matters in the classification-based prompt. the distribution of temporal relations is highly imbalanced in existing datasets, and we find that putting the majority relation, which is before, at the beginning of the candidate list will significantly affect the performance. for the vanilla classification-based prompt, we randomly sample an order of candidates for different cases. in contrast, +before first order and +before last order use a fixed order, which put before at the beginning or end of the candidate list, respectively. 6 as shown in tabel 5, compared with the other two orders, putting before at the beginning causes at least 2.7% decline in f1. further analysis shows that in this scenario, the model is more likely to predict before, making up to 55% of all the predictions. single word answer might not determine the direction of relations. when the model is expected to return a single word, like before to represent the temporal relation between two events, it might mean e 1 before e 2 , but it could also mean e 2 before e 1 . this is a common phenomenon when the prompt does not explicitly mention the direction of relation, e.g., "what is the temporal relation between e 1 and e 2 ". this would lead to inferior performance, that the vanilla classificationbased prompt outperforms the -relation direction prompt by 3.3% in f1. a unified framework may help large language models (llms). as shown in ## related work earlier studies have proposed various definitions of the relations between two events, and all of them adopt allen's interval representation. the 13 interval relations together with 1 vague relation form the basis elements for other relation definitions. time-bank @xcite and tempeval-3 (uzzaman et al., 2013) directly use all of the 14 relations and then the researchers find that some relations are too fine-grained for both humans and models. thus they simplify the etre task by aggregating some relations into a coarse one, e.g., @xcite merged all the overlap relations into a single relation, overlap. iso-timeml @xcite pays attention to a special relation, contain, which is composed of three interval relations where one interval is within the other. the focus on relation contain influences many followers, like thyme @xcite , tb-dense @xcite and maven @xcite . all these definitions, though differ in various aspects, can convert to intervals relations and thus could be interpreted using the endpoints of intervals. in other words, the different relations under these definitions could be represented in our unified framework. we just use two most widely used definitions, tb-dense and matres, to evaluate our framework, and our framework can be applied to other definitions. to solve the etre, previous efforts often regarded this task as a classification problem, and focus on learning better representations of event pairs, e.g, incorporating syntactic information @xcite @xcite or discourse information @xcite ) into the encoder. some studies also try to design auxiliary tasks, like event extraction @xcite or relative event time prediction @xcite to further enhance the encoder. different from their work, we focus on helping the model understand temporal relations better after obtaining the event pair representation from the encoder, and thus our work is orthogonal to them. recently, hwang et al. (2022) uses a box embedding to handle the asymmetric relationship between event pairs. however, the box embedding can only handle four types of relations, i.e., before, after, equal and vague, and it cannot generalize to more complex relations, like includes in tb-dense. to solve another task, cheng and miyao (2020) also consider start and end times separately. however, cheng and miyao (2020) directly uses a classifier to determine the relation between time points and cannot understand the relation vague well. ## conclusion in this paper, we interpret temporal relations as a combination of start and end time points of two events. using this interpretation, we could not only explicitly convey temporal information to the model, but also represent relations of different task definitions in a unified framework. our experimental results in tb-dense and matres demonstrate the effectiveness of our proposed method, significantly outperforming previous state-of-the-art models in full data setting and providing large improvements on both few-shot and transfer-learning settings. in the future, we will investigate the potential of our approach in cross-document scenarios. taylor cassidy, bill mcdowell, nathanael chambers, and steven bethard. 2014. an annotation framework for dense event ordering. in proceedings of the 52nd annual meeting of the association for computational linguistics (volume 2: short papers), pages 501-506. fei cheng and yusuke miyao. 2017. classifying temporal relations by bidirectional lstm over dependency paths. in proceedings of the 55th annual meeting of the association for computational linguistics (volume 2: short papers), pages 1-6, vancouver, canada. association for computational linguistics. fei cheng and yusuke miyao. 2020. predicting event time by classifying sub-level temporal relations induced from a unified representation of time anchors. arxiv preprint arxiv:2008.06452. prafulla kumar choubey and ruihong huang. 2017. a sequential model for classifying temporal relations between intra-sentence events. in proceedings of the 2017 conference on empirical methods in natural language processing, pages 1796-1802, copenhagen, denmark. association for computational linguistics. rujun han, qiang ning, and nanyun peng. 2019. joint event and temporal relation extraction with shared representations and structured prediction. arxiv preprint arxiv:1909.05360. rujun han, xiang ren, and nanyun peng. 2020. econet: effective continual pretraining of language models for event temporal reasoning. arxiv preprint arxiv:2012.15283. zhiting hu, xuezhe ma, zhengzhong liu, eduard hovy, and eric xing. 2016. harnessing deep neural networks with logic rules. in proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: long papers), pages 2410-2420. eunjeong hwang, jay-yoon lee, tianyi yang, dhruvesh patel, dongxu zhang, and andrew mccallum. 2022. event-event relation extraction using probabilistic box embedding. in proceedings of the 60th annual meeting of the association for computational linguistics (volume 2: short papers), pages 235-244, dublin, ireland. association for computational linguistics. hieu man, nghia trung ngo, linh ngo van, and thien huu nguyen. 2022. selecting optimal context sentences for event-event relation extraction. proceedings of the aaai conference on artificial intelligence, 36(10):11058-11066. puneet mathur, rajiv jain, franck dernoncourt, vlad morariu, quan hung tran, and dinesh manocha. 2021a. timers: document-level temporal relation extraction. in proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 2: short papers), pages 524-533, online. association for computational linguistics. puneet mathur, rajiv jain, franck dernoncourt, vlad morariu, quan hung tran, and dinesh manocha. 2021b. timers: document-level temporal relation extraction. in proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 2: short papers), pages 524-533. yuanliang meng, anna rumshisky, and alexey romanov. 2017. temporal information extraction for question answering using syntactic dependencies in an lstm-based architecture. in proceedings of the 2017 conference on empirical methods in natural language processing, pages 887-896, copenhagen, denmark. association for computational linguistics. qiang ning, sanjay subramanian, and dan roth. 2019. an improved neural baseline for temporal relation extraction. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages vague 𝑡 @xmath1 & ∧ 𝑡 @xmath2 & 𝑄 @xmath0 &