Extracting Event Temporal Relations via Hyperbolic Geometry (2021)

Abstract: Detecting events and their evolution through time is a crucial task in natural language understanding. Recent neural approaches to event temporal relation extraction typically map events to embeddings in the Euclidean space and train a classifier to detect temporal relations between event pairs. However, embeddings in the Euclidean space cannot capture richer asymmetric relations such as event temporal relations. We thus propose to embed events into hyperbolic spaces, which are intrinsically oriented at modeling hierarchical structures. We introduce two approaches to encode events and their temporal relations in hyperbolic spaces. One approach leverages hyperbolic embeddings to directly infer event relations through simple geometrical operations. In the second one, we devise an end-to-end architecture composed of hyperbolic neural units tailored for the temporal relation extraction task. Thorough experimental assessments on widely used datasets have shown the benefits of revisiting the tasks on a different geometrical space, resulting in state-of-the-art performance on several standard metrics. Finally, the ablation study and several qualitative analyses highlighted the rich event semantics implicitly encoded into hyperbolic spaces.

## introduction successful understanding of natural language depends, among other factors, on the capability to accurately detect events and their evolution through time. this has recently led to increasing interest in research for temporal relation extraction @xcite with the aim of understanding events and their temporal orders. temporal reasoning has been proven beneficial, for example, in understanding narratives @xcite , answering questions @xcite , or summarizing events @xcite . however, events that occurred in text are not just simple and standalone predicates, they rather form complex and hierarchical structures with different granularity levels (fig. 1 ), a characteristic that still challenges existing models and restricts their performance on real-world datasets for temporal relation extraction @xcite . addressing such challenges requires models to not only recognize accurately the events and their hierarchical and chronological properties but also encode them in appropriate representations enabling effective temporal reasoning. although this has prompted the recent development of neural architectures for automatic feature extraction @xcite @xcite , which achieves better generalization and avoids costly design of statistical methods leveraging hand-crafted features @xcite @xcite , the inherent complexity of temporal relations still hinders approaches that just rely on the scarce availability of annotated data. some of the intrinsic limitations of the mentioned approaches are due to the adopted embedding space. existing approaches to temporal relation extraction typically operate in the euclidean space, in which an event is represented as a point. although euclidean embeddings exhibit a linear algebraic structure which captures co-occurrence patterns among events, they are not able to reveal richer asymmetric relations, such as event temporal order (e.g., 'event a happens before event b' but not vice versa). inspired by recent works on learning non-euclidean embeddings, such as poincarè embeddings @xcite showing superior performance in capturing asymmetrical relations of objects, we propose to learn event embeddings in hyperbolic spaces @xcite . hyperbolic spaces can be viewed as continuous versions of trees, thus naturally oriented to encode hierarchical and asymmetrical structures. for instance, @xcite showed that hyperbolic spaces with just two dimensions (i.e., poincaré disk) could efficiently embed tree structures with arbitrarily low distortion @xcite , while euclidean spaces cannot achieve a comparable distortion even with an unbounded number of dimensions @xcite . despite the hierarchical properties arising in modelling event relations, there are still very few studies on how to leverage those models for temporal relation extraction. we propose two hyperbolic-based approaches with different strengths: a lightweight and efficient embedding learning method based on the poincaré ball model, and an end-to-end deep hyperbolic neural network based on the riemannian optimization. our contributions can be summarized as follows: ## related work our work is related to at least two lines of research: one about event temporal relation extraction and another on hyperbolic neural models. ## preliminaries in this section, we give a brief introduction to hyperbolic geometry and hyperbolic neural networks. ## event temporal relation extraction in the hyperbolic space in this section, we propose two approaches to leverage a hyperbolic space for temprel extraction. the first approach learns event embeddings that encode temporal order via a hyperbolic space, while the second one is an end-to-end hyperbolic neural network tailored for the temprel extraction task. ## experimental setup we describe the datasets, methodologies used in the recent literature for the temprel extraction tasks. we also briefly present the parameter setup of our experiments. a description of the evaluation metrics can be found in the appendix d. dataset matres @xcite . matres consists of documents from three sources: timebank (183 documents), aquaint (72 documents), and platinum (20 documents). we follow the official split in which timebank and aquaint are used for training, while platinum is used for testing. we further split 20% of the training data as the validation set. temporal and causal reasoning (tcr) @xcite is another dataset that adopts the annotation scheme defined in matres. it is a much smaller dataset, with just 25 documents and 2.6k matres tcr cogcomptime @xcite 61.6 72.5 61.6 66.6 --68.1 70.7 lstm* @xcite 70.2 76.5 67.3 73.2 79.6 75.7 75.7 77.6 lstm+knowledge* @xcite 70.2 80.1 70.1 74.8 79.3 76.9 76.9 78.1 lstm+knowledge+ilp @xcite --71.7 76.7 --80.8 78.6 jcl-base @xcite 67.7 80.3 -73.5 ----jcl+multi-task+logic @xcite 72.2 83.8 -77.6 ----jcl-all @xcite 73 parameter setup based on the results of preliminary experiments, the contextualized embeddings are produced from roberta in both of our proposed approaches. we conducted preliminary experiments to determine the best dimension for the poincaré embeddings but found the variations in performance had no statistical significance. thus, we adopted the 2d poincaré embeddings for the sake of simplicity and ease of visualization. more details about the model architecture and hyperparameter setting can be found in the appendix a. ## experimental results overall comparison existing methodologies adopted for temprel extraction commonly leverage several auxiliary components, such as external commonsense knowledge and multi-task objectives. therefore, to better understand the impact made by the adoption of the hyperbolic geometry, we conduct additional experiments over ablated versions of the baseline models. in particular, in using our proposed alternative end-to-end hgru model, hgru (static elmo) outperforms both the standard lstm model and the variant incorporating commonsense knowledge (lstm+knowledge) on matres. this verifies that the hyperbolic-based method is more efficient than its euclidean counterparts. in order to fairly compare with the state-of-the-art model @xcite on this task, we utilize roberta and the auxiliary temporal commonsense knowledge since @xcite also fine-tunes roberta on matres and uses external commonsense knowledge. the results show that hgru (roberta) + knowledge outperforms jcl-base @xcite significantly by 7% in f 1 . it even outperforms jcl-all, which further incorporates logic constraints and multi-task learning. on the tcr dataset, the proposed hyperbolicbased methods also see similar improvement over existing methods. in terms of the difference between the two proposed methods, poincaré events embeddings achieve a higher f 1 score on tcr. the reason is that hgru tends to predict more vague labels, but there is no vague in tcr. interestingly, both proposed methods predict less vague labels when using static roberta. a detailed breakdown of results for each temporal relation and training cost is presented in the appendix b. ## conclusion in this paper, we proposed to model event temporal relations overcoming the limitations of euclidean representations, and designing two temprel extraction methods using hyperbolic geometry. the first approach highlighted the convenience of learning event embedding in the poincaré ball, achieving performance on par with recent methodologies using a simple rule-based classifier. then, we designed a hyperbolic neural network, incorporating temporal commonsense, outperforming state-ofthe-art models on the standard datasets. finally, a qualitative analysis pointed out the inherent advantage of employing hyperbolic spaces to encode asymmetric relations. in the future, we plan to extend our approaches to a wider spectrum of event relations, including causal and sub-event relations.