Event Temporal Relation Extraction with B ayesian Translational Model (2023)

Abstract: Existing models to extract temporal relations between events lack a principled method to incorporate external knowledge. In this study, we introduce Bayesian-Trans, a Bayesian learning-based method that models the temporal relation representations as latent variables and infers their values via Bayesian inference and translational functions. Compared to conventional neural approaches, instead of performing point estimation to find the best set parameters, the proposed model infers the parameters’ posterior distribution directly, enhancing the model’s capability to encode and express uncertainty about the predictions. Experimental results on the three widely used datasets show that Bayesian-Trans outperforms existing approaches for event temporal relation extraction. We additionally present detailed analyses on uncertainty quantification, comparison of priors, and ablation studies, illustrating the benefits of the proposed approach.

## introduction understanding events and how they evolve in time has been shown beneficial for natural language understanding (nlu) and for a growing number of related tasks @xcite @xcite @xcite . howeover, events often form complex structures with each other through various temporal relations, which is challenging to track even for humans @xcite . one of the main difficulties is the wide variety of linguistic expressions of temporal relations across different contexts. although many of them share some linguistic similarities, most of the topics in which they occur are characterized by some shared but unspoken knowledge that determines how temporal information is expressed. for example, when it comes to health, prevention is widely 1 experimental source code is available at https:// github.com/xingwei-warwick/bayesian-trans practised, with many treatments (e.g., vaccinations) being effective only if administered before the onset of a disorder. on the contrary, in the automotive industry, it is common that most people repair their car after a problem occurs. however, despite its simplicity, such commonsense knowledge is rarely stated explicitly in text and varies greatly across different domains. for example, in figure 1 , a detection model lacking the commonsense knowledge that vaccination can protect people from infection, tends to get confused by the complex linguistic structures in the excerpt and returns the wrong prediction entailing that 'died' happens after 'vaccinated'. instead, with the consideration of prior temporal knowledge involving the vaccination event from an external knowledge source atomic @xcite , a model gives the correct prediction that 'died' occurs before 'vaccinated'. methods proposed in recent studies for event relation extraction are mostly end-to-end neural architectures making rather limited use of such commonsense knowledge @xcite . only a few works have explored the incorporation of external knowledge to mitigate the scarcity of event annotations @xcite . nevertheless, these approaches typically update the event representations with knowledge features derived from external sources, lacking a principled way of updating models' beliefs in seeing more data in the domains of interests. in this work, we posit that the bayesian learning framework combined with translational models can provide a principled methodology to incorporate knowledge and mitigate the lack of annotated data for event temporal relations. translational models, such as transe @xcite , are energybased models based on the intuition that the relations between entities can be naturally represented by geometric translations in the embedding space. more concretely, a relation between a head entity and a tail entity holds if there exists a translational operation bringing the head close to the tail vector. specifically, we introduce a novel bayesian translational model (bayesian-trans) for event temporal relation extraction. compared to conventional neural translational models, which only yield a point estimation of the network parameters, the bayesian architecture can be seen as an ensemble of an infinite number of neural predictors, drawing samples from the posterior distribution of the translational parameters, refining its belief over the initial prior. as a result, event temporal relations are determined by the stochastic translational parameters drawn from posterior distributions. additionally, such posteriors are conditioned upon the prior learned on external knowledge graphs, providing the commonsense knowledge required to interpret more accurately the temporal information across different contexts. as shown in the results obtained from the experimental evaluation on three commonly used datasets for event temporal relation extraction, the combination of translational models and bayesian learning is particularly beneficial when tailored to the detection of event relations. moreover, a favorable by-product of our bayesian-trans model is the inherent capability to express degrees of uncertainty, avoiding the overconfident predictions on out-of-distribution context. our contributions are summarized in the following: ## related work this work is related to at least three lines of research: event temporal relation detection, prior knowledge incorporation, and graph embedding. ## bayesian-trans model in identifying temporal relations between events, we aim at predicting the relation type of two events given in text, commonly denoted as head event x h and tail event x t : here, p(λ|g) denotes the prior distribution of λ derived from an existing knowledge graph encoded as g. directly inferring eq. ( 2 ) is intractable. but we can resort to amortised variational inference to learn model parameters. in what follows, we present our proposed bayesian learning framework built on translational models for event temporal relation extraction, called bayesian-trans, with its architecture shown in figure 2 . in particular, the context s in which the two events occur is the input to our bayesian-trans. first, we encode s via a pre-trained language model generating the contextual embeddings e h and e t for the triggers of the head and tail events, respectively. the contextualised event trigger representations, e h and e t , are fed as input into a bayesian translational module. this module, by means of variational inference, determines the parameters of the translational model, encoding the posterior distribution of the temporal relations conditioned upon the input events. finally, we use a score function on the translated head and tail triggers to predict their temporal relation. we provide a more detailed description in the following. ## experimental setup datasets we evaluated the proposed bayesian-trans model on three event temporal relation datasets: matres @xcite , temporal and causal reasoning (tcr) @xcite , and timebank-dense (tbd) @xcite . timebank-dense is a densely annotated dataset focusing on the most salient events and providing 6 event temporal relations. matres follows a new annotation scheme which focuses on main time axes, with the temporal relations between events determined by their endpoints, resulting in a consistent inter-annotator agreement (iaa) on the event annotations @xcite . tcr follows the same annotation scheme, yet with a much smaller number of event relation pairs than in matres. baselines we compare the proposed bayesian-trans foot_2 with the following baselines: cogcomptime @xcite bilstm is a basic relation prediction model built by @xcite . lstm + knowledge @xcite incorporates knowledge features learnt from an external source and optimize global consistency by ilp. deep structured @xcite adds a structured support vector machine on top of a bilstm. joint constrained learning @xcite constrains the training of a roberta-based event pair classifier using predefined logic rules, while knowledge incorporation and global optimization are also included. poincaré event embedding @xcite learns event embeddings based on a poincaré ball and determines the temporal relation base on the relative position of events. hgru + knowledge @xcite is a neural architecture processing temporal relations via hyperbolic recurrent units which also incorporates knowledge features like lstm + knowledge. relative event time @xcite is a neural network classifier combining an auxiliary task for relative time extraction over an event timeline. uast @xcite is an uncertainty-aware self-training model. we show the result of the model which is trained on all the labeled data. ## experimental results temporal relation classification we first compare bayesian-trans with the most recent approaches for temporal event classification in ## conclusion we propose a principled approach to incorporate knowledge for event temporal relation extraction named bayesian-trans, which models the relation representations in the mure translational model, as latent variables. the latent variables are inferred through variational inference, during which commonsense knowledge is incorporated in the form of the prior distribution. the experiments on ma-tres, tcr, and tbd show that bayesian-trans achieves state-of-the-art performance. comprehensive analyses of the experimental results also demonstrate the characteristics and benefits of the proposed model.