Multi-Relational Probabilistic Event Representation Learning via Projected G aussian Embedding (2023)

Abstract: Event representation learning has been shown beneficial in various downstream tasks. Current event representation learning methods, which mainly focus on capturing the semantics of events via deterministic vector embeddings, have made notable progress. However, they ignore two important properties: the multiple relations between events and the uncertainty within events. In this paper, we propose a novel approach to learning multi-relational probabilistic event embeddings based on contrastive learning. Specifically, the proposed method consists of three major modules, a multi-relational event generation module to automatically generate multi-relational training data, a probabilistic event encoding module to model uncertainty of events by Gaussian density embeddings, and a relation-aware projection module to adapt unseen relations by projecting Gaussian embeddings into relation-aware subspaces. Moreover, a novel contrastive learning loss is elaborately designed for learning the multi-relational probabilistic embeddings. Since the existing benchmarks for event representation learning ignore relations and uncertainty of events, a novel dataset named MRPES is constructed to investigate whether multiple relations between events and uncertainty within events are learned. Experimental results show that the proposed approach outperforms other state-of-the-art baselines on both existing and newly constructed datasets.

## introduction events, carrying world knowledge, are the major research targets in natural language processing (nlp) for decades. distributed event representation learning has been shown beneficial in various nlp tasks, such as sentiment analysis @xcite , event detection @xcite ) and text generation @xcite . early event representation learning methods mainly focused on the way of composing event components, such as by multilayer perceptrons @xcite , recurrent neural networks @xcite , and tensor networks @xcite . latter work tried to incorporate various external knowledge into event representation learning, such as knowledge graphs @xcite , extra event features @xcite , or commonsense knowledge @xcite . recently, @xcite showed the effectiveness of incorporating contrastive learning @xcite in event representation learning by simultaneously utilizing weakly supervised contrastive learning and prototype-based clustering. so far, similar to word representation learning @xcite , current approaches for event representation learning mainly aim to capture the semantics of events based on largescale co-occurrence training data by making the semantically-similar events closer in embedding space. though notable progress has been made, most existing methods still have two limitations. on the one hand, they ignore the multiple relations between events, which means every event pair that occurs together will be pushed closer whatever the actual relation between them. as shown in @xcite , both event (man go to hospital) and event (man be healthy) will be pushed closer to event (he contracted disease), which means they will be pushed closer too. it is problematic as they are semantically different. on the other hand, the inherent uncertainty or polysemy of events is ignored. a more general event is used to describe more situations, reflecting higher uncertainty in its meaning @xcite . as shown in figure 1(a) , event (he contracted disease) is a general description for illness, which is semantically similar to two specific events (man have headache) and (person catch cold) which are different from each other. however, restricted by the triangle inequality, it is hard for a vector embedding to be close to the other two points that are apart from each other. in this paper, we propose a multi-relational probabilistic event embedding method based on contrastive learning (more-cl) to solve the above limitations. specifically, we utilize comet @xcite , a commonsense generative model, to generate multi-relational positive samples for contrastive training. a probabilistic event encoder based on bert @xcite is proposed to generate gaussian event embeddings by estimating the mean vector and variance matrix. to deal with unseen relations, a relation-aware projector is employed to determine the relation-based event pair context automatically with an attention mechanism and project the density embeddings into relation-specific subspaces. finally, the original infonce loss @xcite is modified to learn multi-relational probabilistic embedding. to investigate the effectiveness of the proposed method, a multi-relational probabilistic event similarity dataset named mr-pes is constructed. experimental results show that more-cl outperforms other baselines by a large margin on both original and new benchmark datasets. in conclusion, our contributions are three-fold: ## method as shown in figue 2, more-cl consists of three modules. firstly, the training events extracted from a large corpus are fed into the multi-relational event generation module to generate positive sample events for contrastive learning. then, the training events as well as their multi-relational positive samples are encoded as multivariate gaussian density embeddings by the probabilistic event encoding module. after that, the density embeddings are projected into relation-specific subspaces by the relation-aware event projection module. finally, three modules are jointly optimized by the modified contrastive learning loss. the details of each step are discussed as follows. ## experiments in this section, we investigate the effectiveness of more-cl by comparing it with several competitive baselines both on the conventional event similarity task and the proposed multi-relation probabilistic event similarly task. ## conclusion in this paper, to model the multiple relations between events and uncertainty within events, we propose a multi-relational probabilistic event representation learning method, more-cl, based on the projected gaussian embedding with contrastive learning. to be more specific, more-cl consists of three modules, a multi-relational event generation module to incorporate relational knowledge of events, a probabilistic event encoding module to model uncertainty with gaussian density embeddings, and a relation-aware projection module to adapt to unseen relations. what's more, we also present a new dataset to test the knowledge of multiple relations and uncertainty learned by event representation methods. the experimental results for both existing and new datasets show the effectiveness of the proposed method.