Improving Event Representation via Simultaneous Weakly Supervised Contrastive Learning and Clustering (2022)

Abstract: Representations of events described in text are important for various tasks. In this work, we present SWCC: a Simultaneous Weakly supervised Contrastive learning and Clustering framework for event representation learning. SWCC learns event representations by making better use of co-occurrence information of events. Specifically, we introduce a weakly supervised contrastive learning method that allows us to consider multiple positives and multiple negatives, and a prototype-based clustering method that avoids semantically related events being pulled apart. For model training, SWCC learns representations by simultaneously performing weakly supervised contrastive learning and prototype-based clustering. Experimental results show that SWCC outperforms other baselines on Hard Similarity and Transitive Sentence Similarity tasks. In addition, a thorough analysis of the prototype-based clustering method demonstrates that the learned prototype vectors are able to implicitly capture various relations between events.

## introduction distributed representations of events, are a common way to represent events in a machine-readable form and have shown to provide meaningful features for various tasks @xcite @xcite @xcite . obtaining effective event representations is challenging, as it requires representations to capture various relations between events. figure 1 presents four pairs of events with different relations. two events may share the same event attributes (e.g. event types and sentiments), and there may also be a causal or temporal relation between two events. early works @xcite exploit easily accessible co-occurrence relation of events to learn event representations. although the use of cooccurrence relation works well, it is too coarse for deep understanding of events, which requires finegrained knowledge @xcite . recent works focus on fine-grained knowledge, such as discourse relations @xcite and commonsense knowledge (e.g. sentiments and intents) @xcite . concretely, @xcite and @xcite leverage 11 discourse relation types to model event script knowledge. @xcite incorporate manually labeled commonsense knowledge (intents and sentiments) into event representation learning. however, the types of fine-grained event knowledge are so diverse that we cannot enumerate all of them and currently adopted fine-grained knowledge fall under a small set of event knowledge. in addition, some manually labeled knowledge @xcite is costly and difficult to apply on large datasets. in our work, we observe that there is a rich amount of information in co-occurring events, but previous works did not make good use of such information. based on existing works on event relation extraction @xcite @xcite , we find that the co-occurrence relation, which refers to two events appearing in the same document, can be seen as a superset of currently defined explicit discourse relations. to be specific, these relations are often indicated by discourse markers (e.g., "because", capturing the casual relation) @xcite . therefore, two related events must exist in the same sentence or document. more than that, the co-occurrence relation also includes other implicit event knowledge. for example, events that occur in the same document may share the same topic and event type. to learn event representations, previous works @xcite based on cooccurrence information usually exploit instancewise contrastive learning approaches related to the margin loss, which consists of an anchor, positive, and negative sample, where the anchor is more similar to the positive than the negative. however, they share two common limitations: (1) such marginbased approaches struggle to capture the essential differences between events with different semantics, as they only consider one positive and one negative per anchor. (2) randomly sampled negative samples may contain samples semantically related to the anchor, but are undesirably pushed apart in embedding space. this problem arises because these instance-wise contrastive learning approaches treat randomly selected events as negative samples, regardless of their semantic relevance. we are motivated to address the above issues with the goal of making better use of cooccurrence information of events. to this end, we present swcc: a simultaneous weakly supervised contrastive learning and clustering framework for event representation learning, where we exploit document-level co-occurrence information of events as weak supervision and learn event representations by simultaneously performing weakly supervised contrastive learning and prototype-based clustering. to address the first issue, we build our approach on the contrastive framework with the infonce objective @xcite , which is a self-supervised contrastive learning method that uses one positive and multiple negatives. further, we extend the infonce to a weakly supervised contrastive learning setting, allowing us to consider multiple positives and multiple negatives per anchor (as opposed to the previous works which use only one positive and one negative). co-occurring events are then incorporated as additional positives, weighted by a normalized co-occurrence frequency. to address the second issue, we introduce a prototype-based clustering method to avoid semantically related events being pulled apart. specifically, we impose a prototype for each cluster, which is a representative embedding for a group of semantically related events. then we cluster the data while enforce consistency between cluster assignments produced for different augmented representations of an event. unlike the instance-wise contrastive learning, our clustering method focuses on the cluster-level semantic concepts by contrasting between representations of events and clusters. overall, we make the following contributions: ## preliminaries event representation model. in the early works @xcite , neural tensor networks (ntns) @xcite are widely adopted to compose the representation of event constitutions, i.e., (subject, predicate, object). however, such methods introduced strong compositional inductive bias and can not extend to events with more additional arguments, such as time, location etc. several recent works @xcite replaced static word vector compositions with powerful pretrained language models, such as bert @xcite , for flexible event representations and achieved better performance. following them, we also take the bert as the backbone model. the bert encoder can take as input a free-form given an input event x i , we obtain three augmented representations z i , z a1 and z a2 of the same event x i using the bert model with different dropout masks. using the same approach, we obtain the representation set {z k } @xmath0. to be the input sequence of length l, where x 0 and x l are the [cls] token and the [sep] token respectively. given x, the bert returns a sequence of contextualized vectors: instance-wise contrastive learning. event representation models learn representations with contrastive learning, which aims to pull related events together and push apart unrelated events. margin loss @xcite is a widely used contrastive loss in most of the existing works on event representation learning @xcite @xcite . most recently, an alternative contrastive loss function, called in-fonce @xcite , has been proposed and shown effective in various contrastive learning tasks @xcite @xcite . @xcite further demonstrate that infonce works better than the margin loss. in this work, we explore the use of infonce to train our event representation model. formally, given a set of n paired events @xmath1, x + i } n @xmath2, where x + i is a positive sample for x i , the infonce objective for (x i , x + i ) is presented in a softmax form with in-batch negatives @xcite : ## the proposed approach in this section, we will present technical details of our proposed approach and our goal is to learn event representations by making better use of cooccurrence information of events. figure 2 presents an overview of our proposed approach, which contains two parts: the weakly-supervised contrastive learning method (left) and the prototype-based clustering method (right). in the following sections, we will introduce both methods separately. ## experiments following common practice in event representation learning @xcite @xcite , we analyze the event representations learned by our approach on two event similarity tasks ( § 4.2) and one transfer task ( § 4.4). ## analysis and visualization in this section, we further analyze the prototypebased clustering method. ## conclusion in this work, we propose a simple and effective framework (swcc) that learns event representations by making better use of co-occurrence information of events, without any addition annotations. in particular, we introduce a weakly supervised contrastive learning method that allows us to consider multiple positives and multiple negatives, and a prototype-based clustering method that avoids semantically related events being pulled apart. our experiments indicate that our approach not only outperforms other baselines on several event related tasks, but has a good clustering performance on events. we also provide a thorough analysis of the prototype-based clustering method to demonstrate that the learned prototype vectors are able to implicitly capture various relations between events.