Semi-supervised New Event Type Induction and Description via Contrastive Loss-Enforced Batch Attention (2023)

Abstract: Most event extraction methods have traditionally relied on an annotated set of event types. However, creating event ontologies and annotating supervised training data are expensive and time-consuming. Previous work has proposed semi-supervised approaches which leverage seen (annotated) types to learn how to automatically discover new event types. State-of-the-art methods, both semi-supervised or fully unsupervised, use a form of reconstruction loss on specific tokens in a context. In contrast, we present a novel approach to semi-supervised new event type induction using a masked contrastive loss, which learns similarities between event mentions by enforcing an attention mechanism over the data minibatch. We further disentangle the discovered clusters by approximating the underlying manifolds in the data, which allows us to achieve an adjusted rand index score of 48.85%. Building on these clustering results, we extend our approach to two new tasks: predicting the type name of the discovered clusters and linking them to FrameNet frames.

## introduction discovering new event types is an important step for adapting information extraction (ie) methods to unseen domains. existing work @xcite @xcite @xcite traditionally uses a predefined list of event types and their respective annotations to learn an event extraction model. however, these annotations are both expensive and time-consuming to create. this problem is amplified when considering specialization-intensive domains such as scientific literature, which requires years of specialized experience to understand even a specific niche. for example, there are a wide range of otherwise obscure events in biomedical literature @xcite , and better ie techniques can empower life-changing breakthroughs in these domains. to adapt ie to these specialized domains, it is critical to discover new event types automatically. there are two primary approaches in event type induction. the first is completely unsupervised induction. it includes recent neural techniques @xcite , as well as ad-hoc clustering techniques @xcite and probabilistic generative methods @xcite @xcite . the second approach, semi-supervised event type induction, was recently introduced by @xcite . it proposes leveraging annotations for existing types to learn to discover new types; this enables taking advantage of existing resources. in this work, we pursue the second approach. current state-of-the-art work in event type induction @xcite uses reconstruction-based losses to find clusters of new types. motivated by recent success in learning representations with contrastive loss @xcite , we propose a novel alternative approach using batch attention and contrastive loss, which achieves state-of-the-art results. essentially, we consider the attention weight between two event mentions as a learned similarity, and we ensure that the attention mechanism learns to align similar events using a semi-supervised contrastive loss. by doing this, we are able to leverage the large variety of semantic information in pretrained language models for clustering unseen types by using a trained attention head. this reveals our first key motivation: unlike @xcite , we are able to separate clustering from learning, allowing specific task-suited clustering algorithms to be selected. this allows us to test multiple clustering strategies after training once. additionally, this easily allows the use of hierarchical clustering, which may be beneficial to some downstream tasks. batch attention is an attention mechanism taken over a minibatch of samples rather than a sequence. previous uses of batch attention have been limited. primarily, it has been used for image classification @xcite and satellite imagery @xcite . in this work, we apply batch attention to natural language instead, which we use for clustering, and we propose the novel idea of enforcing the attention mechanism using contrastive loss. to enable our discovered event types to be used in larger ie systems, it is important to extract information regarding the clusters. previous work has looked to describe clusters-for a given cluster, @xcite uses the nearest trigger to the cluster centroid as its name. however, this approach is nebulous and not easily measurable (because the same trigger can correspond to different event types and there is not a quantitative method to determine if the selected trigger defines the cluster well). this is our second key motivation: we instead introduce two new information retrievalstyled tasks for describing the cluster -type name prediction and framenet @xcite frame linking. type prediction predicts a name for each cluster and is a relatively easy task. framenet linking builds on this by linking event types to relevant frames, and is significantly more useful for downstream applications. our attention-based approach is especially useful here, since it uses the attention mechanism to produce "clustered" features which can have auxiliary task-specific losses applied (prior work is not well-suited for this because the loss is applied to individual data points). this allows our method to build clusters which are more amenable to downstream tasks. the major novel contributions of this paper are: ## experimental results generally, we used default hyperparameters. we split the learning rates into bert and non-bert parameters following @xcite with 2e-5 for bert as in @xcite and 1e-4 for other parameters as in @xcite . for the margin parameter, we examined silhouette scores to select 0.5. for back-translation, we used four languages, german, french, spanish, and chinese, and randomly sampled which language to use for each data point every epoch. we obtained back-translations using the marianmt translation models @xcite . ablations are shown in section j. for our main experiments, we only use the contrastive loss. we take the average of 5 runs to show that our method consistently outperforms @xcite . we also calculate clusters using an ensemble of the 5 runs which shows slightly increased performance, which is an expected result in deep neural networks @xcite . huang and ji (2020) evaluate these clusters using geometric nmi, fowlkes mallows (fowlkes and mallows, 1983), completeness, homogeneity, and v-measure @xcite . we additionally consider adjusted rand index (ari) @xcite . in the downstream tasks, given a clustering we also report the average cluster purity and type representation. given a cluster i of size n i with most frequent type numbering @xcite . note that this average cluster purity is slightly different than traditional purity; it weights small clusters more which is desirable in our case (like macro vs. micro f1 score). type representation is the number of unique frequent subtypes, n t , divided by total types, in this case 23. ## related work although event extraction has long been studied @xcite @xcite @xcite @xcite , recent focus has turned towards discovering events without annotations. it includes recent neural techniques @xcite @xcite , as well as ad-hoc clustering techniques @xcite @xcite and probabalistic generative methods @xcite @xcite . semi-supervised event type induction was recently introduced by @xcite . zero-shot event extraction frameworks, such as @xcite , can be used to perform event extraction on the newly discovered types. recent work by @xcite uses a weakly-supervised contrastive learning-based clustering approach for event representation learning. several new unsupervised deep clustering approaches use contrastive loss for clustering images @xcite ) and text @xcite . these methods require data augmentation to create positive example pairs. contrastive loss has also been applied to learn representations. simclr @xcite uses image augmentations for unsupervised representation learning. follow-up work has applied this loss to natural language @xcite @xcite , with some augmentations being back-translated text @xcite . @xcite use fully supervised contrastive loss to finetune language models. batch attention has been investigated a little in the literature, such as for satellite imagery prediction @xcite or image classification @xcite ; however, it has not been used to learn clustered features. @xcite recently proposed a related idea for a messagepassing network weighted by attention for clustering images. we instead directly use (contrastive loss-enforced) attention weights for clustering. ## conclusion and future work in this work, we present an exciting new approach for event type induction, where we use contrastive loss to control the learning of a batch attention mechanism for both finding and learning about new cluster types. we also consider manifold approximation for clustering, and we introduce two new downstream tasks: type name prediction and framenet linking. this new approach opens several interesting problems for future work. first, this method can potentially be incorporated with reconstruction loss-based approaches, which might improve results or obviate the early stopping criterion. alternatively, the stopping criterion can be integrated into a loss function for better stopping control. notably, this would enable a two-step process of learning clusters and then performing knowledge distillation using those clusters (or an ensemble) while learning other desired losses. future work can investigate the interaction of manifold approximation with large language models and integrate it directly into the clusterer subnetwork. it may be possible to use a heirarchy of event types for knowledge transfer from data-rich types to long-tail types. finally, framenet linking can be extended to wikidata q-node linking, which contains millions of nodes. our approach may also be applicable in other modalities with strong pretrained models.