Shorten the Long Tail for Rare Entity and Event Extraction (2023)

Abstract: The distribution of knowledge elements such as entity types and event types is long-tailed in natural language. Hence information extraction datasets naturally conform long-tailed distribution. Although imbalanced datasets can teach the model about the useful real-world bias, deep learning models may learn features not generalizable to rare or unseen expressions of entities or events during evaluation, especially for rare types without sufficient training instances. Existing approaches for the long-tailed learning problem seek to manipulate the training data by re-balancing, augmentation or introducing extra prior knowledge. In comparison, we propose to handle the generalization challenge by making the evaluation instances closer to the frequent training cases. We design a new transformation module that transforms infrequent candidate mention representation during evaluation with the average mention representation in the training dataset. Experimental results on classic benchmarks on three entity or event extraction datasets demonstrates the effectiveness of our framework.

## introduction long-tailed distributions are common in natural language processing tasks. this natural phenomenon of long-tailed distribution is formulated as zipf's law @xcite . for information extraction, knowledge elements such as entities, relations and events typically conform a long-tailed distribution in natural language. this leads to the imbalanced distribution of types in the benchmark datasets unless manual manipulation is performed to balance the dataset. we show the entity type distribution of an entity extraction dataset (few-nerd @xcite ), and the event type distribution of two event extraction datasets (ace 2005 @xcite and maven @xcite ) in figure 1 . all three datasets have similar long-tailed distributions. among them, maven and few-nerd are two relatively largescale datasets, but the training mentions are still concentrated on a small number of frequent types and majority of the entity/event types are rare types without sufficient training examples. long-tailed distribution is a generic problem not limited to nlp research. in fact, a lot of previous work on the long-tailed learning is in the computer vision domain @xcite . these approaches consider the imbalance in the type distribution as the major problem and are mostly based on balancing the datasets by upweighting the rare types and downweighting the frequent types. however, we argue that this line of research faces significant challenges when applying to the information extraction task. first, balancing the dataset breaks the real-world long-tailed prior in the datasets and may lead the model to mistakenly predict the long-tailed types too often. we observe this phenomenon in our experiments in section 3 especially for the classifier re-training baseline @xcite . second, due to the lack of effective data augmentation approaches as in the computer vision domain, upweighting the training mentions will only make the model to repetitively learn the same set of instances and aggravate the overfitting problem. instead of the imbalance in distribution, the real challenge of long-tailed learning for nlp is knowledge insufficiency, i.e. long-tailed datasets don't contain enough examples to acquire generalizable knowledge for the rare types. learned models tend to capture features such as event trigger words for rare types and fail to generalize well to rare or even unseen cases (unseen trigger words) of those types during evaluation. for instance, if an event extraction model is trained on a dataset where all acquit events are triggered by the keyword acquit, it may not identify the acquit event triggered by walk free in the adjudicator allowed the criminal to walk free. under this perspective, existing work tackling the long-tailed problem in information extraction @xcite incorporates external structural knowledge to help learning rare types. however, the selection of the structural knowledge requires expertise in the target ontology and is not easily transferable to other ontologies. another more recent line of work @xcite considers the causal inference @xcite approach to solve the long-tailed problem. they aim to avoid the learning of spurious correlations between the input features and the labels in the limited training data for rare types. this is usually achieved by removing the effect of a manually selected confounding factor in prediction through deconfounded learning methods such as backdoor adjustment and inference based on total direct effect @xcite . although the model is guided to avoid spurious correlation through causalinference-based approaches, learning the generalizable features with real causalities is still challenging due to the knowledge insufficiency problem. in this work instead of confronting the knowledge insufficiency problem directly, we propose to bypass the problem by transforming the evaluation instances that require additional knowledge into more familiar instances that are closer to the frequent training instances. we decompose the input sentence {t 0 , t 1 , . . . , t n } for the event/entity prediction task into two parts: a candidate token @xmath0. instead of making predictions for an event/entity type y k solely based on p (y k |e, c), we propose to also consider predictions on averaged training inputs p (y k |c, r k ), where r k is the average training hidden representation of the type y k . our framework combines these two predictions using a transformation module. the transformation module computes weights g e for the original candidate token e and g k for each event type representation r k . moreover, instead of simply combining probabilities as g e p (y k |e, c) + g k p (y k |c, r k ), we use two weights g e and g k to combine in the representation layer for e and r k to produce p (y k |g e e + g k r k , c). the transformation module decides the weights g e and g k based on the frequency of the candidate mention in the training dataset and the cosine similarity between the candidate mention representation and all event types' average training mention representations in the ontology. experimental results on three benchmark datasets demonstrate the effectiveness of our framework. additionally, we found our approach, though derived from a different motivation, can be interpreted as a backdoor adjustment approach @xcite as shown in section 2.5, which gives another interpretation that our approach improves long-tail learning by facilitating the model to learn the true correlation between the candidate text span and the event/entity type. we also empirically found that our approach facilitates the model to capture more generalizable features, which aligns with the goal of causal inference approaches. to summarize, we propose a new approach for learning from the long-tailed datasets for entity extraction and event extraction. our contributions are two-fold: ## conclusions and future work in this work we propose a new long-tailed learning framework for entity and event extraction by candidate transformation. we design a novel transformation module to convert representations of rare or unseen mentions during evaluation into representations of average training mentions. experimental results have validated the effectiveness of our framework. our framework can significantly improve the performance on long-tailed types, and outperform other long-tailed learning methods especially for moderately long-tailed types. moreover, our framework does not sacrifice the performance on frequent types. we also discover the connections between our learning framework and the backdoor adjustment in the causal inference theory we empirically observe that our training strategy can improve the model's capability in capturing more generalizable features, which aligns with the causal inference theory. in the future, we will explore: (1) adapt the concept of transformation module to other nlp tasks;