NEXT : An Event Schema Extension Approach for Closed-Domain Event Extraction Models (2023)

Abstract: Event extraction from textual data is a NLP research task relevant to a plethora of domains. Most approaches aim to recognize events from a predefined event schema, consisting of event types and their corresponding arguments. For domains, such as disinformation, where new event types emerge frequently, there is a need to adapt such fixed event schemas to accommodate for new event types. We present NEXT (New Event eXTraction) - a resource-sparse approach to extending a close-domain model to novel event types, that requires a very small number of annotated samples for fine-tuning performed on a single GPU. Furthermore, our results suggest that this approach is suitable not only for extraction of new event types, but also for recognition of existing event types, as the use of this approach on a new dataset leads to improved recall for all existing events while retaining precision.

## introduction event extraction from text is a research task with applications in a wide range of domains @xcite , including finance @xcite , social @xcite , biomedical @xcite and security @xcite . the goal of the event extraction task is to determine the event type, extract the trigger -the most relevant word to the event, as well as any event arguments -other words and phrases relevant to the event @xcite . this is often approached as a closed-domain problem where the model aims to detect events from a predefined event schema consisting of a fixed set of event types and their corresponding argument types @xcite . in contrast, when the set of event types is not fixed or is not completely known at the onset, an opendomain approach is more suitable @xcite . we explore the task of event extraction within the field of fake news and disinformation as a closed-domain problem. nonetheless, the highly dynamic nature of the field implies that a methodology for easy extension of an existing closed domain event extraction approach to new event types is necessary. ideally, such methodology would perform well with a small number of annotated samples, as producing a large annotated dataset for each newly emerging event type would be a very long and expensive process. in this paper, we present a work-in-progress methodology which satisfies the requirements mentioned above. we select an existing model and extend it for a novel event type identified in fake news debunks with minimal resources when it comes to annotated data. we present how we define and annotate a new event, followed by how we fine-tune an existing model. next, we provide a detailed analysis of how well the model learns the new event type, as well as how well it retains the ability to predict the event types for which it was previously trained. ## related work event extraction is a widely studied topic and many different approaches towards it exist. @xcite identify two main paradigms to solving the event extraction task -the pipeline paradigm, where event type, trigger and argument classification are done in sequence @xcite @xcite , and the joint paradigm, where event and arguments are classified simultaneously @xcite @xcite . the latter paradigm prevents error propagation from one classification sub-task to the next. other notable approaches to event extraction are as a classification task @xcite , question answering task (or machine reading comprehension task) @xcite @xcite , sequence labelling task @xcite or sequence-to-structure generation task @xcite . another interesting approach to the event detection task is the presented in peng et al. (2023) reinforcement learning one. @xcite point out that most event extraction methods, among them pipeline and joint paradigm approaches, apply a decomposition strategy where event extraction sub-tasks are solved independently and their results are then combined to predict the whole event entity. this strategy has some drawbacks, such as the need for annotations for different sub-tasks and the difficulty of composing an optimal architecture for different sub-tasks. @xcite addresses both of these by modelling all sub-tasks in a uniform sequenceto-structure generative model called text2event, which extracts events from a text in an end-to-end manner. another advantage of the model is being able to easily transfer to new event types. in our study we aim to find a transferable lowresource solution to event extraction, such that it adapts well to new corpora and new event types with small amounts of annotated data and can be run on a single gpu. while there are other approaches to event extraction with little annotated data such as semi-supervised @xcite , few-shot @xcite and zero-shot @xcite @xcite ) learning, we chose the text2event model for its reported high performance in both supervised and transfer learning settings. for these purposes we extend the text2event model @xcite with a novel event type by fine-tuning it on a small annotated sample set and then evaluate how well the model retains its performance on its original event types on a novel dataset of fake news debunks. ## discussion our proposed approach next to extend an existing event schema with new event types has a few advantages, but also limitations. a notable advantage of this approach is that a dozen annotated samples are sufficient for achieving high precision given a sufficient number of fine-tuning epochs. learning good recall, however, is a more challenging task and requires a larger number of samples -about 50. fine-tuning the model also does not require significant computational resources. all reported experiments were performed on a single cudaenabled gpu. each fine-tuning of an individual model took a few minutes. as expected, we observe that fine-tuning for many epochs leads to overfitting on the new event type. namely, the precision of predictions for the new event type decreases with a larger number of epochs, while the number of predictions grows simultaneously. this problem can be mitigated by pre-filtering the sentences or claims on which the model is used, with a rule as simple as checking whether they contain likely triggers for the event type (e.g. "cure", "treat", "heal" for cure-claim events), as seen in figure 5 . another solution would be to adopt a voting approach by considering only predictions made by both next100 and next500 models (or majority rule if a third finetuned model is used as well) as predictions shared between models tend to be more accurate compared to predictions made by individual models (table with comparison is available in the appendix). despite the large number of fine-tuning epochs for the new event type, this approach retains the model's capability of predicting existing event types. we showed that the majority of such predictions made by the baseline text2event model are also made by the fine-tuned models. furthermore, the fine-tuned models generate two to four times as many predictions as the baseline model. this has only a minor effect on precision -a small drop in performance compared to the baseline model can be observed in we attribute this rise in recall to the baseline model not having been trained or fine-tuned on samples from our claim debunks dataset. though this dataset consists of texts from the same do-main as ace2005 (news media / publishing), the datasets differ on other parameters such as the average number of sentences per text. we observe that a few annotated samples for fine-tuning on one event type are sufficient to boost recall of all other event types. ## conclusion and further work in this work we presented an approach to extend an existing event schema with new event types for closed-domain event extraction. our approach uses a very small number of annotations containing full event structures (event type, trigger and arguments are all annotated). the proposed approach also leads to improvement in the recall of existing event types, on which the model was pre-trained while retaining precision. it can thus be used not only to fine-tune the event extraction model for a new event type but to also simultaneously fine-tune the model for the existing event types on a new dataset without the need for annotation for all event types. an interesting direction for future research would be evaluating whether this boost in performance would also be observed when the task is transferred to a dataset from a less related domain, e.g. biomedical, manufacturing, energy, etc. further pre-training might also be of interest. in terms of evaluation, it would be interesting to explore how our proposed approach compares to alternatives, such as open-domain approaches. also, more documents from the initial dataset could be annotated for the original event types, in order to obtain a clearer picture of the baseline's model performance on them.