EDM 3: Event Detection as Multi-task Text Generation (2024)

Abstract: We present EDM3, a novel approach for Event Detection (ED) based on decomposing and reformulating ED, and fine-tuning over its atomic subtasks. EDM3 enhances knowledge transfer while mitigating prediction error propagation inherent in pipelined approaches. EDM3 infers dataset-specific knowledge required for the complex primary task from its atomic tasks, making it adaptable to any set of event types. We evaluate EDM3 on multiple ED datasets, achieving state-of-the-art results on RAMS (71.3% vs 65.1% F1), and competitive performance on WikiEvents, MAVEN (∆ = 0.2%), and MLEE (∆ = 1.8%). We present an ablation study over rare event types (<15 instances in training data) in MAVEN, where EDM3 achieves ~90% F1. To the best of the authors’ knowledge, we are the first to analyze ED performance over non-standard event configurations (i.e., multi-word and multi-class triggers). Experimental results show that EDM3 achieves ~90% exact match accuracy on multi-word triggers and ~61% prediction accuracy on multi-class triggers. This work establishes the effectiveness of EDM3 in enhancing performance on a complex information extraction task.

## introduction event detection (ed) involves characterizing events occurring in unstructured text, by recognizing their event triggers and classifying their event types. ed is used extensively for downstream tasks such as information retrieval @xcite , event prediction @xcite , and argument detection @xcite . existing methods for ed @xcite cannot easily leverage pre-trained semantic knowledge (lai et al., 2020; @xcite , failing to identify complex events or function in low-resource scenarios @xcite . additionally, they lack the ability to generalize across domains such as biomedicine or cybersecurity. @xcite . as a result, they may handicap comprehensive event extraction @xcite . 2 to overcome these challenges, we propose edm3 (event detection by multi-task text generation over three subtasks), a novel approach based on decomposing an intricate primary task (ed) into its constituent atomic subtasks. we hypothesize that these subtasks (ei, ec) are less reliant on domain-specific knowledge than on semantic similarities @xcite , and hence simpler to learn. edm3 involves training on these subtasks simultaneously in a non-pipelined, multi-task fashion. this diverges from the traditional discriminative token classification paradigm (fig. 1 ). unlike concurrent works such as instructuie @xcite and @xcite , which propose a unified model over multiple disparate language tasks, edm3 focuses on a single complex task. this approach thus provides a framework adapt- approaches datasets tasks covered domain generalization comparative performance identification classification detection liu et al. (2022) ace, maven ✗ ✗ ✓ ✗ sota on maven veyseh et al. (2021) ace, rams, cyseced ✓ ✗ ✓ ✓ sota on ace and cyseced competitive on rams he et al. (2022) mlee ✗ ✗ ✓ ✗ sota on mlee edm3 (ours) maven, mlee wikievents, rams ✓ ✓ ✓ ✓ sota on rams competitive on mlee & maven benchmark on wikievents to evaluate edm3, we conduct extensive experiments on rams, wikievents, maven, and mlee datasets. edm3 achieves an f1 score of 71.3% on rams, surpassing the sota score by 6.2% points. edm3 also achieves a competitive macro f1 score of 60.1% on maven, compared to 60.3% (sota). we benchmark ed performance on wikievents with 60.7% f1 score. finally, edm3 achieves a competitive result of 78.1% f1 against 79.9% (sota) on the biomedical mlee dataset. while other approaches use domain-specific embeddings and hand-crafted features, edm3 uses a vanilla t5 model to obtain these results, supporting our hypothesis. we conduct investigations along multiple lines of inquiry to explore the efficacy of edm3. we observe that our multi-tasking approach improves ed performance by 3-6%. we explore the efficacy of edm3 in low-resource scenarios (evaluating rare event types). experimental results reveal scores of ∼ 90% f1 achieved over rare event types. we also evaluate its performance over multi-word and multiclass triggers, which while lacking in benchmark datasets, are common in real-world data. edm3 achieves ∼ 90% exact match accuracy on multiword triggers and ∼ 61% prediction accuracy on multi-class triggers. finally, we discuss the importance of multi-sentence context. in summary, our contributions are as follows: ## proposed method given an input instance containing diversely-typed event triggers, we aim to capture all triggers present. we reformulate ed and its subtasks as sequence generation tasks. we use instructional prompts to train a model on all 3 generative tasks jointly to create a single multi-task model. task decomposition ed is a multi-level task requiring both event identification and classification, which sequence labeling approaches conduct in a single step. we manually decompose ed into independent tasks to be carried out in parallel with the primary task, to augment the training process. ## conclusion in this paper, we propose edm3, a domainagnostic generative approach to the event detection task. edm3 leverages a multi-tasking strategy that incorporates instructional prompts to improve model performance on imbalanced data and complex event instances. our analysis shows an improvement in f1 score over single-task performance, supporting our main hypothesis viz. the effectiveness of breaking down complex generation tasks into subtasks that can support model learning on the primary task. furthermore, our results highlight the potential for generative models in traditionally discriminative tasks like ed, paving the way for future advancements in the field.