Few-shot Event Detection: An Empirical Study and a Unified View (2023)

Abstract: Few-shot event detection (ED) has been widely studied, while this brings noticeable discrepancies, e.g., various motivations, tasks, and experimental settings, that hinder the understanding of models for future progress. This paper presents a thorough empirical study, a unified view of ED models, and a better unified baseline. For fair evaluation, we compare 12 representative methods on three datasets, which are roughly grouped into prompt-based and prototype-based models for detailed analysis. Experiments consistently demonstrate that prompt-based methods, including ChatGPT, still significantly trail prototype-based methods in terms of overall performance. To investigate their superior performance, we break down their design elements along several dimensions and build a unified framework on prototype-based methods. Under such unified view, each prototype-method can be viewed a combination of different modules from these design elements. We further combine all advantageous modules and propose a simple yet effective baseline, which outperforms existing methods by a large margin (e.g., 2.7% F1 gains under low-resource setting).

## introduction event detection (ed) is the task of identifying event triggers and types in texts. for example, given "cash-strapped vivendi wants to sell universal studios", it is to classify the word "sell" into a transferownership event. ed is a fundamental step in various tasks such as successive event-centric information extraction @xcite @xcite , knowledge systems @xcite , story generation @xcite , etc. however, the annotation of event instances is costly and labor-consuming, which mo- † corresponding author. 1 our code will be publicly available at https://github.com/mayubo2333/fewshot_ed . tivates the research on improving ed with limited labeled samples, i.e., the few-shot ed task. extensive studies have been carried out on fewshot ed. nevertheless, there are noticeable discrepancies among existing methods from three aspects. in this paper, we argue the importance of a unified setting for a better understanding of few-shot ed. first, based on exhaustive background investigation on ed and similar tasks (e.g., ner), we con- the second contribution is a unified view of prototype-based methods to investigate their superior performance. instead of picking up the best-performing method as in conventional empirical studies, we take one step further. we break down the design elements along several dimensions, e.g., the source of prototypes, the aggregation form of prototypes, etc. from this perspective, five prototype-based methods on which we conduct experiment are instances of distinct modules from these elements. and third, through analyzing each effective design element, we propose a simple yet effective unified baseline that combines all advantageous elements of existing methods. experiments validate an average 2.7% f 1 gains under low-resource setting and the best performance under class-transfer setting. our analysis also provides many valuable insights for future research. ## preliminary event detection (ed) is usually formulated as either a span classification task or a sequence labeling task, depending on whether candidate event spans are provided as inputs. we brief the sequence labeling paradigm here because the two paradigms can be easily converted to each other. given a dataset d annotated with schema e (the set of event types) and a sentence @xmath0, ..., x n ] @xmath1, where x i is the i-th word and n the length of this sentence, ed aims to assign a label y @xmath2.a.}) for each x i in x. here n.a. refers to either none events or events beyond pre-defined types e. we say that word x i triggering an event y i if y @xmath3. ## a prototype-based unified view due to the superior performance (sections 5 and 6), we zoom into prototype-based methods to provide a unified view towards a better understanding. we observe that they share lots of similar components. as shown in ## results: class-transfer learning in this section, we evaluate existing methods and the unified baseline under class-transfer setting. here we do not consider in-context learning because previous expetiments show it still lags far from both prompt-and prototype-based methods. ## conclusion we have conducted a comprehensive empirical study comparing 12 representative methods under unified low-resource and class-transfer settings. for systematic analysis, we proposed a unified framework of promising prototype-based methods. based on it, we presented a simple and effective baseline that outperforms all existing methods significantly under low-resource setting, and is an ideal choice as the target model under class-transfer setting. in the future, we aim to explore how to leverage unlabeled corpus for fewshot ed tasks, such as data augmentation, weaklysupervised learning, and self-training.