Zero-shot Event Extraction via Transfer Learning: C hallenges and Insights (2021)

Abstract: Event extraction has long been a challenging task, addressed mostly with supervised methods that require expensive annotation and are not extensible to new event ontologies. In this work, we explore the possibility of zero-shot event extraction by formulating it as a set of Textual Entailment (TE) and/or Question Answering (QA) queries (e.g. “A city was attacked” entails “There is an attack”), exploiting pretrained TE/QA models for direct transfer. On ACE-2005 and ERE, our system achieves acceptable results, yet there is still a large gap from supervised approaches, showing that current QA and TE technologies fail in transferring to a different domain. To investigate the reasons behind the gap, we analyze the remaining key challenges, their respective impact, and possible improvement directions.

## introduction event extraction (ee) has long been an important and challenging nlp task. figure 1 exemplifies a transfer-ownership event from the ace-2005 dataset (walker et al., 2006), where the trigger is "purchased" and the arguments include "china" (buyer), "russia" (seller), etc. the subtasks of ee involve identifying and classifying event triggers and their corresponding arguments. the predominant approaches normally require supervision (e.g. lin et al., 2020), which is both expensive and inflexible when moving to new event ontologies. recent works (chen et al., 2020; du and cardie, 2020) have pointed out the connection between question answering (qa) and ee in developing supervised systems. meanwhile, several efforts have explored unsupervised methods. peng et al. (2016) first attempted to extract event triggers with minimal supervision using similarity-based heuristics. huang et al. (2018) and lai et al. (2020) explored both trigger and argument extraction under a slightly different setting: training on some event types and testing on unseen ones. recently, liu et al. (2020) proposed a qa-based zero-shot argument extraction method, which did not handle triggers. so far, no method has been proposed to extract both event triggers and arguments without any ee training data foot_0 . moreover, the performance of existing zero-shot attempts, especially on arguments, is still far from satisfactory, yet little is known about possible underlying reasons. in this work, we investigate the possibility of zero-shot ee via transfer learning from textual entailment (te) and qa. observe that given pretrained te/qa models, extracting events can be viewed as answering questions/verifying hypotheses about a text. for example, the sentence in figure 1 , taken as the premise, would entail the hypothesis "there is a transfer of ownership", therefore providing the event type. then, by asking q 1 "who bought something?", we obtain "china" as the buyer. similarly, q 2 , q 3 will yield the seller and artifact, and so on. based on the observation above, we propose an intuitive zero-shot ee approach. it does not require any event training data, but we still make several design choices based on the development set. to demonstrate the level of generalization, we choose the optimal model with the ace development set, and evaluate it on both ace and ere (ldc2015e29) test sets. the performance surpasses previous zero-shot approaches on every subtask when the gold trigger span is given, yet is still unsatisfying compared to supervised methods, revealing a large gap in using off-the-shelf te/qa models for direct transfer. to shed light on why it is the case, we identify the key challenges behind the gap, and attribute each of them to the intrinsic weakness of pretrained models, our usage of them, or the task itself. we then anatomize their individual impact with an ablation study. our contributions are: (1) we propose the first te/qa-based event extraction system that tackles both triggers and arguments without any event training data; (2) we show that existing te/qa models do not support direct domain transfer well; and (3) we provide insights into the remaining challenges, their individual influence, and possible directions for future research. ## approach our pipeline consists of two modules, trigger extraction and argument extraction, both relying on pretrained te/qa models for direct transfer. the pretrained models we use are all bertbased (devlin et al., 2019; liu et al., 2019; lewis et al., 2020), including a te model trained on mnli (williams et al., 2018), a yes/no qa model trained on boolq (clark et al., 2019), and an extractive qa model trained on qamr (michael et al., 2018) and/or squad2.0 foot_1 (rajpurkar et al., 2018) foot_2 . the te model, when given a premise and a hypothesis, predicts the relation between them ("entailment", "contradiction", or "neutral"). the yes/no qa model takes as input a context and a yes/no question, and returns either yes or no. finally, the extractive qa model is also given a context but with a wh-question, and the answer is a span in the context. with these models, we design the two modules for event extraction. ## experimental setup we evaluate our system on the ace-2005 dataset. its event ontology has 7 types and 33 subtypes, and we evaluate t-ext directly on the subtypes. the same train/development/test split from lin et al. (2020) is used. we make several design choices foot_7 on the development and report results on the test, ignoring the training set. to demonstrate how our model generalizes, we also directly evaluate the optimal model on the ere dataset (ldc2015e29). to adapt to ere, we define a query for each new event type. there are four subtasks of event extraction: trigger identification (ti), trigger classification (tc), argument identification (ai), and argument classification (ac). we experiment under three settings: scratch, where the system performs all subtasks without any gold annotation; gold ti, where gold trigger spans are given; gold tc, where gold trigger spans and types are given @xcite . following ji and grishman (2008), precision, recall, and f1 are used for evaluation foot_9 . we evaluate argument spans on the head level, consistent with most prior work (huang et al., 2018; wadden et al., 2019; lin et al., 2020; zhang et al., 2021). ## results as shown in ## analysis using the results on ace, we now present an analysis of the remaining core challenges of the task, along with an ablation study on their individual impact. to further understand the challenges, we attribute each to the fragility of the pretrained models (m-error), our usage of the models (u-error), or the task itself (t-error). ## conclusions we propose the first complete zero-shot event extraction system via transfer learning from te and qa. while qa/te models perform exceptionally well on standard benchmarks (squad, qamr, mnli), they do not generalize as expected when being used on ee datasets. we analyze the limited success and several main challenges of the current approach, and provide insights for future improvements. chapter of the association for computational linguistics: human language technologies, volume 1 (long papers), pages 1112-1122, new orleans, louisiana. association for computational linguistics. wenpeng yin, jamaal hay, and dan roth. 2019. benchmarking zero-shot text classification: datasets, evaluation and entailment approach. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 3914-3923, hong kong, china. association for computational linguistics. hongming zhang, haoyu wang, and dan roth. 2021. unsupervised label-aware event trigger and argument classification. in proceedings of the 59th annual meeting of the association for computational linguistics (acl) findings.