Efficient Zero-shot Event Extraction with Context-Definition Alignment (2022)

Abstract: Event extraction (EE) is the task of identifying interested event mentions from text.Conventional efforts mainly focus on the supervised setting. However, these supervised models cannot generalize to event types out of the pre-defined ontology. To fill this gap, many efforts have been devoted to the zero-shot EE problem. This paper follows the trend of modeling event-type semantics but moves one step further. We argue that using the static embedding of the event type name might not be enough because a single word could be ambiguous, and we need a sentence to define the type semantics accurately. To model the definition semantics, we use two separate transformer models to project the contextualized event mentions and corresponding definitions into the same embedding space and then minimize their embedding distance via contrastive learning. On top of that, we also propose a warming phase to help the model learn the minor difference between similar definitions. We name our approach Zero-shot Event extraction with Definition (ZED). Experiments on the MAVEN dataset show that our model significantly outperforms all previous zero-shot EE methods with fast inference speed due to the disjoint design. Further experiments also show that can be easily applied to the few-shot setting when the annotation is available and consistently outperforms baseline supervised methods.

## introduction event extraction, the task of identifying event mentions from documents and classifying them into pre-defined event types, is a fundamental nlp problem @xcite . as a centric information extraction task, event extraction is the foundation of a series of event-centric nlp applications @xcite including event relation extraction @xcite , event schema induction @xcite , and missing event prediction @xcite . traditional event extraction efforts @xcite @xcite mostly focus on learning to identify and classify events under a supervised learning setting, where a pre-defined event ontology and large-scale expert annotations is available. however, the learned supervised models cannot be easily applied to new event types out of the pre-defined ontology, limiting these models' usage in real applications. recently, large-scale pre-trained language models have demonstrated strong semantics representation capabilities and motivated a series of works to extract events in a zero-shot setting. for example, @xcite propose to manually design templates for each event type to convert the event extraction problem into a question-answering (qa) task and then leverage qa models to extract events. following that, @xcite propose to verbalize candidate triggers and event types into hypothesis and premises and leverage pre-trained textual entailment models to extract events. however, as analyzed in @xcite , these models heavily rely on the template design and often suffer from the domain-shifting problem between the original training task and the new task. moreover, as these models require jointly encoding the event mentions and event types, the time complexity is o(n * t ), where n is the number of event mention candidates and t is the number of event types. considering the low inference speed and high computation cost of inference with a deep model, such complexity could be a massive burden for real-time ee systems. to avoid manually designing templates and to improve the inference efficiency, another line of work @xcite tries to leverage pre-trained language representation models (i.e., bert @xcite ) to acquire a contextualized event type representation. the model can decouple the mention and label representations during the inference time and predict the candidate trigger to the most similar event type based on the cosine similarity. as a result, this method could significantly reduce the inference time complexity from o(n * t ) to o(n + t ). however, as the experiments show, using only the label name might not lead to a good event-type representation because the selected words could be ambiguous. in this work, we follow the trend of representation learning @xcite and move forward from representing each event type with a single name to a definition sentence. specifically, we propose a three-stage event representation learning framework. in the offline pretraining phase, we leverage auto-extracted contextdefinition alignments to learn a definition encoding model that can encode the contextualized mentions and definitions into the same embedding space. in the second warming phase, we use the target event types to retrieve hard negative examples to further polish the model. in the end, we identify and classify event mentions based on the cosine similarity between the mention representation and corresponding event-type representations. as our system is a disjoint model, the inference time complexity is also o(n + t ). experiments on maven @xcite , the largest ee dataset to the best of our knowledge, show that zed outperforms all previous zero-shot approaches with high inference efficiency. further experiments show that zed could also be applied to the supervised setting, where it achieves comparable performance in the fully supervised setting and consistently outperforms baseline supervised models in the data-scarce learning settings. specifically, with 10% of the training data, zed could achieve over 95% of the full performance. all the collected alignment data, created definitions, and the code are available at: https://github.com/tencentailab/zed. ## related works in this section, we introduce related works about event extractions, contrastive representation learning, and definition modeling. ## task definition we define the zero-shot event extraction task as follows. given a document in the format of a sentence set s and event type set e. each event type @xmath0. the task is to identify all mentions m e in s that satisfy the definition of e for each @xmath1. ## model we present the model framework in figure 2 . motivated by the success of the "pre-training + finetuning" learning paradigm, we propose to address the zero-shot event extraction problem with a threestage framework. technical details are as follows. ## experiments this section introduces experiment details, including the selected baseline methods, experiment datasets, and implementation details. ## zero-shot performance the zero-shot performance of all models is presented in ## warming with gold annotation zed can also be adapted to a fully supervised or few-shot learning setting when the annotation is available. specifically, during the warming phase of our model, we can replace the auto-retrieved examples with the annotated ones and fine-tune the model. in this section, we follow the benchmark paper @xcite the recent language model-driven baselines foot_3 : dm-bert @xcite and bert @xcite + crf @xcite , which also achieved the previous best performance. please refer to the original papers for technical details of these baseline models. we implement all models with the officially released code foot_4 and report the average performance of five trials on the development set. all models are trained for ten epochs, and the final model is evaluated. like the zero-shot setting, we also report the micro precision, recall, and f1 for both the "identification" and "identifica-tion+classification" settings. all hyper-parameters are based on the officially released code. results in however, we argue that such a large-scale annotation is often expensive in terms of money and time. the data-scarce learning setting might be more applicable in real applications. thus, we also test the performance of these supervised settings under the data-scarce learning setting. specifically, we randomly select 1% to 10% of the training sentences to be sampled from the training data and report the performances in figure 5 . our model can constantly outperform baseline models with a small number of annotations. especially when only 1% of the data is available, we only have 7.07 mentions per event type, zed could achieve over 50 f1. with 10% of the training data, zed could achieve over 95% of full supervised performance. these observations show that our framework could be applied to broader applications where limited or enough annotations are available besides the zero-shot setting. besides that, another interesting finding is that even though "bert+crf" could outperform "dmbert" slightly when enough annotation is available, which is consistent with the observations in @xcite , its performance is worse under the data-scarce setting. this observation indicates that using crf might not be the optimal option when the annotation scale is limited. ## conclusion this paper proposes a novel zero-shot event extraction framework zed. given a set of interested event types in the format of definitions, zed could automatically extract all the event mentions that fit the definitions from raw documents much better than previous methods. experiments show that the proposed warming phase and the mixed strong negative examples sampling strategies contribute to the success of zed. additional experiments also show that zed could be applied to the supervised setting. thanks to the pre-training phase, it could achieve good performance under both the fully supervised and data-scarce settings.