Textual Entailment for Event Argument Extraction: Zero- and Few-Shot with Multi-Source Learning (2022)

Abstract: Recent work has shown that NLP tasks such as Relation Extraction (RE) can be recasted as a Textual Entailment tasks using verbalizations, with strong performance in zero-shot and few-shot settings thanks to pre-trained entailment models. The fact that relations in current RE datasets are easily verbalized casts doubts on whether entailment would be effective in more complex tasks. In this work we show that entailment is also effective in Event Argument Extraction (EAE), reducing the need of manual annotation to 50% and 20% in ACE and WikiEvents, respectively, while achieving the same performance as with full training. More importantly, we show that recasting EAE as entailment alleviates the dependency on schemas, which has been a roadblock for transferring annotations between domains. Thanks to entailment, the multi-source transfer between ACE and WikiEvents further reduces annotation down to 10% and 5% (respectively) of the full training without transfer. Our analysis shows that key to good results is the use of several entailment datasets to pre-train the entailment model. Similar to previous approaches, our method requires a small amount of effort for manual verbalization: only less than 15 minutes per event argument types is needed; comparable results can be achieved from users of different level of expertise.

## introduction building information extraction (ie) systems for real-world applications is very costly and has suffered from data-scarcity problems, due in part to the expertise and time required to annotate training data at a large scale with sufficient consistency, but also due to poor transfer between domains: ie annotations depend on the schema used in each domain, and moving to new domains requires new schemas, new annotation guidelines and the manual annotation of new data. in many cases, there is some information overlap between schemas, but performing transfer learning to leverage such overlap (i.e. learning from multiple sources) can be difficult: it often requires manually mapping labels between schemas, which is typically brittle, cumbersome and requires costly domain expertise @xcite . in order to save annotation effort, recent work recasts ie tasks as textual entailment tasks @xcite @xcite . for instance, @xcite manually verbalize each relation type in the relation extraction (re) dataset tacred @xcite to generate hypotheses for each test example, and then apply an entailment model to output the relation type of the hypothesis with highest entailment probability. the entailment model is typically based on large language models pre-trained on entailment datasets such as mnli @xcite . the approach obtains very strong results on zero-shot and few-shot scenarios, but we note that tacred contains relations between two entities that are easily verbalizable, foot_0 casting doubts on whether entailment would be effective in more complex ie tasks. event argument extraction (eae) involves more complex contexts, higher ambiguity in the words that trigger events, and depends on the event type in addition to the relation (see figure 1 ). in this work, we present the first system for eae that addresses the task as an entailment problem. we empirically show the robustness of the method on the zero-shot, few-shot and full training regimes, obtaining state-of-the-art results on ace @xcite and wikievents @xcite . in addition, we make the following contributions: prompt and pivot task based learning has emerged as a candidate solution for data-scarcity problems @xcite @xcite . the use of discrete @xcite or continuous @xcite prompts allowed language models to perform significantly better on many text classification tasks. closely related to our approach, several works make use of a highresource supervised task such as question answering or entailment as pivot tasks @xcite @xcite @xcite . in the case of entailment, @xcite converted qa data to entailment manually and @xcite did it automatically. other semantic tasks such as named entity recognition, relation extraction and semantic role labelling have also been reformulated as entailment by automatically converting data into the entailment format @xcite @xcite . multi-task learning reformulates multiple tasks to a single and common task via prompting large pre-trained language models, leveraging multiple data sources to improve each task of interest. such approaches have shown improvements in supervised @xcite @xcite and zero-shot scenarios @xcite . while using the language modelling task as a pivot shows strong performance with very large language models, it is not clear that smaller models can benefit from this strategy in the same way. @xcite and @xcite obtained contradictory results. in a similar way, question answering has been proposed as a pivot task for multi-task learning but without promising results @xcite . in this work, we explore multi-source learning, where datasets from different or similar tasks are used to build a model for the target task. event argument extraction is a sub-task of event extraction. the goal is to identify arguments or fillers for a specific slot (a.k.a., role) in an event template. this task has been largely explored on the message understanding conference (muc, @xcite ) and later on automatic content evaluation (ace). ace focused mainly on sentence level evaluation due to the difficulty of the task at the time. recently, new benchmarks such as rams @xcite and wikievents have emerged with the aim of addressing document level information extraction similar to muc. however, most of the interest is still focused on the sentence level. eae has been recently addressed by end-to-end event extraction models @xcite @xcite , instead of treating it as an independent task @xcite ), as we do, or as a subtask in a pipeline @xcite . lately, with the recent paradigm shift to prompt design learning @xcite , several works reformulated the task as a question answering problem @xcite @xcite @xcite @xcite or as a constrained text generation problem @xcite @xcite using predefined prompts, questions or templates. we instead reformulate the task as a textual entailment problem. ## approach in order to cast eae as an entailment task, we verbalize event argument instances using a set of intuitive and linguistically motivated templates to capture the event argument roles, and then per-figure 1 : entailment-based event argument extraction. on the left, input information: the context, the event trigger (hired) and the argument candidate (john d. idol), alongside the types of both. on the middle, some hypothesis verbalized using the templates: the green box is entailed, the yellow box matches the type constraint but it is not entailed, and the rest do not satisfy type constraints. on the right, the output with the inferred role (person). label verbalization is attained using templates that combine the information of the instance and express a specific label. different role verbalizations are shown in figure 1 . a verbalization is generated using templates that have been manually written based on the task guidelines of each dataset. the templates involve the candidate argument, and optionally the event trigger. in some cases, in order to produce a grammatical hypothesis, placeholders corresponding to the agent or theme are also introduced, which can be generic, e.g. someone, or dependent of the argument role, e.g. defendant. we defined several template types (see ## entailment for multi-source learning we hypothesize that two similar ie tasks can benefit from each other even if they do not share the same schema or domain. although this hypothesis is very intuitive and it has been demonstrated on several works for tasks other than ie (see multitask learning on section 2), actual ie models are limited by schema dependency, which makes it almost impossible to learn from datasets annotated with different ie schemas. one option is to perform a manual mapping between schemas, which is costly and often inaccurate @xcite . our approach instead is domain and schema agnostic, and therefore allows to learning from multiple sources seamlessly. given that the sources are recast into a single format in a common entailment formulation, it suffices to fine-tune the model in sequence across the sources. to check our hypothesis we split tasks according to the following criteria: (1) ie sources like relation extraction that are different from eae (e.g. tacred), and (2) eae sources using different schemas (e.g. wikievents and ace). figure 2 summarizes the tasks and datasets used in this work, including the four natural language understanding datasets. ## experimental setup in this section, we describe the methodology for template development, evaluation setting, the baselines used in our experiments, and the computation infrastructure specifications. ## analysis after performing the main experiments we did some additional analysis. ## conclusions this paper shows the entailment-based approach for event argument extraction is extremely effective in zero-shot, few-shot and full train scenarios both on ace and wikievents, outperforming previous methods. first of all, recasting eae as an entailment task allows it to reuse annotations from different event schemas, achieving large gains when transferring annotations between ace and wikievents, and also some gains in the zero-shot performance when transferring annotations from a relation extraction model such as tacred. secondly, we show that using additional training entailment datasets improves results significantly over just using mnli, not only on eae but also on ta-cred. thirdly, we show that the relatively short time spent writing manual templates is much more effective than the time spent on doing annotations, with a sweet spot where the annotation effort is split between the two, with large savings in manual labour. lastly, we show that an independent linguist is able to write templates with comparable performance without any special training. we think that our results and analysis support the potential of entailment models for other nlp tasks. our work paves the way for a new paradigm for ie, where the expert defines the schema using natural language and directly runs those specifications, annotating a handful of examples in the process, and allowing for quick trial-and-error iterations. @xcite propose a user interface alongside this paradigm. more generally, inference capability could be extended, acquired and applied from other tasks, in a research avenue where entailment and task performance improve in tandem.