ZS 4 IE : A toolkit for Zero-Shot Information Extraction with simple Verbalizations (2022)

Abstract: The current workflow for Information Extraction (IE) analysts involves the definition of the entities/relations of interest and a training corpus with annotated examples. In this demonstration we introduce a new workflow where the analyst directly verbalizes the entities/relations, which are then used by a Textual Entailment model to perform zero-shot IE. We present the design and implementation of a toolkit with a user interface, as well as experiments on four IE tasks that show that the system achieves very good performance at zero-shot learning using only 5–15 minutes per type of a user’s effort. Our demonstration system is open-sourced at https://github.com/BBN-E/ZS4IE . A demonstration video is available at https://vimeo.com/676138340 .

## introduction information extraction (ie) systems are very costly to build. the current define-then-annotate-andtrain workflow uses supervised machine learning, where the analyst first defines the schema with the entities and relations of interest and then builds a training corpus with annotated examples. unfortunately, each new domain and schema requires starting from scratch, as there is very little transfer between domains. we present an alternative verbalize-whiledefining workflow where the analyst defines the schema interactively in a user interface using natural language verbalizations of the target entity and relation types. figure 1 shows sample verbalization templates for a simple schema involving an employee relation and a passing away event, as well as a sample output annotated with the schema. the annotation of the employeeof relation requires performing named entity recognition (ner) @xcite and relation * denotes equal contribution. extraction (re) @xcite , while annotating the life.die event involves ner, event extraction (ee), and event argument extraction (eae) @xcite . our toolkit is able to perform those four ie tasks using a single user interface, allowing the analyst to easily model and test the schema without the need to annotate examples. our toolkit leans on recent work which has successfully recast several ie tasks as textual entailment (te) tasks @xcite @xcite . for instance, @xcite model relation types between entity pairs using type-specific verbalization templates that describe the relation, generates a verbalization (hypothesis) automatically using those templates and then uses a pre-trained te model to predict if the premise (the sentence where the pair appears) entails the hypothesis, therefore leading to a prediction of the relation or "no relation". in this paper we thus present zs4ie, a toolkit for zero-shot ie. we show that the four mainstream ie tasks mentioned above can be reformulated as te problems, and that it is possible to achieve strong zero-shot performances leveraging pre-trained te models and a small amount of templates curated by the user. our toolkit allows a novice user to curate templates for each new types of entities, relations, events, and event argument roles, and validate their effectiveness online over any example. we also present strong results on widely used datasets with only 5-15 minutes per type of a user's effort. ## related work textual entailment has been shown to be a reasonable proxy for classification tasks like topic or sentiment analysis @xcite @xcite . to reformulate a classification problem as te, it often starts with defining templates to describe each class label, leading to a natural language text (a "verbalization" of a hypothesis) for each possible label. inference is performed by selecting the most probable candidate hypothesis entailing the premise. te is usually implemented with pre-trained language model finetuned on te datasets, such as mnli @xcite , @xcite , fever @xcite , anli @xcite or xnli @xcite . the results on classification have been particularly strong for zero-shot and few-shot learning, with @xcite hypothesizing that entailment is a true language understanding task, where a model that performs entailment well is likely to succeed on similarlyframed tasks. @xcite reformulated relation extraction as a te task surpassing the state-of-the-art in zero-and few-short learning. a similar approach was previously explored by obamuyide and vlachos (2018), using te models that are not based on pre-trained language models. similar to te, @xcite performs yes/no question answering, in which a model is asked about the veracity of some fact given a passage. @xcite recast the zero-shot event extraction as a te task, using te model to check whether a piece of text is about a type of event. lastly, @xcite showed that te allows to leverage the knowledge from other tasks and schemas. ## ie via textual entailment we first describe how to recast each of the ie tasks (ner, re, ee, eae) as te independently, and leave the workflow between the tasks for the next section. at a high level, the zero-shot te reformulation consists of three steps: candidate generation, label verbalization and te inference (figure 2 illustrates the steps for ner). the first step, candidate generation, identifies text spans (e.g., proper nouns for ner) or span pairs (a pair of entity mentions for relation extraction) in the input sentence as the focus of the prediction. taking a text span (or span pair) as input, the label verbalization step applies a verbalization template to generate a hypothesis, which is a natural language sentence describing the span (or span pair) being an instance of a type of entity, relation, event, or event argument. the verbalization generates hypothesis for each of the target types. finally, the te inference step takes the original sentence (the premise) and each hypothesis as input, and uses a pre-trained te model to predict if the premise entails, contradicts, or is neutral to the hypothesis. the type with the verbalization having the highest entailment probability is selected. we next describe each step in detail. ## zs4ie toolkit zs4ie comprises a pipeline and a user interface. ## experiments we evaluated our system using publicly available datasets. we use conll 2003 @xcite for ner evaluation, ta-cred @xcite for re, and ace for ee and eae @xcite . we evaluate each task independently (not as a pipeline) to make as comparable as possible to existing zero-shot systems. in order to apply our toolkit we made some adaptations as follows: we consider only proper nouns as candidates for ner, and we ignore the misc label because it is not properly defined in the task foot_0 . we evaluate ee as event classification, where the task is to output the events mentioned in the sentence without extracting the trigger words, as we found that deciding which is the trigger word is in many cases an arbitrary decision foot_1 . in the case of re we used the templates from @xcite , which are publicly available. we will release the templates used on the experiments as additional material along with the paper. the analysts spent between 5-15 minutes per type, depending on the task, with ner and ee being the fastest. the results show strong zero-shot performance. note that there is no best entailment model, suggesting that there still exists margin for improvement. however, we see that roberta* performs relatively well in all scenarios except ee (see section 6 for further discussion). the table also shows in the middle three rows the results where we optimize the threshold on development. the results improve in most of the cases, and allow comparison to other zero-shot systems which sometimes optimize a threshold in development data. furthermore, we compare our system with zeroshot task specific approaches from other authors when available. for re, @xcite propose a text-to-triple translation method that given a text and a set of entities returns the existing relations. for ee, @xcite to us, the use of an entailment model, but in their case the input sentence is split in clauses according to the output of a semantic role labelling system. in order to compare their results with ours, we only use the event types, not the trigger information 3 . the results from our system can be seen as an ablation where we do not make use of any srl preprocessing. for eae, @xcite perform zero-shot eae by recasting the task as qa. some of these approaches also optimize a threshold on development data, although it is not always clear. we show that our toolkit with default threshold obtains excellent results despite being an all-in-one method. ## discussion towards post-editing on ie. our internal evaluation suggest that verbalizing-while-defining workflow can have similar impact as post-editing machine translated text, where human translators obtain quality translations with less effort @xcite . the idea of this new framework will bring down the effort required to create larger and higher quality datasets. current ie system are subject to a predefined schema and are useless to classify new types of entities, relations and events. the use interface of zs4ie brings to the annotators the opportunity of defining the schema interactively and manually annotating the dataset with the help of the entailment model. in the future we would like to use the manual annotations to fine-tune the te model, which would further improve the performance, as shown by the excellent few-shot results of @xcite . implicit events extraction. during the development of the ee verbalizations we found out that the 3 output kindly provided by the authors. ## conclusions the zs4ie toolkit allows a novice user to model complex ie schemas, curating simple yet effective templates for a target schema with new types of entities, relations, events, and event arguments. empirical validation showed that reformulating the ie tasks as an entailment problem is easy and effective, as spending only 5-15 minutes per type allows to achieve very strong zero-shot performance. zs4ie brings to the users the opportunity of defining the desired schema on the fly. in addition it allows to annotate examples, similar to post editing mt output. rather than being a finalized toolkit, we envision several exciting directions, such as including further nlp tasks, allowing the user to select custom pre-processing steps for candidate generation and allowing the user to interactively improve the system annotating examples that are used to fine-tune the te model. more generally, we would like to extend the inference capability of our models, perhaps acquired from other tasks or schemas @xcite , in a research avenue where entailment and task performance improve in tandem.