Label Verbalization and Entailment for Effective Zero and Few-Shot Relation Extraction (2021)

Abstract: Relation extraction systems require large amounts of labeled examples which are costly to annotate. In this work we reformulate relation extraction as an entailment task, with simple, hand-made, verbalizations of relations produced in less than 15 min per relation. The system relies on a pretrained textual entailment engine which is run as-is (no training examples, zero-shot) or further fine-tuned on labeled examples (few-shot or fully trained). In our experiments on TACRED we attain 63% F1 zero-shot, 69% with 16 examples per relation (17% points better than the best supervised system on the same conditions), and only 4 points short to the state-of-the-art (which uses 20 times more training data). We also show that the performance can be improved significantly with larger entailment models, up to 12 points in zero-shot, allowing to report the best results to date on TACRED when fully trained. The analysis shows that our few-shot systems are specially effective when discriminating between relations, and that the performance difference in low data regimes comes mainly from identifying no-relation cases.

## introduction given a context where two entities appear, the relation extraction (re) task aims to predict the semantic relation (if any) holding between the two entities. methods that fine-tune large pretrained language models (lm) with large amounts of labelled data have established the state of the art @xcite . nevertheless, due to differing languages, domains and the cost of human annotation, there is typically a very small number of labelled examples in real-world applications, and such models perform poorly @xcite . as an alternative, methods that only need a few examples (few-shot) or no examples (zero-shot) have emerged. for instance, prompt based learning proposes hand-made or automatically learned task and label verbalizations @xcite @xcite as an alternative to standard fine-tuning @xcite . in these methods, the prompts are input to the lm together with the example, and the language modelling objective is used in learning and inference. in a different direction, some authors reformulate the target task (e.g. document classification) as a pivot task (typically question answering or textual entailment), which allows the use of readily available question answering (or entailment) training data @xcite . in all cases, the underlying idea is to cast the target task into a formulation which allows us to exploit the knowledge implicit in pre-trained lm (prompt-based) or general-purpose question answering or entailment engines (pivot tasks). prompt-based approaches are very effective when the label verbalization is given by one or two words (e.g. text classification), as they can be easily predicted by language models, but strive in cases where the label requires a more elaborate description, as in re. we thus propose to reformulate re as an entailment problem, where the verbalizations of the relation label are used to produce a hypothesis to be confirmed by an off-the-shelf entailment engine. in our work 1 we have manually constructed verbalization templates for a given set of relations. given that some verbalizations might be ambiguous (between city of birth and country of birth, for instance) we complemented them with entity type constraints. in order to ensure that the manual work involved is limited and practical in real-world applications, we allowed at most 15 minutes of manual labor per relation. the verbalizations are used as-is for zero-shot re, but we also recast labelled re examples as entailment pairs and fine-tune the en-tailment engine for few-shot re. the results on the widely used tacred @xcite re dataset in zero-and few-shot scenarios are excellent, well over state-of-the-art systems using the same amount of data. in addition our method scales well with large pre-trained lms and large amounts of training data, reporting the best results on tacred to date. ## entailment for re in this section we describe our models for zeroand few-shot re. ## experimental setup in this section we describe the dataset and scenarios we have used for evaluation, how we performed the verbalization process, the different pre-trained nli models we have used and the state-of-the-art baselines that we compare with. ## analysis relation extraction can be analysed according to two auxiliary metrics: the binary task of detecting a positive relation vs. no-relation, and the multi-class problem of detecting which relation holds among positive cases (that is, discarding norelation instances from test data). ## conclusions in this work we reformulate relation extraction as an entailment problem, and explore to what ex14 description extracted from the guidelines. tent simple hand-made verbalizations are effective. the creation of templates is limited to 15 minutes per relation, and yet allows for excellent results in zero-and few-shot scenarios. our method makes effective use of available labeled examples, and together with larger lms produces the best results on tacred to date. our analysis indicates that the main performance difference against supervised models comes from discriminating norelation examples, as the performance among positive examples equals that of the best supervised system using the full training data. we also show that our method can be used effectively as a dataaugmentation method to provide additional labeled examples. for the future we would like to investigate better methods for detecting no-relation in zero-shot settings.