Exploring the zero-shot limit of F ew R el (2020)

Abstract: This paper proposes a general purpose relation extractor that uses Wikidata descriptions to represent the relation’s surface form. The results are tested on the FewRel 1.0 dataset, which provides an excellent framework for training and evaluating the proposed zero-shot learning system in English. This relation extractor architecture exploits the implicit knowledge of a language model through a question-answering approach.

## introduction and related works the fewrel dataset @xcite has been tailored for the task of few shot learning, where the model is presented with a limited sample of candidates in english for each relation ( this work aims to improve upon prior systems by leveraging on the advancements of question answering models and by representing relations using their surface forms as they appear on wikidata. the fewrel dataset is an ideal playground for this task: it forces the system to generalize by evaluating on unseen relations while at the same time mapping every relation to a wikidata identifier. the contributions of this paper are as in the following wikidata id description aliases p140 religion of a person, organization or religious building, or associated with this subject religious affiliation, faith, life stance, denomination p931 territorial entity or entities served by this transport hub (airport, train station, etc.) serves city, city served, train station serves ## problem statement the overarching goal is to classify a relation that connects two entities, i.e. to build a system that -given a pair of entities e 1 , e 2 , and a sentence s -predicts the probability of the entities being connected by a relation r. in this work, the relation is represented through its surface form. consider the example sentence "john smith receives an obe" with entities e 1 as "john smith" and e 2 as "obe". with the proposed method the correct relation is found by iterating over all the available surface forms and choosing the most probable one: r * = argmax r p (r, e 1 , e 2 , s) . zero-shot relation extraction is notoriously difficult, due to the challenge of generalizing the surface form to every possible semantic equivalent. in order to improve generalization, this paper aims to utilize the implicit knowledge contained in a pre-trained language model fine-tuned on squad 1.1, a questionanswering dataset @xcite . the challenge is therefore to implement a model that is compatible with eq. ( 1 ) and the question-answering task. a key insight is to notice that relation extraction can be understood as a question answering problem @xcite , where the surface form is the query and the entities connected by the relation are the "answer" to the query. in this work, however, the entities e 1 and e 2 are an input of the model. our solution is to signal that something is different about the relevant words by doubling the entities' tokens foot_1 : for example, the entity "john smith" becomes "john john smith smith" and "obe" appears as "obe obe". please note that -as the results show -the system does not just learn to associate repeating words to boundaries: in a nutshell we are asking the model if the entities are in the right place given a relation. using this technique the pre-trained question answering model is seen to vastly improve the generalization in the task at hand. an adversarial approach to the training is shown in fig. 1 : the model is taught to generate the entities given a sentence and a surface form (left of fig. 1 ); in the "fake" examples the system should return an adversarial configuration (right of fig. 1 ), where no entity is found: in this scenario all the output items are vanishing except the first. the system thus learns to quantify an adversarial score a. the probability in eq. ( 1 ) can then be defined as notice that the real output of the system is this adversarial score. the tagging of relevant entities is an expedient used to exploit the implicit knowledge of a pre-trained question answering system. ## model the model is based on a straightforward extension of the bert architecture for question answering on squad @xcite (fig. 1 ): as an input the system is presented with a text. this text is a concatenation of the relation's surface form (e.g. "a person has a title") and the sentence we want to extract the relation from (e.g. "john smith receives an obe"). the input is first tokenized according to the wordpiece algorithm @xcite , then passed onto a pre-trained bert model @xcite . the last hidden layer of bert is directed onto a dense layer with a 4-dimensional output per token. finally, a softmax filter is applied horizontally. in this way each row gives a unique position for the start and end of the relation's subject and object. the softmax is only applied for the length of the sentence, consistently with the task of marking the entities' boundaries. as shown in fig. 1 if the relation does not connect the relevant entities, the first token (corresponding to a [sent] tag) is trained to be 1 for all the rows of the output. ideally this method enforces that when no relation is found in the sentence no entity is tagged. each row of the output is therefore trained to give an adversarial score a i . the adversarial score a for eq. ( 2 ) is then defined as the minimum of all the four adversarial scores. @xmath0. ## training the model is implemented in pytorch @xcite . the original fewrel set is augmented with adversarial examples, generated by choosing a random relation from the training set excluding the correct one. the surface form itself is chosen randomly from a list made by concatenating the relation description and its aliases, as seen in ## results and discussion the zero-shot accuracy is compared with prior 1-shot results: while the 5-shot regime scores better, the number of aliases per wikidata relation is inconsistent, thus there cannot be a guarantee that a relation will have 5 different surface forms to choose from. only the wikidata description is used in validation and testing. 0-shot 5-ways 0-shot 10-ways 1-shot 5-ways 1-shot 10-ways (4) bert + squad 82.72 ± 0.02 67.9 ± 0.7 -- validation: as shown in both the distillbert and bert models improve after being fine tuned on squad (3 and 4). it is worth noticing that the smaller model (3) improves dramatically, achieving better accuracy than the vanilla bert model (2): clearly the question answering dataset provides a relevant template for zeroshot relation extraction. this phenomenon suggests a clear way to increase the accuracy: fine-tune the question answering model on a dataset with more examples than squad 1.1. in the end, the best results on the validation set are achieved with the larger bert model (4) after fine tuning on squad. remarkably, the accuracy is shown to be competitive with 1-shot results from recent models, although the limited number of relations on the validation set -only 16 -cannot guarantee generalization on the set of all possible relations. testing: the results on the test set are less competitive ( ## conclusions and future work this paper built and tested a general purpose system that only needs a wikidata-style description as a surface form to extract relations. the results are evaluated on the fewrel 1.0 set and compared to recent works. the model seems competitive on the validation set, whereas it is found struggling on the test set due to the intrinsic challenge of generalizing from a relation's surface form. an intriguing future line of research is to combine the zero-shot description with few-shot examples. this hybrid approach should be able to leverage on this paper's findings and achieve competing results in the task of relation extraction.