ZS - BERT : Towards Zero-Shot Relation Extraction with Attribute Representation Learning (2021)

Abstract: While relation extraction is an essential task in knowledge acquisition and representation, and new-generated relations are common in the real world, less effort is made to predict unseen relations that cannot be observed at the training stage. In this paper, we formulate the zero-shot relation extraction problem by incorporating the text description of seen and unseen relations. We propose a novel multi-task learning model, Zero-Shot BERT (ZS-BERT), to directly predict unseen relations without hand-crafted attribute labeling and multiple pairwise classifications. Given training instances consisting of input sentences and the descriptions of their seen relations, ZS-BERT learns two functions that project sentences and relations into an embedding space by jointly minimizing the distances between them and classifying seen relations. By generating the embeddings of unseen relations and new-coming sentences based on such two functions, we use nearest neighbor search to obtain the prediction of unseen relations. Experiments conducted on two well-known datasets exhibit that ZS-BERT can outperform existing methods by at least 13.54% improvement on F1 score.

## introduction relation extraction is an important task in the natural language processing field, which aims to infer the semantic relation between a pair of entities within a given sentence. there are many applications based on relation extraction, such as extending knowledge bases (kb) @xcite and improving question answering task @xcite . existing approaches to this task usually require large-scale labeled data. however, the labeling cost is a considerable difficulty. some recent studies generate labeled data based on distant supervision @xcite . nevertheless, when putting the relation extraction task in the wild, existing supervised models cannot well recognize the relations of instances that are extremely rare or even never covered by the training data. that said, in the real-world setting, we should not presume the relations/classes of newcoming sentences are always included in the training data. thus it is crucial to invent new models to predict new classes that are not defined or observed beforehand. such a task is referred as zeroshot learning (zsl) @xcite @xcite . the idea of zsl is to connect seen and the unseen classes by finding an intermediate semantic representation. unlike the common way to train a supervised model, seen and unseen classes are disjoint at training and testing stages. hence, zsl models need to generate transferable knowledge between them. with a model for zsl relation extraction, we will be allowed to extract unobserved relations, and to deal with new relations resulting from the birth of new entities. existing studies on zsl relation extraction are few and face some challenges. first, while the typical study @xcite cannot perform zeroshot relation classification without putting more human effort on it, as they solve this problem via pre-defining question templates. however, it is infeasible and impractical to manually create templates of new-coming unseen relations under the zero-shot setting. we would expect a model that can produce accurate zero-shot prediction without the effort of hand-crafted labeling. in this work, we take advantage of the description of relations, which are usually publicly available, to achieve the goal. second, although there exists studies that also utilize the accessibility of the relation descriptions @xcite , they simply treat zero-shot prediction as the text entailment task and only output a binary label that indicates whether the entities in the input sentence can be depicted by a given relation description. such problem formulation requires the impractical execution of multiple classifications over all relation descriptions, and cannot make relations comparable with each other. this paper presents a novel model, zero-shot bert (zs-bert), to perform zero-shot learning for relation extraction to cope with the challenges mentioned above. zs-bert takes two model inputs. one is the input sentence containing the pair of target entities, and the other is the relation description, i.e., text describing the relation of two target entities. the model output is the attribute vector 1 depicting the relation. the attribute vector can be considered as a semantic representation of the relation, and will be used to generate the final prediction of unseen relations. we think a better utilization of relation descriptions by representation learning is more cost-effective than collecting tons of instances with labeled relations. therefore, an essential benefit of zs-bert is free from heavycost crowdsourcing or annotation, i.e., annotating what kind of attribute does a class have, which is commonly used in zero-shot learning problem @xcite . figure 1 depicts the overview of the proposed zs-bert, which consists of five steps. each training instance is a pair of input sentence x i and its corresponding relation's description d j . first, we learn a projection function f that projects the input sentence x i to its corresponding attribute vector, i.e., sentence embedding. second, we learn another mapping function g that encodes the relation description d j as into its corresponding attribute vector, which is the semantic representation of d j . third, given the training instance (x i , d j ), we train zs-bert by minimizing the distance be-1 the terms, "attribute vector", "embedding", and "representation", are used interchangeably throughout this paper. bert-based relation extraction. contextual representation of words is effective for nlp tasks. bert (devlin et al., 2019) is a pre-training language model that learns useful contextual word representations. bert can be moderately adopted for supervised or few-shot relation extraction. r-bert (wu and he, 2019) utilize bert to generate contextualized word representation, along with entities' information to perform supervised relation extraction and have shown promising result. bert-pair (gao et al., 2019) makes use of the pre-train bert sentence classification model for few-shot relation extraction. by pairing each query sentence with all sentences in the support set, they can get the similarity between sentences by pretrained bert, and accordingly classify new classes with a handful of instances. these models aim to solve the general relation extraction task, which are more or less having ground truth, rather than having it under the zero-shot setting. zero-shot relation extraction. relevant studies on zero-shot relation extraction are limited. to the best of our knowledge, there are two most similar papers, which consider zero-shot relation extraction as two different tasks. levy et al. (2017) treat zero-shot relation extraction as a question answering task. they manually define 10 question templates to represent relations, and generate the prediction by training a reading comprehension model to answer which relation satisfies the given sentence and question. however, it is required to have human efforts on defining question templates for unseen relations so that zsl can be performed. such annotation by domain knowledge is unfeasible in the wild when more unseen relations come. on the contrary, the data requirement of zs-bert is relatively lightweight. for each relation, we only need one description that could express the semantic meaning. the descriptions of relations are easier to be collected as we may access them from open resources. under such circumstances, we may be free from putting additonal effort to the annotation. obamuyide and vlachos (2018) formulate zsl relation extraction as a textual entailment task, which requires the model to predict whether the input sentence containing two entities matches the description of a given relation. they use enhanced sequential inference model (esim) @xcite and conditioned inference model (cim) @xcite as their entailment methods. by pairing each input sentence with every relation description, they train the models to answer whether the paired texts are contradiction or entailment. this allow the model to inference on input sentence and unseen relation description pair, thus is able to predict unseen relation accordingly. ## problem definition let y @xmath0, ..., y n s } and y @xmath1, ..., y m u } denote the sets of seen and unseen relation labels, respectively, in which @xmath2. such two sets are disjoint, i.e., y s ∩ y @xmath3. for each relation label in seen and unseen sets, we denote the corresponding attribute vector as a i @xmath4, respectively. given the training set with n samples, consisting of input sentence x i , entities e i1 and e i2 , and the description d i of the corresponding seen relation y j s , denoted as our goal is to train a zero-shot relation extraction model m, i.e., m(s i ) → y i @xmath5, based on the training set such that using m to predict the unseen relation y k u of a testing instance s , i.e., m(s ) → y j @xmath6, can achieve as better as possible performance. we train the model m so that the semantics between input sentence and relation description can be aligned. we learn m by minimizing the distance between two embedding vectors f (x i ) and g(d i ), where learnable functions f and g project x i and d i into the embedding space, respectively. when new unseen relation y j u and its description is in hand, we can project the description of y j u to the embedding space by function g. when testing, new instance @xmath7, e j1 , e j2 , d j ) is input, in which z i denotes new sentence containing entities e j1 and e j2 , we project z i to the embedding space by our learned function f , and find the nearest neighboring unseen relation y j u , where z i and y i u are both unknown at the training stage. ## the proposed zs-bert model we give an overview of our zs-bert in figure 2 . the input sentence x i is tokenized and sent into the upper-part zs-bert encoder to obtain contextual representation. we specifically extract the representation of [cls], h 0 , and two entities' representations h 1 e , h 2 e , and then concatenate them to derive sentence embeddings âi s , by a fully-connected layer and activation operation. in the bottom part, we use sentence-bert @xcite to obtain attribute vector a i s for seen relations by encoding the corresponding description of relation d i . we train zs-bert under a multitask learning structure. one task is to minimize the distance between attribute vector a i s and sentence embedding âi s . the other is to classify the seen relation y j s at the training stage, in which a softmax zs-bert encoder 𝐶𝐿𝑆 𝑇 … 𝑇 … 𝑇 … … 𝑇 … 𝑇 𝑇 𝑆𝐸𝑃 𝐸 𝐸 … 𝐸 … 𝐸 … … 𝐸 … 𝐸 𝐸 𝐸 𝐻 𝐻 … 𝐻 … 𝐻 … … 𝐻 … 𝐻 𝐻 𝐻 sentence embedding 𝑎 h ⊕ h ⊕ h h h h input sentence input tokens embedding contextual representation relation prediction sentence-bert encoder 𝐶𝐿𝑆 𝑇 … … … … … 𝑇 𝐸 𝐸 … … … … … 𝐸 𝑆𝐸𝑃 𝐸 𝐻 𝐻 … … … … … 𝐻 𝐻 attribute vector 𝑎 of relation description born of the sea was first published in 2003 by viking press in paperback format. organization or person responsible for publishing books, games or software. relation description input tokens embedding contextual representation layer that accepts relation embedding is used to produce the relation classification probability. at the testing stage, by obtaining the embeddings of new-coming sentences and unseen relations, we use âi s and nearest neighbor search to obtain the prediction of unseen relations. ## conclusions in this work, we present a novel and effective model, zs-bert, to tackle the zero-shot relation extraction task. with the multi-task learning structure and the quality of contextual representation learning, zs-bert can not only well embed input sentences to the embedding space, but also substantially improve the performance. we have also conducted extensive experiments to study different aspects of zs-bert, from hyperparameter sensitivity to case study, and eventually show that zs-bert can steadily outperform existing relation extraction models under zero-shot settings. furthermore, learning effective embeddings for relations might also be helpful to semi-supervised learning or few-shot learning by utilizing prototypes of relations as the auxiliary information.