Pretrained Knowledge Base Embeddings for improved Sentential Relation Extraction (2022)

Abstract: In this work we put forward to combine pretrained knowledge base graph embeddings with transformer based language models to improve performance on the sentential Relation Extraction task in natural language processing. Our proposed model is based on a simple variation of existing models to incorporate off-task pretrained graph embeddings with an on-task finetuned BERT encoder. We perform a detailed statistical evaluation of the model on standard datasets. We provide evidence that the added graph embeddings improve the performance, making such a simple approach competitive with the state-of-the-art models that perform explicit on-task training of the graph embeddings. Furthermore, we ob- serve for the underlying BERT model an interesting power-law scaling behavior between the variance of the F1 score obtained for a relation class and its support in terms of training examples.

## introduction besides large quantities of unstructured textual data, also structured data has become widely available to machine learning researchers in recent years. knowledge bases (kbs), such as wikidata @xcite (formerly freebase @xcite , yago @xcite and umls @xcite , organise various kinds of information in structured form and constantly grow in size and richness of included information. they are represented in terms of relations between entities, forming a graph structure that makes retrieval and processing of the included information easier and finds particular application in various language related tasks, such as question answering (qa), search engine development and knowledge discovery. distant supervision @xcite is another notable example that employs a kb to improve relation extraction (re). for each pair of entities found in a sentence, distantly supervised models check whether a link between the entities in the kb graph exists, and, if there is a match, the sentence is then used as a training example for supervised learning. note that the utility of kbs for re extends beyond being just a useful source of supervision labels. a natural question arises whether one can develop models which combine unstructured textual data with structured information to further improve performance, for general natural language processing tasks. one particular class of approaches that has been gaining momentum is based on the idea of dynamically learning representations of kb entities simultaneously with word representations @xcite @xcite . the motivation behind this class of methods is whether such combined representations could improve performance for a downstream nlp task, due to a more representative embedding. however, although in theory this could result in optimally finetuned word and graph representations for the downstream task, it might be challenging in practice. on-task training of the graph embeddings requires significantly more complex models, and therefore increases the training cost and is more prone to overfitting. therefore, instead of training both, graph and word embeddings, we investigate in this work whether combining static pre-trained graph embeddings, such as those provided in @xcite , with on-task learned word embeddings already achieves a significant performance boost for downstream tasks. the underlying question being whether a neural model is able to transfer the topological information contained in the pre-trained graph embeddings in a useful manner to the task at in the following, we would like to briefly review three particular works in the literature that make use of the information contained in a kb to improve classification performance on the re task, and explain how our work relates to these previous works. in @xcite the authors propose to use kbs as a supplementary source of information to improve on the multi-instance learning paradigm for re. note that in multi-instance re one aims at identifying the relation between two targeted entities, for a given bag of sentences. in particular, the authors of @xcite match the relation predicted by the stanford openie @xcite pipeline with the set of relation aliases found in the kb. out of this they obtain a matched relation embedding, h rel , that is then concatenated to the sentence representation. similarly, they build an entity type embedding, h type , using the entity type found in the kb, which is concatenated as well to the sentence encoding. in @xcite the authors make use of the kb information to improve on the sentential re task. they propose to construct an entity attribute context embedding, h o , by processing several entity properties found in the kb with the help of a bilstm @xcite encoder. additionally, a triplet context embedding, h r , is learned for each relation triplet by imposing the translational property in the embedding space @xcite to the triplet and its kb neighbours. finally, the two different representations obtained are aggregated with the sentence encodings by a gp-gnn @xcite and fed to a classifier for relation prediction. the authors of @xcite , however, suggest that statically adding all the available kb information might be counterproductive in some case. they rather propose to dynamically select the useful information. to do so, for each entity they extract and encode several kb properties with a bilstm @xcite , that are then combined together with the sentence encoding to form a heterogenous information graph. the relevant context information is then obtained by pruning this graph with the help of a combination of graph convolutional neural network @xcite , pooling and self-attention lay- bert mean re softmax head tail entity filter barack obama was born in hawaii [cls] kb pre-trained graph embeddings ffnn ffnn biaffine figure 1 : the re model we make us of, inspired by @xcite . the initial encodings of the sentences are provided by a pre-trained language model. the encodings corresponding to named entities are extracted, and averaged in case of multi-token entities. for the graph embedding augmented model the graph embeddings are in addition concatenated to the relative entity encodings. each entity is then decomposed in a head-tail representation to form the (h, t) candidate pairs passed to the biaffine attention layer @xcite for relation classification. note that the kb embedding is dynamically learned in all above cited works. furthermore, in these examples, it derives from a wide variety of entity properties and information retrieved from the kb, but not directly from the graph structure itself. the only exception being the triplet-context embedding h r of @xcite , which comes closest to the graph embedding @xcite we are going to make use of. however, their embedding is trained only on the downstream task, but not on the full kb. therefore, what is so far missing in the literature, is the basic baseline of simply adding a precomputed topology-based kb embedding in order to improve the re task performance. in this work we aim to fill this gap. we provide a detailed evaluation of such a model, which takes pre-computed kb embeddings into account. as we will show below, such a simple model extension is in fact competitive with the more advanced models mentioned above on standard benchmark datasets. ## model to perform re, we make use of the model introduced in nguyen and verspoor (2019); @xcite . the only difference being that we do not perform named entity recognition (ner), but instead train the model using gold entities, i.e. the correct span of the entity mentions is fed to the model as input. thereby, we are able to evaluate re performance without contamination with wrongly predicted entities. we chose this model for its relative simplicity, while still providing close to state-of-the-art performance in re and its high modularity that allows for easy modifications and extensions. the re classifier we use resembles the one introduced in @xcite , except that it is combined with a pre-trained transformer-based @xcite encoder, as proposed in @xcite . for illustration, the complete model is sketched in figure 1 . the information flows from left to right and the errors are free to backpropagate through all the preceding layers. the input of the model is a sentence of n tokens, w 1 , w 2 , ..., w n , containing two or more known entities, e @xmath0, w j+1 , ..., w j+k . the output is one or more triplets (h, t, r) representing the relations found in the input sentence. h and t represent the head and the tail of the relation respectively, while r is the type of relationship between h and t. a more detailed description of our model pipeline is given in the remainder of this section. for reproducibility, we made our model source codes available on github foot_0 . ## results in order to quantify the benefit of adding pretrained graph embeddings to a re model, we have considered two popular re datasets: the wikidata (sorokin and gurevych, 2017) and nyt @xcite corpora. since we make use of pre-trained graph embeddings, we decided to discard all sentences in the training set containing entities not present in the graph embedding @xcite . note that in order to be able to fairly compare the models with and without graph embeddings, we also exclude these sentences in training the basic model without graph embeddings. for the test set, we keep all the sentences regardless of the available embeddings. in case no embedding for the entity is available, we simply substitute the graph embedding with a zero tensor. for each dataset we train the model with a different random initialization ten times with and without the addition of the pre-trained graph embeddings. hence, in total we train twenty models per experiment. note that we always use the adamw @xcite optimizer with learning rate 2 • 10 -5 . for the implementation of the optimizer and of the whole model depicted in figure 1 we rely on the pytorch library @xcite . for evaluation, we compare the performance of the two models (with and without graph embeddings) using precision, recall and f1 score. both, for each single relation class, and on average with micro-and macro-averaging. in detail we rely on the two following evaluation methods: at first, for each relation class, we collect the f1 score obtained by the ten different trained models and we plot the complete distribution of the results as a violin plot. the mean of the ten f1 values obtained is also reported as a colored dot inside the violin. the same is done for the global micro-and macro-average. we do this for both, the model with added graph embeddings and the baseline model without graph embeddings. then, we generate the micro precision-recall curve for each of the ten models trained. the ten curves obtained are interpolated and averaged to form a single mean pr curve both, for the model provided with the graph embeddings and the baseline model. at each value of recall we compute the standard deviation of the precision over the ten different curves. the deviation is shown in shaded color around the mean curve. ## conclusion in both the experiments discussed in section 4, we measured on average a noticeable improvement over the baseline for the model with included pretrained graph embeddings. tables 2 and 3 summarize the numeric outcomes of our experiments, and also include for comparison results of the stateof-the-art methods obtained by others on the same datasets. in particular, our model is able to reach performance close to the current state-of-the-art in the wikidata dataset under the micro-averaged metrics and sets a new state-of-the-art under the macro f1 metric. similarly, for the nyt dataset our model achieves a new state-of-the-art under the p@10 and p@30 metric. in common with related works in the literature, our model rests on the assumption of having the correct entity identification at hand (gold entities). identifying entities in a sentence, however, is itself a challenging task, usually referred to as named entity recognition. this task is further complicated by the need to map the entities to corresponding nodes in the kb (entity linking). this currently limits the practical applicability of ours, and models akin to it in the literature, and warrants further research. we also analyzed in detail the performance of our model for each relation of the wikidata dataset, finding an interesting power-law scaling of the variance of the f1 score with increasing support of the relation. in particular, this study provided us with an estimate of around ∼ 1000 training occurences per relation needed for good prediction performance with small uncertainty. however, further investigation is needed to explore the validity of this finding in more generality. therefore, we like to highlight that we were not only able to verify that the inclusion of general pre-trained graph embeddings is helpful for the re task, but also that such a simple model extension is competitive with other state-of-the-art models that directly perform on-task training of those embeddings. this implies that the inclusion of such pretrained graph embeddings might be helpful across a wider spectrum of language related tasks to improve performance at a relatively low additional cost of complexity and computational burden. we see this work as giving further support to the wider adoption of pre-computed graph embeddings in natural language processing tasks. we envisage that their adoption may become comparable to the popular glove @xcite and word2vec @xcite pre-trained word embeddings. mike mintz, steven bills, rion snow, and daniel jurafsky. 2009. distant supervision for relation extraction without labeled data. in proceedings of the joint conference of the 47th annual meeting of the acl and the 4th international joint conference on natural language processing of the afnlp, pages 1003-1011, suntec, singapore. association for computational linguistics. makoto miwa and mohit bansal. 2016. end-to-end relation extraction using lstms on sequences and tree structures. in proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: long papers), pages 1105-1116, berlin, germany. association for computational linguistics. abhishek nadgeri, anson bastos, kuldeep singh, isaiah onando mulang', johannes hoffart, saeedeh shekarpour, and vijay saraswat. 2021. kgpool: dynamic knowledge graph context selection for relation extraction.