Knowledge Base Completion Meets Transfer Learning (2021)

Abstract: The aim of knowledge base completion is to predict unseen facts from existing facts in knowledge bases. In this work, we introduce the first approach for transfer of knowledge from one collection of facts to another without the need for entity or relation matching. The method works for both canonicalized knowledge bases and uncanonicalized or open knowledge bases, i.e., knowledge bases where more than one copy of a real-world entity or relation may exist. Such knowledge bases are a natural output of automated information extraction tools that extract structured data from unstructured text. Our main contribution is a method that can make use of a large-scale pretraining on facts, collected from unstructured text, to improve predictions on structured data from a specific domain. The introduced method is the most impactful on small datasets such as ReVerb20K, where we obtained a 6% absolute increase of mean reciprocal rank and 65% relative decrease of mean rank over the previously best method, despite not relying on large pre-trained models like BERT.

## introduction a knowledge base (kb) is a collection of facts, stored and presented in a structured way that allows a simple use of the collected knowledge for applications. in this paper, a knowledge base is a finite set of triples h, r, t , where h and t are head and tail entities, while r is a binary relation between them. manually constructing a knowledge base is tedious and requires a large amount of labor. to speed up the process of construction, facts can be extracted from unstructured text automatically, using, e.g., open information extraction (oie) tools, such as reverb @xcite or more recent neural approaches @xcite . alternatively, missing facts can be inferred from existing ones using knowledge base completion (kbc) algorithms, such as the united kingdom uk alan turing europe lived in part of lived in (?) if the knowledge base is canonicalized, "the united kingdom" and "uk" are known to be the same entity. if the knowledge base is uncanonicalized or open, this information may not be given. conve @xcite , tucker @xcite , or 5 e @xcite . it is desirable to use both oie and knowledge base completion approaches to automatically construct kbs. however, automatic extractions from text yield uncanonicalized entities and relations. an entity such as "the united kingdom" may also appear as "uk", and a relation such as "located at" may also appear as "can be found in". if we fail to connect these occurrences and treat them as distinct entities and relations, the performance of kbc algorithms drops significantly @xcite . if our target data are canonicalized, collecting additional uncanonicalized data from unstructured text is not guaranteed to improve the performance of said models. an illustration of a knowledge base can be found on figure 1 . open knowledge base completion (okbc) aims to mitigate this problem and to predict unseen facts even when real-world entities and relations may appear under several different names. existing work in the area overcomes this either by learning a mapping from entity and relation names to their embeddings @xcite or by using external tools for knowledge base canonicalization @xcite , and using the obtained predictions to enhance the embeddings of the entities @xcite . in this work, we follow the first of these two approaches. we pre-train rnn-based encoders that encode entities and relations from their textual representations to their embeddings jointly with a kbc model on a large okbc benchmark. we use this pre-trained kbc model and encoders to initialize the final model that is later fine-tuned on a smaller dataset. more specifically, kbc parameters that are shared among all inputs are used as an initialization of the same parameters of the fine-tuned model. when initializing the input-specific embeddings, we introduce and compare two approaches: either the pre-trained entity and relation encoders are also used and trained during the fine-tuning, or they are used in the beginning to compute the initial values of all entity and relation embeddings, and then dropped. we evaluate our approach with three different kbc models and on five datasets, showing consistent improvements on most of them. we show that pre-training turns out to be particularly helpful on small datasets with scarce data by achieving sota performance on the reverb20k and reverb45k okbc datasets @xcite and consistent results on the larger kbc datasets fb15k237 and wn18rr @xcite . our results imply that even larger improvements can be obtained by pre-training on a larger corpus. the code used for the experiments is available at https:// github.com/vid-koci/kbctransferlearning. we highlight that @xcite and @xcite use the term "open knowledge graph embeddings" for okbc. we use the term "open knowledge base completion" to also include kbs with non-binary relations, which cannot be viewed as graphs, and to consider methods that may not produce embeddings. our main contributions are briefly as follows: ## model for transfer learning in this section, we introduce the architecture of our model and how a pre-trained model is used to initialize the model for fine-tuning. the model consists of two encoders, one for entities and one for relations, and a kbc model. given a triple h, r, t , the entity encoder is used to map the head h and the tail t into their vector embeddings v h and v t , while the relation encoder is used to map the relation r into its vector embedding v r . these are then used as the input to the kbc algorithm of choice to predict their score (correctness), using the loss function, as defined by the kbc model. the two parts of the model are architecturally independent of each other and will be described in the following paragraphs. an illustration of our approach is given in figure 2 . ## datasets and experimental setup this section describes the used data, the experimental setup, and the baselines. ## experimental results this section contains the outcome of pre-training and fine-tuning experiments. in the second part of this section, we additionally investigate an option of zero-shot transfer. for each model, we report its mean rank (mr), mean reciprocal rank (mrr), and hits at 10 (h@10) metrics on the test set. we selected @xmath0, since it was the most consistently hits@n metric reported in related work. we report the hits@n performance for other values of n, the validation set performance, the running time, and the best hyperparameters in appendix c. all evaluations are performed using the filtered setting, as suggested by @xcite . when evaluating on uncanonicalized datasets with known gold clusters, reverb20k and reverb45k, the bestranked tail from the correct cluster is considered to be the answer, following @xcite . the formal description of all these metrics can be found in appendix a. pre-training results. the performance of the pre-trained models on olpbench is given in okbc results. our results of the models on re-verb20k and reverb45k are given in finally, our best model outperforms the stateof-the-art approach by chandrahas and talukdar (2021) on reverb20k and reverb45k. even when compared to pre-trained gru_conve, which is based on the same kbc model, okgit(conve) and care(conve,lan) lag behind. this is particularly outstanding, because the bert and roberta language models, used by okgit(conve), received by several orders of magnitude more pre-training on unstructured text than our models, making the results more significant. similarly, the initialization of models with big-graph seems to visibly help the performance, however, they are in turn outperformed by a noen-coder model, initialized with pre-trained encoders instead. this indicates that our suggested pretraining is much more efficient, despite the smaller computational cost. ## related work in recent years, there have been several approaches to improve the performance of kbc algorithms through data augmentation, commonly through various levels of connection with unstructured text. @xcite , for example, use pre-trained word embeddings to initialize their entity embeddings. @xcite make use of an encoder that generates an embedding given a description of the entity, and they show that their approach generalizes even to previously unseen entities. @xcite make use of a large-scale pre-trained transformer model to classify whether a fact is true. they rely on costly pre-training and do not generate embeddings for entities that could be used to incorporate background knowledge into natural language understanding systems @xcite . previous attempts at open knowledge base completion are tied to existing work on the canonicalization of knowledge bases. to canonicalize open knowledge bases, automatic canonicaliza-tion tools cluster entities using manually defined features @xcite or by finding additional information from external knowledge sources @xcite . @xcite use clusters obtained with these tools to augment entity embeddings for kbc. we note that @xcite use rnn-based encoders to encode relations, but not to encode entities. @xcite , on the other hand, introduce a model with rnn-based encoders for both entities and relations, similarly to our approach, however, they do not transfer beyond the introduced olpbench dataset. finally, chandrahas and talukdar (2021) use both kb canonicalization tools and large-scale pre-trained model bert, combining their predictions to make a more informed decision. the development of methods that improve predictions of kbc algorithms through data augmentation or transfer is tied to the advances in oie and kbc methods. however, these are beyond the scope of this project; see the works by @xcite and @xcite for an overview. ## summary and outlook in this work, we have introduced a novel approach to transfer learning between various knowledge base completion datasets. the main strength of the introduced method is the ability to benefit from pre-training on uncanonicalized knowledge bases, constructed from facts, collected from unstructured text. scaling the introduced method up would let us train large-scale pre-trained models that have already shown to be incredibly successful in natural language processing. we tested our method on 5 different datasets, showing that pre-training improves the performance of models. pre-training turned out to be particularly beneficial on smallscale datasets, where we were able to obtain the most significant gains, e.g., 6% absolute increase of mrr and 65% decrease of mr over the previously best method on reverb20k, despite not relying on large pre-trained models like bert. there are several directions of future work, such as scaling of pre-training to larger models and datasets and investigating the impact of the encoder architecture.