Learning to Borrow– Relation Representation for Without-Mention Entity-Pairs for Knowledge Graph Completion (2022)

Abstract: Prior work on integrating text corpora with knowledge graphs (KGs) to improve Knowledge Graph Embedding (KGE) have obtained good performance for entities that co-occur in sentences in text corpora. Such sentences (textual mentions of entity-pairs) are represented as Lexicalised Dependency Paths (LDPs) between two entities. However, it is not possible to represent relations between entities that do not co-occur in a single sentence using LDPs. In this paper, we propose and evaluate several methods to address this problem, where we borrow LDPs from the entity pairs that co-occur in sentences in the corpus (i.e. with mentions entity pairs) to represent entity pairs that do not co-occur in any sentence in the corpus (i.e. without mention entity pairs). We propose a supervised borrowing method, SuperBorrow , that learns to score the suitability of an LDP to represent a without-mentions entity pair using pre-trained entity embeddings and contextualised LDP representations. Experimental results show that SuperBorrow improves the link prediction performance of multiple widely-used prior KGE methods such as TransE, DistMult, ComplEx and RotatE.

## introduction knowledge graphs (kgs) are a structured form of information that underline the relationships between real-world entities @xcite @xcite . a kg is represented using a set of relational tuples of the form (h, r, t), where r represents the relation between the head entity h and the tail entity t. for example, the relational tuple (joe biden, president-of, us) indicates that the president-of relation holds between joe biden and us. there exists a large number of publicly available and widely used kgs, such as freebase @xcite , dbpedia @xcite , and yago ontology @xcite . kgs have been effectively applied in various nlp tasks such as, relation extraction @xcite , question answering @xcite , and dialogue systems @xcite . however, most kgs suffer from data sparseness as many relations between entities are not explicitly represented @xcite . to overcome the sparsity problem, knowledge graph embedding (kge) models learn representations (a.k.a. embeddings) for entities and relations in a given kg in a vector space, which can then be used to infer missing links between entities @xcite @xcite . such models are trained to predict relations that are likely to exist between entities (known as link prediction or kg completion) according to some scoring formula. although previously proposed kge methods have shown good empirical performances for kg completion @xcite , the kges are learnt from the kgs only, which might not represent all the relations that exist between the entities included in the kg. to overcome this limitation, prior work has used external text corpora in addition to the kgs @xcite @xcite @xcite . compared to structured kgs, unstructured text corpora are abundantly available, up-to-date and have diverse linguistic expressions for extracting relational information. the co-occurrences of two entities within sentences (a.k.a textual mentions) in a text corpus has shown its success for text-enhanced kges @xcite . for example, the relational tuple in the freebase kg (joe biden, president-of, us) is mentioned in the following sentence "joseph robinette biden jr. is an american politician who is the 46th and current president of the united states." this sentence expresses the president-of relation between the two entities joe biden and us. as the entity-pair (joe biden,us) appears in a single sentence, we call it a with-mention entity-pair. however, even in a large text corpus, not every related entity pair cooccurs in a specified window, which are referred to as without-mention entity-pairs in previous studies. for instance, if we consider the widely used fb15k-237 kg @xcite and the clueweb12 @xcite text corpus with fb entity mention annotations, foot_0 33% of entitypairs in fb15k-237 do not have textual mentions within the same sentences. this sparseness problem limits the generalisation capabilities of using textual mentions for enhancing kges. specifically, @xcite ; @xcite have shown larger improvements in link prediction for with-mention entity-pairs over without-mention pairs. in this paper, we propose a method to augment a given kg with additional textual relations extracted from a corpus and represented as ldps. the augmented kg can then be used to train any kge learning method. this is attractive from both scalability and compatibility point of views because our proposal is agnostic to the kge learning method that is subsequently used for learning kges. our main contribution in this paper is to improve link prediction for without-mention entity-pairs by borrowing ldps from with-mention entity-pairs to overcome the sparseness in co-occurrences of the without-mention entity-pairs. we show that learning a supervised borrowing method, superborrow, that scores suitable ldps to represent withoutmention entity-pairs based on pre-trained entity embeddings and contextualised ldp embeddings boosts the performance of link prediction using a series of kge methods, compared to what would have been possible without textual relations. ## related work kges from a multi-relational graph: typically, kg embedding models consist of two main steps: (a) defining a scoring function for a tuple, and (b) learning entity and relation representations. entities are usually represented as vectors, whereas relations can be represented by vectors (e.g. transe @xcite , distmult @xcite and complex @xcite ) matrices (e.g. rescal @xcite ), or using some form of a representation, scoring functions are then defined to evaluate the strength of a relation r between h and t entities in a triple. transe is one of the earliest and well-known distance-based kge method that performs a linear translation and its scoring function is given in conventional kge models are trained using the facts in the kgs, which are often incomplete. therefore, to overcome the sparsity of structured kgs, we propose to integrate information from a text corpus, thereby augmenting the kg. the augmented kg is then used as the input to existing kge methods to learn accurate entity and relation embeddings. in particular, we do not modify the scoring functions nor optimisation objectives for the respective kge methods, which makes our proposed approach applicable in many existing kge methods without any modifications. text-enhanced kges: recently, a new line of research that combines textual information with relational graphs has emerged @xcite . different combination methods have been proposed for this purpose. @xcite proposed a model to embed both entities and words (using entity names and wikipedia anchors) into the same low-dimensional vector space to capture relational information from a kg and the co-occurrences from the corpus. @xcite control the amount of information shared between the two data sources in the joint embedding space using regularisation. this joint model is further enhanced by incorporating entity descriptions from an external corpus, which are jointly learnt with the kg @xcite @xcite . in a different scenario, the text-enhanced knowledge embedding model by @xcite creates a co-occurrence network of words and entities from an entity-annotated corpus. the authors define point-wise and pair-wise contexts using the co-occurrence frequencies in the network. then, entity and relation embeddings are enhanced using textual point-wise and pair-wise embeddings, respectively. similarly, rezayi et al. ( 2021 ) construct an augmented kg that has nodes from external text. the original and the augmented graphs are then aligned to suppress the noise and distil relevant information. in our work, we focus on adding extra edges to the kg rather than nodes as in @xcite and @xcite . in addition to contextual information and textual descriptions of individual words/entities, sentences where two entities co-occur have been used as contextual evidence to learn kges @xcite @xcite . for example, @xcite extracted ldps by parsing co-occurring sentences in a text corpus, which are then used as textual relations in the kg. this model can be seen as a special case of universal schema @xcite , which combines textual and kg relations in the same entity-pair co-occurrence matrix, subsequently decomposed to obtain entity embeddings. @xcite proposed a novel triple scoring function where textual mentions are used as a source of supporting evidence for a triple. our problem setting differs from prior work on text-enhanced kges in two important ways. first, we do not modify the underlying structure of the kge method, which is attractive from both scalability and compatibility of our proposal. second, rather than considering only entity-pairs that are occurring within a specified context in the corpus (i.e. with-mention entity-pairs), we propose to borrow ldps from with-mention entity-pairs to overcome the data sparseness in without-mention entity-pairs that never co-occur within any sentence in the corpus. ## method a relational kg d consists of a set of entities e and a set of relations r. in d, knowledge is represented by relational tuples (h, r, t) ∈ d, where the head entity h is related to the tail entity t by the kg relation r. in this work, we assume relations to be asymmetric in general (if (h, r, t) ∈ d then it does not necessarily follow that (t, r, h) ∈ d). the goal is to learn representations for entities and relations such that missing tuples can be accurately inferred. as kgs d are often sparse with many missing edges between entities, the learnt kges are affected, which in return impacts the performance of kges on downstream tasks such as link prediction. to address this sparseness problem, we consider the availability of a text corpus t where relational facts are expressed using contexts in which an entity-pair co-occurs. the textual relations that are extracted from t can be injected into d before applying a kge method. to align d with t , entity linking is applied to resolve ambiguous entity mentions in the text with unique entities in the kg @xcite . then, sentences that mentions two entities are considered as textual mentions of relations between entities. assuming that the corpus is annotated using the entities in d, there are multiple possibilities to obtain relational features of sentences that mention the entities. following previous work @xcite , we first run a dependency parser @xcite on each sentence containing an entity-pair to obtain ldps. then, if e contains the head and tail entities of an ldp l (but the entity-pair might not be connected by kg relations), we insert l into r to form a textual triple (h, l, t) ∈ d. the augmented kg is then used to learn embeddings for e and r using different kge methods. during kge processs, we treat both original relations in the kg and the augmented ldps equally. in principle, any existing kge learning method can be applied on the augmented kg as we later see in our experiments. one obvious limitation of the above-described method is that entity-pairs that never co-occur in any contextual window (e.g. a sentence) will not be connected by any ldp during the augmentation process. this is fine if the two entities are truly unrelated. however, this is problematic for entities that are related in the kg but their relations were not sufficiently covered in the text corpus because of the coverage issues and small size of the corpus. as we later see in our evaluations ( § 5), this is indeed the case for the majority of the without-mention entity-pairs. to overcome this limitation of our proposal, next we describe a method to borrow ldps from with-mention entity-pairs to without-mention entity pairs. it is worth noting that we do not connect any two entities by ldps, but only those that are related in the kg and predicted to be associated with an ldp by the proposed method. ## analysis borrowed ldps: to provide examples of ldps injected into fb15k237, ## conclusion we considered the problem of representing withoutmention entity-pairs in kge learning. specifically, we proposed a method (superborrow) to determine which ldps to borrow from with-mention entitypairs to augment a kg using a corpus. our proposed method improves the performance of several kge learning methods in link and relation prediction tasks.