A Hierarchical N-Gram Framework for Zero-Shot Link Prediction (2022)

Abstract: Knowledge graphs typically contain a large number of entities but often cover only a fraction of all relations between them (i.e., incompleteness). Zero-shot link prediction (ZSLP) is a popular way to tackle the problem by automatically identifying unobserved relations between entities. Most recent approaches use textual features of relations (e.g., surface name or textual descriptions) as auxiliary information to improve the encoded representation. These methods lack robustness as they are bound to support only tokens from a fixed vocabulary and unable to model out-of-vocabulary (OOV) words. Subword units such as character n-grams have the capability of generating more expressive representations for OOV words. Hence, in this paper, we propose a H ierarchical N -gram framework for Z ero- S hot L ink P rediction (HNZSLP) that leverages character n-gram information for ZSLP. Our approach works by first constructing a hierarchical n-gram graph from the surface name of relations. Subsequently, a new Transformer-based network models the hierarchical n-gram graph to learn a relation embedding for ZSLP. Experimental results show that our proposed HNZSLP method achieves state-of-the-art performance on two standard ZSLP datasets.

## introduction link prediction models aim to predict relations between entities in knowledge graphs (kgs). majority of these methods learn low-dimensional representations of entities and relations (i.e., knowledge graph embeddings (kge)), which are then used to infer links between entities. traditional approaches @xcite assume that all relation types are known in the kg. this assumption is however unrealistic since kgs are inherently incomplete. to tackle this issue, the zero-shot link prediction (zslp) task has been introduced for identifying unseen relations by leveraging auxiliary information that bridges the gap between seen and unseen relations @xcite . little previous work exists on zslp as the task is relatively new @xcite @xcite . most efforts focus on using textual features @xcite or ontologies @xcite as auxiliary information for the task. particularly, @xcite use surface names of entities and relations while @xcite use the textual descriptions of relations. however, these approaches have two main limitations. first, common knowledge graphs such as wordnet @xcite and freebase @xcite often do not include textual descriptions of the relations. as such, these need to be obtained from other external sources (e.g., wikipedia foot_0 ) and are likely to be noisy, leading to poor performance. second, manually obtaining such relation descriptions is a labor-intensive and time-consuming process due to the large size of kgs. alternatively, @xcite proposed learning word representations from the surface name of relations using a pre-trained language model such as roberta @xcite . as surface names are readily available in the kg, this approach is more robust. however, it faces two fundamental weaknesses. first, context is an essential requirement for any text representation method. surface names on the other hand are represented by short texts, e.g., a relation "teammate" will have a single word representation observed in training and will therefore have little to no association with an unobserved relation for zero-shot. second, neural text encoders lack the ability to capture representations for out-of-vocabulary words. this same problem also applies to "word"-delimited models @xcite ) that aim to learn from textual descriptions of relations. in such cases, the relation representation ability of current methods may be limited significantly, which inadvertently hurts the zero-shot link prediction performance. in this paper, we follow a different direction. instead of simply learning representations from entire words of a relation's surface name, we hypothesize that leveraging character n-grams foot_1 (or n-grams for brevity) information from the relation name will help in generating better representations of unseen relations in zero-shot settings. models built on subword units (e.g., character n-grams) have the intrinsic ability of generating representations for rare or out-of-vocabulary words @xcite . inspired by this, we propose a novel hierarchical n-gram framework for zero-shot link prediction (hnzslp) that learns auxiliary information from character n-grams of the surface name of a relation. hnzslp consists of three main components: (1) a new hierarchical n-gram graph (or n-gram graph for brevity) for representing the relationships between all the character n-grams of a relation; (2) gramtransformer, a new transformerbased @xcite model for encoding the relation n-gram graph; and (3) a kg embedding model which adapts prevalent kge models (e.g., transe @xcite , distmult @xcite , tucker @xcite ) to compute a link prediction score between entities in the zero-shot setting. we perform extensive experiments on two standard datasets for zero-shot link prediction demonstrating the superiority of our method over prior state-of-the-art methods. our contributions are the following: ## problem statement a knowledge graph (kg) is defined as a graph @xmath0, e), where e denotes a set of entities and r denotes the set of relations among these entities. in a kg, the entities and relations are usually organized as facts and each fact is defined as a triplet (h, r, t) where h, @xmath1, tail entity and the relation between the two entities, respectively. in the zero-shot link prediction problem, we assume that there are two disjoint relation sets in the kg, a seen relation set r s and an unseen relation set r u , where r s ∩ r @xmath2. we are given a training set d @xmath3, r s , t)|h, @xmath4, r @xmath5. meanwhile, we define the test set as d @xmath6, r u , t, c (h,ru) )|h, @xmath7, c (h,ru) ⊆ e, r @xmath8, where t is the ground-truth tail entity and c (h,ru) denotes a candidate set corresponding to a query (h, r u ). given a query (h, r u ), the objective of zero-shot link prediction is to find the ground-truth tail entity t from the candidate set c (h,ru) . join and compositional graphs; (3) kg embedding learning module combines the embeddings of the head entity and relation to predict the tail entity and update the embeddings. ## experiments we validate hnzslp by comparing hnzslp's performance with several recent works, including zsgan @xcite and ontozsl @xcite . zsgan exploits the generated description embeddings of unseen relations to predict the tail entity while ontozsl introduces the ontology strategy in the task. following geng et al. (2021) , we further compare with the baselines zsl-transe and zsl-distmult that use word2vec @xcite , and respectively employ transe @xcite and dismult @xcite as kge models for zero-shot link prediction. for a fair comparison, we exclude the results of @xcite as this approach does not show the results in our used datasets. we use four commonly used metrics, mean reciprocal ranking (mrr), hits@10, hits@5, hits@1, and evaluate on two benchmark datasets, including nell-zs and wikidata-zs (wiki-zs) proposed by @xcite . a summary of dataset statistics is given in ## more analysis in order to further explore the effectiveness of our framework, we perform a series of analyses based on different characteristics of our model. first, we explore the effectiveness of our proposed gram-transformer with two latest works that learn text information from the character level or byte level. the contribution of our model components can also be learned from ablated models. so we propose two model variants to help us validate the advantages of the n-gram graph information and gramtransformer. next, we explore the performance of our model with a different number of nodes in the ngram graph. in the last, we did a comparison with the method which applies the language model in this task. for more experiments about the hnzslp performance in the oov problem, the impact of different node selection strategies, please refer to appendix sections 9.2, 9.3. ## conclusion in this paper, we propose a novel zsl framework hnzslp for link prediction. specifically, we proposed a gramtransformer to learn the n-gram graph information of the relation surface name and utilize the kge model to infer the tail entity. experimental results show that our framework achieves consistent improvements over various baselines in two zslp datasets. as the gramtransformer can be considered as a text representation method, in the future, we intend to explore its effectiveness on other nlp tasks including text classification.