Robust Knowledge Graph Completion with Stacked Convolutions and a Student Re-Ranking Network (2021)

Abstract: Knowledge Graph (KG) completion research usually focuses on densely connected benchmark datasets that are not representative of real KGs. We curate two KG datasets that include biomedical and encyclopedic knowledge and use an existing commonsense KG dataset to explore KG completion in the more realistic setting where dense connectivity is not guaranteed. We develop a deep convolutional network that utilizes textual entity representations and demonstrate that our model outperforms recent KG completion methods in this challenging setting. We find that our model’s performance improvements stem primarily from its robustness to sparsity. We then distill the knowledge from the convolutional network into a student network that re-ranks promising candidate entities. This re-ranking stage leads to further improvements in performance and demonstrates the effectiveness of entity re-ranking for KG completion.

## introduction knowledge graphs (kgs) have been shown to be useful for a wide range of nlp tasks, such as question answering @xcite , dialog systems @xcite , relation extraction @xcite , and recommender systems @xcite . however, because scaling the collection of facts to provide coverage for all the true relations that hold between entities is difficult, most existing kgs are incomplete @xcite , limiting their utility for downstream applications. because of this problem, kg completion (kgc) has come to be a widely studied task @xcite @xcite @xcite @xcite . the increased interest in kgc has led to the curation of a number of benchmark datasets such as fb15k @xcite , wn18 @xcite , fb15k-237 @xcite , and @xcite that have been the focus of most of the work in this area. however, these benchmark datasets are often curated in such a way as to produce densely connected networks that simplify the task and are not representative of real kgs. for instance, fb15k includes only entities with at least 100 links in freebase, while yago3-10 is limited to only include entities in yago3 @xcite that have at least 10 relations. real kgs are not as uniformly dense as these benchmark datasets and have many sparsely connected entities @xcite . this can pose a challenge to typical kgc methods that learn entity representations solely from the knowledge that already exists in the graph. textual entity identifiers can be used to develop entity embeddings that are more robust to sparsity @xcite . it has also been shown that textual triplet representations can be used with bert for triplet classification @xcite . such an approach can be extended to the more common ranking paradigm through the exhaustive evaluation of candidate triples, but that does not scale to large kg datasets. in our work, we found that existing neural kgc models lack the complexity to effectively fit the training data when used with the pre-trained textual embeddings that are necessary for representing sparsely connected entities. we develop an expressive deep convolutional model that utilizes textual entity representations more effectively and improves sparse kgc. we also develop a student reranking model that is trained using knowledge dis-tilled from our original ranking model and demonstrate that the re-ranking procedure is particularly effective for sparsely connected entities. through these innovations, we develop a kgc pipeline that is more robust to the realities of real kgs. our contributions can be summarized as follows. prior work has effectively utilized entity names or descriptions to aid kgc @xcite @xcite . in more recent work, @xcite explore the problem of kgc using commonsense kgs, which are much sparser than standard benchmark datasets. they adapt an existing kgc model to utilize bert @xcite embeddings. in this paper, we develop a deep convoluational architecture that is more effective than adapting existing shallow models which we find to be underpowerered for large kg datasets. @xcite developed a triplet classification model by directly fine-tuning bert with textual entity representations and reported strong classification results. they also adapted their triplet classification model to the ranking paradigm by exhaustively evaluating all possible triples for a given query, (e 1 , r, ?). however, the ranking performance was not competitive foot_0 , and such an approach is not scalable to large kg datasets like those explored in this work. exhaustively applying bert to compute all rankings for the test set for our largest dataset would take over two months. in our re-ranking setting, we reduce the number of triples that need to be evaluated by over 7700×, reducing the evaluation time to less than 15 minutes. bert as a knowledge base: recent work @xcite @xcite has utilized the masked-languagemodeling (mlm) objective to probe the knowledge contained within pre-trained models using fill-in-the-blank prompts (e.g. "dante was born in [mask]"). this body of work has found that pre-trained language models such as bert capture some of the relational knowledge contained within their pre-training corpora. this motivates us to utilize these models to develop entity representations that are well-suited for kgc. re-ranking: @xcite introduced cascade re-ranking for document retrieval. this approach applies inexpensive models to develop an initial ranking and utilizes expensive models to improve the ranking of the top-k candidates. reranking has since been successfully applied across many retrieval tasks @xcite @xcite . despite re-ranking's widespread success, recent kgc work utilizes a single ranking model. we develop an entity re-ranking procedure and demonstrate the effectiveness of the re-ranking paradigm for kgc. knowledge distillation: knowledge distillation is a popular technique that is often used for model compression where a large, high-capacity teacher is used to train a simpler student network @xcite . however, knowledge distillation has since been shown to be useful for improving model performance beyond the original setting of model compression. @xcite al. (2017) demonstrated that knowledge distillation improved image classification performance in a setting with noisy dataset # nodes # rels # train # valid # test fb15k-237 14,451 237 272,115 17,535 20,466 snomed ct core 77,316 140 502,224 71,778 143,486 cn-100k 78,088 34 100,000 1,200 1,200 fb15k-237-sparse 14,451 237 18,506 17,535 20,466 ## datasets we examine kgc in the realistic setting where kgs have many sparsely connected entities. we utilize a commonsense kg dataset that has been used in past work and curate two additional sparse kg datasets containing biomedical and encyclopedic knowledge. we release the encyclopedic dataset and the code to derive the biomedical dataset to encourage future work in this challenging setting. the summary statistics for all datasets are presented in ## methods we provide an overview of our model architecture in figure 2 . we first extract feature representations from bert @xcite to develop textual entity embeddings. motivated by our observation that existing neural kg architectures are underpowered in our setting, we develop a deep convolutional network utilizing architectural innovations from deep convolutional vision models. our model's design improves its ability to fit complex relationships in the training data which leads to downstream performance improvements. finally, we distill our ranking model's knowledge into a student re-ranking network that adjusts the rankings of promising candidates. in doing so, we demonstrate the effectiveness of the re-ranking paradigm for kgc and develop a kgc pipeline with greater robustness to the sparsity of real kgs. ## conclusion kgs often include many sparsely connected entities where the use of textual entity embeddings is necessary for strong performance. we develop a deep convolutional network that is better-suited for this setting than existing neural models developed on artificially dense benchmark kgs. we also introduce a re-ranking procedure to distill the knowledge from our convolutional model into a student re-ranking network and demonstrate that our procedure is particularly effective at improving the ranking of sparse candidates. we utilize these innovations to develop a kgc pipeline with greater robustness to the realities of kgs and demonstrate the generalizability of our improvements across biomedical, commonsense, and encyclopedic kgs. the hyperparameter that controls the depth of the convolutional network. this means that our bert-resnet model consists of 3n = 6 sequential bottleneck blocks. we trained the models using a batch size of 64 with a 1vsall strategy @xcite with the binary cross entropy loss function. we use the adam optimizer (kingma and ba, 2015) with decoupled weight decay regularization @xcite and train the model with a learning rate of 1e-3. we use label smoothing with a value of 0.1, clip gradients to a max value of 1, and regularize the model using weight decay with a weight of 1e-4. we apply dropout with drop probability 0.2 after the embedding layer and apply 2d dropout @xcite with the same drop probability before the 2d convolutions. we apply dropout with probability 0.3 after the pooling and fully connected layer. we manually tuned the hyperparameters for this model based on validation performance.