Knowledge Representation Learning with Contrastive Completion Coding (2021)

Abstract: Knowledge representation learning (KRL) has been used in plenty of knowledge-driven tasks. Despite fruitfully progress, existing methods still suffer from the immaturity on tackling potentially-imperfect knowledge graphs and highly-imbalanced positive-negative instances during training, both of which would hinder the performance of KRL. In this paper, we propose Contrastive Completion Coding ( C^3 ), a novel KRL framework that is composed of two functional components: 1. Hierarchical Architecture, which integrates both low-level standalone features and high-level topology-aware features to yield robust embedding for each entity/relation. 2. Normalized Contrasitive Training, which conducts normalized one-to-many contrasitive learning to emphasize different negatives with different weights, delivering better convergence compared to conventional training losses. Extensive experiments on several benchmarks verify the efficacy of the two proposed techniques and combing them together generally achieves superior performance against state-of-the-art approaches.

## introduction knowledge graph (kg), as a well-structured representation of knowledge, plays an important role in a variety of knowledge-driven applications. upon kg, knowledge representation learning (krl) @xcite aims to embed the high-dimension and usually discrete features of entities/relations into a low-dimension vector space. these learned representations, by encoding the underlying semantic relationships among entities/relations, are able to facilitate various downstream tasks, such as question answering @xcite , recommendation @xcite and relation extraction @xcite to name some. as a basic research topic, krl has always attracted many attentions of researchers in relevant domains. previous krl methods generally consider kg completion (a.k.a link prediction) as the learning goal. in particular, they define certain score or energy function to accomplish the training by pushing up the score with respect to the observed positive triplets while simultaneously pushing down the score in terms of those negative ones @xcite . to further take the kg connectivity into account, recent works propose to take advantage of graph neural network (gnn) @xcite to exploit graph topology in krl @xcite . the gnnbased approaches have dominated the state-of-theart performance in popular benchmarks. despite fruitful progress they have achieved, existing methods still suffer from the immature ability on tacking incomplete/noisy kg and imbalanced positive-negative pairs. regarding the first issue, it is hard to construct perfect kg in practice owing to the expensive annotation effort, let alone that the information in kg is dynamically updating and it is difficult to detect the change at any time. in this situation, using gnn to aggregate information among noisy instances will increase the spread of noise and cause detriment to knowledge representation learning. in terms of the second issue, it is common that the number of negative instances is much greater than that of positive instances, and the importance of different negative instances differs greatly. recalling the training loss in previous krl methods (such as the margin-based @xcite and logistic-based @xcite loss), it coequally compares each positive instance with only one negative instance at each training iteration. in this way, it not only restrains the interaction between positive-negative instances, but also overlooks the different weights of different negative samples to each positive instance, which, in general, would lead to bias and slow training convergence. taking the triple (kobe bryant, na-tionality, united states) for example, we replace the tail entity with others to generate negative triple set, including (kobe bryant, nationality, italy) and (kobe bryant, nationality, michael jordan). in fact, for the second triple, michael jordan is not even a nation name and such negative fact should be weighted less compared to others, such as the first triple. to address the both issues as mentioned above, this paper proposes contrastive completion coding (c 3 ), a novel framework to allow robust and efficient krl. c 3 is mainly composed of two functional parts: 1. hierarchical architecture, which is designed to preserve mixed information from both low-level (embedding net) and high-level (gnn) features of each instance. by ensembling different levels of features, we can make full use of topology structure by gnn while effectively suppressing the dispersion of noise over imperfect kg. 2. normalized contrasitive training, which maximizes the normalized probability of the positive instance over all potential candidates that includes more than one negative sample. in this manner, the importance of different negative triples will be automatically reflected with regard to the positive instance during training. indeed, this objective is also known as infonce, a kind of mutual information loss that has been applied widely in machine learning and computer vision @xcite @xcite . we summarize our contributions as follows: our work is closely related to two main branches of study in knowledge representation learning and contrastive loss. ## contrastive completion coding in this section, we first present the problem definition in our task, and then follow it up by providing the details of our architecture framework and the training strategy in figure 1 . ## conclusion in this paper, we present c 3 , a novel knowledge representation learning framework, which is mainly composed of two functional parts: 1) hierarchical architecture, which has also exhibited the effectiveness in suppressing the spread of noise. 2) normalized contrasitive training, which can attend the different importance of different negative samples, giving rise to more effective learning. comparable experimental evaluations reveal that the two proposed techniques are efficient and compatible with each other. the analysis of hierarchical scoring shows that low-level and high-level features are both very necessary for robust krl, and they complement each other. the contrastive training experiments show that infonce loss is more suitable and efficient for the kg completion task, as our c 3 converges faster and performs better. to the best of our knowledge, we are the first to apply infonce to attend to the different importance of different negative samples in krl. our proposed method is simple, yet effective and well-motivated for resolving crucial issues in krl. calculate the high-level score: calculate the total score: {s(z t j i , c i ) ← ρs l (z update θ ← ∇ θ l n end while output: low-level feature z l , high-level feature z h and context representation c.