Improving Knowledge Graph Completion with Structure-Aware Supervised Contrastive Learning (2024)

Abstract: Knowledge Graphs (KGs) often suffer from incomplete knowledge, which restricts their utility. Recently, Contrastive Learning (CL) has been introduced to Knowledge Graph Completion (KGC), significantly improving the discriminative capabilities of KGC models and setting new benchmarks in performance. However, existing contrastive methods primarily focus on individual triples, overlooking the broader structural connectivities and topologies of KGs. This narrow focus limits a comprehensive understanding of the graph’s structural knowledge. To address this gap, we propose StructKGC , a novel contrastive learning framework designed to flexibly accommodate the diverse topologies inherent in KGs. We introduce four contrastive tasks specifically tailored to KG data: Vertex-level CL, Neighbor-level CL, Path-level CL, and Relation composition level CL. These tasks are trained synergistically during the fine-tuning of pre-trained language models (PLMs), allowing for a more nuanced capture of subgraph semantics. To validate the effectiveness of our method, we perform a comprehensive set of experiments on several real-world datasets. The experimental results demonstrate that our approach achieves SOTA performance under standard supervised and low-resource settings. Furthermore, we observe that the various structure-aware tasks introduced can mutually reinforce each other, resulting in consistent performance enhancements.

## introduction knowledge graphs, such as freebase and wikidata, are stores of relational facts that have become crucial sources of knowledge in knowledge-intensive applications. a kg typically comprises a collection of triples, where each triple (h, r, t) signifies the relationship between a head entity, a tail entity, and the corresponding relation. factual knowledge is virtually infinite and subject to frequent changes, leading to concerns about the incompleteness of kgs. to tackle this issue, researchers have focused on knowledge graph completion (kgc) models that automatically fill in missing triples. these models fall into two main categories: embedding-based methods and text-based methods. embeddingbased methods learn low-dimensional vectors for entities and relations by minimizing a loss function @xcite @xcite . textbased methods @xcite @xcite leverage available text to gather textual information for entities and relations. recent approaches have applied contrastive learning (cl) to textbased kgc models, significantly improving their discriminative power @xcite . typically, they use a dual-tower architecture that utilizes pre-trained language models (plms) to produce textual embeddings and then use the in-fonce contrastive objective @xcite to perform instance discrimination. despite their effectiveness, current contrastive approaches are not explicitly designed to identify the graph structure in kgs. they typically pair the entity-relation pair (h, r) with one positive tail entity t from the same triple (as shown in 2a). however, using just one positive sample, this approach fails to capture the broader connectivities and diverse topologies in kgs, which are crucial for understanding complex relation mappings (e.g., one-to-many, many-to-one) @xcite and long-range dependencies @xcite . entities in a kg are often surrounded by multiple neighboring entity-relation pairs that enrich their profiles. additionally, paths between entities can reveal meaningful patterns and dependencies, offering insights into intricate relations. furthermore, we observe that entities with similar positions in a kg share underlying attributes, consistent with the distributional semantics assumption that similar words appear in similar contexts @xcite . assume that entities represented by e i occupy the same structural position as (e i category -----→ animal). regardless of whether these entities pertain to dogs, cats, or lions, they exhibit analogous roles, such as respiration and sustenance requirements. establishing such semantic relevance is essential for inferring incomplete facts. fig. 1 illustrates an example of knowledge graph completion, the cooccurrence of ronaldo and zidane in the context of (p lay_f or, real_m adrid) suggests that they possess analogous conceptual attributes. therefore, inferring zidane's accurate profession becomes more plausible when considering ronaldo's occupation. this principle extends to entity-relation pairs and multi-hop paths. for example, the path p lay_f or ∧ p lay_f or -1 shares similar properties with the relation t eammate, as both involve similar semantic interactions, mirroring the underlying relational patterns. however, current textbased methods primarily focus on internal links within triples, disregarding the external semantic relevance beyond the triples. this results in incomplete representations that miss the holistic structural knowledge. therefore, integrating contextual structural information is crucial for text-based kgc, yet it remains largely underexplored. building on our analysis, we systematically investigate the effects of various structural forms. instead of focusing on individual triples, we consider the structural context around an anchor as weakly positive instances. our key idea is to maximize the mutual information between the central anchor and its context. to achieve this, we explore various subgraph topologies surrounding each triple, including vertex, neighbor, and path structures, as shown in fig. 2 . then, we propose a novel structure-aware contrastive learning framework, named structkgc, which offers the flexibility to effortlessly handle a wide range of topology types and quantities within kgs, without introducing additional parameters. following this, we introduce four supervised contrastive learning tasks tailored for kg data: vertex-level cl, neighborlevel cl, path-level cl, and relation composition level cl, each designed to capture distinct knowledge properties. during the fine-tuning stage of the plms, we jointly train these tasks, which facilitates a collaborative reinforcement effect among the different tasks and enables the model to effectively capture the underlying semantics within subgraphs. in summary, the contributions of this work are as follows: to address this, researchers have proposed methods considering the graph structure in kgs. r-gcn @xcite introduces a graph convolutional network (gcn) variant for relational data. compgcn @xcite uses composition operations to jointly embed entities and relations. the path ranking algorithm (pra) @xcite captures contextual relationships via indirect paths. ptranse @xcite treats multi-hop paths as new relations. these studies have showcased the efficacy of incorporating graph context into knowledge graph completion. nevertheless, these methodologies have limitations as they fail to consider the potential semantic correlations among the knowledge graph contexts. additionally, they struggle considerably in formulating predictive models within an inductive paradigm. ## methodology in this section, we first introduce knowledge graph completion (kgc), and then we investigate the existing contrastive learning loss functions employed in the kgc, analyzing its potential drawbacks. following this, we propose a novel structure-aware contrastive learning framework. fig. 3b presents an overview of our method, which consists of four contrastive learning objectives tailored to the characteristics of knowledge graphs. ## ablation on positive sample quantity in this paper, we use an in-batch strategy to effectively reuse samples, resulting in an increase in both negative and positive samples as the batch size grows. this makes it difficult to analyze the contribution of positive samples alone, as the model's performance is also affected by changes in the number of negatives. to address this, we fix the batch size and limit the maximum sampling of positive samples per batch to analyze their impact on fb15k-237. fig. 5 shows that the model's performance can be significantly enhanced by including more positive samples, especially when the number of positive samples is small. this suggests that integrating further structural context can augment the model's performance. with increasing positive instances, the model can grasp a wider structural context, facilitating a deeper understanding of structural semantics. nonetheless, we have observed that the advantage of adding more positive samples becomes less significant as the number of samples increases. ## conclusion in this work, we propose a simple yet effective framework (structkgc) that learns knowledge representations by efficiently utilizing the structure information. in particular, we propose a novel structure-aware supervised contrastive learning and design four cl tasks specifically designed for kgs. by jointly training these tasks, our structkgc can sufficiently perceive diverse structural knowledge. experiments show that our method achieves overall state-of-the-art performance better than other baselines in the link prediction task on benchmark datasets. the primary focus of this study is to leverage structural information for mining weak positive samples. in the future, we are interested in incorporating the negative sampling strategies, especially in hard negative mining, to further improve the discriminative ability in contrastive learning.