GLTW : Joint Improved Graph Transformer and LLM via Three-Word Language for Knowledge Graph Completion (2025)

Abstract: Knowledge Graph Completion (KGC), which aims to infer missing or incomplete facts, is a crucial task for KGs. However, integrating the vital structural information of KGs into Large Language Models (LLMs) and outputting predictions deterministically remains challenging. To address this, we propose a new method called GLTW , which encodes the structural information of KGs and merges it with LLMs to enhance KGC performance. Specifically, we introduce an improved Graph Transformer ( iGT ) that effectively encodes subgraphs with both local and global structural information and inherits the characteristics of language model, bypassing training from scratch. Also, we develop a subgraph-based multi-classification training objective, using all entities within KG as classification objects, to boost learning efficiency. Importantly, we combine iGT with an LLM that takes KG language prompts as input. Our extensive experiments on various KG datasets show that GLTW achieves significant performance gains compared to SOTA baselines.

## introduction knowledge graphs (kgs) are pivotal resource for a multitude of knowledge-intensive intelligent tasks (e.g., question answering @xcite , recommendation systems @xcite , planning @xcite , and reasoning @xcite , among others). they are composed of a vast number of triplets in the format of (h, r, t), where h and t represent the head and tail entities, respectively, and r denotes the relationship connecting these two entities. however, popular existing kgs, such as freebase @xcite , wordnet @xcite , and wikidata @xcite , suffer from a significant drawback: the presence of numerous incomplete or missing triplets, thereby giving rise to the task of kg completion (kgc). kgc aims to accurately predict the missing triplets by leveraging known entities and relations for effectively enhancing kgs. in recent years, with super-sized training corpora and computational cluster resources, large language models (llms) have developed rapidly and enabled state-of-the-art performance in a wide range of natural language tasks @xcite @xcite . consequently, certain studies have applied llms to kgc tasks. for instance, @xcite @xcite utilize zero/few-shot incontext learning (icl) to accomplish kgc, while @xcite leverage llms to enhance the descriptions of entities and relations in kgs, thereby improving text-based kgc methods @xcite @xcite @xcite . intuitively, integrating nontextual structured information appropriately can augment llms' understanding and representation of kgs. for example, @xcite @xcite combine graph-structured information with llms to boost kgc tasks. yet, they either use traditional embedding-based kgc methods @xcite @xcite that only consider internal links of triplets or rely on graph neural networks (gnns) @xcite ) that merely encode local subgraphs, thus missing out on global structural knowledge. also, llms, typically used for generative tasks, have long been troubled by hallucination @xcite . in contrast, the prediction targets of kgc are generally confined to the given kg, making it unwise to directly integrate llms into kgc tasks 1 . in short, how to encode both local and global structural information of kgs and combine it with knowledge-rich llms to achieve deterministic kgc remains underexplored. to this end, we propose a novel method (named gltw), which effectively encodes kg subgraphs with both local and global structural information and integrates llms in a deterministic fashion to improve the performance of kgc. concretely, we first treat entities and relations within kg as inseparable units, adding them as tokens to the original tokenizer, while referring to triplets as three-word sentences @xcite . subsequently, for each target triple, we extract a subgraph that encompasses both local and global structural information from the given training kg data (section 3.1). to effectively process the subgraph, we introduce an improved graph transformer (igt), which takes the entity and relation embeddings (initialized by a pooling operation), the relative distance matrix, and the relative distinction matrix of the subgraph as inputs, and encodes them using the enhanced graph attention mechanism (section 3.2). furthermore, we construct multiple positive and negative triplet samples from the subgraph, which are used to build the subgraph-based multi-classification training objective with all entities within the kg as classification objects (section 3.3). finally, we merge igt with llm that takes kg language prompt as input (section 3.4). to sum up, we highlight our contributions as follows: ## method in this section, we elaborate on our proposed method, gltw, in four parts: subgraph extraction, improved graph transformer, subgraph-based training objective, and joint igt and llm. figure 1 illustrates the pipeline of gltw. notably, the kg language prompt in this paper directly follows that of mkgl @xcite . ## conclusion in this paper, we propose a novel method, gltw, which aims to encode the structural information of kgs and integrate it with llms to enhance kgc performance. specifically, we formulate an improved graph transformer (igt) that effectively encodes subgraphs with both local and global struc-tural information and inherits the characteristics of language models, thus circumventing the need for training from scratch. also, we develop a subgraphbased multi-classification training objective that treats all entities within kg as classification objects to improve learning efficiency. importantly, we combine igt with an llm that takes kg language prompt as input. finally, we conduct extensive experiments to verify the superiority of gltw.