A Multi-Expert Structural-Semantic Hybrid Framework for Unveiling Historical Patterns in Temporal Knowledge Graphs (2025)

Abstract: Temporal knowledge graph reasoning aims to predict future events with knowledge of existing facts and plays a key role in various downstream tasks. Previous methods focused on either graph structure learning or semantic reasoning, failing to integrate dual reasoning perspectives to handle different prediction scenarios. Moreover, they lack the capability to capture the inherent differences between historical and non-historical events, which limits their generalization across different temporal contexts. To this end, we propose a **M**ulti-**E**xpert **S**tructural-**S**emantic **H**ybrid (MESH) framework that employs three kinds of expert modules to integrate both structural and semantic information, guiding the reasoning process for different events. Extensive experiments on three datasets demonstrate the effectiveness of our approach.

## introduction incorporating real-world knowledge is essential for enhancing natural language processing capabilities @xcite @xcite . directly extracting specific knowledge or facts from various unstructured texts is time-consuming and laborious, hence knowledge graph (kg) is adopted to store some common facts and reduce retrieval cost. however, traditional kgs are limited to static fact storage. to capture the dynamic nature of facts, temporal knowledge graph (tkg) was proposed to record facts changing over time. it can provide certain evidence for many downstream tasks, like situation analysis, political decision making and service recommendation @xcite @xcite @xcite . tkg reasoning aims to predict the missing objects of future events based † corresponding authors. 1 the code and dataset are available at https://github. com/applied-machine-learning-lab/mesh-tkgr . you need to correctly predict the missing object of the last quadruplet: 189: [france, accuse, hezbollah] 189: [france, accuse, iran] 275: [france, demand, ubs] 339: [france, demand, ? ] on existing facts @xcite @xcite , where the formal problem definition is formulated in section 3.1. according to the type of utilized information, existing methods typically consist of two lines. one line relies on structural information (figure 1(a) ), modeling entity interactions through graph neural network (gcn) @xcite @xcite . previous works in this category have explored the utilization of recurrent networks with neighborhood aggregation mechanisms and recursive graph structures to jointly capture temporal dependencies and concurrent relations in tkgs @xcite . the other line (figure 1(b )) focus on semantic reasoning with pre-trained language models (plms) @xcite , particularly applying large language models (llms) to generate interpretable reasoning @xcite . these methods @xcite @xcite typically generate predictions through in-context learning with relevant historical facts. some methods further enhance performance by incorporating retrieval-augmented generation and parameter-efficient tuning, allowing llms to better adapt to the tkg reasoning task. in summary, existing works focus on either structural or semantic information, overlooking the potential benefits of integrating both types of information. however, different types of information can provide complementary insights for reasoning. the absence of structural information leads to insufficient knowledge of entity interaction patterns, while the lack of semantic information prevents understanding of entities' actions. as exemplified in figure 1 (b), france's recurring "accuse" or "demand" actions could inform predictions about future "demand" action. there is a logical progression from "accuse" to "demand", but the graph-based methods simply treat them as two different relations, losing this reasoning evidence. from another viewpoint, events to be predicted can be typically divided into two categories: historical events, which have already taken place in the past, and non-historical events, which have never occurred up to now. there is an inherent reasoning gap between different events @xcite . for historical events, capturing recurrence patterns is crucial while exploring evolution patterns is essential for non-historical events. this inspires several works that attempted to handle historical and non-historical events differently. tirgn @xcite ) models both the periodic patterns (which often appear in recurring historical events) and sequential evolution patterns (which characterize non-historical events), while cenet @xcite employs a binary classifier to separate the two types of events and focus predictions on relevant candidate entities. these methods often overlook that different types of information have their own advantages when handling different types of events. to address the aforementioned limitations, we propose a multi-expert structural-semantic hybrid (mesh) framework that effectively integrates structural and semantic information to model historical patterns for temporal knowledge graph reasoning. this model consists of a feature encoder, two kinds of event-aware expert modules and a prediction expert module. the underlying feature encoder outputs structural information from gcn and semantic information from llm. then we employ two kinds of event-aware expert modules to learn information weight allocation patterns for historical/non-historical events. there is a challenge in distinguishing event types. to address this, we design a prediction expert module which assigns different weights to each event-aware expert module, thereby implicitly distinguishing different types of events. this unified architecture enables adaptive information fusion without requiring explicit event type labels, offering both flexibility and efficiency. to summarize, the contributions of our work are as follows: llm-based tkg reasoning models. some methods formulate tkg reasoning as masked language modeling (mlm) or next-token generation tasks, using language models as the backbone. chapter @xcite uses a pretrained language model as encoder and employs a prefix-based tuning method to obtain good representations. ppt @xcite performs masked token prediction with fine-tuned bert. recently, llms have demonstrated powerful capabil-ities in summarization and reasoning. icl @xcite leverages llms with in-context learning by carefully selecting historical facts as context and decodes the outputs to rank predicted entities. gentkg @xcite proposes a retrieval-augmented generation approach with temporal logic rules, while applying parameterefficient tuning to adapt llms for tkg reasoning. coh @xcite captures key historical interaction from input text and also applies parameter-efficient tuning. although these llmbased methods provide context-based interpretable reasoning, they often struggle to capture the complex structural patterns in tkgs. ## methods in this section, we first introduce the problem definition and notations of the temporal knowledge graph reasoning. then we present the overall architecture of our proposed approach, followed by a detailed description of each model component. ## conclusion in this work, we proposed a multi-expert structural-semantic hybrid (mesh) framework. through the design of expert modules and the event-aware gate function, our model enabled adaptive information fusion based on event types. extensive experiments on three public datasets demonstrated that our approach consistently outperformed existing methods.