G en TKG : Generative Forecasting on Temporal Knowledge Graph with Large Language Models (2024)

Abstract: The rapid advancements in large language models (LLMs) have ignited interest in the temporal knowledge graph (tKG) domain, where conventional embedding-based and rule-based methods dominate. The question remains open of whether pre-trained LLMs can understand structured temporal relational data and replace them as the foundation model for temporal relational forecasting. Therefore, we bring temporal knowledge forecasting into the generative setting. However, challenges occur in the huge chasms between complex temporal graph data structure and sequential natural expressions LLMs can handle, and between the enormous data sizes of tKGs and heavy computation costs of finetuning LLMs. To address these challenges, we propose a novel retrieval-augmented generation framework named GenTKG combining a temporal logical rule-based retrieval strategy and few-shot parameter-efficient instruction tuning to solve the above challenges, respectively. Extensive experiments have shown that GenTKG outperforms conventional methods of temporal relational forecasting with low computation resources using extremely limited training data as few as 16 samples. GenTKG also highlights remarkable cross-domain generalizability with outperforming performance on unseen datasets without re-training, and in-domain generalizability regardless of time split in the same dataset. Our work reveals the huge potential of LLMs in the tKG domain and opens a new frontier for generative forecasting on tKGs. The code and data are released here: https://github.com/mayhugotong/GenTKG .

## introduction forecasting the future lies in the intrinsic nature of humans to take controllability over the futural uncertainty ever since the existence of ancient fortunetellers who predict the future with insights into historical events. as the wave of artificial general intelligence (agi) led by large language models (llms) @xcite showcases a persistent craving for ability to model the complex information evolving in the real world, master the implicit rules and give predictions of what might happen next based on the historical observations @xcite , we term this challenge for llms as generative forecasting. we find temporal knowledge graph (tkg) is a natural instance for investigating such a challenge attributed to the evolving world knowledge it contains and the task performed on it, namely temporal knowledge graph forecasting. in short sentence, tkgs are multi-relational, directed graphs with labeled timestamped edges between entities (nodes) and can be viewed as streaming data sources where events come hourly, daily, or yearly, etc., and tkg forecasting task aims to forecast future events at timestamp t based on past historical events before t. specifically, tkg originates from knowledge graph (kg) @xcite which structures knowledge fact in the real world in the form of triples (e s , r, e o ), such as (paris, the capital of, france), where e s , e o represent the subject and object entity respectively, and r represents the observed predicate between the two entities. as world knowledge evolves constantly over time such as the inaugurated presidents of the usa, the temporal knowledge graph (tkg) was introduced by @xcite to indicate the temporal effectiveness of the world events by extending a timestamp t to form quadruples (e s , r, e o , t). for example, (donald trump, the president of, the usa, 2021) is followed by (joe biden, the president of, the usa, 2023). the tkg forecasting task aims to answer queries (e s , r, ?, t) that predict the missing object given history events before t. in tkg, the first embedding-based representation learning method is introduced by @xcite . the following conventional embeddingbased methods @xcite @xcite @xcite require carefully designed models that embed indexed quadruples into hidden latent space and hence lose the semantic aspects of events in tkgs. besides, they require separate training for different datasets and hence suffer to handle even slight dataset modification and time split adaptation. in stark contrast, the rule-based methods @xcite focus on mining temporal logic rules within the tkg graph structure in a symbolic way using neural networks. however, it possesses limited scalability to only similar datasets sharing similar rules. with the huge advancements emerging with numerous large language models (llms) @xcite , for example utilizing the emergent in-context learning (icl) ability of llms @xcite by sequentializing temporal ascending ordered tkg facts to texts but failed to compete with the above conventional methods @xcite . the question remains open: can pre-trained llms understand structured temporal relational data and replace conventional methods as the foundation model for temporal relational forecasting? to address the above issue, we bring temporal knowledge forecasting into the generative forecasting setting and deliberately prioritize the most influential factors in these two domains: the temporal and structural characteristics of tkgs and the flexible natural language processing abilities of large language models (llms). however, two challenges stand in the middle how to integrate them organically. the first is the modality challenge between data structures. as tkg are complex temporal multi-relational graph data with tens of thousands of quadruples, it is hard to adapt to sequential natural language expressions that llms can process. the second is the computation challenge with the enormous costs of fine-tuning llms especially with tens of thousands of quadruples requiring months of training time on consumable graphic cards. to solve the above two challenges, we propose gentkg, a novel retrieval-augmented generation framework that solves the tkg forecasting task in the generative forecasting setting, outperforming embedding-based, rule-based and icl methods. besides, gentkg serves as an instantiation that sheds light on the promising generative forecasting ability of llms. for the first modality challenge between structured temporal graph data and sequential natural languages, we solve it in the retrieval phase. we utilize a temporal logical rule-based retrieval strategy (tlr) that mines the temporal logic rules of the tkgs and forms a rule bank. these rules serve to retrieve the most temporally and logically relevant historical facts to the given query. these facts are then sequentialized to natural languages in the ascending temporal order and filled in a specialized prompt template for llms. although the prompts are in the form of sequential natural languages, they inherit structural information in the tkg implicitly since the extraction process is highly dependent on learned structural rules. these prompts enable llms to comprehend temporal relational data, and tlr enables the input window of llm to serve as the implicit and decouplable interface for communicating temporal and structural relational data to llms. moreover, tlr delivers improvement over the recent pure icl method, regardless of the backbone llm being used. for the second computation challenge between huge tkg size and high computation costs of llm, we solve it in the generation phase. we propose a few-shot parameter-efficient instruction-tuning strategy (fit) that aligns llm with a temporal relational forecasting task and reforms it into an autoregressive generation task. we further decompose the second computation challenge in two subtasks from the perspective of model and data respectively. the first subtask is to deal with the enormous computation costs and hardware requirements in training llm. we solve this subtask with a parameter-efficient fine-tuning (peft) adaptation method, specifically low-rank adaptation (lora) @xcite . the second subtask is to deal with the enormous size of training data in tkgs. we deliberately think out of the box by bypassing learning the data like conventional methods and instead, letting the llm learn the generative forecasting task on tkg. in other words, we reform data-centric model learning to task-centric llm alignment that aligns llms with tkg forecasting task through instruction tuning. we have specially designed task instructions, retrieved facts as input, and generative predictions as output. besides, we introduce few-shot tuning that further reduces training data to only 1024 prompt-response pairs which is as few as 0.27% of original tens of thousands of training data with exceeding performance. under extreme cases, we could further reduce to as few as 16 samples which is 0.0042% of original data while maintaining comparable performance to conventional methods. our approach offers a foundational framework for future explorations in generative forecasting on temporal knowledge graphs. our contributions are: ## generative forecasting on temporal knowledge graph in this section, we explain our gentkg framework following its two-phase methodology: retrievethen-generate, in two sections. in section 2.1, we explain the retrieval phase, which proposes a temporal logical rule-based retrieval strategy (tlr) to capture historical facts that exhibit high temporal relevance and logical coherence. in section 2.2, we delve into the details of the few-shot parameterefficient instruction-finetuning strategy (fit), an essential component that aligns large language models (llms) to the task of generative forecasting on temporal knowledge graphs. ## experimental setup in this section, we describe the experimental setup of gentkg framework. specifically, we describe four datasets, the evaluation protocols, and the experimental design. datasets four benchmark datasets are used to evaluate gentkg: 1) icews14 @xcite 2) icews18 @xcite 3) gdelt @xcite 4) yago @xcite . the two versions of the integrated crisis early warning system (icews) both consist of timestamped political events, e.g., (angela merkel, visit, india, 2015-03-25). the gdelt and yago datasets are extracted from the subsets of gdelt and yago knowledge bases containing facts and time information. dataset statistics is shown in evaluation since gentkg generates entity predictions directly, we use the temporal-aware filtered @xcite hits@1/3/10 metric to evaluate extrapolated link prediction. hits@1/3/10 denotes the proportion of the actual missing entities ranked within the top 1/3/10. baselines since gentkg is the first method to introduce instruction-tuned generative models into the tkg forecasting domain, it is necessary to include three typical types of existing methods as baselines. the first are embedding-based methods, represented by re-gcn @xcite , xerte @xcite , tango @xcite , and timetraveler @xcite . the rule-based method is tlogic @xcite and the third type is the llm-based icl method with frozen parameters @xcite . ## conclusion this paper raises the question and proves that pretrained llms can understand structured temporal relational data and replace existing tkg models as the foundation model for temporal relational forecasting task. we propose a retrieval-augmented generative framework gentkg that can efficiently align llm with temporal relational forecasting task through two stages: temporal logical rule-based retrieval (tlr) and few-shot parameter-efficient fine-tuning (fit). extensive experimental results demonstrate that gentkg outperforms conventional embedding-based, rule-based and icl methods. moreover, gentkg is training-light through comsumable computation resources with extremely few training data, and exhibits strong cross-domain and in-domain transferability breaking the barriers of conventional data-centric learning.