zr LLM : Zero-Shot Relational Learning on Temporal Knowledge Graphs with Large Language Models (2024)

Abstract: Modeling evolving knowledge over temporal knowledge graphs (TKGs) has become a heated topic. Various methods have been proposed to forecast links on TKGs. Most of them are embedding-based, where hidden representations are learned to represent knowledge graph (KG) entities and relations based on the observed graph contexts. Although these methods show strong performance on traditional TKG forecasting (TKGF) benchmarks, they face a strong challenge in modeling the unseen zero-shot relations that have no prior graph context. In this paper, we try to mitigate this problem as follows. We first input the text descriptions of KG relations into large language models (LLMs) for generating relation representations, and then introduce them into embedding-based TKGF methods. LLM-empowered representations can capture the semantic information in the relation descriptions. This makes the relations, whether seen or unseen, with similar semantic meanings stay close in the embedding space, enabling TKGF models to recognize zero-shot relations even without any observed graph context. Experimental results show that our approach helps TKGF models to achieve much better performance in forecasting the facts with previously unseen relations, while still maintaining their ability in link forecasting regarding seen relations.

## introduction knowledge graphs (kgs) represent world knowledge with a collection of facts in the form of (s, r, o) triples, where in each fact, s, o are the subject and object entities and r is the relation between them. temporal knowledge graphs (tkgs) are introduced by further specifying the time validity. each tkg fact is denoted as a quadruple (s, r, o, t), where t (a timestamp or a time period) * equal contribution. in recent years, there has been an increasing number of works paying attention to forecasting future facts in tkgs, i.e., tkg forecasting (tkgf) or tkg extrapolated link prediction (lp). most of them are embedding-based, where entity and relation representations are learned with the help of the observed graph contexts. although traditional embedding-based tkgf methods show impressive performance on current benchmarks, they share a common limitation. in these works, models are trained on the tkg facts regarding a set of relations r, and they are only expected to be evaluated on the facts containing the relations in r. they cannot handle any zero-shot unseen relation r / ∈ r because no graph context regarding unseen relations exists in the training data and thus no reasonable relation representations can be learned. in the forecasting scenario, as time flows, new knowledge is constantly introduced into a tkg, making it expand in size. this increases the chance of encountering newly-emerged relations, and therefore, it is meaningful to improve embedding-based tkgf methods to be more adaptive to zero-shot relations. with the increasing scale of pre-trained language models (lms), lms become large lms (llms). recent studies find that llms have shown emerging abilities in various aspects @xcite and can be taken as strong semantic knowledge bases (kbs) @xcite . inspired by this, we try to enhance the performance of embeddingbased tkgf models over zero-shot relations with an approach consisting of the following three steps: (1) based on the relation text descriptions provided in tkg datasets, we first use an llm to produce an enriched relation description (erd) with more details for each kg relation (sec. 3.1). (2) we then generate the relation representations by leveraging another llm, i.e., t5-11b @xcite . we input erds into t5's encoder and transform its output into relation representations of tkgf models (sec. 3.1). (3) we design a relation history learner (rhl) to capture historical relation patterns, where we leverage llm-empowered relation representations to better reason over zero-shot relations (sec. 3.2). with these steps, we align the natural language space provided by llms to the embedding space of tkgf models, rather than letting models learn relation representations solely from observed graph contexts. even without any observed associated facts, zero-shot relations can be represented with llm-empowered representations that contain semantic information. we term our approach as zrllm since it is used to enhance zero-shot relational learning on tkgf models by using llms. we experiment zrllm on seven recent embedding-based tkgf models and evaluate them on three new datasets constructed specifically for studying tkgf regarding zero-shot relations. our contribution is three-folded: (1) to the best of our knowledge, this is the first work trying to study zero-shot relational learning in tkgf. (2) we design an llm-empowered approach zrllm and manage to enhance various recent embeddingbased tkgf models in reasoning over zero-shot relations. (3) experimental results show that zr-llm helps to substantially improve all considered tkgf models' abilities in forecasting the facts containing unseen zero-shot relations, while still maintaining their ability in link forecasting regarding seen relations. ## experiments we give details of our new zero-shot tkgf datasets in sec. 4.1. in sec. 4. to exclude this concern, we construct new benchmark datasets on top of the facts happening after the publication date of t5-11b. we first construct two datasets icews21-zero and icews22zero based on the integrated crisis early warning system (icews) @xcite kb. icews21-zero contains the facts happening from 2021-01-01 to 2021-08-31, while all the facts in icews22-zero happen from 2022-01-01 to 2022-08-31. besides, we also construct another dataset acled-zero based on another kb: the armed conflict location & event data project (acled) @xcite . facts in acled-zero take place from 2023-08-01 to 2023-08-31. all the facts in all three datasets are based on social-political events described in english. inspired by @xcite , our dataset construction process consists of the following steps. ## conclusion we study zero-shot relational learning in tkgf and design an llm-empowered approach, i.e., zrllm. zrllm extracts the semantic information of kg relations from llms and introduces it into tkg representation learning. it also uses an rhl module to capture the temporal relation patterns for better reasoning, and meanwhile promote the embedding space alignment between text and tkgs. we couple zrllm with several embedding-based tkgf models and find that zrllm provides huge help in forecasting the facts with zero-shot relations, and moreover, it maintains models' performance over seen relations.