Temporal Knowledge Graph Forecasting Without Knowledge Using In-Context Learning (2023)

Abstract: Temporal knowledge graph (TKG) forecasting benchmarks challenge models to predict future facts using knowledge of past facts. In this paper, we develop an approach to use in-context learning (ICL) with large language models (LLMs) for TKG forecasting. Our extensive evaluation compares diverse baselines, including both simple heuristics and state-of-the-art (SOTA) supervised models, against pre-trained LLMs across several popular benchmarks and experimental settings. We observe that naive LLMs perform on par with SOTA models, which employ carefully designed architectures and supervised training for the forecasting task, falling within the (-3.6%, +1.5%) Hits@1 margin relative to the median performance. To better understand the strengths of LLMs for forecasting, we explore different approaches for selecting historical facts, constructing prompts, controlling information propagation, and parsing outputs into a probability distribution. A surprising finding from our experiments is that LLM performance endures ( \pm 0.4% Hit@1) even when semantic information is removed by mapping entities/relations to arbitrary numbers, suggesting that prior semantic knowledge is unnecessary; rather, LLMs can leverage the symbolic patterns in the context to achieve such a strong performance. Our analysis also reveals that ICL enables LLMs to learn irregular patterns from the historical context, going beyond frequency and recency biases

## introduction knowledge graphs (kgs) are prevalent resources for representing real-world facts in a structured way. while traditionally, kgs have been utilized for representing static snapshots of "current" knowledge, recently, temporal kgs (tkgs) have gained popularity to preserve the complex temporal dynamics eleutherai gpt-j-6b 6b los angeles 3 eleutherai gpt-neox-20b ## problem formulation in-context learning. icl is an emergent capability of llms that aims to induce a state in the model to perform a task by utilizing contextual input-label examples, without requiring changes to its internal parameters @xcite . formally, in icl for classification, a prompt is constructed by linearizing a few input-output pair examples (x i , y i ) from the training data. subsequently, when a new test input text x test is provided, icl generates the output y test ∼ p llm (y test | x 1 , y 1 , . . . , x k , y k , x test ) where ∼ refers to decoding strategy. temporal knowledge graph forecasting. formally, a tkg, @xmath0, r, e, t ), is comprised of a set of entities v, relations r, facts e, and timestamps t . moreover, since time is sequential, g can be split into a sequence of time-stamped snapshots, @xmath1, g 2 , . . . , g t , . . .}, where each snapshot, g @xmath2, r, e t ), contains the facts at a specific point in time t. each fact @xmath3, p, o, t) where s, @xmath4, @xmath5, and @xmath6. the tkg forecasting task involves predicting a temporally conditioned missing entity in the future given a query quadruple, (?, p, o, t) or (s, p, ?, t), and previous graph snapshots here, the prediction typically involves ranking each entity's assigned score. ## in-context learning for temporal knowledge graph forecasting in this work, we focus on 1) modeling appropriate history e q for a given query quadruple q, 2) converting {e q , q} into a prompt θ q , and 3) employing icl to get prediction y q ∼ p llm (y q | θ q ) in a zero-shot manner. here, the history e q is modeled on the facts from the previous graph snapshots ## related works event forecasting. forecasting is a complex task that plays a crucial role in decision-making and safety across various domains @xcite . to tackle this challenging task, researchers have explored various approaches, including statistical and judgmental forecasting @xcite @xcite . statistical forecasting involves leveraging probabilistic models @xcite or neural networks @xcite to predict trends over time-series data. while this method works well when there are many past observations and minimal distribution shifts, it is limited to numerical data and may not capture the underlying causal factors and dependencies that affect the outcome. on the other hand, judgmental forecasting involves utilizing diverse sources of information, such as news articles and external knowledge bases, to reason and predict future events. recent works have leveraged language models to enhance reasoning capabilities when analyzing unstructured text data to answer forecasting inquiries @xcite . yago wiki icews14 icews18 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 unidirection, entity, history @xmath7. lexical yago wiki icews14 icews18 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 index, entity, history @xmath8. bidirection yago wiki icews14 icews18 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 index, unidirection, history @xmath9. pair figure 4: performance (hit@1) analysis on prompt variation. the comparable performance exhibited by both the index and lexical models indicates that these models rely heavily on learning patterns and are less influenced by semantic priors. moreover, the unidirectional model typically outperforms the bidirectional model, suggesting that the robust constraints on historical data enable the model to comprehend observed patterns better. finally, the performance of the entity and pair models varies depending on the dataset. temporal knowledge graph. temporal knowledge graph (tkg) reasoning models are commonly employed in two distinct settings, namely interpolation and extrapolation, based on the facts available from t 0 to t n . (1) interpolation aims to predict missing facts within this time range from t 0 to t n , and recent works have utilized embeddingbased algorithms to learn low-dimensional representations for entities and relations to score candidate facts @xcite @xcite ; in-context learning. in-context learning (icl) has enabled llms to accomplish diverse tasks in a few-shot manner without needing parameter adjustments @xcite . in order to effectively engage in icl, models can leverage semantic prior knowledge to accurately predict labels following the structure of in-context exemplars @xcite @xcite @xcite , and learn the input-label mappings from the in-context examples presented @xcite . to understand the mechanism of icl, recent studies have explored the icl capabilities of llms with regards to the impact of semantic prior knowl-edge by examining their correlation with training examples @xcite @xcite , data distribution @xcite , and language compositionality @xcite in the pre-training corpus. other recent works show that llms can actually learn input-label mappings from in-context examples by showing the transformer models trained on specific linear function class is actually predicting accurately on new unseen linear functions @xcite . more recently, there is a finding that largeenough models can still do icl using input-label mappings when semantic prior knowledge is not available @xcite . ## conclusion in this paper, we examined the forecasting capabilities of in-context learning in large language models. to this end, we experimented with temporal knowledge graph forecasting benchmarks. we presented a framework that converts relevant historical facts into prompts and generates ranked link predictions through token probabilities. our experimental results demonstrated that without any finetuning and only through icl, llms exhibit comparable performance to current supervised tkg methods that incorporate explicit modules to capture structural and temporal information. we also discovered that using numerical indices instead of entity/relation names does not significantly affect the performance, suggesting that prior semantic knowledge is not critical for overall performance. additionally, our analysis indicated that icl helps the model learn irregular patterns from historical facts, beyond simply making predictions based on the most common or the most recent facts in the given context. together, our results and analyses demonstrated that icl can be a valuable tool for predicting future links using historical patterns, and also prompted further inquiry into the potential of icl for additional capabilities.