ONSEP : A Novel Online Neural-Symbolic Framework for Event Prediction Based on Large Language Model (2024)

Abstract: In the realm of event prediction, temporal knowledge graph forecasting (TKGF) stands as a pivotal technique. Previous approaches face the challenges of not utilizing experience during testing and relying on a single short-term history, which limits adaptation to evolving data. In this paper, we introduce the Online Neural-Symbolic Event Prediction (ONSEP) framework, which innovates by integrating dynamic causal rule mining (DCRM) and dual history augmented generation (DHAG). DCRM dynamically constructs causal rules from real-time data, allowing for swift adaptation to new causal relationships. In parallel, DHAG merges short-term and long-term historical contexts, leveraging a bi-branch approach to enrich event prediction. Our framework demonstrates notable performance enhancements across diverse datasets, with significant Hit@k (k=1,3,10) improvements, showcasing its ability to augment large language models (LLMs) for event prediction without necessitating extensive retraining. The ONSEP framework not only advances the field of TKGF but also underscores the potential of neural-symbolic approaches in adapting to dynamic data environments.

## introduction event prediction is a widely researched topic @xcite since accurate prediction of future events allows one to minimize losses associated with certain future events. to model large amounts of real-world event data that represent complex interactions between entities over time, the temporal knowledge graph (tkg) has been introduced @xcite . tkg is used to represent structural relationships among entities through timestamped quadruples (s, r, o, t), where s and o are entities, r is a binary relation † corresponding author. * the data and code of this paper can be found at https: //github.com/aqseabiscuit/onsep . between them, and t specifies the time when the event (s, r, o) occurs. for example, the quadruple (angela m erkel, visit, china, 2014/07/04) indicates that angela merkel visited china on july 4, 2014. in this task, temporal knowledge graph forecasting (tkgf) aims to predict future events of the graph by inferring missing entities in a quadruple for a future time. this involves generating predictions for either the object entity (s, r, ?, t+k) or the subject entity (?, r, o, t + k) by utilizing historical data from previous snapshots, where k represents the number of time steps or intervals into the future beyond the current time t. existing research studies have provided theoretical methodologies for time-sensitive applications such as recommendation systems @xcite , financial analysis @xcite , and social crisis early warning systems @xcite . traditional approaches @xcite @xcite @xcite involve converting event data into tkgs and combining graph neural networks (gnns) and recurrent neural net-works (rnns) capture evolving entity relationships through embeddings. however, these methods must perform model training on specific datasets, which is resource-intensive. with the proven understanding and generative capabilities of large language models (llms), recent studies have explored methods on llms @xcite @xcite . the method proposed by @xcite offers a more adaptable method for tkgf with in-context learning (icl) on llms, allowing llms to adapt to tkgf by using examples in the context, without fine-tuning. nevertheless, due to limitations in the length of history of llm inputs, this approach may not fully capture long-term trends among events and cannot effectively leverage past insights, such as causal relationships between events. imagine a tkg scenario that relates to everyday life. an example quadruple could be: '(sarah, consult, dr. smith, 2022/04/10)', indicating that sarah consulted dr. smith on april 10, 2022. in this case, the icl method may use static key values like 'sarah' and '(sarah, consult)' to extract historical events. however, this method may not account for sarah's tendency to begin consultations with phone calls months before meeting in person, this is evident in events such as 'discuss by telephone' being a preceding cause, for example, '(sarah, discuss by telephone, dr. smith, 2022/01/08)'. this long-standing pattern of initiating consultations with a call is a crucial piece of historical context that short-term data analysis could miss. without considering these longer-term causal interactions, the llmbased method may not make accurate future predictions. besides, in datasets like icews @xcite , the distribution of data containing entities and relations is dynamically changing. the emergence of new relations during test time poses challenges for traditional icl methods, which rely on fixed keyword-key value matching and cannot adapt over time. this highlights the need for methods that can dynamically capture and apply updates in real-time, enhancing adaptability in event prediction. to overcome these limitations, this paper proposes the online neural-symbolic event prediction (onsep) framework. it enhances both accuracy and adaptability in event prediction by addressing inadequate long-term causal relationship capture and enabling real-time adaptability for self-improvement without fine-tuning. figure 1 illustrates a comparison of the onsep and icl approaches. by contrast, onsep mainly has two novel components: 1) dynamic causal rule mining (dcrm): utilizing llms' external knowledge, dcrm semantically detects cause-effect links and dynamically constructs causal rules. this enables onsep to quickly adapt to new data and causal relationships, facilitating real-time updates and leveraging past experiences without extensive retraining. 2) dual history augmented generation (dhag): dhag employs the long short-term bi-branch retriever (lsbbr) and a hybrid model inference (hmi) strategy to merge short-term and long-term historical contexts. the latter benefits from causal rules derived during the dcrm phase, enabling a broader event to be captured within the limits of historical input length. inspired by the multi-branch fusion inference technology described in @xcite , the hmi strategy applies weighted fusion to balance the contributions from both dual historical contexts. these innovations address the limitations of context length constraints in llms and improve the retrieval of relevant historical events over extended periods. onsep shows significant performance gains on various datasets using internlm2-7b model, achieving hit@1 improvements over icl of 9.63%, 9.35%, and 16.28% at history length 100, and 8.14%, 8.64%, and 15.25% at 200, and achieved competitive performance of embedding-based models trained on specific datasets. to summarize, our main contributions include: ## method the onsep framework (figure 2 ) aims for event prediction via two key modules: (1) dcrm, which adaptively adjusts to changing data distributions during single-step prediction testing without requiring extensive training data, and (2) dhag, which integrates patterns from short-term historical events with causality from long-term event developments. detailed explanations are provided in the following sections. ## conclusion in this paper, we introduce a novel online neuralsymbolic framework, onsep, that integrates llms with tkgs to achieve adaptive and precise event forecasting in a dynamic online environment. to overcome the challenges of not utilizing experience during testing and relying on a single shortterm history, which limits adaptation to new data, we propose a dynamic causal rule mining module and a dual history augmented generation module within the onsep framework. this design allows llms to access the most recent history to identify patterns, as well as causal relationships from a broader range of past events. extensive experiments conducted on three benchmark datasets have proven the efficacy of onsep in tkgf, surpassing previous methods and demonstrating its broad applicability across diverse llms. our framework shows great potential for future applications in financial forecasting, public sentiment monitoring, and recommendation systems.