Predicting the Unpredictable: Uncertainty-Aware Reasoning over Temporal Knowledge Graphs via Diffusion Process (2024)

Abstract: Temporal Knowledge Graph (TKG) reasoning seeks to predict future incomplete facts leveraging historical data. While existing approaches have shown effectiveness in addressing the task through various perspectives, such as graph learning and logic rules, they are limited in capturing the indeterminacy in future events, particularly in the case of rare/unseen facts. To tackle the highlighted issues, we introduce a novel approach by conceptualizing TKG reasoning as a sequence denoising process for future facts, namely DiffuTKG. Concretely, we first encodes the historical events as the conditional sequence. Then we gradually introduce Gaussian noise to corrupt target facts during the forward process and then employ a transformer-based conditional denoiser to restore them in the reverse phase. Moreover, we introduce an uncertainty regularization loss to mitigate the risk of prediction biases by favoring frequent scenarios over rare/unseen facts. Empirical results on four real-world datasets show that DiffuTKG outperforms state-of-the-art methods across multiple evaluation metrics.

## introduction temporal knowledge graphs (tkgs) are dynamic, multi-relational structures that encapsulate the progression of events and knowledge in the real world, represented as quadruples (s, r, o, t), such as @xcite . the reasoning tasks over tkgs are classified based on the temporal scope: interpolation involves inferring missing information within the observed timeframe, while extrapolation aims at predicting future events. our research specifically focuses on extrapolation in tkgs, a domain that has more practical implications due to its forward-looking nature. existing studies @xcite @xcite in tkg reasoning typically aggregate adjacent structure information and temporal information to derive the deterministic representations of entities and relations @xcite . these representations are subsequently applied within a scoring function, such as convtranse @xcite , to assess the likelihood of potential future facts (events). despite the significant progress in tkg reasoning techniques, these deterministic methods exhibit inherent deficiencies when it comes to grappling with the uncertainties that arise from the unpredictable nature of future interactions @xcite and the evolving understanding of temporal and structural relationships over time @xcite @xcite . this challenge is particularly evident in scenarios characterized by a scarcity of discriminative information, especially for facts with sparse or even no historical interactions @xcite @xcite . these conventional approaches, which minimize the plausibility scores of unseen relation facts via the maximum likelihood objective, operate under the presumption that all unseen relation facts are false beliefs. as a result, they fail to capture the subtle uncertainty associated with these unseen facts. to illustrate, consider the scenario depicted in figure 1 , biden associated with only three historical facts and is anticipated to engage with the relatively rare or previously unseen facts iraq and zelensky in future scenarios. if we utilize deterministic embeddings derived from historical events to represent biden, biden's position in the embedding space (mapped into a 2d map) may fall somewhere in the middle of donald trump, g20, and iran. in such a setting, if predictions are based on the proximity within this embedding space, biden is more likely to be forecasted to interact with donald trump, g20, and iran, due to the closer embeddings. furthermore, the widespread use of the maximum likelihood objective, such as crossentropy loss, exacerbates prediction biases by favoring historically frequent scenarios over rare or unseen interactions @xcite , thus hindering the model's adaptability to the unpredictable dynamics and emerging relationships inherent in real-world tkg scenarios. to address these challenges, we propose dif-futkg, a novel approach that reformulates tkg reasoning into a sequence prediction task by managing the inherent uncertainties through a sequence denoising method. in the training phase, diffutkg systematically transforms sequences of objects, relationships, and temporal intervals relevant to subject entities into a unified continuous representation. this process is then augmented by the strategic introduction of gaussian noise, simulating the uncertain nature of future events (distribution ranges depicted in figure 1 ). subsequently, diffutkg harnesses a transformer-based framework for the denoising and accurate reconstruction of target entities, with the process intricately conditioned on both relational and temporal insights to mirror the intricate dynamics of tkgs structure. furthermore, diffutkg integrates an uncertainty regularization loss, which aids in distinguishing between seen and rare/unseen events, thereby enhancing the model's predictive clarity and reducing overfitting tendencies. during inference, dif-futkg employs a reverse diffusion step initialized with sampled gaussian noise to predict missing entities, subsequently refining these predictions based on calculated confidence scores. empirical studies conducted on four benchmark datasets demonstrate the effectiveness of diffutkg. in summary, our main contributions are as follows: ## diffusion models for discrete data the continuous diffusion model (dm) is a probabilistic model containing two markov chains, mainly consisting of forward and reverse processes, which diffuse the data with pre-defined noise and reconstruct the desired sample from the noise @xcite . in this article, we center on dms tailored for discrete data @xcite . in the forward diffusion process for discrete dataw, an embedding step first transforms w into a continuous embedding x 0 ∈ r d , parametrized by q(x 0 |w) = n (x 0 , emb(w), β 0 i). in addition, emb(w) ∈ r d is an embedding function that maps each word to a vector in r d . then the diffusion process corrupts x 0 to obtain the latent variables x 1:t by gradually adding noise in t steps, where x t is a standard gaussian noise. the forward transition x t-1 → x t can be attained by the reverse denoising process takes the initial state x t to reconstruct the original data x 0 by learning from a neural network f θ . the process can be formulated as the first expectation is to train the predicted model f θ (x t , t) to approximate x 0 from time step 2 to t. empirically, it can effectively reduce rounding errors. the second expectation consists of two components: the first component aims to bring the predicted x 0 , closer to the embedding emb(w), while the second component focuses on accurately rounding x 0 to the text w. ## our approach in this section, we introduce the details of our framework as shown in figure 2 . we first formulate the task definition of tkg reasoning as follow. definition 1 (temporal knowledge graph) a temporal knowledge graph (tkg), denoted as g, serves as a dynamic, multi-relational network of entities interconnected through time-stamped relations. this structure is conceptualized as a series of chronological kg snapshots, represented as @xmath0, g 2 , . . . , g t-1 }. each snapshot g t @xmath1, encapsulates the facts at a specific time t i , expressed as time-stamped quadruple (s, r, o, t i ), where s, @xmath2, respectively, and @xmath3. to facilitate a bi-directional comprehension of relationships within the tkg (kazemi and poole, 2018), the inverse quadruple (o, r -1 , s, t i ) is systematically appended to g. definition 2 (temporal knowledge graph reasoning) the primary aim of tkg reasoning is to enable extrapolative entity prediction. specifically, this entails predicting either the missing object entity in a future relation (s, r, ?, t) or the absent subject entity (?, r, o, t) utilizing historical tkg snapshots g t-l-1:t-1 = {g t-l , g t-l+1 , . . . , g t-1 } spanning the preceding l timesteps. ## conclusion in this study, we introduce diffutkg, a novel paradigm that reconceptualizes tkg reasoning as a denoising diffusion process, tailored to address the inherent uncertainties within future facts. during the denoising training phase, we initiate the process by generating embeddings from historical data as conditional inputs. following this, we methodically introduce gaussian noise to the target entities, reflecting the uncertainty of future facts, and utilize a conditional denoising decoder for their accurate reconstruction. in addition to reconstruction loss, we incorporate an auxiliary loss aimed at reducing prediction biases, particularly those arising from an overemphasis on historically frequent scenarios at the expense of rare or previously unseen facts. our empirical evaluations across various benchmark datasets confirm diffutkg's superior performance and efficiency in inference. l2tkg @xcite exploits the intratime and inter-time latent relations to alleviate the problem of missing associations in tkg reasoning. cluster @xcite learns a beam search policy via reinforcement learning (rl) to induce multiple clues from historical facts and adopts a gcn-based sequence method to deduce answers from clues. titer @xcite navigates through tkg historical snapshots and searches for the temporal evidence chain to locate the target object. tlogic @xcite generates answers by applying rules to observed events before the query timestamp and scores the answer candidates relying on the rules' confidences and time differences. techs @xcite integrates propositional and first-order reasoning in a logical decoder to achieve explainability.