N eu STIP : A Neuro-Symbolic Model for Link and Time Prediction in Temporal Knowledge Graphs (2023)

Abstract: Neuro-symbolic (NS) models for knowledge graph completion (KGC) combine the benefits of symbolic models (interpretable inference) with those of distributed representations (parameter sharing, high accuracy). While several NS models exist for KGs with static facts, there is limited work on temporal KGC (TKGC) for KGs where a fact is associated with a time interval. In response, we propose a novel NS model for TKGC called NeuSTIP, which performs link prediction and time interval prediction in a TKG. NeuSTIP learns temporal rules with Allen predicates, which ensure temporal consistency between neighboring predicates in the rule body. We further design a unique scoring function that evaluates the confidence of the candidate answers while performing link and time interval predictions by utilizing the learned rules. Our empirical evaluation on two time interval based TKGC datasets shows that our model shows competitive performance on link prediction and establishes a new state of the art on time prediction.

## introduction knowledge graphs (kgs) are factual information repositories, where each fact is encoded as r(s, o), where s and o are the real-world entities and r is the relationship between them. for instance, the fact presidentof(joe biden, usa) represents the fact that joe biden is the president of usa. temporal kgs extend these to entity-entity relations that have a temporal facet, for e.g., workedat @xcite ). in this work, we study temporal kgs that maintain temporal facts, r(s, o, t ), annotating each fact r(s, o) with the time period t for which it holds. while being a popular source of structured information, kgs are often incomplete. to this end, the problem of enriching kgs by inferring missing information is formulated as a kg completion (kgc) task. in the context of static (non-temporal) kgs, a key kgc task is link prediction, viz, given a query r(s, ?), predict o for which the fact r(s, o) holds in the real-world. this problem is fairly well-studied and has been tackled in many different ways. existing works can be broadly categorized into gnn-based solutions @xcite , lm-based approaches @xcite , kg embeddingbased models @xcite , and neuro-symbolic (ns) solutions @xcite . of special interest to us are ns approaches, which bring together humaninterpretable deduction capabilities of symbolic models with parameter sharing and other benefits of distributed representations in neural models. temporal kg completion (tkgc) extends kgc task to tkgs. in addition to link prediction, it involves an additional task of time-interval prediction -given a query r(s, o, ?), infer the time interval when the fact holds true. the majority of existing solutions are kg-embedding based @xcite @xcite , and do not incorporate symbolic rules. ns approaches have received very limited attention for tkgc. tlogic and alre-ir @xcite study ns-tkgc, but for time-instant kgs (i.e., kgs where a fact associated with an instant, not an interval). tilp @xcite is a very recent model for time-interval tkgc, but it can only do link prediction. to the best of our knowledge, no ns-tkgc approach exists that performs both link and time-interval prediction. a key challenge in building an ns-tkgc model is to design a unified rule language, which is intuitive and interpretable to humans, and also effective on both prediction tasks. secondly, ideally, the confidence of a rule should be computed based on both the statistical properties of the rule groundings and also the similarity scores of latent representations (which is not the case in existing models). contributions: we propose neustip (neuro symbolic link and time interval prediction), the first comprehensive ns framework for tkgc, which is effective on both link prediction and time interval prediction tasks. it uses an intuitive rule language that integrates the complete set of allen algebra relations and kg relations, with the goal of enforcing temporal consistency between neighboring predicates in the rule body. it also uses a novel way to compute confidence of temporal rules that combines both symbolic and embedding information. moreover, to the best of our knowledge, it is the first ns-tkgc model to perform timeinterval prediction.we evaluate the performance of neustip using two benchmark time interval kgc datasets, wikidata12k and yago11k @xcite . we find that a hybrid of neustip with a kg-embedding model timeplex @xcite obtains best results on both prediction tasks in both datasets. we release the neustip 1 codebase for further research. ## related work there are three main types of tkgc models in literature: purely embedding-based, multi-hop reasoning based, and rule-based. embedding-based models: all these models learn embeddings of entities and relations and define an associated scoring function to assess the validity of a temporal fact. earlier methods incorporated time within entity embeddings @xcite , and used transe-based scoring @xcite . recent methods @xcite adapt static kge models such as rotate @xcite , boxe @xcite for tkgc task. other approaches such as transe-tae @xcite , timeplex @xcite explicitly learn time embeddings, and additionally model temporal constraints between pairs of tuples in the tkg. these models are generally less interpretable, due to their complete reliance on latent embeddings. multi-hop reasoning models: such models exploit neighborhood information of an entity by employing distinct graph neural network architectures @xcite . models such as temp @xcite , re-net @xcite , xerte @xcite , cygnet @xcite exploit self-attention/gru, rnn, timeaware attention mechanism, and copy-generation 1 https://github.com/dair-iitd/neustip.git model respectively to integrate time information in a gnn. gnn-based models are generally computationally expensive @xcite , and do not scale well to large datasets. neuro-symbolic (rule-based) models: these models combine neural embeddings with explicit rule learning and inference, getting benefits of both interpretable inference of symbolic models and parameter sharing of neural models. tlogic @xcite and alre-ir @xcite are ns link-prediction models designed for time-instant tkgs. probably the closest to our work is a very recent model tilp @xcite that performs all possible constrained walks on time interval datasets to learn temporal logic rules and adopts attention mechanism to score each rule. there are two key differences between tilp and neustip. firstly, tilp's rules require expressing temporal relations between all pairs of intervals mentioned in the rule body. this blows up the number of rules, forcing tilp to use only three allen relations. in contrast, neustip encodes temporal relations only between pairs of neighboring intervals in a rule path, and uses all 13 allen relations. our experiments show that neustip performs better inferences compared to tilp. secondly, and more importantly, tilp is developed only for link prediction, whereas neustip introduces specific score and loss functions for time-interval prediction. ## preliminaries a knowledge graph maintains a set of entities e, and relations r, and each fact r(s, o) is denoted by a directed edge from subject entity s to object entity o, with the label r. a temporal kg (tkg) k additionally maintains the time domain, which is suitably discretized into a set t of discrete time instants, with two special elements t min and t max denoting the minimum and maximum time instants attainable in the tkg. a temporal fact is r(s, o, t ), where @xmath0, t e ] denotes the time interval during which the relation was true (t b , t @xmath1. it is represented in the tkg as an edge s (r, t ) ## the proposed neustip model we first describe neustip's rule language and its algorithm for mining rules. we then discuss its model for scoring a candidate answer, for both link and time interval prediction queries (see figure 2 ). further, we describe the loss function and inference procedures. ## analysis we investigate several research questions to gain further insights into our model: q1. what is the effect of each component of neustip on link and time-interval prediction? q2. how does our model perform in the inductive learning setting where the train and the test entities are disjoint? q3. how does neustip perform when there is limited training data available? q4. are the rules generated by our model for these analysis questions, unless otherwise stated, we use neustip (base) model -this allows us to directly assess the impact of design choices made in our model. ablation study: in order to understand the incremental contribution of each component (q1), we conduct an ablation study by removing three components from neustip: allen predicates (tr), pca score (pca) (equation 4 ) and duration distribution (intv) (section 4.4). please note that duration distribution is employed only during time interval prediction, so it will not change the results of link prediction. likewise, allen predicates (tr) are critical in predicting potential time intervals in our algorithm (section 4.2.2) and cannot be absent from the model while doing time prediction. limited training data: ns models have another advantage that they can learn in the presence of less amount of training data (q3). we test this in figure 3 on wikidata12k dataset where we compare neustip (base) against timeplex (base) on test data while training the models at varied training data sizes. we observe that when limited data is available (e.g. 10% data) neustip's rules are still able to capture the patterns in the data, while timeplex struggles to perform well (appendix m). human interpretability of logical rules: one advantage of our temporal rule-based model is that the predictions are in a human-interpretable form (q4), whereas the predictions of embedding-based @xcite ) ## conclusions in this paper, we develop a novel neuro-symbolic tkgc method that represents the temporal information of tkgs in a temporal rule language it defines, and uses rule groundings to ascribe a confidence score to a candidate answer, which can be an entity or a time interval. for that, it defines a novel scoring function. and, it also uses novel loss formulation for training. the key novelty of the proposed formulation is that it can perform both link prediction and time interval predictions in the neurosymbolic setting. compared to previous methods, our model has made substantial improvements in both link prediction and time interval prediction performance over two benchmark datasets. furthermore, we show that our model is generalizable to new entities, is human-interpretable, and works well with limited data while answering a given link prediction and time prediction query. we release our code foot_0 and datasets for further research. in the future, we will explore combinations of neustip with other orthogonal paradigms such as models that learn box embeddings instead of vector embeddings @xcite , and models based on graph neural network @xcite . exploring different methods of ensembling our rule-based and embedding-based models (apart from linear ensembling) can be another future work direction. numeric features present in the data. although we exploited the gadgets subsumed in timeplex @xcite to capture numeric features present in tkgs, capturing these features directly inside our model is an interesting future direction. further, our proposed model currently deals with closed path rules only and could benefit more if open paths can be exploited for time prediction.