Path Spuriousness-aware Reinforcement Learning for Multi-Hop Knowledge Graph Reasoning (2023)

Abstract: Multi-hop reasoning, a prevalent approach for query answering, aims at inferring new facts along reasonable paths over a knowledge graph. Reinforcement learning methods can be adopted by formulating the problem into a Markov decision process. However, common suffering within RL-based reasoning models is that the agent can be biased to spurious paths which coincidentally lead to the correct answer with poor explanation. In this work, we take a deep dive into this phenomenon and define a metric named Path Spuriousness (PS), to quantitatively estimate to what extent a path is spurious. Guided by the definition of PS, we design a model with a new reward that considers both answer accuracy and path reasonableness. We test our method on four datasets and experiments reveal that our method considerably enhances the agent’s capacity to prevent spurious paths while keeping comparable to state-of-the-art performance.

## introduction knowledge graph (kg), a set of structured facts about real-world human knowledge, is utilized in numerous downstream nlp applications @xcite @xcite . common suffering affecting many downstream tasks is kg incompleteness. a variable amount of facts are missing in practical kgs. kg reasoning, the process to derive new knowledge from kg @xcite @xcite , is the way to address kg completion problem. a prevailing approach for kg reasoning is incorporating kg embedding (kge), which maps entities and relations into a vector space @xcite . embedding-based models have great power in expressing semantic similarity of entities and relations, but usually lack explainability due to the high-dimension representation @xcite . the corresponding author is jianxin li. an alternative approach is multi-hop reasoning, also referred to as path-based models @xcite , which infers new facts along existing paths in kg. multi-hop reasoning offers explanations for its predictions by taking advantage of reasoning paths. recently, reinforcement learning (rl) has been applied to multi-hop reasoning @xcite . rl-based methods train an agent to walk over the kg and search for a path leading to the answer @xcite . they have drawn surging attention in the past few years for their good prediction accuracy and excellent explainability. however, most prevalent rl-based models are suffering from the spurious path problem @xcite . a spurious path reaches a correct answer merely by coincidence and has no logical relevance with its prediction, such as the blue path in figure 1 : we shouldn't say tom was educated at mit just because there exists one person (i.e., jerry) who was born in the same country (i.e., u.s.) with tom and happened to visit mit before. multi-hop reasoning is a typical sparse reward scenario, where all actions except the final one will get no feedback during the decision process. that is to say, after the agent reached the correct answer following a spurious path, all actions along the decision trajectory will get positive rewards even if they are totally irrelevant to the query. as a consequence, the agent will be biased toward those wrong actions @xcite . the spurious path problem causes severe damage to the explainability of rl-based models when paths found by the agent are needed to serve as evidence to explain the answer. moreover, spurious paths may mislead the agent to learn a wrong policy and further harm the generalization ability of the model. although some studies have noticed the spurious path problem and provided instinctive solutions, such as action drop @xcite and rule guider @xcite , a quantitative estimation is still absent. to address the above-mentioned problem, we define a new metric called path spuriousness (ps) to measure to what extent a path is spurious. the inspiration is that a spurious path is not a fake path, but an adventitious path offering the accurate answer for a certain query. that is to say, if we randomly exchange intermediate entities of the path and keep the relation order, it is unlikely to get the correct prediction for the same query. with ps, we can reflect the reasonableness of predictions made by multi-hop reasoning models. specifically, answers obtained by following paths with low ps are much more reasonable and explainable than those by following paths with high ps. with the definition of ps, we put forward a path spuriousness-aware reward for rl-based multi-hop reasoning models. combined with correctnessguided rewards, our new reward leads the agent to not only obtain effective answers but also offer high-quality reasoning paths. our major contributions are concluded as follows: (1) we first propose a quantitative metric ps to measure the spuriousness of reasoning paths. ## related work in this section, we give outlines of two main related areas and discuss their connection to our method. ## preliminaries in this section, we first give crucial definitions of several concepts. then the reinforcement learning formula and reward shaping skills will be introduced, which is the foundation of our model. @xcite , where e is the set of entities, r is the set of relations and ψ is the mapping from relations to pairs of entities. a fact in knowledge graph g is an ordered triple δ =< e s , r, e @xmath0, e @xmath1, @xmath2, and ψ(r) =< e s , e o >. ## methodology in this section, we discuss the feature of spurious paths and give a quantitative definition of ps. then we introduce two kinds of reward functions that take ps into account. ## experiment we evaluate our model on four datasets and compare it with eight common baseline models. ours+min and ours+sum indicate our two approaches with r min and r sum respectively. ## conclusion in this paper, we discuss the spurious path problem which widely exists in the rl-based multi-hop reasoning models. to address this problem, we define a new metric named path spuriousness (ps) to quantitatively evaluate to what extent a path is spurious and consequently propose a new reward that considers both the prediction accuracy and path reasonableness. under the guidance of the reward, the agent can be aware of not only whether its prediction is right, but also the spuriousness of its reasoning path, and thus avoid spurious paths. experiments show our method largely outperforms baseline models in terms of ps, and keep comparable to the state-of-the-art performance of prediction accuracy. detailed analysis indicates that a trade-off between pursuing better prediction accuracy and keeping high path reasonableness exists. its significance varies among different datasets. we provide a method to make a balance by manually pruning hyperparameters. analysis on reasoning hops shows potential in long-hop reasoning tasks. in future work, we would like to further investigate it.