Learning Collaborative Agents with Rule Guidance for Knowledge Graph Reasoning (2020)

Abstract: Walk-based models have shown their advantages in knowledge graph (KG) reasoning by achieving decent performance while providing interpretable decisions. However, the sparse reward signals offered by the KG during a traversal are often insufficient to guide a sophisticated walk-based reinforcement learning (RL) model. An alternate approach is to use traditional symbolic methods (e.g., rule induction), which achieve good performance but can be hard to generalize due to the limitation of symbolic representation. In this paper, we propose RuleGuider, which leverages high-quality rules generated by symbolic-based methods to provide reward supervision for walk-based agents. Experiments on benchmark datasets shows that RuleGuider clearly improves the performance of walk-based models without losing interpretability.

## introduction while knowledge graphs (kgs) are widely adopted in natural language processing applications, a major bottleneck hindering its usage is the sparsity of facts @xcite , leading to extensive studies on kg completion (or reasoning) @xcite @xcite @xcite . many traditional approaches on the kg reasoning task are based on logic rules @xcite @xcite . these methods are referred to as symbolic-based methods. although they showed good performance @xcite , they are inherently limited by their representations and generalizability of the associated relations of the given rules. to ameliorate such limitations, embeddingbased methods @xcite @xcite @xcite @xcite @xcite were proposed. they learn distributed representations for entities and relations and make predictions using the representations. despite their superior performance, they fail to make human-friendly interpretations. to improve the interpretability, many recent efforts formulate the task as a multi-hop reasoning problem using reinforcement learning (rl) techniques @xcite @xcite @xcite , referred to as walk-based methods. a major issue of these methods is the reward function. a "hit or not" reward is too sparse while a shaped reward using an embedding-based distance measurement lin et al. (2018) may not always result in desirable paths. in this paper, we propose ruleguider to tackle the aforementioned reward issue in walk-based methods with the help of symbolic rules. we aim to improve the performance of walk-based methods without losing their interpretability. the ruleguider is composed of a symbolic-based model fetching logic rules and a walk-based agent searching reasoning paths with the guidance of the rules. we also introduce a way to separate the walkbased agent to allow for further efficiency. we experimentally show the efficiency of our model without losing the interpretability. ## problem and preliminaries in this section, we review the kg reasoning task. we also describe the symbolic-based and walkbased methods used in ruleguider. problem formulation. a kg consisting of fact triples is represented as @xmath0, r, e j )} ⊆ e × r × e, where e and r are the set of entities and relations, respectively. given a query (e s , r q , ?) where e s is a subject entity and r q is a query relation, the task of kg reasoning is to find a set of object entities e o such that (e s , r q , e o ) , where e @xmath1, is a fact triple missing in g. we denote the queries (e s , r q , ?) as tail queries. we note that we can also perform head queries (?, r q , e o ). to be consistent with most existing works, we only consider tail queries in this paper. however, these methods have limitations. for example, rules mined from different kgs may have different qualities, which makes the reasoner hard to select rules. figure 1 ## proposed method: ruleguider ruleguider consists of a symbolic-based method (see section 2), referred to as rule miner, and a walk-based method, referred to as agent. the rule miner first mines logic rules and the agent traverses over the kg to learn the probability distribution of reasoning paths with the guidance (via the reward) of the rules. as the agent walks through relations and entities alternatively, we propose to separate the agent into two sub-agents: a relation and entity agents. after the separation, the search space is significantly pruned. figure 2 shows the structure of these two agents in detail. ## experiments in this section, we compare ruleguider with other approaches on three datasets. we describe the experiment setting, results, and analysis. ## conclusions in this paper, we proposed a collaborative framework utilizing both symbolic-based and walk-based models. we separate the walk-based agent into an entity and relation agent to effectively leverage the symbolic rules and significantly reduce the action space. experimentally, our approach improved the performance of the state-of-the-art walk-based models on two benchmark kgs. in future work, we would like to study how to introduce acyclic rules to the walk-based systems.