Ruleformer: Context-aware Rule Mining over Knowledge Graph (2022)

Abstract: Rule mining is an effective approach for reasoning over knowledge graph (KG). Existing works mainly concentrate on mining rules. However, there might be several rules that could be applied for reasoning for one relation, and how to select appropriate rules for completion of different triples has not been discussed. In this paper, we propose to take the context information into consideration, which helps select suitable rules for the inference tasks. Based on this idea, we propose a transformer-based rule mining approach, Ruleformer. It consists of two blocks: 1) an encoder extracting the context information from subgraph of head entities with modified attention mechanism, and 2) a decoder which aggregates the subgraph information from the encoder output and generates the probability of relations for each step of reasoning. The basic idea behind Ruleformer is regarding rule mining process as a sequence to sequence task. To make the subgraph a sequence input to the encoder and retain the graph structure, we devise a relational attention mechanism in Transformer. The experiment results show the necessity of considering these information in rule mining task and the effectiveness of our model.

## introduction people built different knowledge graphs, such as freebase @xcite , to store complex structured information and knowledge about abstract and real world. the facts in kg are usually represented in the form of triplets, e.g., (new york, iscityof, usa). kgs have been widely applied in various intelligent systems such as question answering @xcite , recommender system @xcite @xcite and zero-shot learning @xcite . although these kgs already contain a large number of relations and entities, they still suffer from the incompleteness of facts, whether constructed automatically or manually. in order to further expand kgs, many reasoning methods have been proposed to automated fact exploration, such as knowledge graph embedding (kge) @xcite @xcite @xcite , graph neural networks @xcite , and rule mining methods @xcite @xcite @xcite . compared with deep learning approaches like kge, rule mining methods are preferred due to their interpretability for reasoning and robustness for domain knowledge transfer. to mine the structure and confidence of rules at the same time in a fast way, differentiable rule learning methods @xcite are introduced and attract many research interests. the learning targets of existing methods is determining the confidence and structure of rules, according to which reasoning tasks are conducted. however, the uniform way of applying rules ignores the context of specific triplets, for which rules should be applied in different order. specifically, for a query (h, r, ?), there might be multiple rules of relation r (not only one) that could be used to get the target answer. for a specific query triplet, not all rules of r are convincing for reasoning, but how to select appropriate rule has not been studied. for example, to infer the missing fact, (localemployee/ambassador, nationality, ?) as figure 1 illustrated, previous rule mining methods only rely on the head relation nationality, so rules of nationality will be used in a fixed order. the two queries will get the same results, and whatever the order is, one of answers will be wrong. more precious rules can be observed if we pay attention to the head entity's identity, which is localemployee/ambassador. based on the observation, when the head entity is a localemployee, the rule nationality(x, y) ← livesin(x, z) ∧ iscityof(z, y) may derive more accurate result. in another case, if we know the head entity is an ambassador, then rule nationality(x, y) ← worksat(x, z) ∧ belongsto(z, y) will be a better choice for inferring. this example shows that the order and choice of rules to be applied for reasoning will significantly affect the results, and context information could help determine proper order. in this paper, we investigate making rule mining methods not only learn confidence and structures of rules but also learn to choose suitable rules based on the context of the head entity in completion tasks, which is challenging because of the diversity of context. we propose a transformer-based model to aggregate the context around the head entity because of its excellent performance on information interaction. essentially, the model consists of two blocks, 1) an encoder block aggregating the information of the head entity's subgraph and completing the information intersection of context, 2) a decoder block utilizing the aggregated entity embedding to generate the probability of relations for each step in differentiable rule mining process. since transformer framework is a sequence to sequence model, we design a converter that can turn graph structure into sequence. moreover, to maintain the information of subgraph, we modify the attention mechanism of transformer by adding the relation information to the attention calculation process, which we call relational attention mechanism. experimentally, we evaluate our model on several datasets (umls @xcite , fb15k-237 @xcite , wn18rr @xcite ) on link prediction and rule parsing task. the improvement in both experiments demonstrate the effectiveness of ruleformer. a case study is also analyzed, and the case proves our assumption and the ability of ruleformer to select suitable rules for different triples. in summary, our contributions are as follows: ## methodology knowledge graph g is composed by a set of triplets like {(e s , r, e t )|@xmath0, e s , ∈ e, e @xmath1, where e is a countable set of entities and r is a set of relations, respectively. for the task of rule mining, all the triplets that belong to g are given to the model, and for each head relation r, the model is supposed to find meaningful rules that are interpretable and understandable for humans. the rule we want is in the following form: in this section, to provide an intuition about each part of our model, we introduce the details about ruleformer. firstly, an encoder is designed for converting the subgraph structure to a sequence, so that the context information of head entity can be input to the transformer framework with relational attention mechanism which is used to retain the structure information of the graph completely. then we present how our model is deployed to utilize the output of encoder to support the rule mining process and generate the target sequence that represents rule body. finally, we propose a case-based rule parsing algorithm to get symbolic rules from the parameters. ## conclusion in this paper, we draw attention to using context to assist rule mining. we regard it as a sequence generation problem, and design a converter turning graph structure into a sequence. we propose a transformer-based model, ruleformer, which utilizes the subgraph of head entity when learning rules over knowledge graph. the experiment results show that in a specific environment, our proposed model can mine and select different suitable rules. the performance on link prediction task and rule parsing also improves with our model and parsing algorithm. future work may focus on a more effective way of rule mining and consider more complex forms of rules.