Logical Neural Networks for Knowledge Base Completion with Embeddings & Rules (2022)

Abstract: Knowledge base completion (KBC) has benefitted greatly by learning explainable rules in an human-interpretable dialect such as first-order logic. Rule-based KBC has so far, mainly focussed on learning one of two types of rules: conjunction-of-disjunctions and disjunction-of-conjunctions. We qualitatively show, via examples, that one of these has an advantage over the other when it comes to achieving high quality KBC. To the best of our knowledge, we are the first to propose learning both kinds of rules within a common framework. To this end, we propose to utilize logical neural networks (LNN), a powerful neuro-symbolic AI framework that can express both kinds of rules and learn these end-to-end using gradient-based optimization. Our in-depth experiments show that our LNN-based approach to learning rules for KBC leads to roughly 10% relative improvements, if not more, over SotA rule-based KBC methods. Moreover, by showing how to combine our proposed methods with knowledge graph embeddings we further achieve an additional 7.5% relative improvement.

## introduction knowledge bases (kb), e.g. freebase @xcite , are inherently incomplete thus techniques have been proposed to identify facts missing from a kb. knowledge base completion (kbc) can be broadly classified into knowledge graph embeddings (kge) and rule-based knowledge base completion (rule-based kbc). while kge has achieved highly accurate kbc by learning a low dimensional hyperspace, rule-based kbc offers superior explainability by learning rules in a humaninterpretable dialect such as first-order logic (fol). in this work, we take a closer look at rulebased kbc. a knowledge base consists of facts denoted by triples of the form ⟨h, r, t⟩ where tail t and head h denote entities while r denotes a relation. figure 1 shows a kb containing two facts ⟨al pacino, bornin, new york city⟩, ⟨new york city, partof, u. s.⟩ denoted by (solid) directed edges. the goal of rule-based kbc is to infer missing facts using fol rules such as the one shown on top of figure 1 that combines a person p 's birth city c and the country in which c is present in to ascertain p 's citizenship. by substituting p/al pacino, c/new york city, and n/u. s. into the rule we can infer that al pacino is a u. s. citizen which is a fact missing from the kb (denoted by a dashed edge in figure 1 ). most approaches that learn fol rules for kbc fall into one of two categories: edge-based (eb) and path-based (pb). we compare their relative strengths via an example. neurallp @xcite ) is perhaps the oldest example of an eb kbc model. intuitively, eb approaches learn rules by breaking paths connecting the tail and head of an existing fact into their constituent edges. for instance, to predict missing facts for r 0 in g train (figure 2 (a) ), eb learns that in the two paths connecting head and tail of ⟨v 1 , r 0 , u 1 ⟩, viz. this kind of rule is called conjunction-ofdisjunctions since the conjunction operator (∧) with and semantics is outside and the disjunction operators (∨) with or semantics are inside. in contrast to eb, recently rule-based kbc has opted for a path-based approach (pb) where paths are kept intact. given the paths connecting u 1 and v 1 , pb learns that in a missing r 0 fact the tail and head must either be connected by a path with an r 1 edge followed by an r 3 edge, or an r 2 edge followed by an r 4 edge which is expressed in fol as: in contrast to the previous rule, this kind of rule is called disjunction-of-conjunctions since the disjunction operator (∨) is outside while the conjunction operators (∧) are inside. both eb and pb's learned rules correctly predict the missing fact ⟨u 2 , r 0 , v 2 ⟩ (figure 2 -→ v 3 , is a missing fact while pb's rule predicts false in this case correctly indicating that it is a non-fact (indicated by a dashed edge). eb's claim is especially disconcerting given that u 3 and v 3 's connecting path contains an r 1 edge followed by an r 4 edge which is a relation sequence that was never observed in g train ! previous rule-based kbc approaches have more often than not restricted themselves to learning only one kind of rule, e.g., @xcite learns conjunction-of-disjunctions while minerva @xcite and rnnlogic @xcite learn disjunction-of-conjunctions. in this paper, we propose to utilize logical neural networks (lnn) @xcite , a recently proposed framework for neuro-symbolic ai (nesy) that extends boolean logic to the continuous domain while harboring close ties to fol semantics thus enabling learning fully explainable rules for kbc via gradient-based optimization. more importantly, lnn's representation power surpasses that of simpler nesy variants used in eb rule-based kbc approaches such as neurallp @xcite and drum @xcite allowing learning of both conjunction-of-disjunctions and disjunction-of-conjunctions. we propose an enhanced lnn framework that improves its applicability to kbc and propose lnn extensions of eb and pb rule-based kbc within the same nesy framework that can then be compared on kbc benchmarks fairly to determine whether the kind of rule has any effect on kbc quality (as implied by our example). lastly, since lnns extend logic to the continuous-valued domain, it is relatively simple to combine our proposals with kge to achieve further improvements in kbc quality. in the next section, we introduce notation and define the kbc task. section 3 proposes our lnn extensions highlighting its advantages over other nesy frameworks. in section 4, we apply lnn to kbc. in section 5, we present experiments comparing our models against rule-based kbc baselines before concluding with section 6. ## enhanced logical neural networks while previous work has utilized neuro-symbolic ai to learn rules for kbc, these have relied on simplistic, parameter-free alternatives to logical operators and harbor tenuous connections to boolean logic's semantics. lnns improve upon both aspects. let us take the specific example of logical conjunction ∧, a core operator in propositional logic. since ∧ is not differentiable and thus cannot be used to learn rules via gradient-based optimization, neurallp @xcite replaces it with product t-norm: prod(x, y) = xy, ∀x, @xmath0, 1] prod(x, y) is depicted in figure 3 (left) whose output smoothly interpolates from 0 to 1 as the inputs x and y increase. this is distinct from ∧ whose truth table is defined to return true (1) only when both its inputs are true (1), and false (0) otherwise. in contrast, lnn conjunction is defined as: intuitively, ⊗ divides [0, 1] into three regions: [0, 1α] representing false, (1α, α) denoting a transitionary state, and [α, 1] representing true (see figure 4 ). from the interpretability perspective, we would like the transitionary state to be as small as possible since it is unclear whether ⊗ is true or false when its output is in this range. one way to affect the size of the transitionary state is by adjusting α which acts as a tunable knob. a higher (lower) α corresponds to a larger (smaller) transitionary state with α = 1/2 representing the ideal case where the size reduces to 0 and every output of ⊗ can be interpreted either as true or false. unfortunately, the following proposition shows that this may not be possible for kbc: a system of equations is said to be feasible if there exists at least one solution for it, otherwise it is infeasible. the proof of proposition 3.1 appears in appendix a and involves manipulating the constraints in equation 1. it implies that unless we set α above a certain lower bound, we cannot learn w, β since no such parameter setting exists. on the other hand, even if we set α ≥ n/(n + 1) to satisfy the lower bound, then the size of the transition state cannot be reduced below (1 -n n+1 , n n+1 ). this has stark implications for kbc since some of the models we introduce in the next section entail feeding all relations appearing in the kg to an lnn operator, such as ⊗, to learn which relations from r are useful for predicting missing facts. in this case our main enhancement to the lnn framework is to enable it to handle high-arity inputs. to this end, we propose to exploit the fact that lnns being neural networks, allows composition of different operators. instead of burdening operators such as ⊗ to learn from high arity inputs, we use the following operator: ## kbc with logical neural networks in this section, we introduce scoring functions that can perform tail prediction using lnn operators developed so far. all our scoring functions rely on path counts. let we refer to the sequence of relations r 1 , . . . r m as its relation path. furthermore, given an m-length relation path @xmath1, . . . r m , p r (u, v) denotes the set of all paths connecting h to t in g via relation path r. ## experimental results in this section, we compare the proposed methods with other rule-based kbc approaches. recall that, in section 2.2 we pose kbc as a ranking task. thus, we adopt ranking metrics to compare the quality of the learned rules (section 5.1). to illustrate interpretability, we also list rule examples in section 5.2. for brevity, we next describe the salient points of our experimental setup next, while appendix c provides a full details. datasets, metrics and baselines: popular kbc benchmarks include kinship & umls @xcite , wn18rr @xcite , and fb15k-237 @xcite . in addition to their standard splits, @xcite define their own splits for kinship and umls which we also report results on. please see appendix b for more details underlying all of these approaches for kbc, including an extended discussion on other closely related works. implementation: following previous work @xcite , we introduce inverse triples, i.e., for each ⟨h, r, t⟩ ∈ e we add a new triple ⟨t, r -1 , h⟩ where r -1 denotes a new, inverse relation. we use adagrad @xcite to train our lnns and tune hyperparameters on the validation set with step @xmath3.1, 1.0}, margin γ ∈ {0.1, 0.5, 1.0, 2.0}, α ∈ [5/6, 1) and batch size 8. we learn rules up to length 4 for fb15k-237, 5 for wn18rr, and 3 for kinship and umls. we combine our rule-learning approach with pre-trained cp-n3 embeddings @xcite of dimension 4k for fb15k-237, 3k for wn18rr, and 8k for kinship and umls. ## conclusion our goal is to lend structure to the area of kbc. an ideal approach 1) should be interpretable, 2) can accurately identify missing facts, and 3) can be further improved using successful kbc techniques (e.g., kge). we categorized previous works into edge-based or path-based rule-based kbc, and illustrating via carefully constructed examples how one of these may lead to rules better suited for kbc. while previous rule-based kbc has utilized neurosymbolic ai (nesy), we proposed the use of logical neural networks, a particularly powerful nesy framework, that is not only differentiable but also harbors strong connections to boolean logic's semantics resulting in improved interpretability. we further enhanced the lnn framework to improve its handling of high-arity inputs that is useful for learning kbc rules. our exhaustive experiments confirm that 1) eb-lnn and pb-lnn outperform their respective counterparts available in the literature, 2) pb-lnn is the more accurate of the two, 3) combining with kge leads to further improvements in kbc quality, and 4) learned rules can be easily interpreted. @xcite 's original proposal of lnns mostly evaluated the framework on synthetic tasks with limited size datasets. besides our work in this paper, @xcite and @xcite have recently applied lnns to learning rules for short-text entity linking and helping an agent solve text-world games @xcite . since knowledge bases may benefit diverse downstream applications including but not limited to question-answering @xcite , semantic search @xcite , and dialogue generation @xcite , we expect that in future more efforts will be devoted to adapting and utilizing expressive nesy frameworks, such as lnns, to solve real-world applications. other avenues of future work include learning kbc rules more efficiently and combining eb and pb into one unified, approach for improved kbc.