Neural-Symbolic Commonsense Reasoner with Relation Predictors (2021)

Abstract: Commonsense reasoning aims to incorporate sets of commonsense facts, retrieved from Commonsense Knowledge Graphs (CKG), to draw conclusion about ordinary situations. The dynamic nature of commonsense knowledge postulates models capable of performing multi-hop reasoning over new situations. This feature also results in having large-scale sparse Knowledge Graphs, where such reasoning process is needed to predict relations between new events. However, existing approaches in this area are limited by considering CKGs as a limited set of facts, thus rendering them unfit for reasoning over new unseen situations and events. In this paper, we present a neural-symbolic reasoner, which is capable of reasoning over large-scale dynamic CKGs. The logic rules for reasoning over CKGs are learned during training by our model. In addition to providing interpretable explanation, the learned logic rules help to generalise prediction to newly introduced events. Experimental results on the task of link prediction on CKGs prove the effectiveness of our model by outperforming the state-of-the-art models.

## introduction commonsense reasoning refers to the ability of capitalising on commonly used knowledge by most people, and making decisions accordingly @xcite . this process usually involves combining multiple commonsense facts and beliefs to draw a conclusion or judgement @xcite . while human trivially performs such reasoning, current artificial intelligence models fail, mostly due to challenges of acquiring relevant knowledge and forming logical connections between them. recent attempts in empowering machines with the capability of commonsense reasoning are mostly centred around large-scale commonsense knowledge graphs (ckg), such as atomic and conceptnet @xcite . unlike conventional knowledge graphs (kg), ckgs usually contain facts about arbitrary phrases. for instance, "personx thanks persony" is connected to " to express gratitude" via the link " because x wanted". this non-canonicalised freeform text representation has resulted in having conceptually related nodes with different representation, which forms large sparse ckgs @xcite . therefore, established reasoning models on conventional kgs perform poorly on cgks @xcite @xcite . in addition, the nature of commonsense reasoning encourages dynamic ckgs, where new sets of facts and phrases are introduced frequently. most existing models in this realm are based on a static set of facts and phrases, which results in poor generalisation @xcite . nevertheless, the inference process in existing approaches is like a black box, where internal behaviour of the model is hardly interpretable. to overcome these limitations, we propose a neural-symbolic reasoning model based on backward-chaining. while traditional theorem proving algorithms @xcite ) work based on a set of predefined rules and unification over discrete symbols, we leverage a continuous relaxation of weak unification and a rule learner module. the weak unification over continuous embedding representation helps to address the challenges of unseen sparsity of ckgs. the rule learner module, in addition to providing interpretability, is used to generalise prediction to unseen data points to mitigate the problem of large-scale dynamic ckgs. the experiments on the task of link prediction confirm the superiority of our model, by a margin of up to 22 points, over the state-of-the-art models. ## related works recent approaches in knowledge base completion task have mostly relied on a graph and entityrelation embedding methods @xcite . in these approaches, entities and relations are embedded in a complex space, and using a scoring function plausibility of a triple is estimated @xcite @xcite . in addition to node embedding, graph embedding methods have also been used to capture the structural complexity of knowledge bases @xcite . language generative models also have been applied on knowledge bases in order to use the rich information of pre-trained models to address ckg completion task @xcite . @xcite proposed a method based on using language models and graph networks to solve the problem of the sparsity of ckgs, by taking structural and contextual characteristics of ckgs into account. however, the aforementioned models are highly dependant on training on a set of static entities, and fail to perform when new triples are presented. ## our approach a ckg @xmath0, e), where n is the set of nodes and e is the set of edges in g, consists of triples in form of r(h, t), where h, @xmath1, and @xmath2. the goal of the ckg completion task is to estimate probable t given a query @xmath3, ?). as the target node may not pose a direct link to h via r, this task requires a model capable of multistep reasoning. given a query r q (h q , ?), we try to identify an implication rule and apply it to prove r q (h q , t) for a target entity or event t. a rule r takes the form of r q (x, z) : -r 0 (x, y 0 ), ..., r k (y k-1 , z), where capitalised letters denote variables, r q (x, z) is the rule head, and the rule body is a conjunction of atoms. we apply such a rule by unifying atoms with triples in the given ckg to obtain r since semantically equivalent/similar events or entities in a ckg often have different surface forms, we consider weak unification of an atom with a triple instead of only considering exact match of two atoms, a weak unification operator @xcite unifies two different symbols by measuring the similarity of their representations. given a query, we do not know the target rule in advance. as shown in the example in fig. 1 , we successively create a new rule by appending the body of the previous rule with an atom in the form of r(t k-1 , x). whenever such a new atom is added, we query the ckg to find triples as candidates of unification. this step enables reasoning on large scale kbs. in contrast, the prior works @xcite ) require comparison with each node in a ckg. after applying the weak unification operator to each of the triples, we find topk most similar nodes and use each of the entity/event in the place of x to create a new atom for a new rule. the process is repeated until the maximal rule length is reached. the above mentioned reasoning process is delivered by a a neural-symbolic reasoner. it consists of a query module, a weak unification operator, and a rule creation module. query given a rule with a rightmost atom r k-1 (t k-1 , x) in the rule body, we send the representation of t k-1 as query to the given ckg to retrieve unification candidates. a node in a ckg is a word sequence. to support comparison of nodes w.r.t. their semantic similarities, we encode queries and nodes in a ckg with a pre-trained bert (devlin et al., 2019) into embeddings. to this end, a node is converted into [cls]+node+[sep ], and fed into the model, and we use the representation of [cls] token from the last layer of bert as representation of node node. we apply faiss 1 (john @xcite to index embeddings of an ckg, because it supports fast retrieval of k nearest neighbours of a dense vector. for each node v in the topk list, we collect a set of triples c(v), which are all triples having v as the head in the ckg. as a result, we have k such sets and form a candidate set c by taking the union of them. weak unification from a candidate set c we identify topk most relevant triples to unify r k-1 (t k-1 , x). first, we formulate a set of hypotheses h by replacing x with possible tails. in practice, we use all tails of the triples in c. furthermore, we construct a bipartite graph between c and h, in which an edge denotes the unification between a triple from c and another from h. we measure unification scores by using cosine similarity and obtain an similarity matrix @xmath4. the final unification score of candidate triple i is computed by max j u ij . we keep only topk highest scored candidate triples. rule creation given the topk highest scored candidate triples and a rule r k with a rightmost atom r k-1 (t k-1 , x), we create a new rule based on r k for each triple k by substituting it for r(t k-1 , x) and append another atom r k (t k , x). the relation r k is estimated by a relation predictor f θ (r k-1 , k), where both r k-1 and the current step k are mapped to the corresponding embeddings. given a query r q (h q , ?), we initialise the first rule as r q (h q , x). after reaching the pre-defined maximal rule length, we consider the score of a rule after unification as the lowest unification score associated with the rule, following @xcite . we rank all rules by their scores and select the tails in the rule heads of the topk highest scored rules as the results. another benefit of our reasoner is that humans can easily collect evidences to interpret reasoning results. the model can yield the rules and unified triples in a human-friendly format, which are generated at each step. in contrast, prior work @xcite on commonsense reasoners produces only hard-to-understand distributed representations in intermediate steps. ## experiments to evaluate the performance of our model foot_0 in the task of ckg completion, in this section, we report the results of our model in comparison with the baselines. evaluation metrics: following previous works on knowledge base completion @xcite , we report the results of hits and mean reciprocal rank. similar to @xcite , when computing the scores for a gold target entity, we filter out all remaining valid entities. furthermore, for each triple (h, r, t), the score is the average of scores measured from (h, r, ?) and (t, r -1 , ?). baselines for comparison, we report the performance of state-of-the-art models in ckg and kb completion. we compare our model to dist-mult @xcite , @xcite , conve @xcite , rotate @xcite , and malaviya @xcite . the first four models are high performance models in conventional kb completion, whereas the latter is proposed for ckg completion. ## conclusion in this work, we propose a neural-symbolic reasoning model over commonsense knowledge graphs (ckgs). our proposed model leverages a relation prediction module, which provides capability of multi-step reasoning. this ability, alongside weak unification, helps generalising our model to large-scale unseen data. we showed that our model yields state-of-the-art results when applied to large-scale sparse ckgs, and the inference step is interpretable.