Towards Unified Representations of Knowledge Graph and Expert Rules for Machine Learning and Reasoning (2022)

Abstract: With a knowledge graph and a set of if-then rules, can we reason about the conclusions given a set of observations? In this work, we formalize this question as the cognitive inference problem, and introduce the Cognitive Knowledge Graph (CogKG) that unifies two representations of heterogeneous symbolic knowledge: expert rules and relational facts. We propose a general framework in which the unified knowledge representations can perform both learning and reasoning. Specifically, we implement the above framework in two settings, depending on the availability of labeled data. When no labeled data are available for training, the framework can directly utilize symbolic knowledge as the decision basis and perform reasoning. When labeled data become available, the framework casts symbolic knowledge as a trainable neural architecture and optimizes the connection weights among neurons through gradient descent. Empirical study on two clinical diagnosis benchmarks demonstrates the superiority of the proposed method over time-tested knowledge-driven and data-driven methods, showing the great potential of the proposed method in unifying heterogeneous symbolic knowledge, i.e., expert rules and relational facts, as the substrate of machine learning and reasoning models.

## introduction symbolic reasoning methods such as rule-based expert systems @xcite are reliable and interpretable in solving complex inference problems in specialized domains, but are also difficult to generalize because eliciting a comprehensive set of rules from human experts is costly and time-consuming. recently, knowledge graph (kg) as a flexible representation of symbolic knowledge has been proven successful for knowledge-based reasoning @xcite by utilizing the distributed representations to generalize from known facts to unseen yet probably true facts, which is also known as the knowledge graph completion task @xcite . however, such models can only represent and reason about multi-relational data in the form of (subject, predicate, object) triples @xcite , not conditional if-then rules. therefore, current knowledge graph embedding models are not suited to solve inference problems where conclusions (outcomes) can be inferred from a set of observations. this current work is motivated by one overarching question: can we unify the representation of above heterogeneous symbolic knowledge to perform complex inference tasks? more concretely, we study the following research question. with a large-scale kg with rich relational facts and a moderate set of if-then rules as the prior knowledge, can we reason about the most likely conclusion(s) given a set of observations? with the rapid development of knowledge graph, the knowledge acquisition bottleneck @xcite is greatly alleviated, making it much more practicable to jointly utilize the knowledge and data for learning systems in today than in the past. recent studies have shown great success in integrating the knowledge into data-driven models, and such hybrid learning system normally achieves more favorable performance than traditional methods, as presented in fig. 1 . however, there is a general absence of sufficient labeled data in some high-stake scenarios such as medical diagnosis. moreover, such critical domains' inherent nature strictly mandates the models to be trustworthy and interpretable. these high-demanding characteristics directly challenge existing vulnerable knowledge-driven methods and data-hungry machine learning methods, and the solution still remains underexplored 2 @xcite . in this work, we formalize the above challenge as the cognitive inference problem and introduce three design goals for the model to address this problem: 1) the ability to extensively inherit existing symbolic knowledge. the model is expected to leverage not only if-then rules, but also large number of facts in knowledge graphs. 2) the ability to directly utilize existing symbolic knowledge in the reasoning procedure. this allows the model to make decent predictions based on prior knowledge, even when it is not trained. moreover, it makes the model's reasoning process interpretable. 3) the ability to be continuously optimized when training data is available. this enables the model to improve like any machine learning models. more importantly, it ensures the model's robustness so that it adapts to the nuances of real-world data that are not encoded in prior symbolic knowledge. to achieve the above goals, we first introduce the cognitive knowledge graph (cogkg), which represents relational facts and expert rules in a unified framework. specifically, it is a directed hypergraph with entities as nodes, and the relations or expert rules as edges. then, we propose a novel inference framework called coginfer that bridges the knowledge-driven and data-driven reasoning paradigms, which not only utilize explicit knowledge representations but also harvest knowledge from training examples if applicable. more precisely, it performs reasoning with symbolic knowledge, and the reasoning process could be further optimized with labeled data towards better performance. in this way, we aim to combine the symbolic reasoning and statistical learning in the same general framework coginfer, which 2 see detailed discussion in appendix a. make our method achieve the design goals as stated above and stand out from existing works. to make fair comparisons with existing knowledge-driven and data-driven baselines, we investigate the cognitive inference problem under both unsupervised and supervised settings. extensive experiments on two clinical diagnosis benchmarks show that the coginfer successfully learns from both symbolic knowledge and labeled data to address the proposed new inference task, substantially surpassing strong data-driven baselines. even without any training examples, it still outperforms existing knowledge-driven baselines that only harvests either expert rules or knowledge graph, demonstrating the great potential of the proposed framework. the main contributions of this work are three-fold: ## conclusion in this work, we introduce a new machine reasoning task, namely, cognitive inference problem, which directly challenges existing knowledgedriven and data-driven methods. to address this problem, we also introduce the cognitive knowledge graph (cogkg) that aims to unify the heterogeneous symbolic knowledge of expert rules and relational facts in knowledge graph, and propose a general framework coginfer with two implementations. experimental results on two clinical diagnosis benchmarks demonstrate the superiority of our work over existing methods. tingyu xia, yue wang, yuan tian, and yi chang. 2021. using prior knowledge to guide bert's attention in semantic textual matching tasks. in proceedings of the web conference 2021, pages 2466-2475. vikas yadav and steven bethard. 2018. a survey on recent advances in named entity recognition from deep learning models. in proceedings of the 27th international conference on computational linguistics, pages 2145-2158. zhiwei yang, hechang chen, jiawei zhang, jing ma, and yi chang. 2020. attention-based multi-level feature fusion for named entity recognition. in international joint conference on artificial intelligence. erxin yu, wenjuan han, yuan tian, and yi chang. 2020. tohre: a top-down classification strategy with hierarchical bag representation for distantly supervised relation extraction. in proceedings of the 28th international conference on computational linguistics, pages 1665-1676, barcelona, spain (online). international committee on computational linguistics. zhiyuan zhang, xiaoqian liu, yi zhang, qi su, xu sun, and bin he. 2020. pretrain-kge: learning knowledge representation from pretrained language models. in findings of the association for computational linguistics: emnlp 2020, pages 259-266, online. association for computational linguistics.