End-to-end Case-Based Reasoning for Commonsense Knowledge Base Completion (2023)

Abstract: Pretrained language models have been shown to store knowledge in their parameters and have achieved reasonable performance in commonsense knowledge base completion (CKBC) tasks. However, CKBC is knowledge-intensive and it is reported that pretrained language models’ performance in knowledge-intensive tasks are limited because of their incapability of accessing and manipulating knowledge. As a result, we hypothesize that providing retrieved passages that contain relevant knowledge as additional input to the CKBC task will improve performance. In particular, we draw insights from Case-Based Reasoning (CBR) – which aims to solve a new problem by reasoning with retrieved relevant cases, and investigate the direct application of it to CKBC. On two benchmark datasets, we demonstrate through automatic and human evaluations that our End-to-end Case-Based Reasoning Framework (ECBRF) generates more valid, informative, and novel knowledge than the state-of-the-art COMET model for CKBC in both the fully supervised and few-shot settings. We provide insights on why previous retrieval-based methods only achieve merely the same performance with COMET. From the perspective of CBR, our framework addresses a fundamental question on whether CBR methodology can be utilized to improve deep learning models.

## introduction commonsense knowledge helps humans navigate everyday situations seamlessly @xcite and is required for many intelligent scenarios @xcite . to automatically enlarge the scale of commonsense knowledge base for the benefit of reducing labeling labor and expense, knowledge graph completion (kgc) has become a hot research topic @xcite . the general kgc task is to expand existing knowledge graphs by using well-trained classifiers-they are trained with existing annotated samples and predict whether or not there is a relationship between two existing entities in a knowledge graph @xcite . although kgc methods can automatically find unlabeled relationships, they are always classification or ranking tasks and are limited to existing entities in a knowledge graph and can't extend to new entities @xcite . to extend to new entities, comet @xcite proposes to use text generation for exploring and discovering new entities, which is called commonsense knowledge base completion (ckbc) task, utilizing the knowledge within the pretrained language models (plm), which has been with process in recent years @xcite . specifically, comet uses subject and relation as direct input to plm and aims to generate objects, most of which are novel and unseen entities. however, ckbc is knowledge-intensive, requiring wide-ranging and detailed knowledge; and it is reported that the ability of pretrained language models to access and precisely manipulate knowledge is limited @xcite . one potential solution to this is to provide "non-parametric knowledge" through additional input. @xcite and @xcite , for example, have shown that by retrieving passages that contain knowledge relevant to the current task, performance can be improved. for ckbc, unfortunately, it might be especially difficult to find useful passages that contain relevant commonsense knowledge from the web due to a reporting bias @xcite in which people rarely express the obvious (i.e., commonsense knowledge). an example of reporting bias from in addition, to prevent ecbrf from overfitting to some commonly retrieved cases, we propose random mask as a training strategy that randomly masks the retrieved cases during training, which functions similar to dropout @xcite and further improves the performance of the framework. we also analyze several variations to better understand the process. although past attempts suggest that similar retrieval-based methods cannot improve the performance of ckbc @xcite , on two benchmark datasets, we demonstrate through automatic and human evaluations that our end-toend case-based reasoning framework (ecbrf) foot_1 generates more valid and informative knowledge than (1) the state-of-the-art comet model @xcite for ckbc which employs no case retrieval, and (2) a baseline model that employs random case retrieval -on both fully supervised and few-shot settings. we also provide an analysis on why different conclusions are reached. in addition, our framework draws insight from case-based reasoning (cbr), and also has contributions to the cbr research. cbr is a subject in classical ai that solves a new problem by reusing the solutions of retrieved seen similar problems stored in the case base @xcite ). cbr's methodology has four steps -case retrieval, reuse, revise and retain. past years of accomplishment in deep learning (dl) have led to enthusiasm in the cbr community to apply dl in the service of cbr. however, based on the observation that many challenges remain in dl where cbr has advantages (e.g. few-shot learning), some cbr researchers @xcite advocate using cbr to complement dl. however, past works on using cbr to complement dl only limit to shallow neural networks (nn) @xcite @xcite . the latest work even suggests that in many tasks nn itself outperforms cbr-complemented nn @xcite , which raises fundamental questions on whether cbr methodology is useful for dl. our work addresses this doubt by being the first to show a concrete implementation of the integration of the full methodology of cbr to plm (as one typical model in dl) (we show in §6 that we simulate the third step in the methodology of cbr instead of actually implement it, since it requires huge human efforts) and show that the integration method can benefit from multiple steps in the methodology of cbr, and can lead to better performance over plm itself in both fully supervised settings and few-shot settings on ckbc. notably our proposed framework has a larger advantage in fewshot settings, where cbr methods typically have advantage. we also find that the generation of our framework is largely related to the retrieved case especially when they are similar, which exhibits strong case-based reasoning patterns. in addition, a detailed analysis of our framework from a cbr perspective is provided in §6. our contributions can be summarized as follows: (1) drawing insights from cbr, we introduce a new end-to-end framework for ckbc task. we also propose training strategies that can better utilize the retrieved knowledge. (2) we conduct extensive experiments on the ckbc task in various settings (e.g. fully supervised and few-shot), and the results consistently demonstrate that our proposed framework achieves improvements over the state-of-the-art baseline methods. (3) from the perspective of the cbr community, whether cbr methodology can be used to improve dl models remains a fundamental research question. we address this doubt by being the first to show a concrete implementation of the integration of the full methodology of cbr to plm, and showing that such integration can achieve better performance than single plm. a thorough analysis of the integration from cbr perspective is also provided. reasoning in nlp cbr could be seen as a type of analogical reasoning @xcite , and analogical reasoning belongs to inductive reasoning @xcite . inductive reasoning @xcite is different from deductive reasoning @xcite (both belong to logical reasoning) that the premise in inductive reasoning can not provide conclusive support to its conclusion. ## task definition in the generative ckbc task, a knowledge data instance is represented as a tuple of subject, relation, and object: (sub, rel, obj). all sub and obj are in natural language phrases (figure 1 ). rel can be used as either a special token or the corresponding natural language phrases @xcite . here we use rel as natural language phrases. the task is that given a pair of sub and rel, the goal is to generate the corresponding obj. ## methodology we start by formalizing our framework as a retrievethen-predict generative process. then in §4.2, we describe our ecbrf's modules for the generative process in detail. finally, we present a hybrid training strategy for better regularization. figure 1 describes our method. in the figure, "query" stands for a sub and rel pair which is used as input to ecbrf to generate obj. "case base" is initialized with knowledge triples from the training set. "cases" means the retrieved knowledge triples from the "case base". "in-context demonstrations" stand for the retrieved cases that are used for input augmentation (concatenate with the query). the subject, relation, and object of the retrieved cases are sub-scripted with "r" (e.g., sub r ). ## experiments & analysis in this section, we introduce the experiment datasets and evaluation details, as well as experiment setups and the experiment results, measured with automatic and human evaluations. ## conclusion drawing insights from cbr, we propose an endto-end framework for the ckbc task. we demonstrate through automatic and human evaluations that our framework generates more valid knowledge than the state-of-the-art comet model in both the fully supervised and few-shot settings. from the perspective of cbr, our framework addresses a fundamental question on whether cbr methodology can be utilized to improve deep learning models.