CIKQA : Learning Commonsense Inference with a Unified Knowledge-in-the-loop QA Paradigm (2023)

Abstract: We propose a new commonsense reasoning benchmark to motivate commonsense reasoning progress from two perspectives: (1) Evaluating whether models can distinguish knowledge quality by predicting if the knowledge is enough to answer the question; (2) Evaluating whether models can develop commonsense inference capabilities that generalize across tasks. We first extract supporting knowledge for each question and ask humans to annotate whether the auto-extracted knowledge is enough to answer the question or not. After that, we convert different tasks into a unified question-answering format to evaluate the models’ generalization capabilities. We name the benchmark Commonsense Inference with Knowledge-in-the-loop Question Answering (\name). Experiments show that with our learning paradigm, models demonstrate encouraging generalization capabilities. At the same time, we also notice that distinguishing knowledge quality remains challenging for current commonsense reasoning models.

## introduction understanding human language requires both language knowledge (e.g., grammar and semantics) and world knowledge, which can be further divided into factual and commonsense knowledge @xcite . recently, the community has made great progress in helping machines acquire and apply language and factual knowledge. however, how to help machines acquire and infer over commonsense is still unclear. to answer this question, many commonsense reasoning datasets @xcite @xcite @xcite have been proposed. even though they target different knowledge types, modalities, and formats, they often follow a standard supervised learning setting that aims at helping machines solve a specific task with training data. however, two limitations of this learning paradigm have restricted the development of commonsense reasoning systems. first, there is no clear separation between knowledge and inference. as discussed in @xcite , a common phenomenon is that larger training data will lead to better performance, mainly because richer knowledge is covered. however, due to the large scale of commonsense knowledge, it is infeasible to annotate a large enough training set for each task, and the responsibility of the training data should be teaching models how to make inferences rather than acquire commonsense knowledge. several recent works have explored using structured knowledge for commonsense reasoning tasks @xcite @xcite . however, as these works did not clearly analyze the coverage of the structured knowledge (i.e., knowledge graphs (kgs)), it is still unclear what the performance means, better knowledge coverage, or better inference capability. to investigate what is behind this learning process, we propose to equip each question with auto-extracted knowledge and ask humans to annotate whether the knowledge is sufficient to answer the question. by doing so, we could evaluate whether models can know if the provided knowledge is good or not and how well they can conduct inference over the provided knowledge to solve the task. second, supervised learning may force the model to learn the distribution of the training data rather than a universal inference model. as a result, the model may perform well on the test set that follows the same distribution but fail to generalize @xcite . previously, as different tasks have different formats, it is hard to evaluate the generalization ability of commonsense reasoning models. following the trend of figure 1 : cikqa demonstration. all tasks are converted into a unified format such that we can easily evaluate the generalization capability of all models. we also equip all questions with auto-extracted knowledge graphs from existing kgs and ask humans to annotate whether the knowledge is gold or not. in this example, we expect models to first identify the quality of the knowledge and then conduct inference over the knowledge to solve the question. using a unified format (i.e., question answering) for different tasks @xcite , we propose to convert various commonsense reasoning tasks into a unified qa format such that we can easily and fairly evaluate the generalization ability of learned commonsense reasoning models. combining these two lines of effort, we propose a new commonsense inference benchmark commonsense inference with knowledge-in-theloop qa (cikqa). an example is shown in figure 1 . we first convert several popular commonsense reasoning tasks into a unified qa format and equip them with the relevant knowledge from existing commonsense knowledge graphs. we leverage human annotation to label whether the provided knowledge is correct and enough foot_0 to answer the question. the cikqa benchmark can motivate us to answer two questions: (1) whether current models can distinguish the knowledge is gold or not; (2) can current commonsense inference models generalize across different commonsense reasoning tasks? experiments with several recent knowledgebased commonsense reasoning models show that even though current deep models could learn to conduct simple inferences after training with a few examples when gold knowledge is provided, they still cannot learn to distinguish gold knowledge very well. moreover, although current mod-els demonstrate encouraging generalization ability across the three tasks we consider, they still struggle with complex inference (e.g., abductive reasoning). we hope that our benchmark foot_1 can motivate more advanced commonsense inference methods in the future. ## dataset construction in cikqa, to encourage a generalizable commonsense inference model, we follow previous work @xcite @xcite to unify all selected tasks as a binary question answering problem, and equip each question with a supporting knowledge graph g retrieved from existing commonsense kgs. we leverage crowd-sourcing workers to annotate whether the knowledge is gold (i.e., accurate and enough) for answering the question. with that, we can evaluate whether models know how to distinguish gold and knowledge and whether they can learn the generalizable inference with the help of the knowledge. in total, cikqa contains 15 thousand instances from four kinds of commonsense reasoning tasks. details about task selection, format unification, knowledge extraction, and annotation are as follows. ## experiment setup we present the performance of the following commonsense inference models on cikqa: for each candidate answer, we concatenate it with the question and feed it to the model. after getting the sentence representation, a linear layer is used to obtain a score and trained with a cross-entropy loss. to further utilize the knowledge, mhka @xcite uses a transformer network to model the paths from the question nodes and answer nodes, then concatenates the knowledge and context representation for the final prediction. ## result analysis we first conduct analysis experiments to evaluate to what extent the provided knowledge could help existing models. for each model, we train it with different numbers of training instances and report the average performance and standard deviation of five trials. experiment results with all instances and the gold subset of cikqa, where only instances with gold knowledge are used for training and testing, are presented in figure 2 and 3, respectively. from the results, we can make the following observations. first, when explicitly including the knowledge, all inference models outperform the baseline model with no supporting knowledge, especially g2t. when the autoextracted and gold knowledge is provided, g2t outperforms the baseline vanilla lm model by 4.17 and 15.34 accuracy, respectively. it supports our assumption that learning all knowledge from the limited training data is hard, and external structured knowledge could help. at the same time, we also notice a significant gap between autoextracted knowledge and gold knowledge. for example, if gold knowledge is available, models could learn to answer the questions with only a few examples. this indicates that the knowledge quality can significantly impact models' performance, which further shows the importance of distinguishing whether the knowledge is gold or not automatically. last but not least, we can see that g2t outperforms other inference models in most settings, which shows that with the help of current large-scale lms, jointly encoding questions and knowledge is more efficient and a more effective strategy than acquiring them separately. due to the simplicity and efficiency of g2t, we will conduct the rest analysis experiments with g2t. ## related work to help machines understand commonsense, the community has devoted great efforts to constructing commonsense knowledge bases with either crowdsourcing (e.g., conceptnet @xcite and atomic @xcite ) or information extraction techniques (e.g., aser @xcite ). typically, crowd-sourced knowledge bases are of higher quality, and the autoconstructed ones have broader coverage. besides acquiring commonsense knowledge, the community also developed many commonsense reasoning datasets to train and test models' commonsense reasoning abilities. even though these datasets may have different formats (e.g., slot fitting in winogrande @xcite and question answering in commonsenseqa @xcite ), knowledge types (e.g., causal commonsense in copa @xcite and numerical commonsense in numersense @xcite ), or modalities (e.g., visual commonsense in vcr @xcite and textual commonsense in many others), they follow a standard supervised learning setting, and aim at helping machines to solve a specific commonsense task in an end-to-end manner. given this setting, it is often difficult to tell what has been learned during the training. was it used to acquire commonsense knowledge, learn to conduct commonsense inference, or both? such ambiguity limits our progress in solving these commonsense reasoning tasks. in this work, we connect the efforts on commonsense acquisition and inference by creating a commonsense inference benchmark cikqa , where models can focus on learning to identify the gold knowledge and perform inference over the sup-porting commonsense knowledge. answering questions in natural language based on a knowledge base (kb) is a mature research topic in the nlp community, which is also known as the kbqa problem @xcite @xcite @xcite . previous work mainly focuses on factual knowledge, which is stored in the triplets format. the main challenge is to parse the question and then precisely and effectively identify the correct path over a large-scale kb to make the inference. compared with inference over factual knowledge, inference over commonsense knowledge brings the following unique challenges: (1) commonsense is a kind of preference rather than fixed knowledge. as a result, the ideal commonsense reasoning process could involve the comparison of multiple candidates. for example, both "drink coffee" and "drink bear" could happen in the morning, but a normal person will prefer "drink coffee;" (2) beyond named entities, commonsense knowledge also covers daily entities and events, and thus it is difficult to find an exact node from the commonsense kb that matches the question, and we may need to conduct inference based on the partial match (i.e., the extracted nodes are relevant but not identical). ## conclusion in this paper, we present cikqa, a unified commonsense inference benchmark. specifically, we first convert several popular commonsense tasks into a unified qa format and then equip each question with a supporting commonsense knowledge graph. we also leverage humans to annotate the quality of auto-extracted knowledge. experiments show that even though models can better learn how to perform commonsense inference with a few ex-amples and significantly outperform the baseline method that does not use structured knowledge in the data-scarce setting, identifying the gold knowledge is still a challenging problem. more interestingly, with our unified formulation, models demonstrate the encouraging generalization ability across tasks. as both the format unification and supporting graph extraction are automatic, we can easily extend to other commonsense reasoning tasks in the future.