Identify, Align, and Integrate: Matching Knowledge Graphs to Commonsense Reasoning Tasks (2021)

Abstract: Integrating external knowledge into commonsense reasoning tasks has shown progress in resolving some, but not all, knowledge gaps in these tasks. For knowledge integration to yield peak performance, it is critical to select a knowledge graph (KG) that is well-aligned with the given task’s objective. We present an approach to assess how well a candidate KG can correctly identify and accurately fill in gaps of reasoning for a task, which we call KG-to-task match. We show this KG-to-task match in 3 phases: knowledge-task identification, knowledge-task alignment, and knowledge-task integration. We also analyze our transformer-based KG-to-task models via commonsense probes to measure how much knowledge is captured in these models before and after KG integration. Empirically, we investigate KG matches for the SocialIQA (SIQA) (Sap et al., 2019b), Physical IQA (PIQA) (Bisk et al., 2020), and MCScript2.0 (Ostermann et al., 2019) datasets with 3 diverse KGs: ATOMIC (Sap et al., 2019a), ConceptNet (Speer et al., 2017), and an automatically constructed instructional KG based on WikiHow (Koupaee and Wang, 2018). With our methods we are able to demonstrate that ATOMIC, an event-inference focused KG, is the best match for SIQA and MCScript2.0, and that the taxonomic ConceptNet and WikiHow-based KGs are the best match for PIQA across all 3 analysis phases. We verify our methods and findings with human evaluation.

## introduction recently, several datasets @xcite @xcite have been released to tackle the challenge of commonsense reasoning. while deep pretrained language-models (lms) @xcite @xcite have been at the top of most leaderboards, they still have shortcomings when it comes to commonsense reasoning @xcite @xcite . thus, incorporating knowledge graph (kg) information into these models is an active area of research @xcite @xcite . however, when selecting a kg match for a task, it is often difficult to quantitatively assess what kind of knowledge is missing from these models and how much of the missing knowledge required for the task is available in a candidate kg. it is also critical to examine how easily transformer-based models can learn commonsense knowledge, to determine the benefits of integrating a kg. we investigate how well a kg matches with a task objective, referred to as kg-to-task match. we use a 3-step process that examines knowledge identification, alignment, and integration. we utilize a modular pipeline approach to allow for interpretable results and easy replacement of new and different modules. our approach reveals features such as: how often a kg identifies a knowledge gap in a question-answer pair (identification), whether a kg identifies the correct knowledge gap (alignment), and whether the inserted knowledge correctly fills the knowledge gap required for the task (integration). these steps are depicted in fig. 1 . we also compare the effects of knowledge content, structure, and shape. the results of this analysis are impacted by the model we use, and thus we also develop probes to examine how much commonsense knowledge lms already know and how easy it is for them to learn. we evaluate our kg-to-task models in a qa probe setup to examine how much commonsense is learned with and without the matched kg. our probes are automatically built from atomic, enabling us to leverage existing knowledge sources as a probing base without relying on expensive collection methods. we also include an mlm probe setup to obtain zero-shot and fine-tuned results on probes for social relations, agent-patient assignment, and world knowledge. we present detailed empirical results on three diverse datasets: socialiqa (siqa) task @xcite , which requires social knowledge; physical iqa (piqa) @xcite ) which requires physical knowledge; and mcscript2.0 @xcite , which requires commonsense script knowledge, not restricted to a particular domain. since both siqa and piqa require a particular domain of commonsense knowledge, these tasks allow us to draw strong conclusions about kg integration, as knowledge must be well aligned with the tasks to yield performance gains. analyzing mcscript2.0, on the other hand, allows us to understand how this analysis applies to a task where the best match is not obvious. we compare kg-totask match with three diverse kgs: atomic @xcite , conceptnet @xcite , and automatically extracted subgraphs from wikihow. each kg is tailored for a different commonsense domain: atomic focuses on social commonsense, conceptnet on taxonomic commonsense, and wik-ihow on instruction-based commonsense. this allows us to see how different tasks require different types of commonsense knowledge. to investigate kg-to-task match, we follow three phases: identify, align, and integrate. in our first phase, we examine knowledge gap iden-tification by analyzing our extraction quantities. in our second phase, we examine alignment by utilizing a 'knowledge-surrounded' (ks) model, in which we replace task candidate answers with knowledge-surrounded answers. we found that atomic is the best match for siqa across both identification and alignment: 11% more atomic data is extracted for question-answer knowledge gaps than conceptnet data, with a 4.8% performance increase over bert using our atomic ks model. we use our third phase, integration, to investigate the classification change distributions from bert to the ks model, finding that our model is more confident about correct classification changes, supporting the atomic-siqa match. additionally, both conceptnet and wikihow graphs outperformed atomic on piqa: 8% more conceptnet data is extracted than atomic and a 17.4% performance increase is achieved with our concept-net ks model, whereas we get a 15.5% increase with our wikihow ks model. finally, we find that atomic is the best match for mcscript2.0, with a 2.7% increase with our atomic ks model. we also perform human evaluation and show important connections between the analysis phases. we see that if our ks model shows improvement for high quality settings, our extraction step is a valid knowledge-gap identification metric between 74% and 89% of the time, depending on the dataset. we also show that our best alignment strategy for atomic-siqa fills knowledge gaps 66% of the time, outperforming the best alignment strategy for conceptnet-siqa, which supports our ks model performance results. we find similar trends for piqa alignment and also find that the amount of information available at inference time may affect alignment results for mcscript2.0. human evaluation shows that 93% of atomic-siqa ks model prediction changes (with respect to the baseline) select the answer with the highest knowledge quality, verifying our integration phase as a quality metric. our commonsense qa probes before and after kg integration show that our ks model only considerably outperforms the bert baseline on certain relational probes, indicating the type of knowledge gaps atomic is better at resolving, e.g., relational knowledge such as feelings, reactions, etc. overall, our methods not only illustrate the type of knowledge that current transformer-based models are missing to approach human-level commonsense reasoning but also how we can identify, align, and integrate knowledge between a kg and a task to find the best match to fill in these missing gaps of reasoning. ## mlm commonsense probes we evaluate the transformer-based models used in our setup to assess how much knowledge lms already know and how easy it is for them to learn. ## conclusion we proposed a method to analyze how well a candidate kg can correctly identify and accurately fill in gaps of reasoning for a given task. we presented a three step approach for analyzing this kg-to-task match via identification, alignment, and integration. we found that the atomic kg aligns best with the siqa task, and quantitatively analyze the quality of the extracted commonsense. we also found that the conceptnet and wikihow based kgs match best with the piqa task. finally, we see that the atomic kg also aligns best with mcscript2.0, which was a novel discovery and is most likely a result of the task's script knowledge requirement. wikihow title, and each sentence in the title's corresponding paragraph.