Complex Reasoning over Logical Queries on Commonsense Knowledge Graphs (2024)

Abstract: Event commonsense reasoning requires the ability to reason about the relationship between events, as well as infer implicit contextunderlying that relationship. However, data scarcity makes it challenging for language models to learn to generate commonsense infer-ences for contexts and questions involving interactions between complex events. To address this demand, we present COM2 (COMplexCOMmonsense), a new dataset created by sampling multi-hop logical queries (e.g., the joint effect or cause of both event A and B, or theeffect of the effect of event C) from an existing commonsense knowledge graph (CSKG), and verbalizing them using handcrafted rules andlarge language models into multiple-choice and text generation questions. Our experiments show that language models trained on COM2 exhibit significant improve ments in complex reasoning ability, resulting in enhanced zero-shot performance in both in-domain and out-of-domain tasks for question answering and generative commonsense reasoning, without expensive human annotations

## introduction large language models struggle to effectively perform reasoning when presented with complex tasks, such as reasoning about multiple events and their relationships. this shortcoming is due to both the inherent difficulty of reasoning over multiple pieces of information, as well as a lack of adequate-scale, supervised training datasets for learning @xcite . unfortunately, complex and multihop commonsense reasoning benchmarks @xcite are both technically challenging and financially expensive to curate. consequently, previous efforts either constructed datasets (a) with simpler reasoning structures, such as single-hop figure 1 : an example of conjunctive logical queries and their verbalization as complex commonsense inferences. to alleviate this training data bottleneck, recent works have explored extracting and formulating questions from existing commonsense knowledge graphs (cskgs; @xcite , which store commonsense triples. however, using cskgs to produce high-quality reasoning datasets poses several challenges. first, while the shared entities in commonsense triples encode a complex, interconnected graph structure, the sparsity of this structure limits the number of potential questions that encode more than one reasoning hop @xcite . second, triples in cskgs are represented in a context-free manner, such as the event "personx gets tired of it" in fig. 1 , yielding ambiguous (and sometimes incorrect) human annotations in the cskg, e.g., atomic @xcite has an error rate of over 10%. these errors propagate when triples are naively combined to construct reasoning questions. finally, also because triples in cskgs are represented in a context-free manner, additional context must be added to make questions fluent, a problem exacerbated in multi-hop settings where the entities of multiple reasoning hops must be coherently verbalized together. in this paper, we construct com 2 (complex commonsense), a novel commonsense reasoning dataset using multi-hop queries in commonsense knowledge graphs to construct question answer pairs requiring complex narrative reasoning. to build this dataset, we use conjunctive logical queries @xcite , a subset of first-order logical queries that use existential quantifiers and conjunction. the multi-hop projection operation involves inferring hidden contexts, while the intersection operation enables reasoning among multiple events, encompassing common cause or effect, and abduction. for example, in fig. 1 , an intersection of two triples can be verbalized to a short narrative, and the process of inferring the common tail can be seen as an abduction of the hidden cause between the two heads. to address the challenges above, we propose to first densify the cskg to merge nodes with high semantic similarity, increasing the connectivity of the graph. then, we use an off-the-shelf plausibility scorer to filter out low quality triples, avoiding error propagation as we construct more complicated queries. finally, we verbalize the queries to a natural language context with handcrafted rules and large language models to derive coherent and informative narrative contexts for our questions. our final com 2 dataset comprises 790k questionanswer pairs (both with multiple-choice and generative answer settings), including 1.3k examples that we manually verify for evaluation. our results demonstrate the challenges faced by even powerful llms and supervised question answering models on the com 2 dataset, underscoring the difficulty of performing complex multihop reasoning. moreover, fine-tuning question answering models and generative commonsense inference models on com 2 leads to substantial improvements across eight commonsense reasoning datasets, showing the efficacy of our framework for boosting commonsense reasoning ability. to conclude, our contributions are three-fold. first, we present a pipeline for sampling and verbalizing complex logical queries from cskgs, to form a complex commonsense reasoning benchmark, com 2 , with minimal human effort. second, we benchmark the complex reasoning ability of various state-of-the-art language models and question answering models on com 2 . finally, we validate the benefit of fine-tuning on com 2 on eight zeroshot commonsense reasoning datasets. previous efforts on answering logical queries on knowledge graphs focus on constructing box embeddings @xcite , embeddings based on beta distributions @xcite , particle simulations @xcite , and computation tree optimization @xcite . other related works focus on leveraging two-hop projection and intersection queries in conceptnet to improve commonsense question answering @xcite , inferring missing entities in verbalized complex queries on factual knowledge graphs @xcite , and developing an llm agent for complex operators within the kg @xcite . instead of relying on embeddings or limited query types for matching synthetic logical queries, we leverage the concept of logical queries to effectively acquire complex reasoning data from cskgs with minimum human efforts. complex commonsense reasoning recent advances in commonsense reasoning have been driven by the construction of human-annotated @xcite @xcite , 2021; @xcite @xcite and humanvalidated @xcite com-monsense knowledge graphs (cskg). a common approach to create challenges for commonsense reasoning involves constructing tasks in the form of question-answering @xcite , knowledge base completion @xcite and population @xcite , grounding @xcite , and daily dialogue @xcite , based on cskgs. however, most of those previous benchmarks are based on one-hop triples. in contrast, real-world situations in narratives usually involve more complicated reasoning across multiple events, sentences, and paragraphs @xcite . previous works learn representations of narrative chains @xcite and draw inferences @xcite . to address more complex paragraph-level or multievent reasoning, paracomet @xcite proposed to pre-train on distantly supervised onehop paragraph-level commonsense inferences, and comet-m @xcite was fine-tuned on a crowdsourced corpus focusing on reasoning on multiple events. instead of crowdsourcing or using language models to distill complex inferences, we provide narrative-level inference by verbalizing complex logical queries over cskgs, to effectively acquire grounded inferences at scale. ## methodology in this section, we introduce the construction details of com 2 , including the pre-processing, sampling, and verbalization of complex queries, as well as the details of human annotations. an ovewview of the pipeline is presented in fig. 2 . ## experiments we conduct experiments on the evaluation set of com 2 , formulated as a multi-choice question answering (mcqa) task. specifically, we examine the performance of state-of-the-art off-the-shelf language models on com 2 , and also study the effect of training a question answering model on the distantly supervised training set of com 2 . ## downstream evaluation in addition to benchmarking complex commonsense reasoning, we also study the effect of leveraging com 2 as training data to generalize to other downstream commonsense reasoning tasks. as tasks, we use zero-shot commonsense question answering (csqa), and generative commonsense inference, including one-hop, multi-event, and paragraph-level settings. ## conclusion in this paper, we leverage the concept of conjunctive logical queries to create a complex commonsense reasoning dataset derived from cskgs. the dataset, com 2 , comprises a human-annotated evaluation set and a distantly supervised training set without further annotations. our experiments highlight the challenging nature of complex commonsense reasoning that involves multiple events or multi-hop scenarios, even for advanced language models such as gpt-4. additionally, we train question answering models and generative commonsense reasoning models using com 2 . the re-sults show significant improvements across eight diverse downstream commonsense reasoning tasks, highlighting the potential of leveraging cskgs to acquire complex reasoning signals inexpensively, without relying on extra human effort.