Zero-shot Commonsense Reasoning over Machine Imagination (2024)

Abstract: Recent approaches to zero-shot commonsense reasoning have enabled Pre-trained Language Models (PLMs) to learn a broad range of commonsense knowledge without being tailored to specific situations. However, they often suffer from human reporting bias inherent in textual commonsense knowledge, leading to discrepancies in understanding between PLMs and humans. In this work, we aim to bridge this gap by introducing an additional information channel to PLMs. We propose Imagine (Machine Imagination-based Reasoning), a novel zero-shot commonsense reasoning framework designed to complement textual inputs with visual signals derived from machine-generated images. To achieve this, we enhance PLMs with imagination capabilities by incorporating an image generator into the reasoning process. To guide PLMs in effectively leveraging machine imagination, we create a synthetic pre-training dataset that simulates visual question-answering. Our extensive experiments on diverse reasoning benchmarks and analysis show that Imagine outperforms existing methods by a large margin, highlighting the strength of machine imagination in mitigating reporting bias and enhancing generalization capabilities.

## introduction commonsense reasoning has been considered a crucial milestone in the pursuit of artificial general intelligence @xcite . while pre-trained language models (plms; @xcite often exhibit near-human reasoning capabilities after being fine-tuned on specific commonsense datasets, they face challenges in zeroshot scenarios where examples differ significantly from their training data distribution @xcite ., existing model (car) imagine (ours) q. how do you butter toast? 2019; @xcite . overcoming this limitation is crucial for achieving human-level proficiency in natural language understanding. one promising approach to this limitation is injecting commonsense knowledge from external knowledge bases (kbs; @xcite into plms. specifically, this involves transforming knowledge entities into a questionanswering (qa) format, resulting in a synthetic qa dataset. this constructed dataset is then used to train plms similarly to the pre-training phase. since the knowledge bases can cover a wide spectrum of commonsense knowledge, this approach leads to substantial improvements in reasoning ability across diverse situations without specializing in specific knowledge @xcite . however, they often suffer from human reporting bias @xcite , as textual commonsense knowledge only captures the most frequently occurring scenarios, thereby neglecting less common but equally critical knowledge necessary for comprehensive reasoning. figure 1 illustrates a case where a recent model @xcite fails to accurately reason about the question "how do you butter toast?". since the existing models rely solely on textual inputs, they often neglect contextual details, such as the fact that butter is typically too solid to be dipped. in contrast, humans can easily answer such questions by visually imagining the shape, solidity, and interactions of butter with other objects. this observation motivates us to explore additional modalities to complement textual commonsense knowledge. in this paper, we introduce imagine (machine imagination-based reasoning), a novel zero-shot commonsense reasoning framework designed to circumvent the reporting bias inherent in textual inputs. inspired by the cognitive studies highlighting the beneficial effects of visual imagery on language understanding @xcite , imagine is designed to leverage visual signals to complement textual inputs. to achieve this, we integrate plms with a conditional image generator, enabling machine imagination capabilities. to guide the model in learning to utilize visual and textual inputs jointly, we create a synthetic vqa dataset, which is then used to optimize plms. by acquiring a broad spectrum of commonsense knowledge along with visual signals, imagine enhances reasoning capabilities while circumventing human reporting bias. to verify the effectiveness of imagine, we perform extensive experiments, encompassing diverse reasoning benchmarks, architectures, and scales. the experimental results convincingly demonstrate that imagine surpasses existing methods, including large language models, in reasoning capabilities. moreover, our in-depth analysis reveals that imagine effectively enables plms to adaptively leverage machine imagination capabilities in a beneficial manner. the contributions of this paper include the following: ## machine imagination-based reasoning in this section, we elaborate on the proposed method, namely imagine (machine imaginationbased reasoning), for zero-shot commonsense reasoning. the core strategy is to complement textual commonsense knowledge with visual signals derived from machine-generated images. to achieve this, we first couple the plms with a text-to-image generator ( §3.1), enabling machine imagination in text-based plms. we then construct a large-scale synthetic vqa dataset to learn the joint use of textual and visual signals in the reasoning process ( §3.2). by optimizing the model with additional signals that encapsulate commonsense knowledge, imagine can effectively perform commonsense reasoning while avoiding human reporting bias inherent in textual inputs ( §3.3, §3.4). the overall procedure is depicted in figure 2 . ## experiments in this section, we demonstrate the effectiveness of imagine. specifically, we conduct extensive experiments and analysis to answer the following research questions: sentence composition (qasc; @xcite , science questions (sciq; @xcite , and the ai2 reasoning challenge (arc-easy, arc-challenge; @xcite . given that science qa datasets often contain various types of reporting bias, such as color and shape biases, we selected these datasets to verify the efficacy of imagine in mitigating reporting bias. baselines. we mainly compare imagine with the following zero-shot commonsense reasoning frameworks: mr @xcite , smlm (banerjee and baral, 2020), zero-shot fusion (kim et al., 2022), car @xcite , and the stateof-the-art framework, candle @xcite . to confirm the efficacy of training with machine imagination in imagine, we also compare it with z-lavi @xcite , which leverages machine imagination but does not include the training process. beyond the reasoning framework based on kbs, we evaluate the recent llms, which include llama2 13b @xcite , mistral 7b (v0.1) @xcite , opt 30b @xcite , flan 137b @xcite , and the gpt families (i.e., gpt-3.5, chatgpt (gpt-3.5-turbo), gpt-4). backbones. to verify the general applicability of imagine, we apply our method to the both encoder and decoder models. specifically, following the previous works, we utilize roberta-large @xcite and deberta-v3-large @xcite . each model has 362m and 443m parameters, respectively. as for the decoder model, we use gpt-2-large that involves 792m parameters. implementation details are in appendix b. a1. paper a2. exam [piqa] q: brush dust off eyebrows a1: use toothbrush to groom a2: use dental floss to groom [anli] q. everyone laughed at the funny video. a1. they took a study break to film videos a2. beth found a funny cat video. [siqa] q. after starting the bar fight kai told riley that he had better go quickly. how would you describe riley? a1. a trouble maker a2. full of adrenaline a3. a peace maker [csqa] q. what part of a table would you put a ruler in? a1. drawer a2. desk a3. the backside a4. office a5. measure distance existing model: a2(x) imagine: a1(o) existing model: a3(x) imagine: a3(x) existing model: a1(x) imagine: a2(o) existing model: a1(o) imagine: a1(o) existing model: a2(x) imagine: a2(x) [csqa] q. where usually lacks an elevator but sometimes has a telephone book? a1. at hotel a2. kitchen a3. library a4. telephone booth a5. house existing model: a4(o) imagine: a5(x) ## conclusion in this paper, we have proposed imagine, a novel zero-shot commonsense reasoning framework that leverages visual signals to mitigate reporting bias in textual inputs. to steer imagine in effectively utilizing visual information, we have created a largescale synthetic vqa dataset and optimized the model to use both textual and visual information. our extensive experiments have shown that imag-ine establishes new state-of-the-art results on zeroshot commonsense reasoning tasks compared to strong baselines (including large language models), demonstrating the efficacy of machine imagination. moreover, the in-depth analysis clearly supports the strength of the proposed method by showing that the model tends to utilize visual information beneficially.