Revisiting Generative Commonsense Reasoning: A Pre-Ordering Approach (2022)

Abstract: Pre-trained models (PTMs) have lead to great improvements in natural language generation (NLG). However, it is still unclear how much commonsense knowledge they possess. With the goal of evaluating commonsense knowledge of NLG models, recent work has proposed the problem of generative commonsense reasoning, e.g., to compose a logical sentence given a set of unordered concepts. Existing approaches to this problem hypothesize that PTMs lack sufficient parametric knowledge for this task, which can be overcome by introducing external knowledge or task-specific pre-training objectives. Different from this trend, we argue that PTM’s inherent ability for generative commonsense reasoning is underestimated due to the order-agnostic property of its input. In particular, we hypothesize that the order of the input concepts can affect the PTM’s ability to utilize its commonsense knowledge. To this end, we propose a pre-ordering approach to elaborately manipulate the order of the given concepts before generation. Experiments show that our approach can outperform the more sophisticated models that have access to a lot of external data and resources.

## introduction pre-trained models (ptms), such as bart @xcite and t5 @xcite , have achieved significant progress in many natural language generation tasks. however, their ability to reason with common sense while generating text is questionable. to push research in this direction, @xcite proposed the task of generative commonsense reasoning (gcr), where the goal is to compose a fluent and rational sentence from a set of concepts. figure 1 shows an example of this problem. to achieve this goal, the model must do commonsense reasoning to build connections between the given concepts and produce a logically sound sentence (e.g., it is the pitcher who throws the ball to the batter rather than the other way). prior works hypothesize that the vanilla ptms are not capable of solving this challenging task @xcite @xcite partly because their self-supervised objectives do not explicitly capture the relational commonsense knowledge @xcite . these works enhance the ptms' performance by explicitly introducing knowledge during fine-tuning or implicitly teaching the model during further pretraining. however, we observe that in some cases, even without external knowledge, ptms can create reasonable output for this task, indicating that ptms may already have the commonsense reasoning ability to some degree. therefore the challenge turns out to be how to make it easier for ptms to fully utilize the inherent commonsense knowledge. one potential solution of this challenge is to make the order of input concepts more natural and aligned with commonsense. for example, in figure 1 , taking {pitcher, throw, ball, batter} as the input is better than {batter, throw, ball, pitcher}, since the order of concepts in the former input is more close to that in the outputs. models that are not pre-trained, such as lstm and gru, prefer a pre-ordering of input tokens to align them with the (expected) output @xcite . for ptms, recent works @xcite @xcite show that they can achieve reasonable performance on graph-to-text tasks without pre-ordering. however, the impact of pre-ordering on ptms, in general, is not well analyzed. in this work, we revisit ptms' ability of generative commonsense reasoning without access to ex-ternal knowledge or task-specific pre-training. we choose bart and t5, two state-of-the-art ptms, as our underlying models. to analyze the utility of pre-ordering the concepts on models' performance, we introduce planned-bart and planned-t5 to manipulate the input concept order before generation, which helps to make the order of input concepts more natural (more close to the order of concepts in the output sentence). we experimentally show that via pre-ordering, planned-bart and planned-t5 exceed the more sophisticated models that have access to external knowledge or training data. it indicates that ptm's inherent ability for generative commonsense reasoning was underestimated while a simple pre-ordering step can help ptms better use this ability. ## analysis in this section, we analyze the impact of input permutation on the model and the generated sentences. ## conclusion in this work, we revisit the ptm's inherent ability of generative commonsense reasoning. we use bart and t5 as underlying generators and propose their planned variants to manipulate the order of the given concepts before generation. experiments on commongen dataset demonstrate that this simple pre-ordering approach can outperform the previous pre-trained or knowledge-enhanced models. besides that, planned models can leverage the pre-ordered concepts to create more succinct and diverse sentences. in conclusion, our work suggests that ptm's inherent ability for generative commonsense reasoning is underestimated due to the unordered input, and the pre-ordering step can help ptms to improve the generation quality.