C ommon G en: A Constrained Text Generation Challenge for Generative Commonsense Reasoning (2020)

Abstract: Recently, large-scale pre-trained language models have demonstrated impressive performance on several commonsense-reasoning benchmark datasets. However, building machines with commonsense to compose realistically plausible sentences remains challenging. In this paper, we present a constrained text generation task, CommonGen associated with a benchmark dataset, to explicitly test machines for the ability of generative commonsense reasoning. Given a set of common concepts (e.g., dog, frisbee, catch, throw); the task is to generate a coherent sentence describing an everyday scenario using these concepts (e.g., “a man throws a frisbee and his dog catches it”). The CommonGen task is challenging because it inherently requires 1) relational reasoning with background commonsense knowledge and 2) compositional generalization ability to work on unseen concept combinations. Our dataset, constructed through a combination of crowdsourced and existing caption corpora, consists of 77k commonsense descriptions over 35k unique concept-sets. Experiments show that there is a large gap between state-of-the-art text generation models (e.g., T5) and human performance (31.6% v.s. 63.5% in SPICE metric). Furthermore, we demonstrate that the learned generative commonsense reasoning capability can be transferred to improve downstream tasks such as CommonsenseQA (76.9% to 78.4 in dev accuracy) by generating additional context.

## introduction commonsense reasoning, the ability to make acceptable and logical assumptions about ordinary scenes in our daily life, has long been acknowledged as a critical bottleneck of artificial intelligence and natural language processing @xcite . most recent commonsense reasoning challenges, such as commonsenseqa (tal- -the gym owner decided to tie a rope to the wall so people could make a wave in it for exercise. ## task formulation and key challenges we formulate the proposed commongen task with mathematical notations and discuss its inherent challenges with concrete examples. the input is an unordered set of k concepts @xmath0, c 2 , . . . , c k } ∈ x (i.e. a concept-set), where each concept c @xmath1. we use x to denote the space of all possible concept-sets and use c to denote the concept vocabulary (a subset of conceptnet's unigram concepts). the expected output is a simple, grammatical sentence @xmath2, using all given concepts in x (morphological inflections are allowed). a scenario can depict either a static situation or a short series of actions. the commongen task is to learn a function f ∶ @xmath3, which maps a concept-set x to a sentence y. the unique challenges of this task come from two aspects: relational reasoning with commonsense. expected generative reasoners should prioritize the most plausible scenarios over many other less realistic ones. as shown in figure 2 , models need to recall necessary relational commonsense facts that are relevant to the given concepts, and then reason an optimal composition of them for gener-ating a desired sentence. in order to complete a scenario, generative commonsense reasoners also need to reasonably associate additional concepts (e.g., 'woman', 'gym') as agents or background environments for completing a coherent scenario. this not only requires understanding underlying commonsense relations between concepts, but also incrementally composing them towards a globally optimal scenario. the underlying reasoning chains are inherently based on a variety of background knowledge such as spatial relations, object properties, physical rules, temporal event knowledge, social conventions, etc. however, they may not be recorded in any existing knowledge bases. ## dataset construction and analysis figure 3 illustrates the overall workflow of our data construction for the proposed commongen task. we utilize several existing caption corpora for sampling frequent concept-sets (sec. 3.1) for reflecting common scenarios. we employ amt crowd workers for collecting human-written sentences (sec. 3.2) for the development and test set, while we carefully monitor the quality of crowd workers and refine them dynamically. finally, we present the statistics of the commongen dataset, and the analysis on the challenges (sec. 3.4). ## methods we briefly introduce the baseline methods that are tested on the commongen task. encoder-decoder models. bidirectional rnns and transformers @xcite are two most popular architectures for seq2seq learning. we use them with the addition of attention mecha2 relation definitions are at https://github.com/ commonsense/conceptnet5/wiki/relations . ## 5 evaluation we first introduce the automatic evaluation metrics, then present main experimental results with manual analysis, and finally introduce the potential application in transferring commongen-trained models for other downstream tasks. ## conclusion our major contribution in this paper are threefold: figure 7 : our annotation interface on the amt platform. the upper part is the instruction for the annotators and we provide an example for them. note that we give the part-of-speech hints (from the captain corpora) to boost the speed of annotation, but we do not remove sentences with wrong part-of-speech as long as they also make sense.