Language Models of Code are Few-Shot Commonsense Learners (2022)

Abstract: We address the general task of structured commonsense reasoning: given a natural language input, the goal is to generate a graph such as an event or a reasoning-graph.To employ large language models (LMs) for this task, existing approaches ‘serialize’ the output graph as a flat list of nodes and edges.Although feasible, these serialized graphs strongly deviate from the natural language corpora that LMs were pre-trained on, hindering LMs from generating them correctly. In this paper, we show that when we instead frame structured commonsense reasoning tasks as code generation tasks, pre-trained LMs of code are better structured commonsense reasoners than LMs of natural language, even when the downstream task does not involve source code at all.We demonstrate our approach across three diverse structured commonsense reasoning tasks. In all these natural language tasks, we show that using our approach, a code generation LM (codex) outperforms natural-LMs that are fine-tuned on the target task (T5) and other strong LMs such as GPT-3 in the few-shot setting.

## introduction the growing capabilities of large pre-trained language models (llms) for generating text have enabled their successful application in a variety of tasks, including summarization, translation, and question-answering @xcite @xcite . nevertheless, while employing llms for natural language (nl) tasks is straightforward, a major remaining challenge is how to leverage llms for structured commonsense reasoning, including tasks such as generating event graphs @xcite , reasoning graphs @xcite , scripts @xcite , and argument explanation graphs @xcite . unlike tradi-tional commonsense reasoning tasks such as reading comprehension or question answering, structured commonsense aims to generate structured output given a natural language input. this family of tasks relies on the natural language knowledge learned by the llm, but it also requires complex structured prediction and generation. to leverage llms, existing structured commonsense generation models modify the output format of a problem. specifically, the structure to be generated (e.g., a graph or a table) is converted, or "serialized", into text. such conversions include "flattening" the graph into a list of node pairs (figure 1d ), or into a specification language such as dot (figure 1c ; @xcite . while converting the structured output into text has shown promising results @xcite , llms struggle to generate these "unnatural" outputs: lms are primarily pre-trained on free-form text, and these serialized structured outputs strongly diverge from the majority of the pre-training data. further, for natural language, semantically relevant words are typically found within a small span, whereas neighboring nodes in a graph might be pushed farther apart when representing a graph as a flat string. thus, a language model which was trained on natural language text is likely to fail to capture the topology of the graph. consequently, using llms for graph generation typically requires a large amount of task-specific training data, and their generated outputs show structural errors and semantic inconsistencies, which need to be further fixed either manually or by using a secondary downstream model @xcite . despite these struggles, the recent success of large-language models of code @xcite @xcite for tasks such as code generation from natural language @xcite , code completion @xcite , and code translation @xcite take the pies out to cool open cabinet drawer take out several plates begin putting pies on plate fill pies onto plates evenly serve the potpies on a plate (a) the script g class tree: @xmath0: # nodes @xmath1... # edges take_pies_out_to_cool.@xmath2.@xmath3... (b) g converted to python code g c using our approach digraph g { begin -> take_pies_out_to_cool; begin -> open_cabinet_drawer; take_pies_out_to_cool -> take_out_several_plates; open_cabinet_drawer -> take_out_several_plates; take_out_several_plates -> begin_putting_pies_on_plates; begin_putting_pies_on_plates -> serve_potpies_on_plate; fill_pies_onto_plates_evenly -> serve_potpies_on_plate; serve_potpies_on_plate -> end; } (c) straightforward encodings of the graph using the "dot" [ (take_pies_out_to_cool, take_out_several_plates), (open_cabinet_drawer, take_out_several_plates), (take_out_several_plates, begin_putting_pies_on_plates), (take_out_several_plates, fill_pies_onto_plates_evenly), (begin_putting_pies_on_plates, serve_potpies_on_plate), (fill_pies_onto_plates_evenly, serve_potpies_on_plate), (serve_potpies_on_plate, end) ] 1a ) is typically represented using the dot format (1c) or as a list of edges (1d), which allows modeling the graph using standard language models. these popular choices are sufficient in principle; however, these formats are loosely structured, verbose, and not common in text corpora, precluding language models from effectively generating them. in contrast, cocogen converts structures into python code (1b), allowing to model them using large-scale language models of code. @xcite , show that code-llms are able to perform complex reasoning on structured data such as programs. thus, instead of forcing llms of natural language (nl-llms) to be fine-tuned on structured commonsense data, an easier way to close the discrepancy between the pre-training data (free-form text) and the task-specific data (commonsense reasoning graphs) is to adapt llms that were pre-trained on code to structured commonsense reasoning in natural language. thus, our main insight is that large language models of code are good structured commonsense reasoners. further, we show that code-llms can be even better structured reasoners than nl-llms, when converting the desired output graph into a format similar to that observed in the code pre-training data. we call our method cocogen: models of code for commonsense generation, and it is demonstrated in figure 1 . our contributions are as follows: ## cocogen: representing commonsense structures with code we focus on tasks of structured commonsense generation. each training example for such tasks is in the form (t , g), where t is a text input, and g is the structure to be generated (typically a graph). the key idea of cocogen is transforming an output graph g into a semantically equivalent program g c written in a general-purpose programming language. in this work, we chose python due to its popularity in the training data of modern @xcite , but our approach is agnostic to the programming language. the codetransformed graphs are similar in their format to the pre-training data of code-llms, and thus serve as easier to generalize training or few-shot examples than the original raw graph. cocogen uses code-llms to generate g c given t , which we eventually convert back into the graph g. we use the task of script generation (proscript, figure 1 ) as a running example to motivate our method: script generation aims to create a script (g) to achieve a given high-level goal (t ). ## evaluation we experiment with three diverse structured commonsense generation tasks: (i) script generation (proscript, section 3.2), (ii) entity state tracking (propara, section 3.3), and (iii) explanation graph generation (explagraphs, section 3.4) dataset details are included in appendix d. despite sharing the general goal of structured commonsense generation, the three tasks are quite diverse in terms of the generated output and the kind of required reasoning. ## analysis in to answer this, we experimented with a text prompt with a code-llm codex, and a code prompt with an nl-llm, davinci. of each of these solely. ## conclusion we present the first work to employ large language models of code for structured commonsense gen-eration. by converting the output commonsense structures to python code, cocogen provides a simple and effective method for leveraging the code-generation abilities of code-llms for structured generation. these results open a promising direction for structural commonsense reasoning. we believe that the principles and the methods presented in this paper are applicable to additional nlp tasks that require "language understanding" and structured prediction. ers for valuable feedback. this material is partly based on research sponsored in part by the air force research laboratory under agreement number fa8750-19-2-0200. the u.s. government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright notation thereon. the views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the air force research laboratory or the u.s. government. this project was also partially supported by a gift from aws ai.