Reasoning Like Program Executors (2022)

Abstract: Reasoning over natural language is a long-standing goal for the research community. However, studies have shown that existing language models are inadequate in reasoning. To address the issue, we present POET, a novel reasoning pre-training paradigm. Through pre-training language models with programs and their execution results, POET empowers language models to harvest the reasoning knowledge possessed by program executors via a data-driven approach. POET is conceptually simple and can be instantiated by different kinds of program executors. In this paper, we showcase two simple instances POET-Math and POET-Logic, in addition to a complex instance, POET-SQL. Experimental results on six benchmarks demonstrate that POET can significantly boost model performance in natural language reasoning, such as numerical reasoning, logical reasoning, and multi-hop reasoning. POET opens a new gate on reasoning-enhancement pre-training, and we hope our analysis would shed light on the future research of reasoning like program executors.

## introduction recent breakthroughs in pre-training illustrate the power of pre-trained language models (lm) on a wide range of natural language (nl) tasks. pretraining on self-supervised tasks, such as masked language modeling @xcite using large amounts of nl sentences, boosts the language understanding of models by a large margin @xcite . however, existing pre-training paradigms have primarily focused on language modeling and paid little attention to advanced reasoning capabilities ( ## related work since we focus on reasoning over natural language, our work is closely related to previous work on reasoning skills in nl tasks. regarding methods to inject reasoning skills into lms, our method is related to two lines of work contributing to the topic: specialized models and pre-training. last, our work is also related to program execution since we employ program executors during pre-training. reasoning skills the literature focuses on reasoning skills, including numerical reasoning @xcite , multi-hop reasoning @xcite , reasoning in hybrid context @xcite and logical reasoning @xcite . our work concentrates on improving the above reasoning skills, leaving the other reasoning abilities such as commonsense reasoning @xcite @xcite for future work. reasoning via specialized models early works typically design specialized models and augment them into lms for different types of questions @xcite @xcite . taking @xcite as an example, they first predicted the answer type of a given question (e.g., "how many"), and then adopted the corresponding module (e.g., count module) to predict the answer. although these methods work well on a specific dataset, it is challenging for them to scale to complex reasoning scenarios @xcite . differently, our work follows the line of reasoning via pre-training, which enjoys better scalability. reasoning via pre-training this line of work focuses on the continued pre-training of lms using large-scale data which involves reasoning. the pretraining data are generally nl text, which are either crawled from web with distant supervision @xcite , generated by a model-based generator @xcite , or synthesized via human-designed templates @xcite @xcite . however, large-scale high-quality textual data involving reasoning is difficult to collect @xcite . meanwhile, as the complexity of desired reasoning operations increases, synthesizing high-quality (e.g., fluent) nl sentences becomes more challenging. different from the above pre-training methods relying on nl data, our pre-training is performed on programs. these programs can be synthesized at any scale with high quality, and thus are much easier to collect. reasoning in giant language models recent works demonstrate that with proper prompting (e.g., chain-of-thoughts prompting), giant language models (e.g., gpt-3) can perform well on reasoning tasks @xcite @xcite . for example, @xcite find that giant language models have the ability to perform complex reasoning step by step with few-shot examples. although these prompting strategies do not need further fine-tuning, the basic assumptions of them and poet are similar, i.e., it is difficult to obtain large amounts of clean sentences involving complex reasoning. however, these prompting strategies do not work well for non-giant language models, while poet is simultaneously applicable to language models ranging from millions (e.g., bart) to billions (e.g., t5-11b). it is also interesting to investigate how these prompting strategies and poet can be combined. ## reasoning like program executors reasoning is the process where deduction and induction are sensibly applied to draw conclusions from premises or facts @xcite . as a supreme feature of intelligence, humans apply reasoning across modalities. taking numerical reasoning as an example, humans can tell how many chocolates are consumed from a math word problem description, or from a real-world event where a mother gets off work and finds the choco-can empty, aside standing their guilty-looking kids with brownish stains on their faces. through detachment of information from their superficial modality and symbolic abstraction, humans manage to unify input formats and condense their numerical reasoning knowledge into one executable symbolic system -this is the origin of an arithmetic program executor. if a model can master these reasoning skills by imitating program executors, we believe in the possibility of transferring those reasoning skills to different modalities. in our case, we expect language models to transfer reasoning to nl-related tasks. given this motivation, we discuss the fundamental components of poet in the rest of this section and present its instantiations later. program refers to a finite sequence of symbols that can be understood and executed by machines. for example, a program can be a logical form (e.g., prolog), a piece of code (e.g., python), or a math expression. compared with nl sentences, programs are more formal. each well-established program follows a specific set of grammar rules and can thus be synthesized systematically. the generalizability of poet framework is free from assumption and derived from the set of grammar rules on which a program follows. in poet, as long as a program returns meaningful output to reflect its computational procedure, it is an acceptable program. program context is the environment in which a program is running, which holds numerous variables accessible to the program. these variables serve as pivot points that anchor the program context with the program. in the same sense, the question and the passage in reading comprehension hold a similar relationship. this suggests a natural analogy between the program-to-program context and the sentence-to-natural context in figure 1 . executor is a black-box software that can execute a given program within the program context. an example could be the python interpreter that executes each line of code, with its specific input data structures as program context. for poet, program executors play the role of teachers to educate students (i.e., lms) on reasoning knowledge they contain. poet expects program executors to deterministically execute an input program with respect to a specific program context. execution result is obtained from the program executor, given a program and program context as input. it is much analogous to the answer part in nl downstream tasks. the execution result is the primary observable data reflecting the intermediate reasoning process and serves as the supervision provided by the program executor. ## poet with singleton executors we first instantiate poet with two singleton (i.e., a single type of reasoning capability) executors and then move on to poet with integrated executors. ## poet with integrated executors poet-math and poet-logic each focus on one specific reasoning skill, making the pre-training task heavily dependent on the downstream task. different from them, poet-sql is proposed to allow lms to master different reasoning skills simultaneously. in our implementation, poet-sql is pre-trained with an integrated sql executor, since we believe that sql queries are complex enough to encompass a wide variety of computational procedures ( pre-training task given a sql query as the program and a database as the program context, the pre-training task of poet-sql is to mimic the query result generation. as shown on the right side of figure 5 , given the concatenation of the program and the program context, the model is pre-trained to output the query result. since the encoder-decoder lms can generate arbitrary tokens, they are well suited for the task. on the other hand, encoderonly lms have insufficient expressiveness to produce out-of-context query results. to allow them to benefit from the sql execution, we tailor the task into a query result selection task for encoderonly lms, which only utilizes query results that can be found in the database. specifically, the task requires encoder-only lms to perform an io sequence tagging process to find the query results in the database, as shown on the left side of figure 5 . note that the tag i is for tokens in the query results (e.g., athens), while o is for other tokens. pre-training corpus each example in the corpus contains a sql query, a database, and a query result. notably, following @xcite , each database is flattened into a sequence when it is fed into lms. meanwhile, to avoid databases being too large to fit into memory, we randomly drop the rows of large databases until their flattened sequences contain less than 450 tokens. for the query result generation task, we follow the same corpus construction strategy as described in @xcite . concretely, by instantiating sql templates from squall @xcite over databases provided by wikisql @xcite , 5 million examples are synthesized for pre-training. for the query result selection task, the pre-training corpus is constructed in a similar way as above, except that only the examples whose query results are suitable for encoder-only are retained. this filtering results in a corpus containing nearly 2 million examples. ## conclusion & future work we introduce poet, a new pre-training paradigm for boosting reasoning capability of language models via imitating program executors. experimental results on six datasets demonstrate that poet can significantly boost existing language models on several reasoning skills, including numerical, logical and multi-hop reasoning. our best language model under poet can reach highly competitive performance with previous specialized models. in the future, we hope our work could inspire more transference of reasoning knowledge from program executors to models. and we will also investigate the causes of the reasoning transfer with more insightful experiments, since we still do not know how the reasoning transfer occurs.