Chain of Thought Prompting Elicits Reasoning in Large Language Models
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We explore how generating a chain of thought —a series of intermediate reasoningsteps—signiﬁcantly improves the ability of large language models to performcomplex reasoning.
In particular, we show how such reasoning abilities emergenaturally in sufﬁciently large language models via a simple method called chainof thought prompting , where a few chain of thought demonstrations are providedas exemplars in prompting.
Experiments on three large language models showthat chain of thought prompting improves performance on a range of arithmetic,commonsense and symbolic reasoning tasks.
The empirical gains can be striking.
For instance, prompting a 540B-parameter language model with just eight chain ofthought exemplars achieves state of the art accuracy on the GSM8K benchmark ofmath word problems, surpassing even ﬁnetuned GPT-3 with a veriﬁer.
Introduction
The NLP landscape has recently been revolutionized by language models (Peters et al., 2018; Devlinet al., 2019; Brown et al., 2020, inter alia ).
Scaling up the size of language models has been shown toconfer a range of beneﬁts, such as improved performance and sample efﬁciency (Kaplan et al., 2020;
Brown et al., 2020, inter alia ).
However, scaling up model size alone has not proved sufﬁcient forachieving high performance on challenging tasks such as arithmetic, commonsense, and symbolicreasoning (Rae et al., 2021).
This work explores how the reasoning ability of large language models can be enhanced by a simplemechanism motivated by two key ideas.
First, in neuro-symbolic computing it has been recognizedthat generating intermediate results signiﬁcantly improves accuracy when a desired input-outputmapping involves multiple computational steps (Chen et al., 2019;
Dong et al., 2019; Mao et al.,2021).
That is, in multi-step reasoning tasks such as program synthesis or arithmetic reasoning,complex input–output mappings are more accurately recovered by training on (cid:104) input, intermediateresults , output (cid:105) triples, rather than simple (cid:104) input, output (cid:105) pairs (Ling et al., 2017; Chen et al., 2021;
Cobbe et al., 2021).
Second, large language models offer the exciting prospect of in-context few-shotlearning via prompting .
That is, instead of ﬁnetuning a language model for each new task, one cansimply “prompt” the model with a few (cid:104) input, output (cid:105) exemplars demonstrating the task.
Remarkably,this has proved successful for a range of simple question-answering tasks (Brown et al., 2020).
In this paper, we combine these two ideas and explore the ability of large language models toperform few-shot prompting for reasoning tasks, given a prompt that consists of triples: (cid:104) input,chain of thought , output (cid:105) .
A chain of thought is a series of intermediate natural language reasoningsteps that lead to the ﬁnal output, inspired by how humans use a deliberate thinking process toperform complicated tasks.
As we will see, large language models can perform such natural languagereasoning through a simple approach we call chain of thought prompting , where chain of thoughtdemonstrations are included in the exemplars for few-shot prompting (illustrated in Figure 1).
[Table] [Table]
Standard Prompting
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
uses language and is therefore applicable to any
We present empirical evaluations on arithmetic, commonsense, and symbolic reasoning benchmarks,showing that chain of thought prompting outperforms standard prompting, sometimes to a strikingdegree.
Figure 2 illustrates one such result—on the GSM8K benchmark of math word
problems(Cobbe et al., 2021), chain of thought prompting with LM-2 540B outperforms standard promptingby a large margin and achieves new state-of-the-art performance.
A prompting only approach isimportant because it does not require a large training dataset and because a single model checkpointcan perform many tasks without loss of generality.
This work underscores how large language modelscan learn via a few examples with natural language data about the task (c.f. automatically learningthe patterns underlying inputs and outputs via a large training dataset).
Chain of Thought
[Figure]
[Figure]
Finetuned GPT-3 and are Cobbe et al.
Consider one’s own thought process when solving a complicated reasoning task such as a multi-step math wordproblem.
It is typical to decompose the problem into intermediate steps and solve each before giving the ﬁnalanswer: “After Jane gives 2 ﬂowers to her mom she has10 . . .
then after she gives 3 to her dad she will have 7 . . .so
the answer is 7.”
The goal of this paper is to endow language models with the ability to generate a similar chain ofthought —a coherent series of intermediate reasoning stepsthat lead to the ﬁnal answer for a problem.
We will showthat sufﬁciently large language models can generate chainsof thought if demonstrations of chain of thought reasoningare provided in the exemplars for few-shot prompting.
Figure 1 shows an example of a model producing a chainof thought to solve a math word problem that it wouldhave otherwise gotten incorrect.
The chain of thought inthis case resembles a solution and can interpreted as one,but we still opt to call it a chain of thought to better capturethe idea that it mimics a step-by-step thought process forarriving at the answer (and also, solutions/explanations typically come after the ﬁnal answer (Narang et al., 2020; Wiegreffe et al., 2022; Lampinen et al.
,2022, inter alia )).
Chain of thought prompting has several attractive properties as an approach for facilitating reasoningin language models.
First, chain of thought, in principle, allows models to decompose multi-step problems intointermediate steps, which means that additional computation can be allocated to problemsthat require more reasoning steps.2.
Second, a chain of thought provides an interpretable window into the behavior of the model,suggesting how it might have arrived at a particular answer and providing opportunitiesto debug where the reasoning path went wrong (although fully characterizing a model’scomputations that support an answer remains an open question).3.
Third, chain of thought reasoning can be used for tasks such as math word problems,commonsense reasoning, and symbolic manipulation, and is applicable (in principle) to anytask that humans can solve via language.4.
Finally, chain of thought reasoning can be readily elicited in sufﬁciently large off-the-shelflanguage models simply by including examples of chain of thought sequences into theexemplars of few-shot prompting.
In empirical experiments, we will observe the utility of chain of thought prompting for arithmeticreasoning (Section 3), commonsense reasoning (Section 4), and symbolic reasoning (Section 5).
Arithmetic Reasoning
We begin by considering math word problems of the form in Figure 1, which measure the arithmeticreasoning ability of language models.
Though simple for humans, arithmetic reasoning is a task wherelanguage models often struggle (Hendrycks et al., 2021;
Patel et al., 2021, inter alia ).
Strikingly, chainof thought prompting when used with the a 540B parameter language model performs comparablywith task-speciﬁc ﬁnetuned models on several tasks, even achieving new state of the art on thechallenging GSM8K benchmark (Cobbe et al., 2021).
3.1 Experimental Setup
We explore chain of thought prompting for various language models on multiple benchmarks.
Benchmarks.
We consider the following ﬁve math word problem benchmarks: (1) the GSM8Kbenchmark of math word problems (Cobbe et al., 2021), (2) the SVAMP dataset of math wordproblems with varying structures (Patel et al., 2021), (3) the ASDiv dataset of diverse math wordproblems (Miao et al., 2020), (4) the AQuA dataset of algebraic word problems, and (5) the MAWPSbenchmark (Koncel-Kedziorski et al., 2016).
Example problems are given in Appendix Table 12.
Standard prompting.
For the baseline, we consider standard few-shot prompting, popularized by
Brown et al.
(2020), in which a language model is given in-context exemplars of input–output pairsbefore outputting a prediction for a test-time example.
Exemplars are formatted as questions andanswers.
The model gives the answer directly, as shown in Figure 1 (left).
Chain of thought prompting.
Our proposed approach is to augment each exemplar in few-shotprompting with a chain of thought for an associated answer, as illustrated in Figure 1 (right).
As mostof the datasets only have an evaluation split, we manually composed a set of eight few-shot exemplarswith chains of thought for prompting—Figure 1 (right) shows one chain of thought exemplar, and thefull set of exemplars is given in Appendix Table 20.
(These particular exemplars did not undergoprompt engineering; robustness is studied in Section 3.4 and Appendix A.2.)
To investigate whetherchain of thought prompting in this form can successfully elicit successful reasoning across a range ofmath word problems, we used this single set of eight chain of thought exemplars for all benchmarksexcept AQuA, which is multiple choice instead of free response.
For AQuA, we used four exemplarsand solutions from the training set, as given in Appendix Table 21.
Language models.
We evaluate three collections of dense left-to-right, decoder-only transformerlanguage models.
The ﬁrst is GPT-3 (Brown et al., 2020), for which we use text-ada-001, textbabbage-001, text-curie-001, and text-davinci-002, which presumably correspond to InstructGPT
[Table]
Math Word Problems (free response)
Math Word Problems (multiple choice)
CSQA (commonsense)
StrategyQA
Date Understanding
Sports
[Figure]
[Table]
[Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Figure] models of 350M, 1.3B, 6.7B, and 175B parameters (Ouyang et al., 2022).
The second is LM- (name and citation redacted for anonymity), 2 which has models of 422M, 2B, 8B, 68B, and 137Bparameters.
The third is LM-2 (also redacted for anonymity), which has models of 8B, 62B, and540B parameters.
We sample from the models via greedy decoding (though follow-up work showschain of thought prompting can be improved by taking the majority ﬁnal answer over many sampledgenerations (Wang et al., 2022a)).
For LM-1, we report averaged results over ﬁve random seeds,where each seed had a different randomly shufﬂed order of exemplars.
To save compute, we reportresults for a single exemplar order for GPT-3 and LM-2, as LM-1 experiments did not show largevariance among different seeds.
3.2 Results
The strongest results of chain of thought prompting are summarized in Figure 4, with all experimentaloutputs for each model collection, model size, and benchmark shown in Table 2 in the Appendix.
There are three key takeaways.
First, Figure 4 shows that chain of thought prompting is an emergent ability of model scale.
That is,chain of thought prompting does not positively impact performance for small models, and only yieldsperformance gains when used with models of ∼ 100B parameters.
We qualitatively found that modelsof smaller scale produced ﬂuent but illogical chains of thought, leading to lower performance thanstandard prompting.
1 https://blog.eleuther.ai/gpt3-model-sizes/#:~:text=Ada%2C%20Babbage%2C%20Curie% 20and%20Davinci,sizes%2C%20but%20it’s%20pretty%20suggestive.
This anonymity decision was recommended by the NeurIPS chairs.
[Figure]
Second, chain of thought prompting has largerperformance gains for more-complicated problems.
For instance, for GSM8K (the dataset withthe lowest baseline performance), performancemore than doubled for the largest GPT and LM2 models.
On the other hand, for SingleOp, theeasiest subset of MAWPS which only requires asingle step to solve, performance improvementswere either negative or very small (see Appendix
Table 3).
Third, the performance of chain of thoughtprompting via GPT-3 175B and LM-2 540Bcompares favorably to prior state of the art,which typically ﬁnetunes a task-speciﬁc modelon a labeled training dataset.
Figure 4 showshow LM-
540B uses chain of thought prompting to achieve new state of the art on GSM8K,
SVAMP, and MAWPS (though note that standard prompting already passed the prior best for
SVAMP).
On the other two datasets, AQuA and
ASDiv, LM-2 with chain of thought promptingreaches within 2% of the state of the art (Appendix Table 2).
To better understand why chain of thoughtprompting works, we manually examined modelgenerated chains of thought by LM-1 137B for
GSM8K. Of 50 random examples where themodel returned the correct ﬁnal answer, all ofthe generated chains of thought were also logically and mathematically correct except twothat coincidentally arrived at the correct answer(see Appendix D.1, and Table 8 for examplesof correct model-generated chains of thought).
We also randomly examined 50 random samples for which the model gave the wrong answer.
The summary of this analysis is that 46% of thechains of thought were almost correct, barringminor mistakes (calculator error, symbol mapping error, or one reasoning step missing), and that theother 54% of the chains of thought had major errors in semantic understanding or coherence (see
Appendix D.2).
To provide a small insight into why scaling improves chain of thought reasoningability, we performed a similar analysis of errors made by LM-2 62B and whether those errors wereﬁxed by scaling to LM-2 540B.
The summary is that scaling LM-2 to 540B ﬁxes a large portion ofone-step missing and semantic understanding errors in the 62B model (see Appendix A.1).
3.3 Ablation Study
The observed beneﬁts of using chain of thought prompting raises the natural question of whether thesame performance improvements can be conferred via other types of prompting.
Figure 5 shows anablation study with three variations of chain of thought described below.
Equation only.
One reason for why chain of thought prompting might help is that it produces themathematical equation to be evaluated, and so we test a variation where the model is promptedto output only a mathematical equation before giving the answer.
Figure 5 shows that equationonly prompting does not help much for GSM8K, which implies that the semantics of the questionsin GSM8K are too challenging to directly translate into an equation without the natural languagereasoning steps in chain of thought.
For datasets of one-step or two-step problems, however, we ﬁndthat equation only prompting does improve performance, since the equation can be easily derivedfrom the question (see Appendix Table 6).
LMLM-
G S
[Figure]
Variable compute only.
Another intuition is that chain ofthought allows the model to spend more computation (i.e.,intermediate tokens) on harder problems.
To isolate the effectof variable computation from chain of thought reasoning, wetest a conﬁguration where the model is prompted to output aonly sequence of dots ( . . . )
equal to the number of characters inthe equation needed to solve the problem.
This variant performsabout the same as the baseline, which suggests that variablecomputation by itself is not the reason for the success of chainof thought prompting, and that there appears to be utility fromexpressing intermediate steps via natural language.
Chain of thought after answer.
Another potential beneﬁt ofchain of thought prompting could simply be that such promptsallow the model to better access relevant knowledge acquiredduring pretraining.
Therefore, we test an alternative conﬁguration where the chain of thought prompt is only given after theanswer, isolating whether the model actually depends on theproduced chain of thought to give the ﬁnal answer.
This variantperforms about the same as the baseline, which suggests thatthe sequential reasoning embodied in the chain of thought isuseful for reasons beyond just activating knowledge.
3.4 Robustness of Chain of Thought
GSM8K
S o l v e r a t e ( % )
Standard prompting
Chain of thought prompting · different annotator (B) · different annotator (C) · intentionally concise style · exemplars from GSM8K ( α ) · exemplars from GSM8K ( β ) · exemplars from GSM8K
[Figure]
has variance for different prompt examples (as expected) but outperforms standard prompting for various annotators as well as for different exemplars.
Sensitivity to exemplars is a key consideration of prompting approaches—for instance, varying the permutation offew-shot exemplars can cause the accuracy of GPT-3 on
SST-2 to range from near chance (54.3%) to near state ofthe art (93.4%)
(Zhao et al., 2021).
In this ﬁnal subsection, we evaluate robustness to chains of thought writtenby different annotators.
In addition to the results above,which used chains of thought written by an Annotator
A, two other co-authors of this paper (Annotators B and
C) independently wrote chains of thought for the samefew-shot exemplars (shown in Appendix H).
Annotator Aalso wrote another chain of thought that was more concisethan the original, following the style of solutions given in
Cobbe et al.
Figure 6 shows these results for LM-1 137B on GSM8Kand MAWPS (ablation results for other datasets are givenin Appendix Table 6 / Table 7).
Although there is varianceamong different chain of thought annotations, as would beexpected when using exemplar-based prompting (Le Scaoand Rush, 2021; Reynolds and McDonell, 2021; Zhaoet al., 2021), all sets of chain of thought prompts outperform the standard baseline by a large margin.
This resultimplies that successful use of chain of thought does notdepend on a particular linguistic style.
To conﬁrm that successful chain of thought promptingworks for other sets of exemplars, we also run experimentswith three sets of eight exemplars randomly sampled from the GSM8K training set, an independent source (examples in this dataset already included reasoning steps like a chain of thought).
4 Figure 6 shows that these prompts performed comparably with our manually written exemplars, alsosubstantially outperforming standard prompting.
In addition to robustness to annotators, independently-written chains of thought, different exemplars,and various language models, we also ﬁnd that chain of thought prompting for arithmetic reasoningis robust to different exemplar orders and varying numbers of exemplars (see Appendix A.2).
Commonsense Reasoning
Although chain of thought is particularly suitable for math word problems, the language-based natureof chain of thought actually makes it applicable to a broad class of commonsense reasoning problems,which involve reasoning about physical and human interactions under the presumption of generalbackground knowledge.
Commonsense reasoning is key for interacting with the world and is stillbeyond the reach of current natural language understanding systems (Talmor et al., 2021).
Benchmarks.
We consider ﬁve datasets covering a diverse range of commonsense reasoning types.
The popular CSQA (Talmor et al., 2019) asks commonsense questions about the world involvingcomplex semantics that often require prior knowledge.
StrategyQA
(Geva et al., 2021) requiresmodels to infer a multi-hop strategy to answer questions.
We choose two specialized evaluation setsfrom the BIG-bench effort (BIG-bench collaboration, 2021): Date Understanding, which involvesinferring a date from a given context, and Sports Understanding, which involves determining whethera sentence relating to sports is plausible or implausible.
Finally, the SayCan dataset (Ahn et al.,2022) involves mapping a natural language instruction to a sequence of robot actions from a discreteset.
Figure 3 shows examples with chain of thought annotations for all datasets.
Prompts.
We follow the same experimental setup as the prior section.
For CSQA and StrategyQA,we randomly selected examples from the training set and manually composed chains of thought forthem to use as few-shot exemplars.
The two BIG-bench tasks do not have training sets, so we selectedthe ﬁrst ten examples as exemplars in the evaluation set as few-shot exemplars and report numbers onthe rest of the evaluation set.
For SayCan, we use six examples from the training set used in Ahn et al.(2022) and also manually composed chains of thought.
Results.
Figure 7 highlights these results for LM-2 (full results for LM-1, GPT-3, and differentmodel scales are shown in Table 4).
For all tasks, scaling up model size improved the performance ofstandard prompting; chain of thought prompting led to further gains, with improvements appearing tobe largest for LM-2 540B. With chain of thought prompting, LM-
540B achieved strong performancerelative to baselines, outperforming the prior state of the art on StrategyQA (75.6% vs 69.4%) andoutperforming an unaided sports enthusiast on sports understanding (95.4% vs 84%).
These resultsdemonstrate that chain of thought prompting can also improve performance on tasks requiring a rangeof commonsense reasoning abilities (though note that gain was minimal on CSQA).
[Figure]
Symbolic Reasoning
[Figure]
Our ﬁnal experimental evaluation considers symbolic reasoning, which is simple for humans but potentially challenging for language models.
We show that chain ofthought prompting not only enables language models toperform symbolic reasoning tasks that are challenging inthe standard prompting setting, but also facilitates lengthgeneralization to inference-time inputs longer than thoseseen in the few-shot exemplars.
Tasks.
We use the following two toy tasks.
• Last letter concatenation.
This task asks the model
to concatenate the last letters of words in a name (e.g.,
“Amy Brown” → “yn” ).
It is a more challenging version
of ﬁrst letter concatenation,
which language models can
already perform without chain of thought.
5 We generate
full names by randomly concatenating names from the
top one-thousand ﬁrst and last names from name census
data ( https://namecensus.com/ ).
• Coin ﬂip.
This task asks the model to answer whethera coin is still heads up after people either ﬂip or don’tﬂip the coin (e.g., “A coin is heads up.
Phoebe ﬂips thecoin.
Osvaldo does not ﬂip the coin.
Is the coin stillheads up?”
→ “no” ).
As the construction of these symbolic reasoning tasks iswell-deﬁned, for each task we consider an in-domain testset for which examples had the same number of steps asthe training/few-shot exemplars, as well as an out-of-domain (OOD) test set, for which evaluationexamples had more steps than those in the exemplars.
For last letter concatenation, the model onlysees exemplars of names with two words, and then performs last letter concatenation on names with 3and 4 words.
6 We do the same for the number of potential ﬂips in the coin ﬂip task.
Our experimentalsetup uses the same methods and models as in the prior two sections.
We again manually composechains of thought for the few-shot exemplars for each task, which are given in Figure 3.
Results.
The results of these in-domain and OOD evaluations are shown in Figure 8 for LM-2, withresults for LM-1 shown in Appendix Table 5.
With LM-2 540B, chain of thought prompting leadsto almost 100% solve rates (note that standard prompting already solves coin ﬂip with LM-2 540,though not for LM-1 137B).
Note that these in-domain evaluations are “toy tasks” in the sense thatperfect solution structures are already provided by the chains of thought in the few-shot exemplars;all the model has to do is repeat the same steps with the new symbols in the test-time example.
Andyet, small models still fail—the ability to perform abstract manipulations on unseen symbols for thesethree tasks only arises at the scale of 100B model parameters.
As for the OOD evaluations, standard prompting fails for both tasks.
With chain of thought prompting,language models achieve upward scaling curves (though performance is lower than in the in-domainsetting).
Hence, chain of thought prompting facilitates length generalization beyond seen chains ofthought for language models of sufﬁcient scale.
Discussion
We have explored chain of thought prompting as a simple mechanism for eliciting multi-step reasoning behavior in large language models.
We ﬁrst saw that chain of thought prompting improvesperformance by a large margin on arithmetic reasoning, yielding improvements that are much strongerthan ablations and robust to different annotators, exemplars, and language models (Section 3).
Next, experiments on commonsense reasoning underscored how the linguistic nature of chain of thoughtreasoning makes it generally applicable (Section 4).
Finally, we showed that for symbolic reasoning,chain of thought prompting facilitates OOD generalization to longer sequence lengths (Section 5).
Inall experiments, chain of thought reasoning is elicited simply by prompting an off-the-shelf languagemodel.
No language models were ﬁnetuned in the process of writing this paper.
The emergence of chain of thought reasoning as a result of model scale has been a prevailing theme.
For many reasoning tasks where standard prompting has a ﬂat scaling curve, chain of thoughtprompting leads to dramatically increasing scaling curves.
Chain of thought prompting appearsto expand the set of tasks that sufﬁciently large language models can perform successfully—inother words, our work underscores that standard prompting only provides a lower bound on thecapabilities of large language models in principle.
This observation likely raises more questionsthan it answers—for instance, how much more can we expect reasoning ability to improve with afurther increase in model scale?
What other prompting methods might expand the range of tasks thatlanguage models can solve?
As for limitations, we ﬁrst we qualify that although chain of thought emulates the thought processes ofhuman reasoners, this does not answer whether the neural network is actually “reasoning,” which weleave as an open question.
Ironically, the dependence on chain of thought prompting and sufﬁcientlylarge models are both key components and major limitations of our proposed approach.
Although thecost of manually augmenting exemplars with chains of thought is minimal in the few-shot setting,the annotation costs could be prohibitive for ﬁnetuning (though this could potentially be surmountedwith synthetic data generation, or zero-shot generalization).
The emergence of chain of thoughtreasoning only at large model scale makes it costly to serve in real-world applications; future workcould explore how to induce reasoning at smaller model scales.
Related Work
This work is inspired by many research areas, which we detail in an extended related work section(Appendix C).
Here we describe two directions and associated papers that are perhaps most relevant.
The ﬁrst relevant direction is using intermediate reasoning steps, either as programs or in naturallanguage.
Work on reasoning via programs includes both neuro-symbolic architectures (Chen et al.
,2019; Dong et al., 2019) and program synthesis/execution (Chen et al., 2021; Nye et al., 2021).
Ourwork generalizes such domain-speciﬁc primitives to natural language, which is open-domain andapplicable for any text-based task in principle.
In terms of natural language reasoning, prior work hasshown that using intermediate reasoning steps confers beneﬁts for tasks such as text classiﬁcation(Zaidan et al., 2007), natural language inference (Camburu et al., 2018;
Marasovi ´ c et al., 2022),commonsense reasoning (Zelikman et al., 2022), and arithmetic reasoning (Cobbe et al., 2021),among other settings.
However, these approaches require ﬁnetuning, typically on a large labeleddataset.
In contrast, we show that such reasoning abilities can be elicited in large language modelsthrough prompting alone.
This prompting setting is important because it works with only a fewlabeled examples, and a single model checkpoint can be used for many tasks.
Naturally, this paper also relates closely to the large body of recent work on prompting.
Since thepopularization of few-shot prompting as given by Brown et al.
, several general approacheshave improved the prompting ability of models, such as automatically learning prompts (Lester et al.,2021) or giving models instructions describing a task (Sanh et al., 2022; Wei et al., 2022; Ouyanget al., 2022).
Whereas these approaches improve or augment the input part of the prompt (e.g.,instructions that are prepended to inputs), our work takes the orthogonal direction of augmenting theoutputs of language models with a chain of thought.
Conclusions
We have explored chain of thought prompting as a simple and broadly applicable method for enhancingreasoning in language models.
Through experiments on arithmetic, symbolic, and commonsensereasoning, we ﬁnd that chain of thought processing is an emergent property of model scale that allowssufﬁciently large language models to perform reasoning tasks that otherwise have ﬂat scaling curves.
Broadening the range of reasoning tasks that language models can perform will hopefully inspirefurther work on language-based approaches to reasoning. A Frequently Asked Questions
[Figure] A.1 Why does increasing model scale improve chain of thought prompting? [Figure]
The ﬁnding that successful chain of thought reasoning predictably emerges only at certain modelscales is intriguing.
Scaling up language models has been shown to confer beneﬁts such as improvedperformance and sample efﬁciency (Kaplan et al., 2020), but chain of thought reasoning is emergentin the sense that its success cannot be predicted only by extrapolating the performance of small scalemodels, as chain of thought actually hurts performance for most models smaller than 10B parameters.
The question of why model scale improves chain of thought prompting is certainly multi-faceted, andwe made a preliminary attempt to shed insight into it via error analysis.
This small analysis involvedmanually reading 45 errors made by LM-2 62B and categorizing them into semantic understanding(20 errors), one step missing (18 errors), and other errors (7 errors).
The “other category” includedhallucinations, repetitive outputs, and symbol mapping errors.
This categorization is a coarse oneborrowed from the initial error analysis done on LM-1 in Appendix D.2, for which categories wereconceived based on what improvements were needed to make the chain of thought correct.
As shown in Figure 9, scaling LM-2 to 540B parameters ﬁxed a substantial portion of errors in allthree categories.
Examples of semantic understanding and one-step missing errors that were ﬁxed byscaling LM-2 to 540B are given in Figure 10.
This result appears consistent with a hypothesis thatlanguage models acquire a range of semantic understanding and logical reasoning skills as a functionof model scale (though note that model scale is often conﬂated with other factors, such as amount oftraining compute).
[Table]
Types of errors made by a 62B language model:
[Table]
by
[Table]
from
[Table]
There are also three notable points regarding why small language models fail.
The ﬁrst observationis that small language models fail at even relatively easy symbol mapping tasks.
As demonstratedin Section 5, for even symbolic reasoning tasks that only require generalization to new examplesusing the same chain of thought logical structure that was given in the few-shot exemplars, smalllanguage models still failed.
The second observation is that small language models seem to haveinherently weaker arithmetic abilities, as shown by Brown et al.
, the ability to do simplearithmetic operations (without semantic understanding) requires sufﬁcient model scale.
Finally, wenoticed qualitatively that small language models often did not generate a ﬁnal answer that could beparsed, due to either repetitions or logic that never arrived at a ﬁnal answer.
In summary, the success of chain of thought reasoning as a result of model scale is a complicatedphenomena that likely involves a variety of emergent abilities (semantic understanding, symbolmapping, staying on topic, arithmetic ability, faithfulness, etc).
Future work could more thoroughlyinvestigate what properties of pretraining data, model architecture, and optimization objective causallyenable such reasoning capabilities.
[Table]
bill was $40.00.
The delivery vendor added 25% to the bill.
This means the bill was increased by 25% of $40.00.
This is 0.25 * $40.00 = $10.00.
The delivery fee was $3.00.
The tip was $4.00.
So the total bill was $40.00 + $10.00 + $3.00 + 540b model $4.00 = $57.00.
The answer is 57.
(Correct)
There are four schools competing at a basketball tournament.
Each school has sent a girls’ basketball team and a boys’ basketball team and each team hasplayers each.
Each school has also sent a coach for each team.
In total, how many people have all of the schools sent?
There are 4 schools.
Each school has sent a girls’ basketball team and a boys’ basketball team.
Each team has 5 players each.
That means there are 4 * 2 * 5 = 40 players.
Each team has a coach.
That means there are 40 + 4 = 44 coaches.
The answer is 44.
(Incorrect: one step missing error)
Each school has sent 2 teams, each with 5 players.
This means each school has sent 2 * 5 = 10 players.
Each school has also sentcoaches.
This means each school has sent 10 + 2 = 12 people.
There are 4 schools, so in total all of the schools have sent 4 * 12 =people.
The answer is 48.
(Correct)
62B
Model Output 62B Model Output
62B Model Output 540B Model Output 540B Model Output 540B Model Output
Question
Question Question
A.2 What is the role of prompt engineering?
One of the key considerations of prompting is sensitivity to the exact prompt.
There is no shortageof work showing that prompts affect language models in unexpected ways (Min et al., 2022).
Thegeneral way that we created chain of thought annotations was by taking eight exemplars from thetraining set and decomposing the reasoning process into multiple steps leading to the ﬁnal answer.
Examples of chain of thought
annotations are provided in Figure 3, with full prompts given inAppendix G.
To analyze how sensitive chain of thought is to prompt engineering, we performed robustness experiments with respect to various factors. • Different annotators.
We ﬁrst analyze robustness to three different annotators (Section 3.4 andFigure 6).
Although there is notable variance in performance (which we will discuss later), chainof thought performed better than the baseline by a large margin for all three annotators on eightdatasets in arithmetic, commonsense, and symbolic reasoning (Table 6 and Table 7).
• Annotators without machine learning background.
The GSM8K dataset (Cobbe et al., 2021)conveniently provides a training set with reasoning chains written by crowd compute workers,which enables us to investigate whether chain of thought still works with reasoning chains from an
[Table]
independent source without a background in machine learning.
So we randomly sampled three setsof eight exemplars with chains of thought from GSM8K. These chain of thought annotations alsooutperformed the baseline by a large margin for all four arithmetic datasets (Table 6), indicatingthat chain of thought is not dependent on a particular set of annotators.
[Table]
• Different exemplars.
The different GSM8K exemplars experiment above (Table 6) also showsthat chain of thought prompting works for different sets of exemplars.
Notably, we test every set ofexemplars on all four arithmetic datasets (instead of picking exemplars from the training set for
[Table]
each dataset), which suggests that the exemplars do not necessarily have to come from the same
[Table]
dataset distribution as the test examples.
[Table]
• Different order of exemplars.
Prior work has shown that in some cases (e.g., classiﬁcation)even
the order of prompts matter—varying the permutation of few-shot exemplars can cause theaccuracy of GPT-3 on SST-2 to range from near chance (54.3%) to near SOTA (93.4%)
(Zhaoet al., 2021).
We show the standard deviation of performance from different exemplars in Table 6and Table 7.
Standard deviations with respect to prompt order are relatively minimal in almostall cases.
The one exception is the coin ﬂip task, for which exemplar orders have high standarddeviation, likely for the reason cited in Zhao et al.
(2021)—for classiﬁcation, many exemplars ofthe same category in a row biases the model outputs).
[Table]
• Different number of exemplars.
We also found that gains from chain of thought prompting
[Table]
generally still held when there was a varying number of few-shot exemplars.
This is shown for ﬁve datasets in Figure 11 (we did not have the compute to run this for all datasets).
We also found in
preliminary experiments that further increasing the number of exemplars in standard prompting
did not lead to signiﬁcant gains (e.g., increasing from 8 to 16 exemplars did not improve the
performance of standard prompting enough to catch up with chain of thought prompting).
• Different language models.
Another interesting question is whether certain prompts that workbetter for one model work better for other large language models.
We ﬁnd that with the same
prompts, chain of thought prompting improves performance across all three models (LM-1, GPT-3,and LM-2) for all datasets except CSQA and StrategyQA for GPT-
(Table 1, Table 4, Table 5).
The fact that gains from chain of thought did not transfer perfectly among models is a limitation;
[Table]
further work could investigate why how different pre-training datasets and model architectures
[Table]
affect the performance gain from chain of thought prompting.
[Table]
Prompt engineering still matters, though.
Although the results are relatively robust to the promptfor arithmetic reasoning, we want to be clear that prompt engineering still does matter, and can
[Table]
improve performance signiﬁcantly in many cases.
Though most chain of thought annotationsoutperform standard prompting, there is large variation in many cases.
For instance, for the coinﬂip task, the performance varied from 99.6% for Annotator A to 71.4% for Annotator C, thoughboth were above standard prompting = 50.0%
(see Table 7).
There are even tasks where promptengineering is a requirement for good performance.
In preliminary experiments, we tried using chainof thought to enable language models to reverse the order of a list of 5 items.
While two co-authorswere not able to write chain of thought prompts that solved the task despite their best attempts, a thirdco-author was able to write a chain of thought that perfectly solved the task.
[Table]
How to generate chain of thought annotations in a robust fashion could be an interesting directionfor future work.
For instance, an idea here could be to use a large language model to automaticallygenerate chains of thought via prompting (and potentially optimize this over a validation set)
A.3 Will chain of thought prompting improve performance for my task of interest?
While chain of thought prompting is in principle applicable for any text-to-text task, it is morehelpful for some tasks than others.
Based on the experiments in this paper, our intuition is that chainof thought helps the most when three conditions are met: (1) the task is challenging and requiresmulti-step reasoning, (2) a large language model is used, and (3) the scaling curve is relatively ﬂat.
Conversely, the beneﬁts are smaller when one or more of these conditions are not met.
These intuitions are perhaps supported by the arithmetic reasoning results.
The performance gainfrom chain of thought prompting is largest for LM-2 540B on GSM8K (challenging multi-step problems, ﬂat scaling curve), which meets these conditions.
The performance gain is small for thesubsets of MAWPS that only require one or two steps (SingleOP, SingleEq, and AddSub), for which
LM-
540B already achieves performance of 90% or higher (and it is also generally true that there isless headroom for improvement when performance is already strong).
A.4 Why is prompting with the equation only not enough for some arithmetic reasoningdatasets?
Prompting with the equation only as an intermediate step does help on many datasets, especially whenthe datasets only require a few reasoning steps (SVAMP, ASDiv, MAWPS).
For GSM8K, however,using the equation only did not improve performance substantially.
Based on qualitative analysis, webelieve that these questions are too semantically challenging for the model to directly translate theminto a math equation.
Consider this example from LM-1 137B:
Q UESTION : Mike plays ping pong for 40 minutes. In the ﬁrst 20 minutes, he scores 4points. In the second 20 minutes, he scores 25% more points. How many total points did hescore?
E QUATION ONLY ( WRONG ANSWER ): (4 + 20 * 0.25) = 6. The answer is 6.
C HAIN OF THOUGHT ( CORRECT ): Mike played ping pong for 40 minutes. In the ﬁrst20 minutes, he scored 4 points. In the second 20 minutes, he scored 25% more points. Sohe scored 25% more in the second 20 minutes.
4 x 1.25 = 5.
So he scored 5 points in thesecond 20 minutes.
So he scored 9 points in total. The answer is 9.
It is hard for the model to directly translate all of the semantics into a single equation, but chain ofthought allows it to better reason about each part of the question via intermediate steps in naturallanguage.
B All Experimental Results
This section contains tables for experimental results for varying models and model sizes, on allbenchmarks, for standard prompting vs. chain of thought prompting.
For the arithmetic reasoning benchmarks, some chains of thought (along with the equations produced)were correct, except the model performed an arithmetic operation incorrectly.
A similar observationwas made in Cobbe et al.
Hence, we can further add a Python program as an externalcalculator (using the Python eval function) to all the equations in the generated chain of thought.
When there are multiple equations in a chain of thought
[Table]
we propagate the external calculator resultsfrom one equation to the following equations via string matching.
As shown in Table
[Table]
we see thatadding a calculator signiﬁcantly boosts performance of chain of thought prompting on most tasks.
[Table]
[-(+7.8) 37.5 (+8.0) 46.
 (+6.5) 20.6 (-4.9) 57.9 (+14.7) + ext. calc 17. GPT-3 175B-]-davinci
[Table]
[-Standard 19. (code-davinci-002) Chain of thought 63.1 (+43.4) 76.4 (+6.5) 80.4 (+6.4) 45.3 (+15.8) 92.
 (+13.9) + ext.-][Table]
[-calc 65. LM-
 540B Standard 17.-]1.3B 2.
6.7B 4.
175B 15.6 46.
[-LM-
-]LM-2 8B 4.
62B 9.6 29.
540B 17.9 56.
[Table] [Table]
[Table]
[Table]
equation only
[Table]
· variable compute only
[Table]
different annotator (B)
[Table]
· different annotator (C)
[Table]
intentionally concise style
[Table]
· exemplars from GSM8K ( α )
[Table]
Chain of thought generally outperforms ablations by a large amount.
Chain of thought prompting has variance (as expected) when used with prompts written by different annotators or when using other exemplars, but still outperforms standard prompting by a large margin.
Standard deviation shown is for different order of few-shot prompting exemplars, with ﬁve different random seeds.
Results here are shown for LM-1 137B, as additional queries for GPT-3 and LM-2 are both limited and expensive.
The exception is that we run SayCan using LM-2 here, as the SayCan evaluation set is onlyexamples and therefore less expensive to run multiple times.
[Table]
C Extended Related Work
Chain of thought prompting is a general approach that is inspired by several prior directions: prompting, natural language explanations, program synthesis/execution, numeric and logical reasoning, andintermediate language steps.
C.1 Prompting
The recent success of large-scale language models has led to growing interest in improving theircapability to perform tasks via prompting (Brown et al.
(2020), and see Liu et al.
(2021) for asurvey).
This paper falls in the category of general prompting approaches, whereby input prompts areoptimized to allow a single large language model to better perform a variety of tasks (Li and Liang,2021; Lester et al., 2021; Reif et al., 2022, inter alia ).
One recent line of work aims to improve the ability of language models to perform a task by providinginstructions that describe the task (Raffel et al., 2020; Wei et al., 2022;
Ouyang et al., 2022; Sanh et al.
,2022; Wang et al., 2022b).
This line of work is related because it also augments input–output pairswith meta-data.
But whereas an instruction augments the input to a task (instructions are typicallyprepended to the inputs), chain of thought prompting augments the outputs of language models.
Another related direction is sequentially combining the outputs of language models; human–computerinteraction (HCI) work (Wu et al., 2022a,b) has shown that combining sequential generations oflanguage models improves task outcomes in a 20-person user study.
C.2 Natural language explanations
Another closely related direction uses natural language explanations (NLEs), often with the goal ofimproving model interpretability (Zhou et al., 2020; Wiegreffe and Marasovi´c, 2021, inter alia ).
Thatline of work typically focuses on natural language inference (Camburu et al., 2018;
Yordanov et al.
,2021; Bostrom et al., 2021), and produces explanations either simultaneously to or after the ﬁnalprediction (Narang et al., 2020; Majumder et al., 2021;
Wiegreffe et al., 2021, 2022).
By contrast,the chain of thought processing considered in this paper occurs before the ﬁnal answer.
And while
NLE aims mostly to improve neural network interpretability (Rajagopal et al., 2021), the goal ofchain of thought prompting is to allow models to decompose multi-hop reasoning tasks into multiplesteps—interpretability is just a side effect.
Marasovi ´ c et al.
(2022) show that prompt-based ﬁnetuningwith NLE improves NLI and classiﬁcation performance, though they largely focus on evaluatingexplanation plausibility.
In comparison, our work focuses on a range of arithmetic, commonsense,and symbolic tasks that require multi-hop reasoning.
C.3 Program synthesis and execution
Using intermediate reasoning steps has a long history in program synthesis and execution (Zarembaand Sutskever, 2014, inter alia ).
Recent work along in this direction has included a number ofarchitectural innovations (Cai et al., 2017; Dong et al., 2019; Yan et al., 2020), as well as the use oflarge language models (Chen et al., 2021;
Austin et al., 2021).
The program execution work closest toours is perhaps Nye et al.
(2021), which show that large language models can perform up to 10-digitaddition, evaluate polynomials, and execute python programs.
Whereas generating a program andthen executing it can be viewed as a type of reasoning, our work generalizes such domain-speciﬁcprimitives to natural language, which is open-domain and relevant to any text-to-text NLP task inprinciple.
C.4 Numeric and logical reasoning
Numeric and logical reasoning has been a long-studied task in machine learning and natural languageprocessing (Lev et al., 2004, inter alia ).
Recent work has also aimed to inject numeric reasoningabilities in language models in various ways, such as augmenting BERT with a predeﬁned set ofexecutable operations (Andor et al., 2019), including a graph neural network (Ran et al., 2019), andusing specialized training procedures (Pi˛ekos et al., 2021).
Another line of work aims to enablelanguage models to perform logical or formal reasoning, often by verablizing the rules in naturallanguage formal rules using language (Clark et al., 2020; Saeed et al., 2021; Liang et al., 2021).
Perhaps the most-related work here is Recchia (2021), which shows that ﬁnetuning enables longhandmodule operations, which has previously been difﬁcult for performers.
Whereas work in this directionis often task-speciﬁc and uses ﬁnetuning, we show that chain of thought prompting works for a broadrange of tasks without any ﬁnetuning.
C.5 Intermediate language steps
Extensive prior work has shown the beneﬁts of endowing neural networks with the ability to produceintermediate steps via training or ﬁnetuning confers various beneﬁts in a range of scenarios.
Asexamples, it has been shown that natural language intermediate steps can improve performance(Zaidan et al., 2007;
Yao et al., 2021; Hase and Bansal, 2022; Gu et al., 2022), improve robustness(Chen et al., 2022), speed up training (Hancock et al., 2018), mitigate bias (Dua et al., 2020), andeven help in image and reinforcement learning settings (Andreas et al., 2018).
To endow models withthe ability to produce intermediate steps, prior work typically ﬁnetunes models on either manuallyannotated training datasets (Camburu et al., 2018; Rajani et al., 2019, inter alia ) or generates syntheticdatasets (Talmor et al., 2020; Zelikman et al., 2022).
Compared with these training or ﬁnetuningmethods, our work shows that various natural language reasoning abilities can be elicited in off-theshelf language models of sufﬁcient scale simply via prompting.
This prompting setup is importantbecause it allows for intermediate step reasoning without a large number of labeled annotations, andbecause a single model can perform a range of reasoning tasks without any gradient updates.
D Appendix: Additional Analysis
D.1 Correct Chain of Thought Analysis
As mentioned in the main text, we analyze 50 chains of thought from LM-1 137B that led to correctanswers in the GSM8K dataset.
Of these 50, only one
arrived at the correct answer through incorrect
reasoning (shown in Table
“correct by chance”
The other 49 had correct logic and math
,with examples shown in Table 8.
Five had minor imperfections while maintaining coherent andunderstandable logic:
• One had underspeciﬁed statements (shown in Table 9: “correct but underspeciﬁed statements”).• One made an unrelated but correct statement (shown in Table 9: “correct but unrelated statement”).• Two had the correct logic but omitted speciﬁc steps in the equation, though it was still understandable (shown in Table 9: “correct but imperfect equation”).• One had the correct math but inverted the semantics (shown in Table 9: “correct but invertedsemantics”)
[Table]
Although we ﬁnd that chain of thought reasoning is mostly correct for math questions where the ﬁnalanswer was correct, this is likely because it is hard to arrive at the correct ﬁnal answer by coincidencefor free response questions.
For multiple choice or binary classiﬁcation problems, it was much more likely that models could arrive at the correct answer via an incorrect reasoning path (e
[Table]
g., all thecommmonsense reasoning datasets we evaluate).
This is a limitation
[Table]
and future work should performan analysis of the factuality of such chains of thought.
[Table]
D.2 Incorrect Chain of Thought Analysis
We also manually analyze 50 randomly sampled outputs of the model that were incorrect on GSM8Kfor LM-
137B. There are many ways that a chain of thought can be incorrect, making the designof error categorization non-trivial.
We decided to categorize errors into what changes are needed tomake the chain of thought correct, with the goal of elucidating how the model can be improved in thefuture.
We found that many chains of thought can be made correct with one of the following three classes ofmodiﬁcation.
• Calculator error only.
We found that 8% of the chains of thought were completely correct exceptfor a calculator error—in other words, applying an external calculator to equations, as done inCobbe et al.
(2021), would make the chain of thought correct.
An example of this type of error isshown in Table 10: “calculator error only”.
Indeed, the solve rate of chain of thought prompting
on for LM-1 137B GSM8K went up from 14.3% to 17.3% when we added a Python programas an external calculator, as shown in Table 2.
Also, 34% of the examples contained calculatorerrors in addition to other types of errors.
However, we perform the rest of the error categorizationindependently of calculator errors.• Symbol mapping error.
We next found that 16% percent of the chains of thought were correctexcept for what we call symbol mapping errors.
We deﬁne a symbol mapping error as when thechain of thought is correct except for the number symbols, and it could be made totally correct bymodifying only the equations and not the words.
As one might argue that they could simply placethe correct ﬁnal equation in any chain of thought, we constrain this category to chains of thoughtwhere the chain of thought can be modiﬁed to be a completely correct reasoning process (not justﬁnal answer).
An example of this error category is shown in Table 10: “symbol mapping error”.
• One step missing error
Our next category of error is chains of thought which were correct exceptthat they were missing a single step.
In other words, these chains of thoughts could be rewritten tobe correct by adding in an additional reasoning step that was missed by the model.
An example ofthis error category is shown in Table 10: “one step missing error”.
We found that 22% percent ofthe errors fell into this category.
[Table]
[Table]
We found that the remaining chains of thought (27 of 50; 54%) would require substantial edits to makeinto a correct chain of thought.
Almost all cases here involved some error in semantic understanding(see Table 11: “semantic understanding error”), and 8 of the 27 also had incoherent chain of thoughts,meaning that some statements in the generated chain of thought did not follow from prior ones orviolated basic world knowledge (see Table 11: “incoherent chain of thought error”).
[Table]
E RROR C ATEGORY
[Table] thought is incoherent in that the percent of entire students enrolled in hip-hope dance cannot be the percent of student enrolled in hip-hop dance minus another term.
D.3 Additional Robustness Analysis
[Figure] As the experiments in the main paper use a ﬁxed number of few-shot exemplars (8; as constrained bythe input length of 1024 tokens), we verify that the chain of thought prompting is robust to variousnumbers of few-shot exemplars.
We run experiments for LM-1 137B, comparing chain of thoughtprompting with standard prompting for the ﬁve datasets where standard prompting had a mostly ﬂatscaling curve (the largest model did not achieve high performance).
As shown in Figure 11, theimprovement of chain of thought prompting over standard prompting remains robust to varying thenumber of few-shot exemplars in the prompt.
[Figure]
[Table]
[Table]
Each pack of dvds costs 76 dollars.
If there is a discount of 25 dollars on each pack.
How much do you have to pay to buy each pack?
[Table]
A car is being driven, in a straight line and at a uniform speed, towards the base of a vertical tower.
The top of the tower is observed from the car and, in the process, it takes 10 minutes for the angle of elevation to change from 45 ◦ to 60 ◦ .
After how much more time will this car reach the base of the tower?
Answer
Choices: (a) 5 √ 3 + 1 (b) 6 √ 3 + √ 2 (c) 7 √ 3 - 1 (d) 8 √ 3 - 2 (e) None of these
MAWPS: SingleOp
If there are 7 bottle caps in a box and Linda puts 7 more bottle caps inside, how many bottle caps are in the box?
MAWPS: SingleEq
Benny bought a soft drink for 2 dollars and 5 candy bars.
He spent a total ofdollars.
How much did each candy bar cost?
MAWPS: AddSub
There were 6 roses in the vase.
Mary cut some roses from her ﬂower garden.
There are now 16 roses in the vase.
How many roses did she cut?
MAWPS: MultiArith
The school cafeteria ordered 42 red apples and 7 green apples for students lunches.
But, if only 9 students wanted fruit, how many extra did the cafeteria end up with?
E Additional DetailsE.1 Reproducibility Statement
As our results make use of two sets of large language models that is not publicly available, we takethe following actions to facilitate reproducibility.
First, we provide the exact input prompts for alltasks in Table 20–Table 27 in Appendix G (and emphasize that we do not perform any ﬁnetuning andonly apply prompting to off-the-shelf language models).
Second, we conduct experiments using thepublicly available GPT-3 API for four model scales text-ada-001, text-babbage-001, text-curie-001,text-davinci-002).
Finally, we make exact inputs, targets, and predictions for LM-1 137B for eachtask available as a zip ﬁle in the supplementary material.
E.2 Computational Resources
For all three language models we evaluated, we did prompting-based inference only.
No ﬁnetuningwas done for this paper.
For inference on LM-1 137B we use TPU v3 (8x8 conﬁguration, 64 chips /128 cores), and for inference on LM-2 540B we use TPU v4 (4x4x12 conﬁguration, 192 chips / 384cores).
GPT-3 experiments were done using the public API.
E.3 Dataset Details and Licenses
We list the details and licenses for all arithmetic and commonsense datasets used in this paper.
Thesymbolic reasoning datasets were created synthetically, as described in Section 4.
Arithmetic reasoning
• Math Word Problem Repository (Koncel-Kedziorski et al., 2016):
AddSub (Hosseiniet al., 2014): https://www.cs.washington.edu/nlp/arithmetic ;
MultiArith (Royand Roth, 2015), license: CC BY 4.0. • ASDiv (Miao et al., 2020): https://github.com/chaochun/nlu-asdiv-dataset .•
AQuA (Ling et al., 2017): https://github.com/deepmind/AQuA , license: https:// github.com/deepmind/AQuA/blob/master/LICENSE .
• GSM8K (Cobbe et al., 2021): https://github.com/openai/grade-school-math , MIT license: https://github.com/openai/grade-school-math/blob/master/
LICENSE .
• SVAMP (Patel et al., 2021): https://github.com/arkilpatel/SVAMP , MIT license:
https://github.com/arkilpatel/SVAMP/blob/main/LICENSE .
Commonsense reasoning
• CSQA (Talmor et al., 2019):
https://www.tau-nlp.org/commonsenseqa , https://
github.com/jonathanherzig/commonsenseqa .
• StrategyQA (Geva et al., 2021): we use the open-domain setting (question-only set)
from BIG-bench collaboration (2021): https://github.com/google/BIG-bench/
tree/main/bigbench/benchmark_tasks/strategyqa .
The original dataset is fromhttps://github.com/eladsegal/strategyqa , MIT license: https://github.com/eladsegal/strategyqa/blob/main/LICENSE .
• Date understanding and sports understanding from BIG-Bench (BIG-bench collaboration,2021): Apache License v.2: https://github.com/google/BIG-bench/blob/main/LICENSE .
• SayCan (Ahn et al., 2022): SayCan dataset can be accessed at https://say-can.github.
io/ under CC BY 4.0 license.
making-requests
[Table] [Table]
[Table] [Table]
[Table] [Table]
So the answer is yes
[Table]
(cid:55) (Ground truth is “no”, being out at home is part of baseball.)
[Table]
P ROMPT FOR M ATH W ORD P ROBLEMS
[Table]
parking
[Table]
and
[Table]
more
[Table]
and her sister had
[Table]
If they
[Table]
many
[Table]
they
[Table]
total
[Table]
Originally, [Table] had 32 chocolates.
Her sister had 42.
So in total they had [Table] [Table]
After eating 35, they had 74 - [Table]
The
[Table]
[Table]
Jason had 20 lollipops.
He gave Denny some lollipops.
Now Jason has 12 lollipops.
How many
[Table] did [Table] to Denny
[Table]
Jason [Table] with 20 lollipops.
Then he had [Table] giving some to Denny.
So he gave Denny 20 -
[Table] [Table]
[Table]
: Shawn has ﬁve toys.
For Christmas, he got two toys each from his mom and dad.
How many toys does he have now?
[Table]
: Shawn started with 5 toys.
If he got 2 toys each from his mom and dad, then that is 4 more toys.
5 + 4 = 9.
[Table]
Q: There were nine computers in the server room.
Five more computers were installed each day, from monday to thursday.
How many computers are now in the server room?
A:
There were originally 9 computers.
For each of 4 days, 5 more computers were added.
So 5 * 4 =computers were added.
9 + 20 is 29.
The answer is 29.
Q: Michael had 58 golf balls.
On tuesday, he
[Table]
P ROMPT FOR AQ U A ALGEBRAIC WORD PROBLEMS
[Table]
P ROMPT FOR L AST L ETTER C ONCATENATION
[Table]
" and concatenate them.
A: The last letter of "Elon" is "n".
The last letter of "Musk" is "k".
Concatenating them is "nk".
The answer is nk.
Q: Take the last letters of the words in "Larry Page" and concatenate them.
A: The last letter of "Larry" is "y".
The last letter of "Page" is "e".
Concatenating them is "ye".
The answer is ye.
Q: Take the last letters of the words in "Sergey Brin" and concatenate them.
A: The last letter of "Sergey" is "y".
The last letter of "Brin" is "n".
Concatenating them is "yn".
The answer is yn.
Q: Take the last letters of the words in "Bill Gates" and concatenate them.
A: The last letter of "Bill" is "l".
The last letter of "Gates" is "s".
Concatenating them is "ls".
The answer is ls.
[Table]
does not ﬂip the coin.
Is the coin still heads up?
A: The coin was ﬂipped by Conception.
So the coin was ﬂipped 1 time, which is an odd number.
The coin started heads up, so after an odd number of ﬂips, it will be tails up.
So the answer is no.
Q: A coin is heads up.
Inga does not ﬂip the coin.
Elanor does not ﬂip the coin.
Is the coin still heads up?
A: The coin was ﬂipped by no one.
So the coin was ﬂipped 0 times.
The coin started heads up, and it was not ﬂipped, so it is still heads up.
So the answer is yes.
Q: A coin is heads up.
Ryan ﬂips the coin.
Shaunda ﬂips the coin.
Is the coin still heads up?
A: The coin was ﬂipped by Ryan and Shaunda.
So the coin was ﬂipped 2 times, which is an even number.
The coin started heads up, so after an even number of ﬂips, it will still be heads up.
So the answer is yes.
P ROMPT FOR CSQA
[Table]
P ROMPT FOR S TRATEGY QA
[Table]
Hydrogen
[Table]
can happen
[Table]
and June
[Table]
winter
[Table]
could
[Table]
Could a llama birth twice during War in
[Table]
War in Vietnam was 6 months.
The gestation period for
[Table]
llama is 11 months
[Table]
is more than
[Table]
Thus
[Table]
llama could not give birth twice during the War in Vietnam.
So the
[Table]
is no.
[Table]
Would a pear sink in water
[Table]
The density of a pear is about 0 .
6 g/cm 3 , which is less than water.
Objects less dense than water ﬂoat.
Thus
[Table]
So
[Table]
is
[Table]
P ROMPT FOR D ATE U NDERSTANDING
Q: 2015 is coming in 36 hours.
What
is
the date one week from
today
in MM/DD/YYYY?
A:
If
is
coming
in 36 hours,
then
it is
coming
in days days
before
/01/2015 is 12/30/
, so today is
So one week from today will be
[Table]
P ROMPT FOR S PORTS U NDERSTANDING
[Table] [Table]
[Table] [Table]
H Appendix: Alternate Annotators for MWP
P ROMPT FOR M ATH W ORD P ROBLEMS
[Table]
P ROMPT FOR M ATH W ORD P ROBLEMS
[Table]