Original: L EAST - TO -M OST P ROMPTING E NABLES C OMPLEX R EASONING IN L ARGE L ANGUAGE M ODELS
Revised: L EAST - TO -M OST P ROMPTING E NABLES C OMPLEX R EASONING IN L ARGE L ANGUAGE M ODELS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Although chain-of-thought prompting has shown impressive results on many natural language reasoning tasks, it often performs poorly on tasks which need to solve problems harder than the demonstration examples.
Revised: Although chain-of-thought prompting has shown impressive results on many natural language reasoning tasks, it often performs poorly on tasks which need to solve problems harder than the demonstration examples.

Original: To tackle such easy-tohard generalization issues, we propose a novel prompting strategy, least-to-most prompting .
Revised: To tackle such easy-tohard generalization issues, we propose a novel prompting strategy, least-to-most prompting .

Original: It is implemented through two stage prompting: reducing a complex problem into a list of  subproblems, and then sequentially solving these subproblems, whereby solving a given subproblem is facilitated by the answers to previously solved subproblems.
Revised: The key idea is to reduce a complex problem into a list of easier subproblems, and then sequentially solve these subproblems, whereby solving a given subproblem is facilitated by the answers to previously solved subproblems.

Original: Experiments on symbolic manipulation, compositional generalization and math reasoning show that least-to-most prompting can generalize to the examples that are harder than those seen in the prompt, and outperform chain-of-thought prompting by a large margin.
Revised: Experiments on symbolic manipulation, compositional generalization and math reasoning show that least-to-most prompting can generalize to the examples that are harder than those seen in the prompt, and outperform chain-of-thought prompting by a large margin.

Original: A notable result is that the GPT3 code-davinci-002 model with least-to-most-prompting solves the SCAN benchmark regardless of splits (such as length split) with an accuracy of 99.7% using 14 examples versus an accuracy of 16.2% by chain-of-thought prompting, and neural-symbolic models in the literature specialized for solving SCAN are trained with the full training set of more than 15,000 examples. 
Revised: A notable result is that the GPT-3 code-davincimodel with least-to-most-prompting solves the SCAN benchmark regardless of splits (such as length split) with an accuracy of 99.7% using 14 examples versus an accuracy of 16.2% by chain-of-thought prompting. Neural-symbolic models in the literature specialized for solving SCAN are trained with the full training set of more than 15,000 examples. The full prompts are included in the Appendix.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Despite the great success of deep learning in the past decade, there still remain huge differences between human intelligence and machine learning: (1) Given a new task, humans usually can learn to accomplish it from only a few demonstration examples, while machine learning requires a large amount of labeled data for model training; (2) Humans can clearly explain the underlying rationale for their predictions or decisions, while machine learning is essentially a black box; (3) Humans can solve problems more difﬁcult than any they have seen before, while for machine learning, examples in training and testing are typically at the same level of difﬁculty.
Revised: Despite the great success of deep learning in the past decade, there still remain huge differences between human intelligence and machine learning: (1) Given a new task, humans usually can learn to accomplish it from only a few demonstration examples, while machine learning requires a large amount of labeled data for model training; (2) Humans can clearly explain the underlying rationale for their predictions or decisions, while machine learning is essentially a black box; (3) Humans can solve problems more difﬁcult than any they have seen before, while for machine learning, examples in training and testing are typically at the same level of difﬁculty.

Original: The recently proposed chain-of-thought prompting approach (Wei et al., 2022; Chowdhery et al., 2022) has taken a signiﬁcant step for narrowing the gap between human intelligence and machine intelligence.
Revised: The recently proposed chain-of-thought prompting approach (Wei et al., 2022; Chowdhery et al., 2022) has taken a signiﬁcant step for narrowing the gap between human intelligence and machine intelligence.

Original: It combines the idea of natural language rationales (Ling et al., 2017; Cobbe et al., 2021) with few-shot prompting (Brown et al., 2020).
Revised: It combines the idea of natural language rationales (Ling et al., 2017; Cobbe et al., 2021) with few-shot prompting (Brown et al., 2020).

Original: By further integrating withthe self-consistency decoding (Wang et al., 2022b) rather than using the typical greedy decoding, few-shot chain-of-thought prompting largely outperforms the state-of-the-art results in the literature on many challenging natural language processing tasks obtained from specially designed neural models trained with hundreds of times more annotated examples, while being fully interpretable.
Revised: When further integrated with self-consistency decoding (Wang et al., 2022b) rather than using the typical greedy decoding, few-shot chain-of-thought prompting largely outperforms the state-of-the-art results in the literature on many challenging natural language processing tasks obtained from specially designed neural models trained with hundreds of times more annotated examples, while being fully interpretable.

Original: However, chain-of-thought prompting has a key limitation—it often performs poorly on tasks that require generalization of solving problems harder than those demonstration examples, such as compositional generalization (Lake & Baroni, 2018; Keysers et al., 2020).
Revised: However, chain-of-thought prompting has a key limitation—it often performs poorly on tasks that require generalization of solving problems harder than the demonstration examples, such as compositional generalization (Lake & Baroni, 2018; Keysers et al., 2020).

Original: To tackle such easy-to-hard generalization issues, we propose least-to-most prompting .
Revised: To tackle such easy-to-hard generalization issues, we propose least-to-most prompting .

Original: It consists of two stages:
Revised: It consists of two stages:

Original: ﬁrst reducing a complex problem into a list of easier subproblems, and then sequentially solving these subproblems, whereby solving a given subproblem is facilitated by the answers to previously solved subproblems.
Revised: ﬁrst reducing a complex problem into a list of easier subproblems, and then sequentially solving these subproblems, whereby solving a given subproblem is facilitated by the answers to previously solved subproblems.

Original: Both stages are implemented by few-shot prompting. So there is no training or ﬁnetuning in either stage.
Revised: Both stages are implemented by few-shot prompting, so that there is no training or ﬁnetuning in either stage.

Original: An example usage of least-to-most prompting is illustrated in Figure 1.
Revised: An example usage of least-to-most prompting is illustrated in Figure 1.

Original: The term least-to-most prompting is borrowed from educational psychology (Libby et al., 2008), where it is used to denote the technique of using a progressive sequence of prompts to help a student to learn a new skill.
Revised: The term least-to-most prompting is borrowed from educational psychology (Libby et al., 2008), where it is used to denote the technique of using a progressive sequence of prompts to help a student to learn a new skill.

Original: Here we apply the technique for teaching humans to teach language models.
Revised: Here we apply this technique for teaching humans to teach language models.

Original: Empirical results on symbolic manipulation, compositional generalization, and math reasoning show that least-to-most prompting can indeed generalize to problems harder than those demonstrated.
Revised: Empirical results on symbolic manipulation, compositional generalization, and math reasoning show that least-to-most prompting can indeed generalize to problems harder than those demonstrated.

Original: LanguageModel
Revised: LanguageModel

Original: Problem [Figure]
Revised: Problem Reduction

Original: Q: It takes Amy 4 minutes to climb to the top of a slide.
Revised: Q: It takes Amy 4 minutes to climb to the top of a slide.

Original: It takes her 1 minute to slide down.
Revised: It takes her 1 minute to slide down.

Original: The water slide closes in 15 minutes.
Revised: The water slide closes in 15 minutes.

Original: How many times can she slide before it closes?
Revised: How many times can she slide before it closes?

Original: A: To solve “How many times can she slide before it closes?”, we need to first solve: “How long does each trip take?”
Revised: A: To solve “How many times can she slide before it closes?”, we need to first solve: “How long does each trip take?”

Original: It takes Amy 4 minutes to climb to the top of a slide.
Revised: It takes Amy 4 minutes to climb to the top of a slide.

Original: It takes her 1 minute to slide down.
Revised: It takes her 1 minute to slide down.

Original: The slide closes in 15 minutes.
Revised: The slide closes in 15 minutes.

Original: Q: How long does each trip take?
Revised: Q: How long does each trip take?

Original: A: It takes Amy 4 minutes to climb and 1 minute to slide down.
Revised: A: It takes Amy 4 minutes to climb and 1 minute to slide down.

Original: 4 + 1 = 5.
Revised: 4 + 1 = 5.

Original: So each trip takes 5 minutes.
Revised: So each trip takes 5 minutes.

Original: It takes Amy 4 minutes to climb to the top of a slide.
Revised: It takes Amy 4 minutes to climb to the top of a slide.

Original: It takes her 1 minute to slide down.
Revised: It takes her 1 minute to slide down.

Original: The slide closes in 15 minutes.
Revised: The slide closes in 15 minutes.

Original: Q: How long does each trip take?
Revised: Q: How long does each trip take?

Original: A: It takes Amy 4 minutes to climb andminute to slide down.
Revised: A: It takes Amy 4 minutes to climb andminute to slide down.

Original: 4 + 1 = 5.
Revised: 4 + 1 = 5.

Original: So each trip takes 5 minutes.
Revised: So each trip takes 5 minutes.

Original: Q: How many times can she slide before it closes?
Revised: Q: How many times can she slide before it closes?

Original: A: The water slide closes in 15 minutes.
Revised: A: The water slide closes in 15 minutes.

Original: Each trip takesminutes.
Revised: Each trip takesminutes.

Original: So Amy can slide 15 ÷ 5 = 3 times before it closes.
Revised: So Amy can slide 15 ÷ 5 = 3 times before it closes.

Original: Sequentially Solve Subquestions
Revised: Sequentially Solve Subquestions

Original: LanguageModel LanguageModel
Revised: LanguageModel LanguageModel

Original: Subquestion
Revised: Subquestion

Original: Append model answer to
Revised: Append model answer to

Original: Subquestion Subquestion
Revised: Subquestion Subquestion

Original: StageStage
Revised: StageStage

Original:  L EAST - TO -M OST P ROMPTING
Revised:  L EAST - TO -M OST P ROMPTING

Original: Least-to-most prompting teaches language models how to solve a complex problem by reducing it to a series of subproblems with increasing complexity to solve.
Revised: Least-to-most prompting teaches language models how to solve a complex problem by reducing it to a series of subproblems with increasing complexity to solve.

Original: It consists of two stages
Revised: It consists of two stages:

Original: The ﬁrst stage is for problem reduction .
Revised: The ﬁrst stage is for problem reduction .

Original: The prompt in this stage contains constant examples that demonstrate the reduction followed by the speciﬁc question to be reduced.
Revised: The prompt in this stage contains constant examples that demonstrate the reduction, followed by the speciﬁc question to be reduced.

Original: The second stage is for problem solving —sequentially solve the generated subproblems from the ﬁrst stage.
Revised: The second stage is for problem solving —sequentially solve the generated subproblems from the ﬁrst stage.

Original: The prompt in this stage consists of three parts: (1) constant examples demonstrating how subproblems are solved; (2) a potentially empty list of previously answered subquestions and generated solutions, and (3) the question to be answered next.
Revised: The prompt in this stage consists of three parts: (1) constant examples demonstrating how subproblems are solved; (2) a potentially empty list of previously answered subquestions and generated solutions, and (3) the question to be answered next.

Original: In the example shown in Figure 1, the language model is ﬁrst asked to reduce the original problem into subproblems.
Revised: In the example shown in Figure 1, the language model is ﬁrst asked to reduce the original problem into subproblems.

Original: The prompt that is passed to the model consists of examples that illustrates how to reduce complex problems (which is not shown in the ﬁgure) followed by the speciﬁc problem to be reduced (as shown in the ﬁgure).
Revised: The prompt that is passed to the model consists of examples that illustrate how to reduce complex problems (which are not shown in the ﬁgure), followed by the speciﬁc problem to be reduced (as shown in the ﬁgure).

Original: The language model’s answer reﬂects that the original problem can be solved via solving an intermediate problem “How long does each trip take?”.
Revised: The language model’s answer reﬂects that the original problem can be solved via solving an intermediate problem “How long does each trip take?”.

Original: In the next phase, we ask the language model to sequentially solve the subproblems from the former problem reduction stage.
Revised: In the next phase, we ask the language model to sequentially solve the subproblems from the former problem reduction stage.

Original: The original problem is appended as the ﬁnal subproblem.
Revised: The original problem is appended as the ﬁnal subproblem.

Original: The solving starts from passing to the language model a prompt that consists of examples that illustrates how problems are solved (not shown in the ﬁgure) followed by the ﬁrst subproblem “How long does each trip take?”.
Revised: The solving starts from passing to the language model a prompt that consists of examples that illustrate how problems are solved (not shown in the ﬁgure), followed by the ﬁrst subproblem “How long does each trip take?”.

Original: We then take the answer generated by the language model (“... each trip takes minutes.”) and construct the next prompt by appending the generated answer to the previous prompt followed by the next subproblem, which happens to be  original problem in this example.
Revised: We then take the answer generated by the language model (“... each trip takes minutes.”) and construct the next prompt by appending the generated answer to the previous prompt, followed by the next subproblem, which happens to be the original problem in this example.

Original: The new prompt is then passed back to the language model, which returns the ﬁnal answer.
Revised: The new prompt is then passed back to the language model, which returns the ﬁnal answer.

Original: Least-to-most prompting can be combined with other prompting techniques like chain-of-thought (Wei et al., 2022) and self-consistency (Wang et al., 2022b) but  not necessary.
Revised: Least-to-most prompting can be combined with other prompting techniques like chain-of-thought (Wei et al., 2022) and self-consistency (Wang et al., 2022b), but does not need to be.

Original: In addition, for some tasks, the two stages in least-to-most prompting could be merged to form a single-pass prompting.
Revised: Also, for some tasks, the two stages in least-to-most prompting can be merged to form a single-pass prompt.

Original:  R ESULTS
Revised:  R ESULTS

Original: We present least-to-most prompting results for symbolic manipulation, compositional generalization, and math reasoning tasks.
Revised: We present least-to-most prompting results for symbolic manipulation, compositional generalization, and math reasoning tasks.

Original: We start with a symbolic manipulation task since it is quite simple while being able to illustrate the key ideas in least-to-most prompting.
Revised: We start with a symbolic manipulation task, since it is quite simple, while being able to illustrate the key ideas in least-to-most prompting.

Original: We will see that language models seem understanding recursion by only observing a two-step demonstration.
Revised: We will see that language models seem to understand recursion by only observing a two-step demonstration.

Original: 3. S YMBOLIC M ANIPULATION
Revised: 3. S YMBOLIC M ANIPULATION

Original: Consider theso-called last-letter-concatenation task (Wei et al., 2022). The input is a list of words, and the output is the concatenation of the last letters of the words in the list.
Revised: Consider the last-letter-concatenation task (Wei et al., 2022), where the input is a list of words, and the output is the concatenation of the last letters of the words in the list.

Original: For example, “thinking, machine” outputs “ge” since the last letter of “thinking” is ”g” and the last letter of “machine” is “e”.
Revised: For example, “thinking, machine” outputs “ge”, since the last letter of “thinking” is ”g” and the last letter of “machine” is “e”.

Original: The technique that we present for solving  last-letter-concatenation task can be directly adapted to solve other symbolic manipulation tasks such as reversing a list of words.
Revised: The technique that we present for solving the last-letter-concatenation task can be directly adapted to solve other symbolic manipulation tasks such as reversing a list of words.

Original: The experiment setup is as follows: (1) only two demonstration examples are provided; (2) the lists used in demonstration contain at most three words, while the test lists contain four or more words.
Revised: The experiment setup is as follows: (1) only two demonstration examples are provided; (2) the lists used for demonstration contain at most three words, while the test lists contain four or more words.

Original: This task is trivial for humans, but it is supposed to be impossible for even most advanced statistical learning methods: (1) models trained with two examples are not expected to generalize; (2) moreover, the length-based train-test split requires out-of-distribution generalization, while machine learning is essentially built on the assumption of identical distributions (Vapnik, 1999).
Revised: This task is trivial for humans, but it is supposed to be hard even for advanced statistical learning methods: (1) models trained with two examples are not expected to generalize; (2) moreover, the length-based train-test split requires out-of-distribution generalization, while machine learning is essentially built on the assumption of identical distributions (Vapnik, 1999).

Original: Chain-of-thought prompting
Revised: Chain-of-thought prompting

Original: Least-to-most prompting (solving stage)
Revised: Least-to-most prompting (solving stage)

Original: Q: “think, machine”
Revised: Q: “think, machine”

Original: A: The last letter of “think” is “k”.
Revised: A: The last letter of “think” is “k”.

Original: The last letter of “machine” is “e”.
Revised: The last letter of “machine” is “e”.

Original: Concatenating “k”, “e” leads to “ke”.
Revised: Concatenating “k”, “e” leads to “ke”.

Original: So, “think, machine” outputs “ke”.
Revised: So, “think, machine” outputs “ke”.

Original: Q: “learning, reasoning, generalization”
Revised: Q: “learning, reasoning, generalization”

Original: A: The last letter of “learning” is “g”.
Revised: A: The last letter of “learning” is “g”.

Original: The last letter of “reasoning” is “g”.
Revised: The last letter of “reasoning” is “g”.

Original: The last letter of “generalization” is “n”.
Revised: The last letter of “generalization” is “n”.

Original: Concatenating “g”, “g”, “n” leads to “ggn”.
Revised: Concatenating “g”, “g”, “n” leads to “ggn”.

Original: So, “learning, reasoning, generalization” outputs “ggn”.
Revised: So, “learning, reasoning, generalization” outputs “ggn”.

Original: Q: “think, machine”
Revised: Q: “think, machine”

Original: A: The last letter of “think” is “k”.
Revised: A: The last letter of “think” is “k”.

Original: The last letter of “machine” is “e”.
Revised: The last letter of “machine” is “e”.

Original: Concatenating “k”, “e” leads to “ke”.
Revised: Concatenating “k”, “e” leads to “ke”.

Original: So, “think, machine” outputs “ke”.
Revised: So, “think, machine” outputs “ke”.

Original: Q: “think, machine, learning”
Revised: Q: “think, machine, learning”

Original: A: “think, machine” outputs “ke”.
Revised: A: “think, machine” outputs “ke”.

Original: The last letter of “learning” is “g”.
Revised: The last letter of “learning” is “g”.

Original: Concatenating “ke”, “g” leads to “keg”.
Revised: Concatenating “ke”, “g” leads to “keg”.

Original: So, “think, machine, learning” outputs “keg”.
Revised: So, “think, machine, learning” outputs “keg”.

Original: Least-to-most prompting .
Revised: Least-to-most prompting .

Original: We need to deﬁne the implementations for both the problem reduction and solving stages.
Revised: We need to deﬁne the implementations for both the problem reduction and solving stages.

Original: Subproblems here are simply the preﬁxes of the list.
Revised: The subproblems here are simply the preﬁxes of the list.

Original: For example, given a list “think, machine, learning, reasoning”, the least-to-most subproblems are: “think, machine”, “think, machine, learning” and “think, machine, learning, reasoning”.
Revised: For example, given a list “think, machine, learning, reasoning”, the least-to-most subproblems are: “think, machine”, “think, machine, learning”, and “think, machine, learning, reasoning”.

Original: Such a trivial reduction can be donestraightforwardly by a manually written script or  prompting language models with several demonstrations.
Revised: Such a simple reduction can be done by a manually written script or by prompting language models (see Appendix A.1 for details about the prompting-based reduction).

Original: The constant prompt context for solving the subproblems is presented in the right column of Table 1.
Revised: The constant prompt context for solving the subproblems is presented in the right column of Table 1.

Original: Essentially, this prompt teaches a language model to build answers to new problems using the answers to previously solved problems: (1) the list in the second example (“think, machine, learning”) is an extension of the list in the ﬁrst example (“think, machine”) rather than an independent example; (2) the answer to the second example copies the output from the ﬁrst one by starting with a sentence that says “think, machine” outputs “ke”.
Revised: Essentially, this prompt teaches a language model to build answers to new problems using the answers to previously solved problems: (1) the list in the second example (“think, machine, learning”) is an extension of the list in the ﬁrst example (“think, machine”) rather than an independent example; (2) the answer to the second example copies the output from the ﬁrst one by starting with a sentence that says “think, machine” outputs “ke”.

Original: Although this copy function is trivial for humans, we found that not all language models (e.g. text-curie-001 in GPT-3) are able to consistently support copying, which may cause least-to-most prompting to fail.
Revised: Although this copy function is trivial for humans, we found that not all language models (e.g. text-curiein GPT-3) are able to consistently support copying, which may cause least-to-most prompting to fail.

Original: Q: “mind, emerge”
Revised: Q: “mind, emerge”

Original: A: The last letter of “mind” is “d”.
Revised: A: The last letter of “mind” is “d”.

Original: The last letter of “emerge” is ”e”.
Revised: The last letter of “emerge” is ”e”.

Original: Concatenating “d”, “e” leads to “de”.
Revised: Concatenating “d”, “e” leads to “de”.

Original: So, “mind, emerge” outputs “de”.
Revised: So, “mind, emerge” outputs “de”.

Original: Q: “mind, emerge, from”
Revised: Q: “mind, emerge, from”

Original: A: “mind, emerge” outputs “de”.
Revised: A: “mind, emerge” outputs “de”.

Original: The last letter of “from” is “m”.
Revised: The last letter of “from” is “m”.

Original: Concatenating “de”, “m” leads to “dem”.
Revised: Concatenating “de”, “m” leads to “dem”.

Original: So, “mind, emerge, from” outputs “dem”.
Revised: So, “mind, emerge, from” outputs “dem”.

Original: Q: “mind, emerge, from, neuron”
Revised: Q: “mind, emerge, from, neuron”

Original: A: “mind, emerge, from” outputs “dem”.
Revised: A: “mind, emerge, from” outputs “dem”.

Original: The last letter of “neuron” is “n”.
Revised: The last letter of “neuron” is “n”.

Original: Concatenating “dem”, “n” leads to “demn”.
Revised: Concatenating “dem”, “n” leads to “demn”.

Original: So, “mind, emerge, from, neuron” outputs “demn”.
Revised: So, “mind, emerge, from, neuron” outputs “demn”.

Original: [Table]
Revised: [Table]

Original:  baselines.
Revised: thought prompting are used as the baselines.

Original: The chainof-thought and least-to-most prompts are listed in Table 1, and for standard prompting, its prompt is formed by removing the rationales from the chain-of-thought prompt.
Revised: The chainof-thought and least-to-most prompts are listed in Table 1. The prompt for standard prompting is formed by removing the rationales from the chain-of-thought prompt.

Original: All the methods are 2-shot with a list of 2 words and  a list of 3 words.
Revised: All the methods are 2shot where one example uses a list of 2 words and the other example uses a list of 3 words.

Original: We randomly sample words in Wiktionary 1 to construct testing lists with lengths varied from 4 to 12.
Revised: We randomly sample words in Wiktionary 1 to construct testing lists with lengths varying from 4 to 12.

Original: For each given length, 500 lists are constructed.
Revised: For each given length, 500 lists are constructed.

Original: The accuracies of different methods with code-davinci-002 in GPT-3 are shown in Table 3.
Revised: The accuracies of different methods with code-davinci-002 in GPT-3 are shown in Table 3.

Original: Standard prompting completely fails all test cases with an accuracy of 0.
Revised: Standard prompting completely fails all test cases with an accuracy of 0.

Original: Chain-of-thought prompting greatly boosts the performance over standard prompting, but it still falls well behind least-to-most prompting, particularly when the lists are long.
Revised: Chain-of-thought prompting signiﬁcantly boosts the performance over standard prompting, but it still falls well behind least-to-most prompting, particularly when the lists are long.

Original: Both chain-of-thought prompting and least-to-most prompting perform worse on longer lists.
Revised: Both chain-of-thought prompting and least-to-most prompting perform worse on longer lists.

Original: However, the performance of chain-of-thought prompting drops much faster with the length increases.
Revised: However, the performance of chain-of-thought prompting drops much faster as the length increases.

Original: By adding two more examples (see Appendix A.1, the lists are still limited to contain 2 or 3 words), we obtain about 2 points higher accuracies for chainof-thought prompting and least-to-most prompting, but the gain is diminishing when more examples are added. We also run the same promptings over text-davinci-002 .
Revised: In Appendices A.2 and A.3, we present additional experiments with different prompt examples and different language models. Among others, these experiments show that 2-shot least-tomost prompting also outperforms 4-shot and 8-shot chain-of-thought prompting.

Original: We observe thatwhen the length of lists is less than 10, text-davinci-002 is comparable to code-davinci-002 . However, after the length of lists increases to 10, text-davinci-002 becomes much worse.
Revised: This demonstrates that the accuracy gain is not caused by the fact that the least-to-most examples are slightly longer. Note that in contrast to least-to-most prompting, we use use two independent prompt examples for chain-of-thought (see Table 1) because this leads to better performance.

Original: We report these additional empirical results in Appendix A.2.
Revised: For completeness, we also present experiments using the same examples as for least-to-most prompting in the Appendix.

Original: Error analysis .
Revised: Error analysis .

Original: Although it signiﬁcantly outperforms chain-of-thought prompting, least-to-most prompting is still far from achieving 100% accuracy for long lists.
Revised: While least-to-most prompting signiﬁcantly outperforms chain-of-thought prompting, it is still far from achieving 100% accuracy for long lists.

Original: In Appendix A.3, we present a detailed error analysis for the results generated from code-davinci-002 .
Revised: In Appendix A.4, we present a detailed error analysis for the results generated from code-davinci-002 .

Original: We ﬁnd that only very few of them are due to incorrect last letters, and most of them are concatenation errors, dropping or adding a letter.
Revised: We ﬁnd that only very few of them are due to incorrect last letters, while most of them are concatenation errors (dropping or adding a letter).

Original: For example, given the list “gratiﬁed, contract, fortitude, blew”, the modelsomehow drops the last letter in the concatenation of “dte” and “w”, and thus predicts the outcome to be “dte” instead of “dtew”.
Revised: For example, given the list “gratiﬁed, contract, fortitude, blew”, the model drops the last letter in the concatenation of “dte” and “w”, and thus predicts the outcome to be “dte” instead of “dtew”.

Original: In another example “hollow, supplies, function, gorgeous”, the model somehow duplicates the last letter “s” in the concatenation of “wsn” and “s” and thus the prediction becomes “wsnss” instead of “wsns”.
Revised: In another example “hollow, supplies, function, gorgeous”, the model somehow duplicates the last letter “s” in the concatenation of “wsn” and “s”, and thus the prediction becomes “wsnss” instead of “wsns”.

Original: Those look somewhat reminiscent of careless mistakes from humans

Original: 3. C OMPOSITIONAL G ENERALIZATION
Revised: 3. C OMPOSITIONAL G ENERALIZATION

Original: SCAN (Lake & Baroni, 2018) is probably the most popular benchmark for evaluating compositional generalization.
Revised: SCAN (Lake & Baroni, 2018) is probably the most popular benchmark for evaluating compositional generalization.

Original: It requires mapping natural language commands to action sequences (Table 4).
Revised: It requires mapping natural language commands to action sequences (Table 4).

Original: Sequence-to-sequence models perform poorly under length split where the action sequences in the training set (about 80% of the full set of over 20,000 commands) are shorter than the action sequences in the testing set.
Revised: Sequence-to-sequence models perform poorly under length split where the action sequences in the training set (about 80% of the full set with over 20,000 examples) are shorter than the action sequences in the testing set.

Original: Many specialized neural-symbolic models have been proposed to solve
Revised: Many specialized neural-symbolic models have been proposed to solve

Original: SCAN (Chen et al., 2020; Liu et al., 2020; Nye et al., 2020; Shaw et al., 2021; Kim, 2021).
Revised: SCAN (Chen et al., 2020; Liu et al., 2020; Nye et al., 2020; Shaw et al., 2021; Kim, 2021).

Original: Here we show howSCAN can be solved by least-to-most prompting with only several demonstrations.
Revised: Here we show how least-to-most prompting solves SCAN using only a few demonstration examples.

Original: Command
Revised: Command

Original: Action Sequence “look thrice after jump”
Revised: Action Sequence “look thrice after jump”

Original: JUMP LOOK LOOK LOOK “run left 
Revised: JUMP LOOK LOOK LOOK “run left and walk”

Original: [Table]

Original: walk” TURN LEFT RUN WALK “look opposite right” 
Revised: TURN LEFT RUN WALK “look opposite right” TURN RIGHT TURN RIGHT LOOK

Original: TURN RIGHT TURN RIGHT LOOK generate we either or prompt language models to expand the Python It is straightforward to teach language models  to expand Python expressions via a few demonstrations.
Revised: Least-to prompting. maps to ority We use Python notation to make prompts and meet the input size limit of language models (usually up to 2048 tokens).

Original: Q: “look
Revised: To generate the ﬁnal results, we either run a postprocessing script or prompt language models to expand the Python expressions, which can be done with near-perfect accuracy of 99.7% (see Appendix B.4 for details). We use the same notation also for the baselines (standard and chain-of-thought prompting) to keep the comparison fair.

Original:  opposite right thrice after walk” A: “look opposite right thrice” can be solved by: “look opposite right”, “look opposite right thrice”
Revised: Q: “look opposite right thrice after walk” A: “look opposite right thrice” can be solved by: “look opposite right”, “look opposite right thrice”.

Original: “walk” can be solved by “walk”.
Revised: “walk” can be solved by “walk”.

Original: So, “look opposite right thrice after walk” can be solved by : “look opposite right”, “look opposite right thrice”, “walk”
Revised: So, “look opposite right thrice after walk” can be solved by: “look opposite right”, “look opposite right thrice”, “walk”.

Original: Q: “jump around right” A: “jump around right” can be solved by: “jump right”, “jump around right”.
Revised: Q: “jump around right” A: “jump around right” can be solved by: “jump right”, “jump around right”.

Original: So, “jump around right” can be solved by: “jump right”, “jump around right”.
Revised: So, “jump around right” can be solved by: “jump right”, “jump around right”.

Original: Q: “look around right thrice and walk” A “look around right thrice” can be solved by: “look right”, “look around right”, “look around right thrice”.
Revised: Q: “look around right thrice and walk” A: “look around right thrice” can be solved by: “look right”, “look around right”, “look around right thrice”.

Original: “walk” can be solved by “walk”.
Revised: “walk” can be solved by “walk”.

Original: So, “look around right thrice and walk” can be solved by: “look right”, “look around right”, “look around right thrice”, “walk”.
Revised: So, “look around right thrice and walk” can be solved by: “look right”, “look around right”, “look around right thrice”, “walk”.

Original: Results .
Revised: Results .

Original: We compare least-to-most prompting with chain-of-thought prompting and standard prompting.
Revised: We compare least-to-most prompting with chain-of-thought prompting and standard prompting.

Original: All the prompting methods share the same example commands, but the regular prompting directly maps them to action sequences without explanations, and chain-of-thought prompting uses the same command-mapping context as least-to-most prompting but does not use command reduction.
Revised: All the prompting methods share the same example commands, but the regular prompting directly maps them to action sequences without explanations, while chain-of-thought prompting uses the same command-mapping context as least-to-most prompting but does not use command reduction.

Original: The accuracies of different prompting methods with different language models are presented in Table 8.
Revised: The accuracies of different prompting methods with different language models are presented in Table 8.

Original: Using code-davinci-002 , least-to-most prompting achieves an accuracy of 99 .
Revised: Using code-davinci-002 , least-to-most prompting achieves an accuracy of 99 .

Original: 7% under length split.
Revised: 7% under length split.

Original: We investigate more than 100 generated solutions which lead to correct outputs.

Original: There is no such a case where the solution is wrong while the ﬁnal output is correct.

Original: A test case of least-to-most prompting is listed in Table 7.
Revised: A test case of least-to-most prompting is listed in Table 7.

Original: More example outputs can be found in Appendix B.3.
Revised: More example outputs can be found in Appendix B.3.

Original: We also test our method on all other splits and the full dataset.
Revised: We also test our method on all other splits and the full dataset.

Original: We ﬁnd that 
Revised: We ﬁnd that the solving rate remains the same.

Original: the
Revised: In addition, it may be interesting to note

Original: [Table]
Revised: [Table]

Original: A: The output of “jump left” concatenates: the output of “turn left”, the output of “jump”.
Revised: A: The output of “jump left” concatenates: the output of “turn left”, the output of “jump”.

Original: “turn left” outputs “TURN LEFT”.
Revised: “turn left” outputs “TURN LEFT”.

Original: “jump” outputs “JUMP”.
Revised: “jump” outputs “JUMP”.

Original: So concatenating the output of “turn left” and the output of “jump” leads to “TURN LEFT” + “JUMP”.
Revised: So concatenating the output of “turn left” and the output of “jump” leads to “TURN LEFT” + “JUMP”.

Original: So the output of “jump left” is “TURN LEFT” + “JUMP”.
Revised: So the output of “jump left” is “TURN LEFT” + “JUMP”.

Original: Q: “run and
Revised: Q: “run and

Original: A: The output of “run and look twice” concatenates: the output of “run”, the output of “look twice”.
Revised: A: The output of “run and look twice” concatenates: the output of “run”, the output of “look twice”.

Original: “run” outputs “RUN”.
Revised: “run” outputs “RUN”.

Original: “look twice” outputs “LOOK” * 2.
Revised: “look twice” outputs “LOOK” * 2.

Original: So concatenating the output of “run” and the output of “look twice” leads to “RUN” + “LOOK” * 2.
Revised: So concatenating the output of “run” and the output of “look twice” leads to “RUN” + “LOOK” * 2.

Original: So the output of “run and look twice” is “RUN” + “LOOK” * 2.
Revised: So the output of “run and look twice” is “RUN” + “LOOK” * 2.

Original: Q: “walk opposite left” A: The output of “walk opposite left” concatenates: the output of “turn opposite left”, the output of “walk”.
Revised: Q: “walk opposite left” A: The output of “walk opposite left” concatenates: the output of “turn opposite left”, the output of “walk”.

Original: “turn opposite left” outputs “TURN LEFT” * 2.
Revised: “turn opposite left” outputs “TURN LEFT” * 2.

Original: “walk” outputs “WALK”.
Revised: “walk” outputs “WALK”.

Original: So concatenating the output of “turn opposite left” and the output of “walk” leads to “TURN LEFT” * 2 + “WALK”.
Revised: So concatenating the output of “turn opposite left” and the output of “walk” leads to “TURN LEFT” * 2 + “WALK”.

Original: So the output of “walk opposite left” is “TURN LEFT” * 2 + ”WALK”.
Revised: So the output of “walk opposite left” is “TURN LEFT” * 2 + ”WALK”.

Original: [Table]
Revised: [Table]

Original: The
Revised: The

Original: [Table]
Revised: [Table]

Original: of “jump
Revised: of “jump

Original: [Table]
Revised: [Table]

Original: The output of
Revised: The output of

Original: [Table]
Revised: [Table]

Original: right twice” concatenates: the output of “jump right”, the output of “jump right”.
Revised: right twice” concatenates: the output of “jump right”, the output of “jump right”.

Original: “jump right” outputs “TURN RIGHT” + “JUMP”.
Revised: “jump right” outputs “TURN RIGHT” + “JUMP”.

Original: So repeating the output of “jump right” two times leads to (“TURN RIGHT” + “JUMP”)
Revised: So repeating the output of “jump right” two times leads to (“TURN RIGHT” + “JUMP”)

Original: So the output of “jump right twice” is (“TURN RIGHT” + “JUMP”)
Revised: So the output of “jump right twice” is (“TURN RIGHT” + “JUMP”)

Original: Q: “jump right twice after walk” A: The output of “jump right twice after walk” concatenates: the output of “walk”, the output of “jump right twice”.
Revised: Q: “jump right twice after walk” A: The output of “jump right twice after walk” concatenates: the output of “walk”, the output of “jump right twice”.

Original: “walk” outputs “WALK”.
Revised: “walk” outputs “WALK”.

Original: “jump right twice” outputs (“TURN RIGHT” + “JUMP”)
Revised: “jump right twice” outputs (“TURN RIGHT” + “JUMP”)

Original: So concatenating the output of “walk” and the output of “jump right twice” leads to “WALK” + (“TURN RIGHT” + “JUMP”)
Revised: So concatenating the output of “walk” and the output of “jump right twice” leads to “WALK” + (“TURN RIGHT” + “JUMP”)

Original: So the output of “jump right twice after walk” is “WALK” + (“TURN RIGHT” + “JUMP”)
Revised: So the output of “jump right twice after walk” is “WALK” + (“TURN RIGHT” + “JUMP”)

Revised: code-davinci-002 consistently performs better than text-davinci-002 , of prompting

Original: [Table]
Revised: [Table]

Revised: The results of text-davinci-002 are based on a random subset of 100 commands.

Original: Error analysis .
Revised: Error analysis .

Original: Under the test  of  length split, there are 13 failures in total from least-to-most prompting: 6 of them incorrectly interpret “twice” and “thrice” following “around”, and the rest incorrectly interpret “after” as “and”.
Revised: In the test set of the length split, there are 13 failures in total from least-to-most prompting: 6 of them incorrectly interpret “twice” and “thrice” following “around”, and the rest incorrectly interpret “after” as “and”.

Original: Let us show a failed examples for each category.
Revised: Let us show a failed example for each category.

Original: In the example “walk opposite right twice after run around right thrice”, code-davinci-002 correctly translates the expression “run around right” to (“TURN RIGHT” + “RUN”)
Revised: In the example “walk opposite right twice after run around right thrice”, code-davinci-002 correctly translates the expression “run around right” to (“TURN RIGHT” + “RUN”)

Original: Then it makes a mistake when applying “thrice” to this expression and produces (“TURN RIGHT” + “RUN”)
Revised: Then it makes a mistake when applying “thrice” to this expression and produces (“TURN RIGHT” + “RUN”)

Original: *instead of (“TURN RIGHT” + “RUN”) * 4 * 3 or (“TURN RIGHT” + “RUN”) 
Revised: * 9 instead of (“TURN RIGHT” + “RUN”) * 4 * 3 or (“TURN RIGHT” + “RUN”) * 12.

Original: In the example “run opposite left thrice after run around left twice”, code-davinci-002 produces the correct translations for both sub-expressions that are connected by “after” but it combines them as if they were connected by “and”.
Revised: In the example “run opposite left thrice after run around left twice”, code-davinci-002 produces the correct translations for both sub-expressions that are connected by “after” but it combines them as if they were connected by “and”.

Original: This means that the model produces (“TURN LEFT” * 2 + “RUN”)
Revised: This means that the model produces (“TURN LEFT” * 2 + “RUN”)

Original: * 3 + (“TURN LEFT” + “RUN”) * 4 * 2 instead of (“TURN LEFT” + “RUN”) * 4 * 2 + (“TURN LEFT” * 2 + “RUN”)
Revised: * 3 + (“TURN LEFT” + “RUN”) * 4 * 2 instead of (“TURN LEFT” + “RUN”) * 4 * 2 + (“TURN LEFT” * 2 + “RUN”)

Original: A detailed error analysis can be found in Appendix B.2.
Revised: A detailed error analysis can be found in Appendix B.2.

Original: 3. M ATH R EASONING Least-to-most prompting (Reducing) Least-to-most prompting (Solving)
Revised: 3. M ATH R EASONING Least-to-most prompting (Reducing) Least-to-most prompting (Solving)

Original: Q: The gender distribution of the population was 50.2% male and 49.8% female.
Revised: Q: The gender distribution of the population was 50.2% male and 49.8% female.

Original: Of the adult population, 29 people or 14.6% of the population are between 20 and 29 years old.
Revised: Of the adult population, 29 people or 14.6% of the population are between 20 and 29 years old.

Original: 28 people or 14.1% are 30 to 39, 36 people or 18.2% are 40 to 49, and 31 people or 15.7% are 50 to 59.
Revised: 28 people or 14.1% are 30 to 39, 36 people or 18.2% are 40 to 49, and 31 people or 15.7% are 50 to 59.

Original: How many percent of people are not 40 to 49?
Revised: How many percent of people are not 40 to 49?

Original: A: To answer the question “How many percent of people are not 40 to 49?’
Revised: A: To answer the question “How many percent of people are not 40 to 49?’

Original: , we need to know “How many percent of people are 40 to 49?”
Revised: , we need to know “How many percent of people are 40 to 49?”

Original: The gender distribution of the population was 50.2% male and 49.8% female.
Revised: The gender distribution of the population was 50.2% male and 49.8% female.

Original: Of the adult population,people or 14.6% of the population are between 20 and
Revised: Of the adult population,people or 14.6% of the population are between 20 and

Original: 29 years old.
Revised: 29 years old.

Original: 28 people or 14.1% are 30 to 39,people or 18.2% are 40 to 49, and 31 people or 15.7% are 50 to 59.
Revised: 28 people or 14.1% are 30 to 39,people or 18.2% are 40 to 49, and 31 people or 15.7% are 50 to 59.

Original: Q: How many percent of people are 40 to 49?
Revised: Q: How many percent of people are 40 to 49?

Original: A: “36 people or 18.2% are 40 to 49”.
Revised: A: “36 people or 18.2% are 40 to 49”.

Original: So the answer is 18.2%.
Revised: So the answer is 18.2%.

Original: Q: How many percent of people are not 40 to 49?
Revised: Q: How many percent of people are not 40 to 49?

Original: A:
Revised: A:

Original: We know that 18.2% are 40 to 49.
Revised: We know that 18.2% are 40 to 49.

Original: So 100% - 18.2% = 81.8% are not 40 to 49.
Revised: So 100% - 18.2% = 81.8% are not 40 to 49.

Original: So the answer is 81.8%.
Revised: So the answer is 81.8%.

Original: We compare different prompting methods using the numerical reasoning subset in DROP (Dua et al
Revised: We compare different prompting methods using the numerical reasoning subset in DROP (Dua et al., 2019).

Original: It contains 5,850 problems of which 1,862 problems are about football games.
Revised: It contains 5,850 problems of which 1,862 problems are about football games.

Original: We design prompts respectively for football and non-football problems.
Revised: We design prompts respectively for football and non-football problems.

Original: The baseline methods include zeroshot, standard prompting and chain-of-thought 
Revised: The baseline methods include zeroshot, standard prompting and chain-of-thought prompting.

Original: The prompts are listed in Appendix C.2 (for non-football problems) and Appendix C.3 (for football problems).
Revised: The prompts are listed in Appendix C.2 (for non-football problems) and Appendix C.3 (for football problems).

Original: An example in the least-to-most prompt is shown in Table 9, where the example on the leftcolumn shows how to reduce a problems to subproblems, and the example on the rightcolumn shows how the subproblems are sequentially solved.
Revised: Examples of the form used in the least-to-most prompt are shown in Table 9, where the one on the left shows how to reduce a problem to subproblems, while the one on the right shows how the subproblems are sequentially solved.

Original: The chain-of-thought prompt is generated by merging the steps in the least-to-most prompt, and the prompt for standard prompting is generated by removing reasoning chains in the chain-of-thought prompt.
Revised: The chain-of-thought prompt is generated by merging the steps in the least-to-most prompt, and the prompt for standard prompting is generated by removing reasoning chains from the chain-of-thought prompt.

Original: All the prompting methods are 3-shot.
Revised: All the prompting methods are 3-shot, except for the least-to-most problem reduction prompt, which is 5-shot.

Original: Postprocessing are applied to model outputs: (1) Normalize numerical answers (e.g., change $1, to 1234); (2) Re-calculate all equations with Python (e.g., change “123 - 22 = 100.
Revised: The following postprocessing is applied to model outputs: (1) Normalize numerical answers (e.g., change $1,234 to 1234); (2) Re-calculate all equations with Python (e.g., change “123 - 22 = 100.

Original: So the answer is 100” to “123 - 22 = 101.
Revised: So the answer is 100” to “123 - 22 = 101.

Original: So the answer is 101”); and (3) Round the predicted result to the same precision of the ground truth (e.g., if the ground truth is 16.7 and our prediction is 16.666, we consider the prediction correct).
Revised: So the answer is 101”); and (3) Round the predicted result to the same precision as the ground truth (e.g., if the ground truth is 16.7 and our prediction is 16.666, we consider the prediction correct).

Original: The accuracies of different methods with the GPT-3 code-davincimodel are listed in Table 10.
Revised: The accuracies of different methods with the GPTcode-davinci-002 model are listed in Table 10.

Original: We randomly pick 20 failures from least-to-most prompting and ﬁnd that: (1) 4 are due to wrong problem reduction.
Revised: We randomly pick 20 failures from least-tomost prompting and ﬁnd that: (1) 4 are due to wrong problem reduction.

Original: Either the reductions are not helpful, or there is no reduction; (2) 13 are due to wrong answers to subproblems; (3) 3 are due to wrong “ground truth” (the predictions are correct).
Revised: Either the reductions are not helpful, or there is no reduction; (2) 13 are due to wrong answers to subproblems; (3) 3 are due to wrong “ground truth” (the predictions are correct).

Original: Example outputs and more results can be found in Appendix C.
Revised: Example outputs and more results can be found in Appendix C.

Original: We also conduct experiments on GSM8K (Cobbe et al., 2021).
Revised: We also conduct experiments on GSM8K (Cobbe et al., 2021).

Original: The results are on the last column of Table 10.
Revised: The results are in the last column of Table 10.

Original: The prompts and sampled results can be found in Appendix D.
Revised: The prompts and sampled results can be found in Appendix D.

Original: We expect that leastto-most prompting could outperform other prompting methods with a larger margin if these math reasoning tasks could take a split such that the test questions need more steps to solve than the demonstration examples.
Revised: We expect that least- to-most prompting could outperform other prompting methods with a larger margin if these math reasoning tasks could take a split such that the test questions need more steps to solve than the demonstration examples.

Original: In addition, we have considered a simpliﬁcation for least-to-most prompting by unifying its two stages, that is, problem reducing and solving, to a single pass (see the prompt in Table 11).
Revised: In addition, we have considered a simpliﬁcation for least-to-most prompting by unifying its two stages, that is, problem reducing and solving, to a single pass (see the prompt in Table 11).

Original: This reduces the inference cost but we do not observe accuracy improvements.
Revised: This reduces the inference cost, but we do not observe accuracy improvements.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: Q

Original: [Table]
Revised: [Table]

Original: [Table] [Table] [Table]
Revised: has

Original: [Table]
Revised: [Table]

Revised: Anna has 2 more apples than Elsa.

Revised: How many apples do they have together?

Original: A:Let’sbreak down this problem: 1.
Revised: A: Let’s break down this problem: 1.

Original: How many apples does Anna have? 2.
Revised: How many apples does Anna have? 2.

Original: How many apples do Elsa and Anna have together?
Revised: How many apples do Elsa and Anna have together?

Original: Anna has 2 more apples than Elsa.
Revised: Anna has 2 more apples than Elsa.

Original: So Anna has 2 + 5 = 7 apples.
Revised: So Anna has 2 + 5 = 7 apples.

Original: Elsa and Anna have 5 + 7 = 12 apples together.
Revised: Elsa and Anna have 5 + 7 = 12 apples together.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Compositional generalization .
Revised: Compositional generalization .

Original: SCAN (Lake & Baroni, 2018) is a widely used benchmark to evaluate compositional generalization.
Revised: SCAN (Lake & Baroni, 2018) is a widely used benchmark to evaluate compositional generalization.

Original: Among all of its splits,  length split, which requires a model to generalize to test sequences longer than training ones, is the most challenging.
Revised: Among all of its splits, the most challenging is the length split, which requires a model to generalize to test sequences longer than training ones.

Original: Prior work with good performance on SCAN mostly proposed neural-symbolic architectures (Chen et al., 2020;
Revised: Prior work with good performance on SCAN mostly proposed neural-symbolic architectures (Chen et al., 2020;

Original: Liu et al., 2020) and grammar induction techniques (Nye et al., 2020; Shaw et al., 2021; Kim, 2021).
Revised: Liu et al., 2020) and grammar induction techniques (Nye et al., 2020; Shaw et al., 2021; Kim, 2021).

Original: Chen et al.
Revised: Chen et al.

Original: (2020) proposed the neural-symbolic stack machine, which contains a neural network as the controller to generate an execution trace for a given input, and a symbolic stack machine to execute the trace and produce the output.
Revised: (2020) proposed the neural-symbolic stack machine, which contains a neural network as the controller to generate an execution trace for a given input, and a symbolic stack machine to execute the trace and produce the output.

Original: The execution trace consists of domain-speciﬁc primitives for sequence manipulation, which allows the machine to break down the input sentence into different components, translate them separately, and compose them together.
Revised: The execution trace consists of domain-speciﬁc primitives for sequence manipulation, which allows the machine to break down the input sentence into different components, translate them separately, and compose them together.

Original: Liu et al.
Revised: Liu et al.

Original: (2020) proposed a framework that cooperatively learns two neural modules, a composer and a solver, to jointly learn the input structure and the symbolic grammar rules.
Revised: (2020) proposed a framework that cooperatively learns two neural modules, a composer and a solver, to jointly learn the input structure and the symbolic grammar rules.

Original: Both Nye et al.
Revised: Both Nye et al.

Original: (2020) and Shaw et al.
Revised: (2020) and Shaw et al.

Original: (2021) inferred the symbolic grammar rules of SCAN, while Kim (2021) proposed to learn a latent neural grammar.
Revised: (2021) inferred the symbolic grammar rules of SCAN, while Kim (2021) proposed to learn a latent neural grammar.

Original: While approaches with symbolic components are able to achieve 100% accuracy on
Revised: While approaches with symbolic components are able to achieve 100% accuracy on

Original: SCAN (Chen et al., 2020; Liu et al., 2020; Nye et al., 2020; Shaw et al., 2021), they require complicated model training and grammar inference algorithms to search in a large grammar space.
Revised: SCAN (Chen et al., 2020; Liu et al., 2020; Nye et al., 2020; Shaw et al., 2021), they require complicated model training and grammar inference algorithms to search in a large grammar space.

Original: Another line of work on SCAN designs data augmentation schemes (Andreas, 2020; Aky¨urek et al., 2021;
Revised: Another line of work on SCAN designs data augmentation schemes (Andreas, 2020; Aky¨urek et al., 2021;

Original: Lake, 2019).
Revised: Lake, 2019).

Original: Both Andreas (2020) and Aky¨urek et al.
Revised: Both Andreas (2020) and Aky¨urek et al.

Original: (2021) construct synthetic training samples by recombining fragments occurred in different training samples, and Aky¨urek et al.
Revised: (2021) construct synthetic training samples by recombining fragments occurring in different training samples, and Aky¨urek et al.

Original: (2021) further designs a sampling scheme that encourages the recombination model to produce rare samples.
Revised: (2021) further designs a sampling scheme that encourages the recombination model to produce rare samples.

Original: On the other hand, Lake (2019) proposed a meta training algorithm, which requires a meta-grammar space to construct training data, and the format of sampled grammars is similar to the SCAN grammar.
Revised: On the other hand, Lake (2019) proposed a meta training algorithm, which requires a meta-grammar space to construct training data, and the format of sampled grammars is similar to the SCAN grammar.

Original: While these data augmentation techniques improve the performance on several compositional generalization benchmarks, they fail to solve the length split of SCAN.
Revised: While these data augmentation techniques improve the performance on several compositional generalization benchmarks, they fail to solve the length split of SCAN.

Original: Other prior works propose neural network architectures to improve compositional generalization, where they encourage the model to learn the word and span mapping (Russin et al., 2019;
Revised: Other prior works propose neural network architectures to improve compositional generalization, where they encourage the model to learn the word and span mapping (Russin et al., 2019;

Original: Li et al., 2019), the alignment of input and output as span trees (Herzig & Berant, 2021), and the permutation equivariance of input and output words (Gordon et al., 2020).
Revised: Li et al., 2019), the alignment of input and output as span trees (Herzig & Berant, 2021), and the permutation equivariance of input and output words (Gordon et al., 2020).

Original: Still, these end-to-end neural networks without symbolic components do not generalize to longer test inputs.
Revised: Still, these end-to-end neural networks without symbolic components do not generalize to longer test inputs.

Original: Unlike these existing work, we demonstrate that without model architectures and symbolic components specially designed to improve compositional generalization, least-to-most prompting achieves 99 .
Revised: Unlike the existing work, we demonstrate that without model architectures and symbolic components specially designed to improve compositional generalization, least-to-most prompting achieves 99 .

Original: 7% accuracy on any split (including length split) with only a handful of demonstration examples, and it does not require any training or ﬁnetuning.
Revised: 7% accuracy on any split (including length split) with only a handful of demonstration examples, and it does not require any training or ﬁnetuning.

Original: Easy-to-hard generalization .
Revised: Easy-to-hard generalization .

Original: In addition to compositional generalization, there are many other tasks where the test cases requires more reasoning steps to solve than thethe training examples, for example, the last-letter-concatenation task where the test lists are longer than the demonstration examples.
Revised: In addition to compositional generalization, there are many other tasks where the test cases require more reasoning steps to solve than the training examples, for example, the last-letter-concatenation task where the test lists are longer than the demonstration examples.

Original: Dong et al.
Revised: Dong et al.

Original: (2019) propose Neural Logic Machines (NLMs) for both inductive learning and logic reasoning.
Revised: (2019) propose Neural Logic Machines (NLMs) for both inductive learning and logic reasoning.

Original: NLMs trained on small-scale tasks (such as small size blocks worlds) can perfectly generalize to large-scale tasks (such as larger size block worlds).
Revised: NLMs trained on small-scale tasks (such as small size block worlds) can perfectly generalize to large-scale tasks (such as larger size block worlds).

Original: Schwarzschild et al.
Revised: Schwarzschild et al.

Original: (2021) show that recurrent networks trained to solve simple problems with few recurrent steps (such as small size mazes or chess puzzles) can solve more complex problems (such as larger size mazes or chess puzzles) by performing additional recurrences during inference.
Revised: (2021) show that recurrent networks trained to solve simple problems with few recurrent steps (such as small size mazes or chess puzzles) can solve more complex problems (such as larger size mazes or chess puzzles) by performing additional recurrences during inference.

Original: In our method, we achieve easy-to-hard generalization by reducing a complex problem into a series of easier problems.
Revised: In our method, we achieve easy-to-hard generalization by reducing a complex problem into a series of easier problems.

Original: Task decomposition .
Revised: Task decomposition .

Original: Perez et al.
Revised: Perez et al.

Original: (2020) decompose a multi-hop question to a number of independent single-hop subquestions which are answered by an off-the-shelf question answering (QA) model.
Revised: (2020) decompose a multi-hop question into a number of independent single-hop subquestions, which are answered by an off-the-shelf question answering (QA) model.

Original: Then those answers are aggregated to form the ﬁnal answer.
Revised: Then those answers are aggregated to form the ﬁnal answer.

Original: Both question decomposition and answer aggregation are implemented by trained models.
Revised: Both question decomposition and answer aggregation are implemented by trained models.

Original: Wang et al.
Revised: Wang et al.

Original: (2022a) conducts multi-hop
Revised: (2022a) conducts multi-hop

Original: QA by modeling prompts as continuous virtual tokens and progressively elicit relevant knowledge from language models via iterative prompting.
Revised: QA by modeling prompts as continuous virtual tokens and progressively eliciting relevant knowledge from language models via iterative prompting.

Original: Unlike these methods, our approach does not involve any training or ﬁnetuning.
Revised: Unlike these methods, our approach does not involve any training or ﬁnetuning.

Original: Moreover, the subquestions generated in least-to-most prompting are usually dependent and have to be sequentially solved in a speciﬁc order so that answers to some subquestions can be usedto as building blocks to solve other subquestions.
Revised: Moreover, the subquestions generated in least-to-most prompting are usually dependent and have to be sequentially solved in a speciﬁc order so that answers to some subquestions can be used as building blocks to solve other subquestions.

Original: Yang et al.
Revised: Yang et al.

Original: (2022) translate natural language questions to SQL queries by decomposing a question into a sequence of slot-ﬁlling natural language prompts corresponding to SQL clauses via a rule-based system.
Revised: (2022) translate natural language questions to SQL queries by decomposing a question into a sequence of slot-ﬁlling natural language prompts corresponding to SQL clauses via a rule-based system.

Original: Wu et al.
Revised: Wu et al.

Original: (2022) propose chaining large language model steps such that the output of one step becomes the input for the next and develop an interactive system for users to construct and modify chains.
Revised: (2022) propose chaining large language model steps such that the output of one step becomes the input for the next and develop an interactive system for users to construct and modify chains.

Original: Least-to-most prompting chains the processes of problem reduction and subproblem solving.
Revised: Least-to-most prompting chains the processes of problem reduction and subproblem solving.

Original:  C ONCLUSION AND D ISCUSSION
Revised:  C ONCLUSION AND D ISCUSSION

Original: We proposed least-to-most prompting to enable language models to solve problems that are harder than those seen in the prompt context.
Revised: We proposed least-to-most prompting to enable language models to solve problems that are harder than those seen in the prompt context.

Original: It involves a top-down problem reduction process and a bottom-up problem solving process.
Revised: It involves a top-down problem reduction process and a bottom-up problem solving process.

Original: Empirical results on symbolic manipulation, compositional generalization, and math reasoning show that least-to-most prompting outperforms standard prompting and chain-of-thought prompting by a large margin.
Revised: Empirical results on symbolic manipulation, compositional generalization, and math reasoning show that least-to-most prompting outperforms standard prompting and chain-of-thought prompting by a large margin.

Original: However, not all problems can be solved by least-to-most prompting.
Revised: However, not all problems can be solved by least-to-most prompting.

Original: Some problems may not be reducible or at least not  easy to reduce.
Revised: Some problems may not be reducible or at least not be easy to reduce.

Original: For example, a math word problem which can be solved using variables and equations may not be that obvious to reduce to subproblems solvable within one arithmetic operation step.
Revised: For example, a math word problem which can be solved using variables and equations may not be that obvious to reduce to subproblems solvable within one arithmetic operation step.

Original: In this case, a possible workaround would be to ﬁrst ﬁnetune the language model to give it knowledge of variables and equations.
Revised: In this case, a possible workaround would be to ﬁrst ﬁnetune the language model to give it knowledge of variables and equations.

Original: How to effectively combine prompting and ﬁnetuning should be a fascinating direction to explore.
Revised: How to effectively combine prompting and ﬁnetuning should be a fascinating direction to explore.

Original: Another limitation of least-to-most prompting stems from its natural language based solution construction.
Revised: Another limitation of least-to-most prompting stems from its natural language based solution construction.

Original: Natural language is expressive and generic but tends to be less precise than computer languages.
Revised: Natural language is expressive and generic but tends to be less precise than computer languages.

Original: For example, a learned computer program may be able to solve symbolic manipulation tasks at any length with an accuracy of 100%.
Revised: For example, a learned computer program may be able to solve symbolic manipulation tasks at any length with an accuracy of 100%.

Original: So we may combine natural language and computer languages in our prompts to make them more powerful.
Revised: So we may combine natural language and computer languages in our prompts to make them more powerful.

Original: This is just like writing a mathematical proof where both natural language texts and abstract symbol based equations are needed.
Revised: This is just like writing a mathematical proof in which both natural language text and abstract symbol based equations are needed.

Original: In the prompt constructed to solve SCAN, we have actually used both natural language and Python notation.
Revised: In the prompt constructed to solve SCAN, we have actually used both natural language and Python notation.

Original: Finally, prompting may not be the best way to enable large language models to reason.
Revised: Finally, prompting may not be the best way to enable large language models to reason.

Original: We can think of prompting as a  one-way communication:  teach a language model new skills without taking into account its response.
Revised: We can think of prompting as a form of one-way communication: we teach a language model new skills without taking into account its response.

Original: It should be natural to extend prompting to full bidirectional conversations, which will be able to allow us to provide immediate feedback for language models such that they could learn better or faster.
Revised: It should be natural to extend prompting to full bidirectional conversations, which would allow us to provide immediate feedback for language models such that they could learn better or faster.

Original: Least-to-most prompting can be considered as a step toward teaching language models through bidirectional conversations.
Revised: Least-to-most prompting can be considered as a step toward teaching language models through bidirectional conversations.

Revised: Appendix Table of Contents

Original: AL AST - LETTER - CONCATENATION A. P ROMPT CONTEXTS WITH 4 EXAMPLES
Revised: A A. context A. Prompt contexts different examples . . . . . . . . . . . . . . . .

Revised: A.2.1 Standard prompting, 4-shot . . . . . . . . . . . . . . . . . . . . . . . .

Revised: A.2.2 Chain-of-thought prompting, 4-shot . . . . . . . . . . . . . . . . . . . .

Revised: A.2.3 Chain-of prompting, shot . . . . . . . . . . . . . . . . . . . .

Revised: A.

Revised: A.2.

Revised: A.

Revised: A.

Revised: A. outputs

Revised: A. prompting

Revised: A.5.

Revised: A.

Revised: A.

Revised: [Table]

Revised: -to-most prompting: Failure

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: contexts

Revised: [Table]

Revised: [Table]

Revised: prompting

Revised: [Table]

Revised: [Table]

Revised: Least-to-most prompting

Revised: [Table]

Revised: [Table]

Revised: of-

Revised: [Table]

Revised: prompting

Revised: [Table]

Revised: Least-to-most prompting

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: -of-thought prompting: Success

Revised: [Table]

Revised: -of-

Revised: [Table]

Revised: [Table]

Revised: 3.3 Least-to-most prompting: Success

Revised: [Table]

Revised: [Table]

Revised: Least-to-most prompting

Revised: Failure

Revised: [Table]

Revised: expressions using prompting

Revised: C DROP

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: .2 Non-

Revised: [Table]

Revised: Subset .

Revised: [Table]

Revised: C.2.1 Zero-shot prompting

Revised: C.2.2 Standard prompting with 3 examples

Revised: C.2.3 Chain-of-thought prompting with 3 examples

Revised: C.2.4 Least-to-most prompting I: problem reducing (5 examples

Revised: C.2.5 Least-to-most prompting II: problem solving (

Revised: examples) . .

Revised: C.3 Football subset

Revised: C

Revised: 3.1 Zero-shot prompting

Revised: C.3.2 Standard prompting with 3 examples

Revised: C

Revised: 3.3 Chain-of-thought prompting with

Revised: examples . .

Revised: C.3.4 Least-to-most prompting I: problem reducing (6 examples

Revised: C.3.5 Least-to-most prompting II: problem solving (3 examples)

Revised: C

Revised: Examples

Revised: where least-to-most succeeded but chain-of-thought failed

Revised: C

Revised: 1 Case

Revised: C

Revised: 4.2 Case

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: The last-letter-concatenation experiments presented in Section 3.1 are based on prompt contexts that consists of 2 demonstration examples.
Revised: [Table] [Table] A L AST - LETTER - CONCATENATION A. P ROMPT CONTEXT FOR REDUCING WORD LIST INTO SUBPROBLEMS In Section 3.1 we mentioned that language model prompting can be used to reduce a word list such as “think, machine, learning, reasoning” into a sequence of subproblems “think, machine”, “think, machine, learning”, and “think, machine, learning, reasoning”.

Original: We also ran experiments with prompt contexts containing twice the number of demonstration examples, which are shown below.
Revised: The following prompt context achieves 100% accuracy on this task when using the text-davinci-002 model. Note that it achieves perfect accuracy on lists up to size 12 (which is the maximum that we tested for our experiment) even though it only contains one exemplar each for lists of sizes 2 and 3. Q: “machine, learning” A: creating sequential sublists of the list “machine, learning”: “machine”“machine, learning” Q: “machine, learning, artiﬁcial” A: creating sequential sublists of the list “machine, learning, artiﬁcial”: “machine”“machine, learning” “machine, learning, artiﬁcial” A.

Revised: P ROMPT CONTEXTS WITH MORE AND DIFFERENT EXAMPLES

Original: The complete set of results is presented in Section A.2.
Revised: The last-letter-concatenation experiments presented in Section 3.1 are based on prompt contexts that consists of 2 demonstration examples. To make sure that the accuracy gain achieved by leastto-most prompting is not caused by the slight increase in example length when compared to chainof-thought, we also performed experiments with more context examples so that we can compare least-to-most vs. chain-of-thought for different prompt sizes. Also, we perform experiments where we use for chain-of-thought prompting the same prompt examples that we use for least-to-most prompting (unlike the situation in Table 1 where we use different examples). All these prompts are shown below, and we present and discuss the corresponding accuracies in Section A.3.

Original: A.1. S TANDARD PROMPTING 
Revised: A.2. S TANDARD PROMPTING , 4- SHOT

Original: Q: “think, machine”
Revised: Q: “think, machine”

Original: A: “ke” Q: “learning, reasoning, generalization” A: “ggn” Q: “artiﬁcial, intelligence” A: “le” Q: “transformer, language, vision” A: “ren”
Revised: A: “ke” Q: “learning, reasoning, generalization” A: “ggn” Q: “artiﬁcial, intelligence” A: “le” [Table]

Original: A.1. C HAIN - OF - THOUGHT PROMPTING 
Revised: A.2. C HAIN - OF - THOUGHT PROMPTING , 4- SHOT

Revised: [Table]

Revised: last letter of

Revised: [Table]

Revised: is

Revised: [Table]

Revised: The last letter of

Revised: [Table]

Revised: is

Revised: [Table]

Revised: Concatenating

Revised: [Table]

Revised: leads to

Original: Q: “think, machine”
Revised: [Table] So [Table] “think

Revised: [Table]

Revised: machine” outputs

Revised: [Table]

Original: A: The last letter of “think” is“k”.
Revised: [Table] The last letter of [Table] is

Revised: [Table]

Revised: The last letter of “generalization” is

Revised: [Table]

Revised: [Table]

Revised: , “g”, “n” leads to “ggn”.

Revised: So

Revised: [Table]

Revised: “learning

Revised: [Table]

Revised: reasoning

Revised: [Table]

Revised: generalization”

Revised: [Table]

Revised: Q: “artiﬁcial, intelligence”

Revised: A: The last letter of “artiﬁcial” is “l”.

Revised: The last letter of “intelligence” is “e”.

Revised: Concatenating “l”, “e” leads to “le”.

Revised: So, “artiﬁcial, intelligence” outputs “le”.

Revised: Q: “transformer, language, vision”

Revised: A: The last letter of “transformer” is “r”.

Revised: The last letter of “language” is “e”.

Revised: The last letter of “vision” is “n”.

Revised: Concatenating “r”, “e”, “n” leads to “ren”.

Revised: So, “transformer, language, vision” outputs “ren”.

Revised: A.2. C HAIN - OF - THOUGHT PROMPTING , 8- SHOT

Revised: [Table]

Revised: A: The last letter of “think” is “k”.

Original: The last letter of “machine” is “e”.
Revised: The last letter of “machine” is “e”.

Original: Concatenating “k”, “e” leads to “ke”.
Revised: Concatenating “k”, “e” leads to “ke”.

Original: So, “think, machine” outputs “ke”. 
Revised: So, “think, machine” outputs “ke”. Q [Table]

Original: Q: “learning, reasoning, generalization” A: The last letter of “learning” is “g”.
Revised: “learning, [Table] , generalization” A: The last letter of “learning” is “g”.

Original: The last letter of “reasoning” is “g”.
Revised: The last letter of “reasoning” is “g”.

Original: The last letter of “generalization” is “n”.
Revised: The last letter of “generalization” is “n”.

Original: Concatenating “g”, “g”, “n” leads to “ggn”.
Revised: Concatenating “g”, “g”, “n” leads to “ggn”.

Original: So, “learning, reasoning, generalization” outputs “ggn”. 
Revised: So, “learning, reasoning, generalization” outputs “ggn”. [Table]

Original: Q: “artiﬁcial, intelligence” A: The last letter of “artiﬁcial” is “l”.
Revised: “artiﬁcial, intelligence” A: The last letter of “artiﬁcial” is “l”.

Original: The last letter of “intelligence” is “e”.
Revised: The last letter of “intelligence” is “e”.

Original: Concatenating “l”, “e” leads to “le”.
Revised: Concatenating “l”, “e” leads to “le”.

Original: So, “artiﬁcial, intelligence” outputs “le”.
Revised: So, “artiﬁcial, intelligence” outputs “le”.

Original: Q: “transformer, language, vision” A: The last letter of “transformer” is “r”.
Revised: Q [Table] “transformer, language, vision” A: The last letter of “transformer” is “r”.

Original: The last letter of “language” is “e”.
Revised: The last letter of “language” is “e”.

Original: The last letter of “vision” is “n”.
Revised: The last letter of “vision” is “n”.

Original: Concatenating “r”, “e”, “n” leads to “ren”.
Revised: Concatenating “r”, “e”, “n” leads to “ren”.

Original: So, “transformer, language, vision”
Revised: So, “transformer, language

Original: [Table]
Revised: [Table]

Revised: So, “school, teacher” outputs “lr”.

Revised: [Table]

Revised: So, “pen,

Revised: [Table]

Revised: “np”.

Original: [Table]
Revised: [Table]

Revised: A.2. C HAIN - OF - THOUGHT PROMPTING , 2- SHOT , SAME EXAMPLES AS FOR LEAST - TO - MOST

Original: [Table]
Revised: [Table]

Original: A: The last letter of “think” is “k”.
Revised: letter of [Table] to [Table]

Revised: So

Revised: [Table]

Original: The last letter of“machine” is “e”.
Revised: machine” outputs [Table] [Table] last letter of

Revised: [Table]

Revised: is

Revised: [Table]

Original:  Concatenating “k”, “e” leads to “ke”.
Revised: The last letter of “learning” is “g”. Concatenating [Table] leads to “keg”.

Original: So, “think, machine” outputs “ke”.
Revised: So, “think, machine [Table] learning” outputs “keg”.

Revised: A.2. L EAST - TO - MOST

Revised: [Table]

Original:  Q: “think, machine, learning” A: “think, machine” outputs “ke”.
Revised: “ke”. So, “think, machine” outputs “ke”. Q: “think, machine, learning” A: “think, machine” outputs “ke”.

Original: The last letter of “learning” is “g”.
Revised: The last letter of “learning” is “g”.

Original: Concatenating “ke”, “g” leads to “keg”.
Revised: Concatenating “ke”, “g” leads to “keg”.

Original: So, “think, machine, learning” outputs “keg”.
Revised: So, “think, machine, learning” outputs “keg”.

Original: Q: “transformer, language” A: The last letter of “transformer” is “r”.
Revised: Q: “transformer, language” A: The last letter of “transformer” is “r”.

Original: The last letter of “language” is “e”.
Revised: The last letter of “language” is “e”.

Original: Concatenating: “r”, “e” leads to “re”.
Revised: Concatenating: “r”, “e” leads to “re”.

Original: So, “transformer, language” outputs “re”.
Revised: So, “transformer, language” outputs “re”.

Original: Q: “transformer, language, vision”
Revised: Q: “transformer, language, vision”

Original: A: “transformer, language” outputs “re”.
Revised: A: “transformer, language” outputs “re”.

Original: The last letter of “vision” is “n”.
Revised: The last letter of “vision” is “n”.

Original: Concatenating: “re”, “n” leads to “ren”.
Revised: Concatenating: “re”, “n” leads to “ren”.

Original: So, “transformer, language, vision” outputs “ren”.
Revised: So, “transformer, language, vision” outputs “ren”.

Original: A. D ATA G ENERATION AND ADDITIONAL RESULTS
Revised: A. D ATA G ENERATION AND ADDITIONAL RESULTS

Original: Data generation.
Revised: Data generation.

Original: The last-letter-concatenation dataset is based on a list of the 10k most common
Revised: The last-letter-concatenation dataset is based on a list of the 10k most common

Original: English words (including proper nouns) used in books that are part of project Gutenberg that is collected in Wiktionary 2 .
Revised: English words (including proper nouns) used in books that are part of project Gutenberg, as collected in Wiktionary 2 .

Original: After eliminating profane words, we ended up with a list of 9694 words (all lowercase).
Revised: After eliminating profane words, we ended up with a list of 9694 words (all lowercase).

Original: For each of the desired list sizes 2, 4, 6, 8, 10, 12, we then generated 500 examples, each of which consists of a random sequence of these words (input) and the corresponding sequence of last letters (output).
Revised: For each of the desired list sizes 2, 4, 6, 8, 10, 12, we then generated 500 examples, each of which consists of a random sequence of these words (input) and the corresponding sequence of last letters (output).

Original: We will release the full dataset upon publication of this paper.
Revised: We will release the full dataset upon publication of this paper.

Original: Below arerandom examples of list size 6:
Revised: Below arerandom examples of list size 6:

Original: • IN: “narrative, celebrate, neighbouring, indebted, stove, calling” OUT: “eegdeg” • IN: “barley, silk, thankful, kiss, logs, silent” OUT: “yklsst” • IN: “knitting, conveyance, receives, represent, cow, shut” OUT: “gestwt” • IN: “olive, dark, limitation, airy, pocket, wondered” OUT: “eknytd” • IN: “apprehensive, exclamation, perspiration, trusting, destiny, tactics”
Revised: • IN: “narrative, celebrate, neighbouring, indebted, stove, calling” OUT: “eegdeg” • IN: “barley, silk, thankful, kiss, logs, silent” OUT: “yklsst” • IN: “knitting, conveyance, receives, represent, cow, shut” OUT: “gestwt” • IN: “olive, dark, limitation, airy, pocket, wondered” OUT: “eknytd” • IN: “apprehensive, exclamation, perspiration, trusting, destiny, tactics”

Original: OUT: “enngys” • IN: “qualiﬁed, envoy, disciple, exert, witnesses, plane” OUT: “dyetse” • IN: “decidedly, dome, france, chris, knowing, peaceful” OUT: “yeesgl” • IN: “deceit, reﬁnement, tips, cord, princes, discovery” OUT: “ttsdsy” • IN: “drops, paste, defective, bohemia, requested, convenient” OUT: “seeadt” • IN: “diverse, christopher, homely, agreeable, fright, suspended” OUT: “eryetd”
Revised: OUT: “enngys” • IN: “qualiﬁed, envoy, disciple, exert, witnesses, plane” OUT: “dyetse” • IN: “decidedly, dome, france, chris, knowing, peaceful” OUT: “yeesgl” • IN: “deceit, reﬁnement, tips, cord, princes, discovery” OUT: “ttsdsy” • IN: “drops, paste, defective, bohemia, requested, convenient” OUT: “seeadt” • IN: “diverse, christopher, homely, agreeable, fright, suspended” OUT: “eryetd”

Original: Complete results.
Revised: Complete results.

Original: Table 12 summarizes all the experiments we performed for the last-letterconcatenation task.
Revised: Table 12 summarizes all the experiments we performed for the last-letterconcatenation task.

Original: In addition to the experiments where prompt contexts contain 2 demonstration examples presented in Section 3.1, this includes experiments where the prompts contain 4 demonstration examples (see above).
Revised: In addition to the experiments where prompt contexts contain 2 demonstration examples presented in Section 3.1, this includes experiments where the prompts contain 4 anddemonstration examples (see above).

Original: These extended prompts increase the accuracyby about 2 percentage points across the board when compared to the basic prompts. Note that even with the extended prompts, the maximum input length is around 1,100 tokens, which is signiﬁcantly below the maximum input length for all models that we tested.
Revised: While more prompt examples have no effect for standard prompting (the accuracy remains at 0), they increase the accuracy across the board for chain-of-thought and least-to-most prompting. However, least-to-most prompting consistently outperforms chain-of-thought prompting. In fact, even if we compare 2-shot least-to-most (prompt size 123 GPT3 tokens) to 8-shot chain-of-thought (prompt size 573 GPT3 tokens), the accuracy for least-to-most prompting is much higher than for chainof-thought prompting.

Original: For the extended prompts, the table also contains theresults from running against two additional GPT-3 models:
Revised: The difference is especially pronounced for long sequences (e.g., for L = 12, we have least-to-most at 74.0% vs. chain-of-thought at 38.4%). This shows that least-to-most prompting is much more data-efﬁcient than chain-of-thought prompting for this problem. Comparing the ﬁrst two rows for chain-of-thought prompting shows that chain-of-thought achieves higher accuracy if we use two independent examples (see prompt in Table 1) instead of the two dependent examples that we use for least-to-most prompting. This demonstrates that the accuracy advantage of least-to-most prompting over chain-

Original: [Equation]
Revised: [Table]

Revised: examples (rather than the full set of 500 exammples).

Revised: The table also contains the results from running against two additional GPT-3 models: text-davinci-002 and codex-davinci-001 .

Original: While text-davinci-002 shows similar accuracy as code-davinci-002 on small list sizes, the accuracy drops off much faster when moving to larger list sizes, both for chain-of-thought prompting as well as for least-to-most prompting.
Revised: While text-davinci-002 shows similar accuracy to code-davinci-002 on small list sizes, the accuracy drops off much faster when moving to larger list sizes, both for chain-of-thought prompting as well as for least-to-most prompting.

Original: This indicates that the code-davinci-002 model has an advantage when it comes to dealing with iteration and recursion.
Revised: This indicates that the code-davinci-002 model has an advantage when it comes to dealing with iteration and recursion.

Original:  code-davinci-001  performs much worse than code-davinci-002 across all dimensions.
Revised: The code-davinci-001 model performs much worse than code-davinci-002 across all dimensions.

Original: Even for the shortest list size (L = 4), the accuracy for least-to-most prompting is only 19.6% compared to 96% for code-davinci-002 .
Revised: Even for the shortest list size (L = 4), the accuracy for least-to-most prompting is only 19.6% compared to 96% for code-davinci-002 .

Original: This indicates that there is a large potential for improvement when using the exact same conﬁguration with new model generations.
Revised: This indicates that there is a large potential for improvement when using the exact same conﬁguration with new model generations.

Original: A. E RROR ANALYSIS : L EAST - TO - MOST PROMPTING
Revised: A. E RROR ANALYSIS : L EAST - TO - MOST PROMPTING

Original: For least-to-most prompting, we analyzed 20 random failures of the code-davinci-002 model on list lengths 4 and 12 for prompt contexts consisting of 2 and 4 examples.
Revised: Error type 2 examples 4 examples L = L = L =

Revised: L =

Revised: Concatenation error

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Copy error

Revised: For least-to-most prompting, we analyzed 20 random failures of the code-davinci-002 model on list lengths 4 and 12 for prompt contexts consisting of 2 and 4 examples.

Original: The results are shown in
Revised: The results are shown in

Original: Table 13.
Revised: Table 13.

Original: Concatenation errors may either be due to dropping a letter, adding a letter or outputting the letters in the wrong order.
Revised: Concatenation errors may either be due to dropping a letter, adding a letter or outputting the letters in the wrong order.

Original: Wrong template means that the model used the extension template instead of the base template to concatenate the last letter of the ﬁrst two words of the list.
Revised: Wrong template means that the model used the extension template instead of the base template to concatenate the last letter of the ﬁrst two words of the list.

Original: Incorrect last letter means that the model got the last letter of a word wrong, and copy error [Table] [Table] Wrong order
Revised: Incorrect last letter means that the model got the last letter of a word wrong, and copy error means that the error was due to making a mistake when copying an intermediate result.

Original: [Table]

Original: Wrong template

Original: [Table] Copy [Table] We observe that for the prompt consisting of 2 examples, the fraction of concatenation errors increases as we go from length 4 to length 12 while the fraction of wrong template errors go down.
Revised: We observe that for the prompt consisting of 2 examples, the fraction of concatenation errors increases as we go from length 4 to length 12 while the fraction of wrong template errors go down.

Original: This makes sense because the number of concatenations grows with  length of the list while the number of times the model needs to use the base template stays constant.
Revised: This makes sense because the number of concatenations grows with the length of the list, while the number of times the model needs to use the base template stays constant.

Original: Note that the template  disappear when we move to the double prompt, which means that adding two more examples helps the  recognize which template to use.
Revised: Note that the template errors disappear when we move to the double prompt, which means that adding two more examples helps the model recognize which template to use.

Original: As a consequence, the double prompt has a  distribution of errors for 
Revised: As a consequence, the double prompt has a similar distribution of errors for both list lengths.

Original: Examples of concatenation errors.
Revised: Examples of concatenation errors.

Original: In the example “gratiﬁed, contract, fortitude, blew”, the model drops the last letter in the concatenation of “dte” and “w”, which means that it predicts the last letter sequence to be “dte” instead of “dtew”.
Revised: In the example “gratiﬁed, contract, fortitude, blew”, the model drops the last letter in the concatenation of “dte” and “w”, which means that it predicts the last letter sequence to be “dte” instead of “dtew”.

Original: In the example “hollow, supplies, function, gorgeous”, the model duplicates the last letter “s” in the concatenation of “wsn” and “s”, which means that it predicts the last letter sequence “wsnss” instead of “wsns”.
Revised: In the example “hollow, supplies, function, gorgeous”, the model duplicates the last letter “s” in the concatenation of “wsn” and “s”, which means that it predicts the last letter sequence “wsnss” instead of “wsns”.

Original: In the example “madly, vengeance, cowardice, monk”, the model drops the last letter “k” in the concatenation of “yee” and ”k” and instead adds the letter “g”.
Revised: In the example “madly, vengeance, cowardice, monk”, the model drops the last letter “k” in the concatenation of “yee” and ”k” and instead adds the letter “g”.

Original: Consequently, the model predicts “yeeg” instead of “yeek”.
Revised: Consequently, the model predicts “yeeg” instead of “yeek”.

Original: In the example “slender, lash, throng, scheme”, the model breaks the order of the letters “h” and “g” in the concatenation of “rh” and “g”, which means that it predicts the last letter sequence “rghe” instead of “rhge”.
Revised: In the example “slender, lash, throng, scheme”, the model breaks the order of the letters “h” and “g” in the concatenation of “rh” and “g”, which means that it predicts the last letter sequence “rghe” instead of “rhge”.

Original: Example of incorrect last letter.
Revised: Example of incorrect last letter.

Original: In the example “modiﬁcation, introducing, schools, lunch”, the model determines the last letter of the word “modiﬁcation” to be “g”.
Revised: In the example “modiﬁcation, introducing, schools, lunch”, the model determines the last letter of the word “modiﬁcation” to be “g”.

Original: Consequently, the predicted last letter sequence is “ggsh” instead of “ngsh”.
Revised: Consequently, the predicted last letter sequence is “ggsh” instead of “ngsh”.

Original: Example of wrong template application.
Revised: Example of wrong template application.

Original: In the example “upper, unexpectedly, speciﬁcally, connection”, the model uses the extension template to determine the output of the ﬁrst two words “upper, unexpectedly”.
Revised: In the example “upper, unexpectedly, speciﬁcally, connection”, the model uses the extension template to determine the output of the ﬁrst two words “upper, unexpectedly”.

Original: I.e., it produces:
Revised: I.e., it produces:

Original: • “upper” outputs “er”.
Revised: • “upper” outputs “er”.

Original: The last letter of “unexpectedly” is “y”.
Revised: The last letter of “unexpectedly” is “y”.

Original: Concatenating “er”, “y” leads to “ery”.
Revised: Concatenating “er”, “y” leads to “ery”.

Original: So, “upper, unexpectedly” outputs “ery”.
Revised: So, “upper, unexpectedly” outputs “ery”.

Original: when it should have produced:
Revised: when it should have produced:

Original: • The last letter of “upper” is “r”.
Revised: • The last letter of “upper” is “r”.

Original: The last letter of “unexpectedly” is “y”.
Revised: The last letter of “unexpectedly” is “y”.

Original: Concatenating “r”, “y” leads to “ry”.
Revised: Concatenating “r”, “y” leads to “ry”.

Original: As a consequence, the model predicts the sequence “eryyn” instead of “ryyn”.
Revised: As a consequence, the model predicts the sequence “eryyn” instead of “ryyn”.

Original: Example of copy error.
Revised: Example of copy error.

Original: In the example “displeased, hawk, healing, anchor”, the model correctly computes the ﬁnal concatenation “dkgr” but then makes a mistake when copying the result to the ﬁnal output and drops the ﬁnal “r”, which leads to the prediction “dkg” instead of “dkgr”:
Revised: In the example “displeased, hawk, healing, anchor”, the model correctly computes the ﬁnal concatenation “dkgr” but then makes a mistake when copying the result to the ﬁnal output and drops the ﬁnal “r”, which leads to the prediction “dkg” instead of “dkgr”:

Original: • “displeased, hawk, healing” outputs “dkg”.
Revised: • “displeased, hawk, healing” outputs “dkg”.

Original: The last letter of “anchor” is “r”.
Revised: The last letter of “anchor” is “r”.

Original: Concatenating: “dkg”, “r” leads to “dkgr”.
Revised: Concatenating: “dkg”, “r” leads to “dkgr”.

Original: So, “displeased, hawk, healing, anchor” outputs “dkg”.
Revised: So, “displeased, hawk, healing, anchor” outputs “dkg”.

Original: A. E XAMPLE OUTPUTS FROM CODE - DAVINCI - 
Revised: A. E XAMPLE OUTPUTS FROM CODE - DAVINCI -

Original: In this section, we present 5 examples (one example for lengths 4, 6, 8, 10, 12) of successes and failures of the code-davinci-002 model for standard prompting, chain-of-thought prompting, and least-to-most prompting.
Revised: In this section, we present 5 examples (one example for lengths 4, 6, 8, 10, 12) of successes and failures of the code-davinci-002 model for standard prompting, chain-of-thought prompting, and least-to-most prompting.

Original: They are all based on the prompt contexts containing 4 demonstration examples.
Revised: They are all based on the prompt contexts containing 4 demonstration examples.

Original: A.4. S TANDARD PROMPTING : F AILURE
Revised: A.5. S TANDARD PROMPTING : F AILURE

Original: “witnesses, arkansas,remaining
Revised: “witnesses, arkansas,

Original: privileged” Golden: “ssgd” Solution: Q: “witnesses, arkansas, remaining,privileged” A: “n” “clan, noun, roy [Table] “conquered, owed, enclosure, indeed, shattered, beads, increased, provided, weak, grind”
Revised: remaining,

Original:  Golden: “ddeddsddkd”
Revised: privileged” Golden: “ssgd”

Original: Solution: Q: “conquered, owed, enclosure, indeed, shattered, beads, increased, provided, weak,

Original: [Table]

Original: A

Original: [Table]

Original: “hope, alison,

Original: [Table]

Original: , beneﬁts, altered,

Original: [Table]
Revised: [Table]

Original: , keen, portuguese, distance,

Original: [Table]

Original: , proof, seamus” Golden: “enssdeneesfs” Solution: Q: “hope, alison, mirrors, beneﬁts, altered,
Revised: A.5. C HAIN - OF - THOUGHT PROMPTING : S UCCESS

Original: [Table]

Original: , keen, portuguese, distance,

Original: [Table]
Revised: [Table]

Original: , proof, seamus”A:

Original: [Table]

Original: A.4.

Original: [Table]

Original: “amply, bestow, maryland, edited, blowing, prosperous” [Table] outputs [Table]
Revised: version”

Original: “needless, courts, reﬂected, curly, glowing, asp, chronicles, anthony”

Original: Golden: “ssdygpsy”

Original: Solution: Q: “needless, courts, reﬂected, curly, glowing, asp, chronicles, anthony”

Original: A
Revised: A

Original: [Table]
Revised: [Table]

Original: “d” [Table] last [Table]
Revised: The last letter of “geese” is “e”.

Original: “y” [Table] last [Table]
Revised: The last letter of “rogers” is “s”.

Original: last [Table] “p”. The last letter of “chronicles” is “s”
Revised: The last letter of “benefactor” is “r”.

Original: [Table]

Original:  last letter [Table] “anthony” is “y”
Revised: The last letter of “version” is “n”.

Original: [Table] Concatenating  “s”, “s”, “d”, “y”, “g”, “p”, “s”, “y” leads to “ssdygpsy”. So, “needless, courts, reﬂected, curly, glowing, asp, chronicles, anthony” outputs “ssdygpsy”.
Revised: Concatenating “e”, “s”, “r”, “n” leads to “esrn”.

Original: “nigel, alaska, rent, hospital, newspaper, medical, nation, dave, associated, rumour”

Original: Golden: “latlrlnedr”

Original: Solution: Q: “nigel, alaska, rent, hospital, newspaper, medical, nation, dave, associated, rumour” A: The last letter of “nigel” is “l”. The last letter of “alaska” is “a”. The last letter of “rent” is “t”.
Revised: So, “geese, rogers,

Original: The last letter of “hospital” is “l”.

Original: The last letter of “newspaper” is “r”.

Original: The last letter of “medical” is “l”. The last letter of “nation” is “n”. The last letter of “dave” is “e”. The last letter of “associated” is “d”.
Revised: [Table]

Original: The last letter of “rumour” is “r”. Concatenating “l”, “a”, “t”, “l”, “r”, “l”, “n”, “e”, “d”, “r” leads to “latlrlnedr”.
Revised: [Table] “needless [Table]

Original: So, “nigel,alaska, rent, hospital, newspaper, medical, nation, dave, associated, rumour” outputs “latlrlnedr”.
Revised: [Table] [Table] [Table] “nigel,

Revised: [Table]

Revised: [Table]

Revised: , hospital, newspaper, medical, nation, dave, associated, rumour”

Revised: [Table]

Revised: “latlrlnedr”

Revised: [Table]

Revised: : Q: “nigel, alaska, rent, hospital, newspaper, medical, nation, dave, associated, rumour”

Revised: [Table]

Revised: The

Revised: [Table]

Revised: letter

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: leads to

Revised: [Table]

Revised: So, “nigel, alaska, rent, hospital, newspaper, medical, nation, dave, associated, rumour” outputs “latlrlnedr”.

Original: “brightness, preparatory, intense, accumulated, reluctantly, nets, inquiries, grip, inquisition, alley, innocent, herbert”
Revised: “brightness, preparatory, intense, accumulated, reluctantly, nets, inquiries, grip, inquisition, alley, innocent, herbert”

Original: Golden: “syedysspnytt”
Revised: Golden: “syedysspnytt”

Original: Solution:
Revised: Solution:

Original: Q: “brightness, preparatory, intense, accumulated, reluctantly, nets, inquiries, grip, inquisition, alley, innocent, herbert”
Revised: Q: “brightness, preparatory, intense, accumulated, reluctantly, nets, inquiries, grip, inquisition, alley, innocent, herbert”

Original: A: The last letter of “brightness” is “s”.
Revised: A: The last letter of “brightness” is “s”.

Original: The last letter of “preparatory” is “y”.
Revised: The last letter of “preparatory” is “y”.

Original: The last letter of “intense” is “e”.
Revised: The last letter of “intense” is “e”.

Original: The last letter of “accumulated” is “d”.
Revised: The last letter of “accumulated” is “d”.

Original: The last letter of “reluctantly” is “y”.
Revised: The last letter of “reluctantly” is “y”.

Original: The last letter of “nets” is “s”. The last letter of “inquiries” is “s”.
Revised: The last letter of “nets” is “s”. The last letter of “inquiries” is “s”.

Original: The last letter of “grip” is “p”.
Revised: The last letter of “grip” is “p”.

Original: The last letter of “inquisition” is “n”.
Revised: The last letter of “inquisition” is “n”.

Original: The last letter of “alley” is “y”.
Revised: The last letter of “alley” is “y”.

Original: The last letter of “innocent” is “t”.
Revised: The last letter of “innocent” is “t”.

Original: The last letter of “herbert” is “t”.
Revised: The last letter of “herbert” is “t”.

Original: Concatenating “s”, “y”, “e”, “d”, “y”, “s”, “s”, “p”, “n”, “y”, “t”, “t” leads to “syedysspnytt”.
Revised: Concatenating “s”, “y”, “e”, “d”, “y”, “s”, “s”, “p”, “n”, “y”, “t”, “t” leads to “syedysspnytt”.

Original: So, “brightness, preparatory, intense, accumulated, reluctantly, nets, inquiries, grip, inquisition, alley, innocent, herbert” outputs “syedysspnytt”.
Revised: So, “brightness, preparatory, intense, accumulated, reluctantly, nets, inquiries, grip, inquisition, alley, innocent, herbert” outputs “syedysspnytt”.

Original: A.4. C HAIN - OF - THOUGHT PROMPTING : F AILURE
Revised: A.5. C HAIN - OF - THOUGHT PROMPTING : F AILURE

Original: “pattern, vote, afterwards, solemn” Golden: “nesn” Solution: Q: “pattern, vote, afterwards, solemn” A: The last letter of “pattern” is “n”.
Revised: [Table]

Original: The last letter of “vote” is “e”.

Original: The last letter of “afterwards” is “s”. The last letter of “solemn” is “n”. Concatenating “n”, “e”, “s”, “n” leads to “ness”. So, “pattern, vote, afterwards, solemn” outputs “ness”.
Revised: [Table]

Original: “disappointment, troublesome, respectfully, monstrous, worry, politics”
Revised: “freezing, arduous, pushing, vitality

Original: Golden: “teysys”
Revised: [Table]

Original: Solution: Q: “disappointment, troublesome, respectfully, monstrous, worry, politics” A: The last letter of “disappointment” is “t”. The last letter of “troublesome” is “e”. The last letter of
Revised: hue

Original: [Table]
Revised: [Table]

Original: is “l”.
Revised: , white

Original: The last letter of “hunting” is “g”. The last letter of “ranch” is “h”. Concatenating “e”, “y”, “h”, “t”, “n”, “s”, “y”, “l”, “g”, “h” leads to “eyhtsynslgh”. So, “sensitive, pray, wreath, betwixt, golden, regions, loyalty, interval, hunting, ranch” outputs “eyhtsynslgh”.
Revised: [Table]

Original: “freezing, arduous, pushing, vitality, hue, generosity, white, motto, matthew, preferred, hounds, breast”
Revised: motto, matthew, preferred, hounds, breast”

Original: Golden: “gsgyeyeowdst”
Revised: Golden: “gsgyeyeowdst”

Original: Solution:
Revised: Solution:

Original: Q: “freezing, arduous, pushing, vitality, hue, generosity, white, motto, matthew, preferred, hounds, breast”A:The last letter of “freezing” is “g”.
Revised: Q: “freezing, arduous, pushing, vitality, hue, generosity, white, motto, matthew, preferred, hounds, breast”A:The last letter of “freezing” is “g”.

Original: The last letter of “arduous” is “s”.
Revised: The last letter of “arduous” is “s”.

Original: The last letter of “pushing” is “g”.
Revised: The last letter of “pushing” is “g”.

Original: The last letter of “vitality” is “y”.
Revised: The last letter of “vitality” is “y”.

Original: The last letter of “hue” is “e”.
Revised: The last letter of “hue” is “e”.

Original: The last letter of “generosity” is “y”.
Revised: The last letter of “generosity” is “y”.

Original: The last letter of “white” is “e”.
Revised: The last letter of “white” is “e”.

Original: The last letter of “motto” is “o”.
Revised: The last letter of “motto” is “o”.

Original: The last letter of “matthew” is “w”.
Revised: The last letter of “matthew” is “w”.

Original: The last letter of “preferred” is “d”.
Revised: The last letter of “preferred” is “d”.

Original: The last letter of “hounds” is “s”.
Revised: The last letter of “hounds” is “s”.

Original: The last letter of “breast” is “t”.
Revised: The last letter of “breast” is “t”.

Original: Concatenating “g”, “s”, “g”, “y”, “e”, “y”, “e”, “o”, “w”, “d”, “s”, “t” leads to “gsgyeyowdst”.
Revised: Concatenating “g”, “s”, “g”, “y”, “e”, “y”, “e”, “o”, “w”, “d”, “s”, “t” leads to “gsgyeyowdst”.

Original: So, “freezing, arduous, pushing, vitality, hue, generosity, white, motto, matthew, preferred, hounds, breast” outputs “gsgyeyowdst”.
Revised: So, “freezing, arduous, pushing, vitality, hue, generosity, white, motto, matthew, preferred, hounds, breast” outputs “gsgyeyowdst”.

Original: A L EAST - TO - MOST PROMPTING : S 
Revised: A.5. L EAST - TO - MOST PROMPTING : S UCCESS

Original: UCCESS “worm
Revised: “worm

Original: , jackson, widow, car”
Revised: , jackson, widow, car”

Original: Golden: “mnwr”
Revised: Golden: “mnwr”

Original: [Table]
Revised: Solution: Q: “worm, jackson” A: The last letter of “worm” is “m”. The last letter of “jackson” is “n”.

Original: [Table]
Revised: Concatenating: “m”, “n” leads to “mn”. So, “worm, jackson” outputs “mn”. Q: “worm, jackson, widow” A: “worm, jackson” outputs “mn”. The last letter of “widow” is “w”.

Original: “osrutysgee”.
Revised: Concatenating: “mn”, “w” leads to “mnw”. So, “worm, jackson, widow” [Table] [Table]

Original: “niece, greedy, anything, retain, moisture, jest, adrian, attic, hugo, accurate, wet, howard”
Revised: last [Table] So [Table]

Original: Golden: “eygnetncoetd”
Revised: [Table] [Table]

Original: Solution:
Revised: Solution:

Original: Q: “niece, greedy”
Revised: [Table] [Table]

Original: A: The last letter of “niece” is “e”.
Revised: A: The last letter of “niece” is “e”.

Original: The last letter of “greedy” is “y”.
Revised: The last letter of “greedy” is “y”.

Original: Concatenating: “e”, “y” leads to “ey”.
Revised: Concatenating: “e”, “y” leads to “ey”.

Original: So, “niece, greedy” outputs “ey”.
Revised: So, “niece, greedy” outputs “ey”.

Original: Q: “niece, greedy anything” A: “niece, greedy” outputs “ey”.
Revised: Q: “niece, greedy, anything” A: “niece, greedy” outputs “ey”.

Original: The last letter of “anything” is “g”.
Revised: The last letter of “anything” is “g”.

Original: Concatenating “ey” “g” leads to “eyg”.
Revised: Concatenating: “ey”, “g” leads to “eyg”.

Original: So, “niece, greedy anything” outputs 
Revised: So, “niece, greedy, anything” outputs “eyg”.

Original: “eyg”. Q: “niece, greedy, anything, retain” A: “niece, 
Revised: Q: “niece, greedy, anything, retain” A: “niece, greedy, anything” outputs “eyg”.

Original: greedy
Revised: The last letter of “retain” is “n”. Concatenating: “eyg”, “n” leads to “eygn”.

Original: , anything” outputs 
Revised: So, “niece, greedy, anything, retain” outputs “eygn”.

Original: “eyg”
Revised: Q: “niece, greedy, anything, retain, moisture” A: “niece, greedy, anything, retain” outputs “eygn”.

Original: The last letter of “retain” is “n”
Revised: The last letter of “moisture” is “e”.

Original: Concatenating : “eyg”, “n” leads to “eygn”.
Revised: Concatenating: “eygn”, “e” leads to “eygne”.

Original: So , “niece , greedy, anything, retain” outputs “eygn”
Revised: So, “niece, greedy, anything, retain, moisture”

Original: Q: “niece,
Revised: [Table]

Original: greedy
Revised: , anything, retain, moisture, jest, adrian, attic, hugo, accurate” outputs “eygnetncoe”. The last letter of “wet” is “t”. Concatenating: “eygnetncoe”, “t” leads to “eygnetncoet”.

Original: anything retain, moisture”
Revised: So, “niece, greedy, anything, retain, moisture, jest, adrian, attic, hugo, accurate, wet” outputs “eygnetncoet”.

Original:  A: “niece, 
Revised: Q: “niece, greedy, anything, retain, moisture, jest, adrian, attic, hugo, accurate, wet, howard” A: “niece, greedy, anything, retain, moisture, jest, adrian, attic, hugo, accurate, wet” outputs “eygnetncoet”. The last letter of “howard” is “d”. Concatenating: “eygnetncoet”, “d” leads to “eygnetncoetd”.

Original: greedy
Revised: So, “niece, greedy, anything, retain, moisture, jest, adrian, attic, hugo, accurate, wet, howard” outputs “eygnetncoetd”. A.5. L EAST - TO - MOST PROMPTING : F AILURE “supper , procession, region, ruby”

Original: anything, retain” outputs
Revised: Golden: “rnny” Solution: Q: “supper, procession” [Table] [Table]

Original: “eygn”
Revised: [Table] [Table] [Table] leads to “rnnn”.

Revised: So, “supper, procession, region, ruby” outputs “rnnn”.

Revised: “release,

Original: The lastletter of
Revised: [Table] [Table] [Table] last

Original: “moisture” is “e”
Revised: [Table] “a” [Table]

Original: Concatenating
Revised: Concatenating

Original: : “eygn”, “e” leads to “eygne” So
Revised: [Table]

Original: , “niece,
Revised: “d”

Original: greedy , anything, retain moisture” outputs
Revised: [Table]

Original: “eygne”.
Revised: “da”

Original: Q: “niece, greedy , anything, retain
Revised: [Table]

Original: , moisture, jest” A: “niece, greedy
Revised: So

Original: , anything,
Revised: [Table]

Original: retain, moisture” outputs “eygne”
Revised: “provoked, columbia” outputs “da”

Original: The last letter of
Revised: [Table]

Original: “jest” is “t”
Revised: sir” [Table] outputs “da”

Original: Concate
Revised: [Table]

Original: - nating: “eygne”, “t” leads
Revised: last letter of “sir” is “r”

Original: to
Revised: [Table] Concatenating [Table]

Original: “eygnet”
Revised: “da” [Table]

Original: So, “niece , greedy, anything retain
Revised: “r”

Original: moisture

Original: jest”

Original: outputs “eygnet”.

Original: Q: “niece,

Original: greedy

Original: anything

Original: retain

Original: moisture

Original: jest

Original: adrian”

Original: A: “niece,

Original: greedy, anything, retain, moisture, jest”

Original: [Table]

Original: [Table]

Original: [Table]

Original: , greedy, anything, retain, moisture, jest, adrian, attic, hugo, accurate, wet,

Original: [Table]
Revised: [Table]

Original: A.4. L EAST - TO - MOST PROMPTING : F AILURE

Original: [Table]

Original: , “d” leads to “eded”.

Original: So, “release, appeared, endure,

Original: [Table]

Original: [Table]

Original: [Table]

Original: leads to

Original: [Table]

Original: So

Original: [Table]

Original: “release

Original: [Table]

Original: appeared

Original: [Table]

Original: [Table]

Original: “mentally

Original: [Table]

Original: Solution: Q

Original: [Table]

Original: “mentally, uncomfortable”

Original: [Table]

Original: [Table]

Original: [Table]

Original: to
Revised: to

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: The

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: “provoked

Original: [Table]
Revised: [Table]

Original: sir seven

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: seven

Original: [Table]
Revised: [Table]

Original: “provoked
Revised: “darleydds”

Original: [Table]
Revised: [Table]

Original: seven timothy submitted”
Revised: “darleyddst”

Original: [Table]
Revised: [Table]

Original: “darleyd”

Original: The last

Original: A: “provoked, columbia, sir, principal, seven, extreme, timothy, submitted, considered” outputs “darleydd”.

Original: The last letter of “spies” is “s”.

Original: Concatenating: “darleydd”, “s” leads to “darleydds”.

Original: So, “provoked, columbia, sir, principal, seven, extreme, timothy, submitted, considered, spies” outputs “darleydds”.

Original: Q: “provoked, columbia, sir, principal, seven, extreme, timothy, submitted, considered, spies, dost”A:“provoked, columbia, sir, principal, seven, extreme, timothy, submitted, considered, spies” outputs “darleydds”.

Original: The last letter of “dost” is “t”.

Original: Concatenating: “darleydds”, “t” leads to “darleyddst”.

Original: So, “provoked, columbia, sir, principal, seven, extreme, timothy, submitted, considered, spies, dost” outputs “darleyddst”.

Original: Q: “provoked, columbia, sir, principal, seven, extreme, timothy, submitted, considered, spies, dost, treasury”

Original: A: “provoked, columbia, sir, principal, seven, extreme, timothy, submitted, considered, spies, dost” outputs “darleyddst”.

Original: The last letter of “treasury” is “y”.

Original: Concatenating: “darleyddst”, “y” leads to “darleyddsty”.

Original: So, “provoked, columbia, sir, principal, seven, extreme, timothy, submitted, considered, spies, dost, treasury” outputs “darleyddsty”.

Original: B SCAN B. P ROMPT CONTEXTS
Revised: B SCAN B. P ROMPT CONTEXTS

Original: In this section we present the prompt contexts used for the SCAN benchmark in Section 3.2.
Revised: In this section we present the prompt contexts used for the SCAN benchmark in Section 3.2.

Original: It includes one context for  standard prompting, least-to-most prompting, and chain-of-thought prompting
Revised: It includes one context for each of standard prompting, least-to-most prompting, and chain-of-thought prompting.

Original: . B.1. S TANDARD PROMPTING
Revised: B.1. S TANDARD PROMPTING

Original: The context for standard prompting consist of a set of commands together with the corresponding action sequences.
Revised: The context for standard prompting consist of a set of commands together with the corresponding action sequences.

Original: Q: “turn left” A: “TURN LEFT” Q: “turn right” A: “TURN RIGHT”Q: “jump
Revised: Q: “turn left” A: “TURN LEFT” Q: “turn right” A: “TURN RIGHT”

Revised: Q: “jump left”

Revised: A: “TURN LEFT” + “JUMP”

Revised: Q: “run right”

Revised: A: “TURN RIGHT” + “RUN”

Revised: Q: “look twice”

Revised: A: “LOOK” *

Revised: Q: “run and look twice”

Revised: A: “RUN” + “LOOK” *

Revised: Q: “jump right thrice”

Revised: A: (“TURN RIGHT” + “JUMP”)

Revised: Q: “walk after run”

Revised: A: “RUN” + “WALK”

Revised: Q: “turn opposite left”

Revised: A: “TURN LEFT” *

Revised: Q: “turn around left”

Revised: A: “TURN LEFT” *

Revised: Q: “turn opposite right”

Revised: A: “TURN RIGHT” *

Revised: Q: “turn around right”

Revised: A: “TURN RIGHT” *

Revised: Q: “walk opposite left”

Original:  L EAST - TO - MOST 
Revised: A: “TURN LEFT” * 2 + “WALK” Q: “walk around left” A: (“TURN LEFT” + “WALK”) * B.1. L EAST - TO - MOST PROMPTING

Original: For least-to-most prompting, we need 2 different prompt contexts: one to reduce commands into a sequence of components and one to map commands to actions.
Revised: For least-to-most prompting, we need 2 different prompt contexts: one to reduce commands into a sequence of components and one to map commands to actions.

Original: Command mapping.
Revised: Command mapping.

Original: The command mapping prompt context consists of a set of commands together with the corresponding action sequences andan explanations of how they were obtained.
Revised: The command mapping prompt context consists of a set of commands together with the corresponding action sequences and explanations of how they were obtained.

Original: “walk” outputs “WALK”. So concatenating the output of “turn opposite left” and the output of “walk” leads to “TURN LEFT” * 2 + “WALK”. So the output of “walk opposite left” is “TURN LEFT” * 2 + “WALK”. Q: “walk around left” A: The output of “walk around left”concatenates: the output of “walk left”, the output of “walk left”, the output of “walk left”, the output of “walk left”.
Revised: Q: “turn left”

Original: “walk left” outputs “TURN LEFT” + “WALK”.
Revised: A: “turn left” outputs “TURN LEFT”.

Original: So repeating the output of “walk around left” four times leads to (“TURN LEFT” + “WALK”)
Revised: Q: “turn right”

Original: So the output of “walk around left” is (“TURN LEFT” + “WALK”)
Revised: A: “turn right” outputs “TURN RIGHT”.

Original: Command reduction.
Revised: Command reduction.

Original: The command reduction prompt context consists of a set of commands together with the corresponding sequence of reduced commands.
Revised: The command reduction prompt context consists of a set of commands together with the corresponding sequence of reduced commands.

Original: Q: “lookright after look twice” A: “look right after look twice” can be solved by: “lookright”, “look twice”.
Revised: Q “look “look

Original: Q: “jump opposite right thrice and walk” A: “jump opposite right thrice” can be solved by: “jump opposite right”, “jump opposite right thrice”.
Revised: A

Original: “walk” can be solved by: “walk”.
Revised: “walk” “run

Original: So, “jump opposite right thrice and walk” can be solved by: “jump opposite right”, “jump opposite right thrice”, “walk”.
Revised: Q

Original: Q: “run left twice and run right” A: “run left twice” can be solved by: “run left”, “run left twice”.
Revised: “look A: “look opposite right thrice” can be solved by “look opposite right” opposite right thrice”.

Original: “run right” can be solved by “run right”.
Revised: “walk” can be solved by “walk”.

Original: So, “run left twice and run right” can.be solved by: “run left”, “run left twice”, “run right”.
Revised: So, “look opposite right thrice after walk” can be solved by “look opposite right”, “look opposite right thrice”, “walk”.

Original: Q: “run opposite right” A: “run opposite right” can be solved by “run opposite right”. Q: “look opposite right thrice after walk” A: “look opposite right thrice” can be solved by: “look opposite right”, “look opposite right thrice”. “walk” can be solved by “walk”.
Revised: Q

Original: So, “look opposite right thrice after walk”
Revised: “jump around right” A: “jump

Original: B.1. C HAIN - OF - THOUGHT PROMPTING
Revised: B.1. C HAIN - OF - THOUGHT PROMPTING

Original: We reuse  command mapping prompt context from least-to-most prompting shown above.
Revised: We reuse the command mapping prompt context from least-to-most prompting shown above.

Original: B. E RROR ANALYSIS : L EAST - TO - MOST PROMPTINGError type
Revised: B. E RROR ANALYSIS : L EAST - TO - MOST PROMPTING

Original: [Table]

Original: Incorrect interpretation of “twice” and “thrice”   - Following “around”   - Following “opposite”   - Other  “after” interpreted as “and”

Original: [Table]

Original: For least-to-most prompting, we analyzed random failures for the models code-davinci-001 and text-davinci-002 , and we analyzed all 13 errors for the model code-davinci-002 .
Revised: For least-to-most prompting, we analyzed random failures for the models code-davinci-001 and text-davinci-002 , and we analyzed all 13 errors for the model code-davinci-002 .

Original: The results are shown in Table 14.
Revised: The results are shown in Table 14.

Original: Errors may either occur during command reduction or during command translation.
Revised: Errors may either occur during command reduction or during command translation.

Original: The translation errors are further split into the following types.
Revised: The translation errors are further split into the following types.

Original: Incorrect interpretation of “twice” and “thrice” means that the model made an error when applying “twice” and “thrice” to an expression.
Revised: Incorrect interpretation of “twice” and “thrice” means that the model made an error when applying “twice” and “thrice” to an expression.

Original: “after” interpreted as “and” means that the model translated an expression containing “after” as if it instead contained “and”.
Revised: “After” interpreted as “and” means that the model translated an expression containing “after” as if it instead contained “and”.

Original: Copy error means that the model made a mistake when copying an intermediate result.
Revised: Copy error means that the model made a mistake when copying an intermediate result.

Original:  We observe that the best model text-davinci-002 made only 2 types of mistakes: it sometimes makes a mistake when applying “twice” and “thrice” to an expression containing “around” and it sometimes interprets “after” as “and”.
Revised: [Table] [Table] [Table] We observe that the best model text-davinci-002 made only 2 types of mistakes: it sometimes makes a mistake when applying “twice” and “thrice” to an expression containing “around”, and it sometimes interprets “after” as “and”.

Original: For text-davinci-001 , which is the older version of the same model, the errors are spread across all types.
Revised: For text-davinci-001 , which is the older version of the same model, the errors are spread across all types.

Original: In particular, it’s worth noting that text-davinci-001 makes a signiﬁcant number of reduction errors and copy errors that were completely eliminated by its successor.
Revised: In particular, it’s worth noting that text-davinci-001 makes a signiﬁcant number of reduction errors and copy errors that were completely eliminated by its successor.

Original: The model text-davinci-002 made most of its mistakes when interpreting “twice” and “thrice” following “around”.
Revised: The model text-davinci-002 made most of its mistakes when interpreting “twice” and “thrice” following “around”.

Original: In addition, it sometimes made a mistake when translating “left” and “right”, which is something we did not observe with the other models.
Revised: In addition, it sometimes made a mistake when translating “left” and “right”, which is something we did not observe with the other models.

Original: In some cases, it dropped the command entirely whereas in other cases it invented a new action such as “LOOK LEFT” (see examples below).
Revised: In some cases, it dropped the command entirely, whereas in other cases it invented a new action such as “LOOK LEFT” (see examples below).

Original: Examples of reduction errors.
Revised: Examples of reduction errors.

Original: In the example “run around left twice after jump around right thrice”, the code-davinci-001 model does not properly reduce the sub-expression “run around left twice”.
Revised: In the example “run around left twice after jump around right thrice”, the code-davinci-001 model does not properly reduce the sub-expression “run around left twice”.

Original: Instead of reducing it to the sequence [“run left”, “run around left”, “run around left twice”], it skips “run around left” and reduces it to [“run left”, “run around left twice”].
Revised: Instead of reducing it to the sequence [“run left”, “run around left”, “run around left twice”], it skips “run around left” and reduces it to [“run left”, “run around left twice”].

Original: Consequently, the model translates this sub-expression to (“TURN LEFT” + “RUN”)
Revised: Consequently, the model translates this sub-expression to (“TURN LEFT” + “RUN”)

Original: * 2 instead of (“TURN LEFT” + “RUN”) * 4 * 2.
Revised: * 2 instead of (“TURN LEFT” + “RUN”) * 4 * 2.

Original: In the example “look around right twice after jump around left twice”, the code-davincimodel does not properly reduce the sub-expression “jump around left twice”.
Revised: In the example “look around right twice after jump around left twice”, the code-davincimodel does not properly reduce the sub-expression “jump around left twice”.

Original: Instead of reducing it to the sequence [“jump left”, “jump around left”, “jump around left twice”], it skips “jump around left” and reduces it to [“jump left”, “jump around left twice”].
Revised: Instead of reducing it to the sequence [“jump left”, “jump around left”, “jump around left twice”], it skips “jump around left” and reduces it to [“jump left”, “jump around left twice”].

Original: Interestingly, the model is able to recover from this mistake and correctly translates the sub-expression to (“TURN LEFT” + “JUMP”)
Revised: Interestingly, the model is able to recover from this mistake and correctly translates the sub-expression to (“TURN LEFT” + “JUMP”)

Original: * 4 * 2 but still produces the wrong ﬁnal action sequence because it interprets “after” like “and”.
Revised: * 4 * 2, but still produces the wrong ﬁnal action sequence because it interprets “after” like “and”.

Original: Examples of incorrect interpretation of “twice” and “thrice”.
Revised: Examples of incorrect interpretation of “twice” and “thrice”.

Original: In the example “walk opposite right twice after run around right thrice”, the code-davinci-002 model correctly translates the expression “run around right” to (“TURN RIGHT” + “RUN”) * 4.
Revised: In the example “walk opposite right twice after run around right thrice”, the code-davinci-002 model correctly translates the expression “run around right” to (“TURN RIGHT” + “RUN”) * 4.

Original: Then it makes a mistake when applying “thrice” to this expression and produces (“TURN RIGHT” + “RUN”)
Revised: Then it makes a mistake when applying “thrice” to this expression and produces (“TURN RIGHT” + “RUN”)

Original: * 9 instead of (“TURN RIGHT” + “RUN”) * 4 * 3 or (“TURN RIGHT” + “RUN”)
Revised: * 9 instead of (“TURN RIGHT” + “RUN”) * 4 * 3 or (“TURN RIGHT” + “RUN”)

Original: In the example “jump opposite right twice and jump around right thrice”, the code-davinci-002 model correctly translates the expression “jump around right” to (“TURN RIGHT” + “JUMP”)
Revised: In the example “jump opposite right twice and jump around right thrice”, the code-davinci-002 model correctly translates the expression “jump around right” to (“TURN RIGHT” + “JUMP”)

Original: Then it makes a mistake when applying “thrice” to this expression and produces (“TURN RIGHT” + “JUMP”)
Revised: Then it makes a mistake when applying “thrice” to this expression and produces (“TURN RIGHT” + “JUMP”)

Original: * 8 instead of (“TURN RIGHT” + “JUMP”)
Revised: * 8 instead of (“TURN RIGHT” + “JUMP”)

Original: ** 3 or (“TURN RIGHT” + “JUMP”)
Revised: ** 3 or (“TURN RIGHT” + “JUMP”)

Original: In the example “walk around left thrice after run opposite left thrice”, the code-davincimodel correctly translates the expression “run opposite left” to “TURN LEFT” * 2 + “RUN”.
Revised: In the example “walk around left thrice after run opposite left thrice”, the code-davincimodel correctly translates the expression “run opposite left” to “TURN LEFT” * 2 + “RUN”.

Original: Then it makes a mistake when applying “thrice” to this expression and produces “TURN LEFT” * 2 + “RUN” * 3 instead of (“TURN LEFT” * 2 + “RUN”) * 3.
Revised: Then it makes a mistake when applying “thrice” to this expression and produces “TURN LEFT” * 2 + “RUN” * 3 instead of (“TURN LEFT” * 2 + “RUN”) * 3.

Original: In the example “walk around left thrice after look right twice”, the code-davinci-001 model correctly translates the expression “look right” to “TURN RIGHT” + “LOOK”.
Revised: In the example “walk around left thrice after look right twice”, the code-davinci-001 model correctly translates the expression “look right” to “TURN RIGHT” + “LOOK”.

Original: Then it makes a mistake when applying “twice” to this expression and produces “TURN RIGHT” + “LOOK” *rather than (“TURN RIGHT” + “LOOK”) * 2.
Revised: Then it makes a mistake when applying “twice” to this expression and produces “TURN RIGHT” + “LOOK” *rather than (“TURN RIGHT” + “LOOK”) * 2.

Original: In the example “walk left and run around right thrice”, the code-davinci-001 model interprets “thrice” as ‘twice”.
Revised: In the example “walk left and run around right thrice”, the code-davinci-001 model interprets “thrice” as ‘twice”.

Original: This means that it produces “TURN LEFT” + “WALK” + (“TURN RIGHT” + “RUN”) * 4 * 2 instead of “TURN LEFT” + “WALK” + (“TURN RIGHT” + “RUN”) * 4 * 3.
Revised: This means that it produces “TURN LEFT” + “WALK” + (“TURN RIGHT” + “RUN”) * 4 * 2 instead of “TURN LEFT” + “WALK” + (“TURN RIGHT” + “RUN”) * 4 * 3.

Original: In the example “jump right twice and look around left thrice”, the text-davinci-002 model correctly translates the sub-expression “look around left” to (“TURN LEFT” + “LOOK”) * 4.
Revised: In the example “jump right twice and look around left thrice”, the text-davinci-002 model correctly translates the sub-expression “look around left” to (“TURN LEFT” + “LOOK”) * 4.

Original: But when applying “thrice” it produces the incorrect translation (“TURN LEFT” + “LOOK”)
Revised: But when applying “thrice”, it produces the incorrect translation (“TURN LEFT” + “LOOK”)

Original: * 3 instead of (“TURN LEFT” + “LOOK”) * 4 * 3.
Revised: * 3 instead of (“TURN LEFT” + “LOOK”) * 4 * 3.

Original: Example of interpreting “after” as “and”.
Revised: Example of interpreting “after” as “and”.

Original: In the example “run opposite left thrice after run around left twice”, the code-davinci-002 model produces the correct translations for both sub-expressions that are connected by “after” but it combines them as if they were connected by “and”.
Revised: In the example “run opposite left thrice after run around left twice”, the code-davinci-002 model produces the correct translations for both sub-expressions that are connected by “after”, but it combines them as if they were connected by “and”.

Original: This means that the model produces (“TURN LEFT” * 2 + “RUN”)
Revised: This means that the model produces (“TURN LEFT” * 2 + “RUN”)

Original: * 3 + (“TURN LEFT” + “RUN”) * 4 * 2 instead of (“TURN LEFT” + “RUN”) * 4 * 2 + (“TURN LEFT” * 2 + “RUN”)
Revised: * 3 + (“TURN LEFT” + “RUN”) * 4 * 2 instead of (“TURN LEFT” + “RUN”) * 4 * 2 + (“TURN LEFT” * 2 + “RUN”)

Original: In the example “walk around left thrice after walk twice”, the code-davinci-002 model produces the correct translations for both sub-expressions that are connected by “after” but it combines them as if they were connected by “and”.
Revised: In the example “walk around left thrice after walk twice”, the code-davinci-002 model produces the correct translations for both sub-expressions that are connected by “after”, but it combines them as if they were connected by “and”.

Original: This means that the model produces (“TURN LEFT” + “WALK”)
Revised: This means that the model produces (“TURN LEFT” + “WALK”)

Original: * 4 * 3 + “WALK” * 2 instead of “WALK” * 2 + (“TURN LEFT” + “WALK”)
Revised: * 4 * 3 + “WALK” * 2 instead of “WALK” * 2 + (“TURN LEFT” + “WALK”)

Original: In the example “look around right twice after jump around left twice”, the code-davincimodel produces the correct translations for both sub-expressions that are connected by “after” but it combines them as if they were connected by “and”.
Revised: In the example “look around right twice after jump around left twice”, the code-davincimodel produces the correct translations for both sub-expressions that are connected by “after”, but it combines them as if they were connected by “and”.

Original: This means that the model produces (“TURN RIGHT” + “LOOK”)
Revised: This means that the model produces (“TURN RIGHT” + “LOOK”)

Original: * 4 * 2 + (“TURN LEFT” + “JUMP”)
Revised: * 4 * 2 + (“TURN LEFT” + “JUMP”)

Original: * 4 * 2 instead of (“TURN LEFT” + “JUMP”)
Revised: * 4 * 2 instead of (“TURN LEFT” + “JUMP”)

Original: * 4 * 2 + (“TURN RIGHT”
Revised: * 4 * 2 + (“TURN RIGHT”

Original: + “LOOK”)
Revised: + “LOOK”)

Original: Examples of incorrect interpretation of “left” and “right”.
Revised: Examples of incorrect interpretation of “left” and “right”.

Original: In the example “look opposite right thrice after look around left thrice”, the text-davinci-002 model translates the component “look left” to “LOOK” instead of “TURN LEFT LOOK”.
Revised: In the example “look opposite right thrice after look around left thrice”, the text-davinci-002 model translates the component “look left” to “LOOK” instead of “TURN LEFT LOOK”.

Original: As a consequence, the whole command is translated to “LOOK” * 4 * 3 + (“TURN RIGHT” * 2 + “LOOK”)
Revised: As a consequence, the whole command is translated to “LOOK” * 4 * 3 + (“TURN RIGHT” * 2 + “LOOK”)

Original: * 3 instead of (“TURN LEFT” + “LOOK”) * 4 * 3 + (“TURN RIGHT” * 2 + “LOOK”) * 3.
Revised: * 3 instead of (“TURN LEFT” + “LOOK”) * 4 * 3 + (“TURN RIGHT” * 2 + “LOOK”) * 3.

Original: In the example “turn around right thrice after look around left twice”, the text-davincimodel makes up the new action “LOOK LEFT” as the translation of the component “look left”.
Revised: In the example “turn around right thrice after look around left twice”, the text-davincimodel makes up the new action “LOOK LEFT” as the translation of the component “look left”.

Original: As a consequence, it translates the whole command to (“LOOK LEFT” * 4) * 2 + (“TURN RIGHT” * 4) * 3 instead of (“TURN LEFT” + “LOOK”)
Revised: As a consequence, it translates the whole command to (“LOOK LEFT” * 4) * 2 + (“TURN RIGHT” * 4) * 3 instead of (“TURN LEFT” + “LOOK”)

Original: * 4 * 2 + (“TURN RIGHT” * 4) * 3.
Revised: * 4 * 2 + (“TURN RIGHT” * 4) * 3.

Original: Example of copy error.
Revised: Example of copy error.

Original: In the example “walk opposite right twice after look around left thrice”, the code-davinci-001 model produces the correct translations for both sub-expressions that are connected with “after”.
Revised: In the example “walk opposite right twice after look around left thrice”, the code-davinci-001 model produces the correct translations for both sub-expressions that are connected with “after”.

Original: In particular, it correctly translates “walk opposite right twice” to (“TURN RIGHT” * 2 + “WALK”)
Revised: In particular, it correctly translates “walk opposite right twice” to (“TURN RIGHT” * 2 + “WALK”)

Original: Then it makes two mistakes when combining them: it treats “after” like an “and”, and it omits the parenthesis when copying the expression (“TURN RIGHT” * 2 + “WALK”)
Revised: Then it makes two mistakes when combining them: it treats “after” like an “and”, and it omits the parenthesis when copying the expression (“TURN RIGHT” * 2 + “WALK”)

Original: This means that the model produces the translation “TURN RIGHT” * 2 + “WALK” * 2 + (“TURN LEFT” + “LOOK”)
Revised: This means that the model produces the translation “TURN RIGHT” * 2 + “WALK” * 2 + (“TURN LEFT” + “LOOK”)

Original: * 4 * 3” instead of (“TURN LEFT” + “LOOK”)
Revised: * 4 * 3” instead of (“TURN LEFT” + “LOOK”)

Original: * 4 * 3 + (“TURN RIGHT” * 2 + “WALK”)
Revised: * 4 * 3 + (“TURN RIGHT” * 2 + “WALK”)

Original:  E XAMPLE OUTPUTS FROM CODE - DAVINCI - 
Revised: B. E XAMPLE OUTPUTS FROM CODE - DAVINCI -

Original: In this section, we present 5 examples of successes and failures of the code-davinci-002 model for chain-of-thought prompting and least-to-most prompting.  
Revised: In this section, we present 5 examples of successes and failures of the code-davinci-002 model for chain-of-thought prompting and least-to-most prompting. B.3.

Original: “jump around left twice opposite
Revised: [Table] LEFT [Table]

Original: Golden: TURN LEFT TURN LEFT WALK TURN LEFT TURN LEFT WALK TURN LEFT TURN LEFT WALK
Revised: LEFT WALK TURN LEFT TURN LEFT WALK TURN LEFT TURN LEFT WALK

Original: TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP 
Revised: TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN RIGHT JUMP

Original: : Q: “jump around left twice after left
Revised: [Table]

Original: A: The output of “jump around twice : the output of “walk opposite left , the output of “jump around left twice”. “walk left outputs
Revised: [Table]

Original: “TURN LEFT” * 2 + “WALK”) So
Revised: [Table] [Table]

Original: “walk twice Golden: TURN LEFT TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT
Revised: [Table]

Original: WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN RIGHT TURN RIGHT TURN RIGHT TURN RIGHT TURN RIGHT TURN RIGHT
Revised: TURN LEFT RUN TURN LEFT RUN TURN LEFT RUN TURN LEFT RUN TURN LEFT RUN

Original: TURN RIGHT TURN RIGHT TURN RIGHT TURN RIGHT TURN RIGHT TURN RIGHT Solution: Q: “walk around left twice and turn around right thrice” A:
Revised: TURN LEFT RUN TURN LEFT RUN TURN LEFT RUN TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK

Original: The So around and walk around thrice” is
Revised: TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK

Original: Golden:
Revised: [Table]

Original: TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT JUMP TURN LEFT
Revised: TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT JUMP TURN LEFT

Original: JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT
Revised: JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT

Original: JUMP
Revised: JUMP

Original: TURN
Revised: Solution

Original: LEFT
Revised: [Table]

Original: WALK WALK TURN
Revised: [Table] [Table]

Original: TURN TURN LEFT WALK TURN
Revised: [Table] [Table]

Original: “walk walk
Revised: [Table]

Original: TURN RIGHT
Revised: TURN LEFT

Original: RIGHT
Revised: TURN LEFT RUN TURN LEFT RUN TURN LEFT RUN TURN LEFT RUN TURN LEFT RUN TURN LEFT RUN TURN LEFT RUN TURN LEFT RUN TURN LEFT RUN [Table] [Table]

Original: TURN RIGHT
Revised: left thrice” and the output of “run left twice” leads to (“TURN LEFT” + “RUN”) *

Original: TURN
Revised: [Table] “TURN LEFT” + “RUN” * 2. So the output of “run around left thrice and run left twice” is (“TURN LEFT” + “RUN”) *+ “TURN LEFT” + “RUN” * 2.

Original:  TURN 
Revised: “walk and walk around left thrice” [Table] : WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK

Original:  TURN 
Revised: [Table] Golden: TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT JUMP

Original: TURN RIGHT RIGHT
Revised: TURN RIGHT JUMP

Original: RIGHT
Revised: [Table] Golden:

Original:  RIGHT  RIGHT RUN TURN RIGHT RUN TURN RIGHT RUN TURN RIGHT
Revised: TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT

Original:  RIGHT 
Revised: WALK TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT WALK

Original:  RIGHTLEFT LEFT LEFT
Revised: TURN RIGHT WALK TURN RIGHT

Original:  LEFT
Revised: WALK TURN RIGHT WALK TURN RIGHT TURN RIGHT RUN B.3.  TURN RIGHT RUN TURN RIGHT RUN TURN RIGHT RUN TURN RIGHT RUN TURN RIGHT RUN TURN RIGHT RUN TURN RIGHT RUN TURN RIGHT RUN TURN RIGHT RUN TURN RIGHT RUN TURN RIGHT RUN TURN RIGHT RUN TURN LEFT TURN LEFT TURN LEFT TURN LEFT

Original: TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT
Revised: TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT

Original: [Table]
Revised: [Table]

Original: [Table] [Table] [Table]
Revised: So,

Original: [Table]
Revised: [Table]

Revised: around left

Original: [Table]
Revised: [Table]

Original: LOOK TURN LEFT LOOK TURN LEFT LOOK TURN LEFT LOOK TURN LEFT LOOK
Revised: can be solved

Original: [Table]
Revised: [Table]

Original: LEFT LOOK TURN LEFT LOOK TURN LEFT LOOK TURN LEFT LOOK TURN LEFT LOOK

Original: [Table]
Revised: [Table]

Original: LEFT LOOK TURN LEFT LOOK TURN
Revised: Solution: Q: “turn around left”

Original: [Table]
Revised: [Table]

Original: So repeating

Original: output “look four times leads “LOOK”
Revised: : TURN LEFT LOOK TURN LEFT LOOK TURN LEFT LOOK TURN LEFT LOOK TURN LEFT LOOK

Original: So output around left” is “LOOK”
Revised: TURN LEFT LOOK TURN LEFT LOOK TURN LEFT LOOK TURN LEFT LOOK TURN LEFT LOOK TURN LEFT LOOK TURN LEFT LOOK TURN LEFT

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original:  TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT
Revised: Golden: TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT

Original: TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT
Revised: TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT

Original: JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT 
Revised: JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP

Original: [Table]
Revised: [Table]

Original: around left”, “jump around left thrice”.
Revised: So,

Original: So, “turn around left thrice and jump around left thrice” can be solved by: “turn left”, “turn around left”, “turn around left thrice”, “jump left”, “jump around left”, “jump around left thrice”. Solution: Q: “turn around left”
Revised: [Table] [Table] [Table]

Original: A: The output of “turn around left” concatenates: the output of “turn left”, the output of “turn left”, the output of “turn left”, the output of “turn left”. “turn left” outputs “TURN LEFT”. So repeating the output of “turn left” four times leads to “TURN LEFT” * 4.
Revised: [Table]

Original: So the output of “turn around left” is “TURN LEFT” * 4.
Revised: [Table] [Table]

Original: Q: “turn around left thrice” A: The output of “turn around left thrice” concatenates: the output of “turn around left”, the output of “turn around left”, the output of “turn around left”. “turn around left” outputs “TURN LEFT” * 4. So repeating the output of “turn around left” three times leads to (“TURN LEFT” * 4) * 3. So the output of “turn around left thrice” is (“TURN LEFT” * 4) * 3.
Revised: [Table]

Original: Q: “jump left” A: The output of “jump concatenates the output of the output of “jump” “turn left” outputs “TURN “jump” outputs “JUMP”
Revised: “turn around left thrice”, the output of “jump around left thrice”.

Original: So the output “turn and the put of “jump” “TURN So the output
Revised: “turn around left thrice” outputs (“TURN LEFT” * 4) * 3.

Original: “jumpis “TURN + “JUMP”. Q: “jump around left” A
Revised: “jump around left thrice” outputs ((“TURN LEFT” + “JUMP”)

Original: The output of “jump around left” concatenates the output “jump the output “jump , the output of “jump left”, the output of “jump left”{+around left thrice” leads to (“TURN LEFT” * 4) * 
+}
Revised: So concatenating the output of “turn around left thrice” and the output of “jump {+around left thrice” leads to (“TURN LEFT” * 4) * 
+}

Original: “jump left” outputs “TURN + “JUMP”.
Revised: + ((“TURN LEFT” + “JUMP”) * 4) * 3.

Original: Sorepeating the output of “jump left” four times leads to (“TURN LEFT” + “JUMP”){+* 4) * 
+}
Revised: So the output of “turn around left thrice and jump around left thrice” is (“TURN LEFT” {+* 4) * 
+}

Original: So the output of “jump around left” is (“TURN LEFT” + “JUMP”) Q: “jump around left thrice” A: The output of “jump around left thrice” concatenates: the output of “jump around left”, the output of “jump around left”, the output of “jump “JUMP” So
Revised: + ((“TURN LEFT” + “JUMP”) * 4) * 3.

Original: So Q “walk around right  after 
Revised: “walk around right twice after walk around right twice”

Original: Golden:
Revised: Golden:

Original: TURN RIGHT WALK TURN RIGHT
Revised: TURN RIGHT WALK TURN RIGHT

Original: WALK TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT WALK
Revised: WALK TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT WALK

Original: TURN RIGHT WALK TURN RIGHT
Revised: TURN RIGHT WALK TURN RIGHT

Original: WALK TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT WALK
Revised: WALK TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT WALK

Original: Reduction:
Revised: Reduction:

Original: Q: “walk around right twice after walk around right twice”
Revised: Q: “walk around right twice after walk around right twice”

Original: A: “walk around right twice” can be solved by: “walk right”, “walk around right”, “walk around right twice”.
Revised: A: “walk around right twice” can be solved by: “walk right”, “walk around right”, “walk around right twice”.

Original: “walk around right twice” can be solved by: “walk right”, “walk around right”, “walk around right twice”.
Revised: “walk around right twice” can be solved by: “walk right”, “walk around right”, “walk around right twice”.

Original: So, “walk around right twice after walk around right twice” can be solved by: “walk right”, “walk around right”, “walk around right twice”, “walk right”, “walk around right”, “walk around right twice”.
Revised: So, “walk around right twice after walk around right twice” can be solved by: “walk right”, “walk around right”, “walk around right twice”, “walk right”, “walk around right”, “walk around right twice”.

Original: Solution:
Revised: Solution:

Original: Q: “walk right”
Revised: Q: “walk right”

Original: A: 
Revised: A: The output of “walk right” concatenates: the output of “turn right”, the output of “walk”. “turn right” outputs “TURN RIGHT”. “walk” outputs “WALK”. So concatenating the output of “turn right” and the output of “walk” leads to “TURN RIGHT” + “WALK”.

Revised: So the output of “walk right” is “TURN RIGHT” + “WALK”.

Original: “jump
Revised: Q: “walk around right” A: The output of “walk around right” [Table] [Table]

Original: TURN RIGHT TURN RIGHT JUMP TURN RIGHT TURN RIGHT JUMP TURN RIGHT JUMP
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT JUMP
Revised: [Table] [Table] [Table] [Table]

Original: TURN RIGHT JUMP TURN RIGHT JUMP
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: So repeating the output of “jump around four times leads to
Revised: TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT

Original: +
Revised: JUMP TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT TURN RIGHT JUMP

Original: TURN RIGHT TURN RIGHT 
Revised: TURN RIGHT TURN RIGHT JUMP

Original: WALK TURN RIGHT  TURN RIGHT WALK TURN RIGHT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK 
Revised: TURN RIGHT WALK TURN RIGHT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT

Original: TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK
Revised: WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK

Original: Reduction:
Revised: Reduction:

Original: [Table]

Original: right twice” A [Table] can be solved by:
Revised: Solution:

Original: [Table]
Revised: [Table]

Original: left”,
Revised: “TURN

Original: [Table]
Revised: “WALK”)

Original: around [Table] [Table] around left thrice”.
Revised: So

Original: “walk around right twice” can be solved by: [Table] right”, [Table]
Revised: B. E XPANDING P YTHON EXPRESSIONS USING PROMPTING

Original: around right”, [Table] around right twice”. So, “walk around left thrice after walk around right twice” can be solved by: “walk left”, “walk around left”, “walk around left thrice”, “walk right”, “walk around right”, “walk around right twice”.
Revised: In Section 3.2, we mention that expanding the Python expressions that we use as an intermediate representation can be done either with a simple postprocessing script or by prompting a language model.

Original: [Table] [Table] : “walk left” [Table]
Revised: In the following, we present a prompt that achieves 99.7% accuracy on a random sample of 1000 Python expressions that are outputted by our solution (using the code-davincimodel).

Original: TURN TURN
Revised: This demonstrates that the L2M method can solve SCAN with a combined accuracy more than 99% (99.7% accuracy for generating the intermediate Python expressions and 99.7% for expanding these expressions), even if we do not use the Python executor and instead perform the expansion of the intermediate representation via prompting.

Original: TURN TURN
Revised: Q: “JUMP” * Rewrite: “JUMP” *

Original: TURN
Revised: A: 1 JUMP 2 JUMP 3 JUMP Q: “RUN” * 4 * Rewrite: “RUN” * A: 1 RUN 2 RUN 3 RUN 4 RUN 5 RUN 6 RUN 7 RUN 8 RUN

Original:  TURN 
Revised: Q: “TURN RIGHT” + “WALK” Rewrite: “TURN RIGHT” + “WALK” A: TURN RIGHT

Original: RUN
Revised: WALK

Original: [Table] [Table] [Table]
Revised: Q: (“TURN LEFT” + “LOOK”) * 2 + “TURN LEFT” + “LOOK”

Original: [Table] [Table] [Table]
Revised: Rewrite: (“TURN LEFT” + “LOOK”) * 2 + “TURN LEFT” + “LOOK” A: 1 (TURN LEFT LOOK) 2 (TURN LEFT LOOK) TURN LEFT LOOK

Original: [Table] Solution: Q: “run left”
Revised: Q: (“TURN RIGHT” * 2 + “JUMP”) *

Original: [Table]
Revised: Rewrite: (“TURN RIGHT” * 2 + “JUMP”) * A: 1 (1 TURN RIGHT 2 TURN RIGHT JUMP) 2 (1 TURN RIGHT 2 TURN RIGHT JUMP) 3 (

Original: TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT 
Revised: TURN RIGHT 2 TURN RIGHT JUMP) 4 (1 TURN RIGHT 2 TURN RIGHT JUMP)

Original: JUMP TURN RIGHT JUMP TURN  RIGHT JUMP TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT TURN RIGHTJUMP
Revised: Q: “TURN LEFT” * 2 + (“TURN RIGHT” + “WALK”) Rewrite: “TURN LEFT” * 2 + (“TURN RIGHT” + “WALK”) * A: 1 TURN LEFT 2 TURN LEFT 1 (TURN RIGHT WALK) 2 (TURN RIGHT WALK) 3 (TURN RIGHT WALK) 4 (TURN RIGHT WALK) 5 (TURN RIGHT WALK) 6 (TURN RIGHT WALK) 7 (TURN RIGHT

Original: TURN RIGHT TURN RIGHT JUMP
Revised: WALK) 8 (TURN RIGHT WALK)

Original: [Table]
Revised: Discussion.

Original: TURN RIGHT WALK TURN RIGHT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT
Revised: The prompt consists of 6 examples, each of which illustrates part of the knowledge needed for this task. Note that we add numbers and parentheses when we unfold multiplication to make it easier for the model to keep track of the repetitions. Multiplication 2.

Original: WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK
Revised: Sequential multiplication 3. Addition 4. Avoid commutativity / associativity in addition 5.

Original: Reduction: Solution:
Revised: Nested multiplication 6. Addition of two multiplications

Original: C DROP C. R ESULTS WITH TEXT - DAVINCI -002 AND LM-540B
Revised: C DROP C. R ESULTS WITH TEXT - DAVINCI -002 AND LM-540B

Original: We reported the results using code-davinci-002 .
Revised: We reported the results using code-davinci-002 .

Original: Here, we report results using the text-davinci-002 model, and a language model with 540 billion parameters ( LM-540B ). 
Revised: Here, we report results using the text-davinci-002 model and a language model with 540 billion parameters ( LM-540B ). [Table]

Original: [Table]
Revised: [Table]

Revised: 002 is consistently worse than code-davinci

Original: [Table] [Table]
Revised: [Table] [Table]

Original: C. N ON - FOOTBALL S UBSET C.
Revised: C. N ON - FOOTBALL S UBSET C.2.

Original: Z ERO - SHOT PROMPTING
Revised: Z ERO - SHOT PROMPTING

Original: For zero-shot, the prompt format is as follows:
Revised: For zero-shot, the prompt format is as follows:

Original: Q: { question }
Revised: Q: { question }

Original: A:
Revised: A:

Original: The answer is
Revised: The answer is

Original: Notice that we add “The answer is” at the beginning of the answer section.
Revised: Notice that we add “The answer is” at the beginning of the answer section.

Original: C.2. S TANDARD PROMPTING WITH 3 EXAMPLES
Revised: C.2. S TANDARD PROMPTING WITH 3 EXAMPLES

Original: Q: Since the 1970s, U.S. governments have negotiated managed-trade agreements, such as the North
Revised: Q: Since the 1970s, U.S. governments have negotiated managed-trade agreements, such as the North

Original: American Free Trade Agreement in the 1990s, the Dominican Republic-Central America Free Trade
Revised: American Free Trade Agreement in the 1990s, the Dominican Republic-Central America Free Trade

Original: Agreement in 2006, and a number of bilateral agreements.
Revised: Agreement in 2006, and a number of bilateral agreements.

Original: In Europe, six countries formed the
Revised: In Europe, six countries formed the

Original: European Coal and Steel Community in 1951 which became the European Economic Community in 1958.
Revised: European Coal and Steel Community in 1951 which became the European Economic Community in 1958.

Original: Two core objectives of the EEC were the development of a common market, subsequently renamed the single market, and establishing a customs union between its member states.
Revised: Two core objectives of the EEC were the development of a common market, subsequently renamed the single market, and establishing a customs union between its member states.

Original: How many years did the European Coal and Steel Community exist?
Revised: How many years did the European Coal and Steel Community exist?

Original: A:
Revised: A:

Original: The answer is 7.
Revised: The answer is 7.

Original: Q: In the county, the population was spread out with 23.50% under the age of 18, 8.70% from 18 to 24, 29.70% from 25 to 44, 24.70% from 45 to 64, and 13.30% who were 65 years of age or older.
Revised: Q: In the county, the population was spread out with 23.50% under the age of 18, 8.70% from 18 to 24, 29.70% from 25 to 44, 24.70% from 45 to 64, and 13.30% who were 65 years of age or older.

Original: How many more percent are under the age of 18 compared to the 18 to 24 group?
Revised: How many more percent are under the age of 18 compared to the 18 to 24 group?

Original: A:
Revised: A:

Original: The answer is 14.8.
Revised: The answer is 14.8.

Original: Q: Playing in their second straight Thanksgiving game, the Eagles struggled especially on defense, where they were unable to stop the much-hyped Lions offense.
Revised: Q: Playing in their second straight Thanksgiving game, the Eagles struggled especially on defense, where they were unable to stop the much-hyped Lions offense.

Original: The worst of it all was how unproven rookie Eric Rowe was tasked with covering wide receiver Calvin Johnson, leading to Johnson catching 3 touchdowns.
Revised: The worst of it all was how unproven rookie Eric Rowe was tasked with covering wide receiver Calvin Johnson, leading to Johnson catching 3 touchdowns.

Original: Stafford’s ﬁve passing touchdowns, including three of them to Johnson was too much for the Eagles to overcome and for the second consecutive time this season, the Eagles gave up 45 points in a game.
Revised: Stafford’s ﬁve passing touchdowns, including three of them to Johnson was too much for the Eagles to overcome and for the second consecutive time this season, the Eagles gave up 45 points in a game.

Original: With the loss, the Eagles drop to 4-7 on the season and 6-1 when playing on
Revised: With the loss, the Eagles drop to 4-7 on the season and 6-1 when playing on

Original: Thanksgiving.
Revised: Thanksgiving.

Original: How many TD passes did Stafford throw other than to Johnson?
Revised: How many TD passes did Stafford throw other than to Johnson?

Original: A:
Revised: A:

Original: The answer is 2.
Revised: The answer is 2.

Original: C.2. C HAIN - OF - THOUGHT PROMPTING WITH 3 EXAMPLES
Revised: C.2. C HAIN - OF - THOUGHT PROMPTING WITH 3 EXAMPLES

Original: Q: Since the 1970s, U.S. governments have negotiated managed-trade agreements, such as the North
Revised: Q: Since the 1970s, U.S. governments have negotiated managed-trade agreements, such as the North

Original: American Free Trade Agreement in the 1990s, the Dominican Republic-Central America Free Trade
Revised: American Free Trade Agreement in the 1990s, the Dominican Republic-Central America Free Trade

Original: Agreement in 2006, and a number of bilateral agreements.
Revised: Agreement in 2006, and a number of bilateral agreements.

Original: In Europe, six countries formed the
Revised: In Europe, six countries formed the

Original: European Coal and Steel Community in 1951 which became the European Economic Community in 1958.
Revised: European Coal and Steel Community in 1951 which became the European Economic Community in 1958.

Original: Two core objectives of the EEC were the development of a common market, subsequently renamed the single market, and establishing a customs union between its member states.
Revised: Two core objectives of the EEC were the development of a common market, subsequently renamed the single market, and establishing a customs union between its member states.

Original: How many years did the European Coal and Steel Community exist?
Revised: How many years did the European Coal and Steel Community exist?

Original: A:
Revised: A

Original:  According to the passage, the European Coal and Steel Community was established in 1951 and became the EEC in 1958.
Revised: : According to the passage, the European Coal and Steel Community was established in 1951 and became the EEC in 1958.

Original: 1958 - 1951 =7.
Revised: =

Original: So the answer is 7.
Revised: So the answer is 7.

Original: Q: In the county, the population was spread out with 23.50% under the age of 18, 8.70% from 18 to 24, 29.70% from 25 to 44, 24.70% from 45 to 64, and 13.30% who were 65 years of age or older.
Revised: Q: In the county, the population was spread out with 23.50% under the age of 18, 8.70% from 18 to 24, 29.70% from 25 to 44, 24.70% from 45 to 64, and 13.30% who were 65 years of age or older.

Original: How many more percent are under the age of 18 compared to the 18 to 24 group?
Revised: How many more percent are under the age of 18 compared to the 18 to 24 group?

Original: A:
Revised: A:

Original: According to the passage, 23.5% are under the age of 18, and 8.7% are from ages 18 to 24.
Revised: According to the passage, 23.5% are under the age of 18, and 8.7% are from ages 18 to 24.

Original: 23.5% - 8.7% = 14.8%.
Revised: 23.5% - 8.7% = 14.8%.

Original: So the answer is 14.8.
Revised: So the answer is 14.8.

Original: Q: Playing in their second straight Thanksgiving game, the Eagles struggled especially on defense, where they were unable to stop the much-hyped Lions offense.
Revised: Q: Playing in their second straight Thanksgiving game, the Eagles struggled especially on defense, where they were unable to stop the much-hyped Lions offense.

Original: The worst of it all was how unproven rookie Eric Rowe was tasked with covering wide receiver Calvin Johnson, leading to Johnson catching 3 touchdowns.
Revised: The worst of it all was how unproven rookie Eric Rowe was tasked with covering wide receiver Calvin Johnson, leading to Johnson catching 3 touchdowns.

Original: Stafford’s ﬁve passing touchdowns, including three of them to Johnson was too much for the Eagles to overcome and for the second consecutive time this season, the Eagles gave up 45 points in a game.
Revised: Stafford’s ﬁve passing touchdowns, including three of them to Johnson was too much for the Eagles to overcome and for the second consecutive time this season, the Eagles gave up 45 points in a game.

Original: With the loss, the Eagles drop to 4-7 on the season and 6-1 when playing on
Revised: With the loss, the Eagles drop to 4-7 on the season and 6-1 when playing on

Original: Thanksgiving.
Revised: Thanksgiving.

Original: How many TD passes did Stafford throw other than to Jhonson?
Revised: How many TD passes did Stafford throw other than to Jhonson?

Original: A:
Revised: A:

Original: According to the passage, Stafford threw 5 TD passes, 3 of which were to Johnson.
Revised: According to the passage, Stafford threw 5 TD passes, 3 of which were to Johnson.

Original: 5 - 3 = 2.
Revised: 5 - 3 = 2.

Original: So the answer is 2.
Revised: So the answer is 2.

Original: C.2. L EAST - TO - MOST PROMPTING I: PROBLEM REDUCING (5 EXAMPLES )
Revised: C.2. L EAST - TO - MOST PROMPTING I: PROBLEM REDUCING (5 EXAMPLES )

Original: Q: The median age in the city was 22.1 years.
Revised: Q: The median age in the city was 22.1 years.

Original: 10.1% of residents were under the age of 18; 56.2% were between the ages of 18 and 24; 16.1% were from 25 to 44; 10.5% were from 45 to 64; and 7% were 65 years of age or older.
Revised: 10.1% of residents were under the age of 18; 56.2% were between the ages of 18 and 24; 16.1% were from 25 to 44; 10.5% were from 45 to 64; and 7% were 65 years of age or older.

Original: Which age group is larger: under the age of 18 or 18 and 24?
Revised: Which age group is larger: under the age of 18 or 18 and 24?

Original: A: To answer the question “Which age group is larger: under the age of 18 or 18 and 24?”, we need to know: “How many percent were under the age of 18?”, “How many percent were between the ages of 18 and 24?”.
Revised: A: To answer the question “Which age group is larger: under the age of 18 or 18 and 24?”, we need to know: “How many percent were under the age of 18?”, “How many percent were between the ages of 18 and 24?”.

Original: Q: Old age pensions were raised by 300 francs per month to 1,700 francs for a single person and to 3,700 francs for a couple, while health insurance beneﬁts were made more widely available to unemployed persons and part-time employees.
Revised: Q: Old age pensions were raised by 300 francs per month to 1,700 francs for a single person and to 3,700 francs for a couple, while health insurance beneﬁts were made more widely available to unemployed persons and part-time employees.

Original: How many francs were the old age pensions for a single person before they were raised?
Revised: How many francs were the old age pensions for a single person before they were raised?

Original: A: To answer the question “How many francs were the old age pensions for a single person before they were raised?”
Revised: A: To answer the question “How many francs were the old age pensions for a single person before they were raised?”

Original: , we need to know: “How many francs were the old age pensions for a single person?”, “How many francs were old age pensions raised for a single person?”.
Revised: , we need to know: “How many francs were the old age pensions for a single person?”, “How many francs were old age pensions raised for a single person?”.

Original: Q: In April 2011, the ECB raised interest rates for the ﬁrst time since 2008 from 1% to 1.25%, with a further increase to 1.50% in July 2011.
Revised: Q: In April 2011, the ECB raised interest rates for the ﬁrst time since 2008 from 1% to 1.25%, with a further increase to 1.50% in July 2011.

Original: However, in 2012-2013 the ECB lowered interest rates to encourage economic growth, reaching the historically low 0.25% in November 2013.
Revised: However, in 2012-2013 the ECB lowered interest rates to encourage economic growth, reaching the historically low 0.25% in November 2013.

Original: Soon after the rates were cut to 0.15%, then on 4 September 2014 the central bank reduced the rates from 0.15% to 0.05%, the lowest rates on record.
Revised: Soon after the rates were cut to 0.15%, then on 4 September 2014 the central bank reduced the rates from 0.15% to 0.05%, the lowest rates on record.

Original: How many percentage points did interest rates drop between
Revised: How many percentage points did interest rates drop between

Original: April 2011 and September 2014?
Revised: April 2011 and September 2014?

Original: A: To answer the question “How many percentage points did interest rates drop between Apriland September 2014?”
Revised: A: To answer the question “How many percentage points did interest rates drop between Apriland September 2014?”

Original: , we need to know: “What was the interest rate in April 2011?”, “What was the interest rate in September 2014?”.
Revised: , we need to know: “What was the interest rate in April 2011?”, “What was the interest rate in September 2014?”.

Original: Q: Non-nationals make up more than half of the population of Bahrain.
Revised: Q: Non-nationals make up more than half of the population of Bahrain.

Original: According to government statistics dated between 2005-2009 roughly 290,000 Indians, 125,000 Bangladeshis, 45,000 Pakistanis, 45,000 Filipinos, and 8,000 Indonesians.
Revised: According to government statistics dated between 2005-2009 roughly 290,000 Indians, 125,000 Bangladeshis, 45,000 Pakistanis, 45,000 Filipinos, and 8,000 Indonesians.

Original: How many Pakistanis and Indonesians are in Bahrain?
Revised: How many Pakistanis and Indonesians are in Bahrain?

Original: A: To answer the question “How many Pakistanis and Indonesians are in Bahrain?”, we need to know: “How many Pakistanis are in Bahrain?”, “How many Indonesians are in Bahrain?”.
Revised: A: To answer the question “How many Pakistanis and Indonesians are in Bahrain?”, we need to know: “How many Pakistanis are in Bahrain?”, “How many Indonesians are in Bahrain?”.

Original: Q: The Census Bureaus 2006-2010 American Community Survey showed that (in 2010 inﬂation adjustment dollars) median household income was $52,056 and the median family income was $58,942.
Revised: Q: The Census Bureaus 2006-2010 American Community Survey showed that (in 2010 inﬂation adjustment dollars) median household income was $52,056 and the median family income was $58,942.

Original: How many years did the Census Bureaus American Community Survey last?
Revised: How many years did the Census Bureaus American Community Survey last?

Original: A: To answer the question “How many years did the Census Bureaus American Community Survey last?”, we need to know: “When did the Census Bureaus American Community Survey start?”, “When did the Census Bureaus American Community Survey end?”.
Revised: A: To answer the question “How many years did the Census Bureaus American Community Survey last?”, we need to know: “When did the Census Bureaus American Community Survey start?”, “When did the Census Bureaus American Community Survey end?”.

Original: C.2. L EAST - TO - MOST PROMPTING II: PROBLEM SOLVING (3 EXAMPLES )
Revised: C.2. L EAST - TO - MOST PROMPTING II: PROBLEM SOLVING (3 EXAMPLES )

Original: The city of Bangkok has a population of 8,280,925.
Revised: The city of Bangkok has a population of 8,280,925.

Original: The census showed that it is home to 81,
Revised: The census showed that it is home to 81,

Original: Japanese and 55,893 Chinese nationals, as well as 117,071 expatriates from other Asian countries, 48,341 from Europe, 23,418 from the Americas.
Revised: Japanese and 55,893 Chinese nationals, as well as 117,071 expatriates from other Asian countries, 48,341 from Europe, 23,418 from the Americas.

Original: Q: How many Chinese nationals are there?
Revised: Q: How many Chinese nationals are there?

Original: A: “The census showed that it is home to 81,570 Japanese and 55,893 Chinese nationals”.
Revised: A: “The census showed that it is home to 81,570 Japanese and 55,893 Chinese nationals”.

Original: So the answer is 55,893 Chinese nationals.
Revised: So the answer is 55,893 Chinese nationals.

Original: Q: How many European nationals are there?
Revised: Q: How many European nationals are there?

Original: A: “48,341 from Europe”.
Revised: A: “48,341 from Europe”.

Original: So the answer is 48,341.
Revised: So the answer is 48,341.

Original: Q: How many more Chinese nationals are there than European nationals?
Revised: Q: How many more Chinese nationals are there than European nationals?

Original: A: We know that there are 55,893 Chinese nationals.
Revised: A: We know that there are 55,893 Chinese nationals.

Original: We also know that there are 48,341 European nationals.
Revised: We also know that there are 48,341 European nationals.

Original: So there are 55,893 - 48,341 = 7552 more Chinese nationals than European nationals.
Revised: So there are 55,893 - 48,341 = 7552 more Chinese nationals than European nationals.

Original: So the answer is 7552.
Revised: So the answer is 7552.

Original: The gender distribution of the population was 2% male and 49.8% female
Revised: The gender distribution of the population was 50.2% male and 49.8% female.

Original: Of the adult population, 29 people or 6% of the population are between 20 and 29 years old
Revised: Of the adult population, 29 people or 14.6% of the population are between 20 and 29 years old.

Original: 28 people or 14.1% are 30 to 39, 36 people or 18.2% are 40 to 49, and 31 people or  are  to 
Revised: 28 people or 14.1% are 30 to 39, 36 people or 18.2% are 40 to 49, and 31 people or 15.7% are 50 to 59.

Original: Q: What is the percentage of people between  to 49?.
Revised: Q: What is the percentage of people between 40 to 49?.

Original: A: “36 people or 18.2% are 40 to 49”.
Revised: A: “36 people or 18.2% are 40 to 49”.

Original: So the answer is 18.
Revised: So the answer is 18.

Original: Q: How many percent of people were not 40 to 49?
Revised: Q: How many percent of people were not 40 to 49?

Original: A:
Revised: A:

Original: We know that 18.2% are 40 to 49.
Revised: We know that 18.2% are 40 to 49.

Original: So 100% - 18.2% = 81.8% are not 40 to 49.
Revised: So 100% - 18.2% = 81.8% are not 40 to 49.

Original: So the answer is 81.8%.
Revised: So the answer is 81.8%.

Original: During the 1970s the oil shocks (1973 oil crisis and 1979 energy crisis) hit Iceland hard.
Revised: During the 1970s the oil shocks (1973 oil crisis and 1979 energy crisis) hit Iceland hard.

Original: Inﬂation rose to 43% in 1974 and 59% in 1980, falling to 15% in 1987 but rising to 30% in 1988.
Revised: Inﬂation rose to 43% in 1974 and 59% in 1980, falling to 15% in 1987 but rising to 30% in 1988.

Original: Iceland experienced moderately strong GDP growth (3% on average) from 1995 to 2004.
Revised: Iceland experienced moderately strong GDP growth (3% on average) from 1995 to 2004.

Original: Growth slowed between 2000 and 2002, but the economy expanded by 4.3% in 2003 and grew by 6.2% in 2004.
Revised: Growth slowed between 2000 and 2002, but the economy expanded by 4.3% in 2003 and grew by 6.2% in 2004.

Original: Growth in 2005 exceeded 6%.
Revised: Growth in 2005 exceeded 6%.

Original: Q: When did the oil crisis occur?
Revised: Q: When did the oil crisis occur?

Original: A: “1973 oil crisis and 1979 energy crisis”.
Revised: A: “1973 oil crisis and 1979 energy crisis”.

Original: So the answer is 1973.
Revised: So the answer is 1973.

Original: Q: When did the energy crisis occur?
Revised: Q: When did the energy crisis occur?

Original: A: “1979 energy crisis”.
Revised: A: “1979 energy crisis”.

Original: So the answer is 1979.
Revised: So the answer is 1979.

Original: Q: How many years passed between the oil and energy crises?
Revised: Q: How many years passed between the oil and energy crises?

Original: A:
Revised: A:

Original: We know that the oil crisis occurred in 1973.
Revised: We know that the oil crisis occurred in 1973.

Original: We also know that the energy crisis occurred in 1979.
Revised: We also know that the energy crisis occurred in 1979.

Original: So 1979 - 1973 = 6 years passed between the oil and energy crises.
Revised: So 1979 - 1973 = 6 years passed between the oil and energy crises.

Original: So the answer is 6.
Revised: So the answer is 6.

Original: C. F OOTBALL SUBSET C.3. Z ERO - SHOT PROMPTING
Revised: C. F OOTBALL SUBSET C.3. Z ERO - SHOT PROMPTING

Original: For zero-shot, the prompt format is as follows:
Revised: For zero-shot, the prompt format is as follows:

Original: Q: { question } A:
Revised: Q: { question } A:

Original: The answer is
Revised: The answer is

Original: Notice that we add “The answer is” at the beginning of the answer section.
Revised: Notice that we add “The answer is” at the beginning of the answer section.

Original: C.3. S TANDARD PROMPTING WITH 3 EXAMPLES
Revised: C.3. S TANDARD PROMPTING WITH 3 EXAMPLES

Original: Q: The Seahawks played the San Francisco 49ers.
Revised: Q: The Seahawks played the San Francisco 49ers.

Original: In the ﬁrst quarter, the Hawks RB Julius Jones got a 27-yard TD run, along with DT Craig Terrill returning a fumble 9 yards for a touchdown.
Revised: In the ﬁrst quarter, the Hawks RB Julius Jones got a 27-yard TD run, along with DT Craig Terrill returning a fumble 9 yards for a touchdown.

Original: In the third quarter, the 49ers almost rallied as RB H. J. Torres made a 12-yard TD pass to Lucas Nelly, along with Mare kicking a 32-yard ﬁeld goal.
Revised: In the third quarter, the 49ers almost rallied as RB H. J. Torres made a 12-yard TD pass to Lucas Nelly, along with Mare kicking a 32-yard ﬁeld goal.

Original: In the ﬁnal quarter, Julius Jones got another 11-yard
Revised: In the ﬁnal quarter, Julius Jones got another 11-yard

Original: TD.
Revised: TD.

Original: How many yards do the shortest touchdown run and the longest touchdown pass combine for?
Revised: How many yards do the shortest touchdown run and the longest touchdown pass combine for?

Original: A:
Revised: A:

Original: The answer is 21.
Revised: The answer is 21.

Original: Q: The Steelers went home for a duel with the Baltimore Ravens.
Revised: Q: The Steelers went home for a duel with the Baltimore Ravens.

Original: Pittsburgh would deliver the opening punch in the ﬁrst quarter with a 1-yard touchdown from running back Rashard Mendenhall.
Revised: Pittsburgh would deliver the opening punch in the ﬁrst quarter with a 1-yard touchdown from running back Rashard Mendenhall.

Original: The Ravens would make it even as running back Willis McGahee got a 9-yard TD.
Revised: The Ravens would make it even as running back Willis McGahee got a 9-yard TD.

Original: The Ravens kicker Billy Cundiff got a 45-yard ﬁeld goal in the second quarter, concluding the ﬁrst half with a 10-7 lead.
Revised: The Ravens kicker Billy Cundiff got a 45-yard ﬁeld goal in the second quarter, concluding the ﬁrst half with a 10-7 lead.

Original: The Steelers brought the game into overtime with a 38-yard ﬁeld goal by Andrew Foster.
Revised: The Steelers brought the game into overtime with a 38-yard ﬁeld goal by Andrew Foster.

Original: The Ravens Billy Cundiff pulled off a winning 33-yard ﬁeld goal in overtime.
Revised: The Ravens Billy Cundiff pulled off a winning 33-yard ﬁeld goal in overtime.

Original: How many points did the Ravens have at halftime?
Revised: How many points did the Ravens have at halftime?

Original: A:
Revised: A:

Original: The answer is 10.
Revised: The answer is 10.

Original: Q: The Vikings ﬂew to Bank of America Stadium to face the Carolina Panthers.
Revised: Q: The Vikings ﬂew to Bank of America Stadium to face the Carolina Panthers.

Original: After a scoreless ﬁrst quarter, Carolina got on the board with quarterback Matt Moore ﬁnding fullback Brad Hoover on a 1-yard TD pass.
Revised: After a scoreless ﬁrst quarter, Carolina got on the board with quarterback Matt Moore ﬁnding fullback Brad Hoover on a 1-yard TD pass.

Original: After yet another scoreless quarter, Carolina sealed the game as Matt Moore completed a 42-yard touchdown pass to wide receiver Steve Smith.
Revised: After yet another scoreless quarter, Carolina sealed the game as Matt Moore completed a 42-yard touchdown pass to wide receiver Steve Smith.

Original: How many scoreless quarters were there?
Revised: How many scoreless quarters were there?

Original: A:
Revised: A:

Original: The answer is 2.
Revised: The answer is 2.

Original: C.3. C HAIN - OF - THOUGHT PROMPTING WITH 3 EXAMPLES
Revised: C.3. C HAIN - OF - THOUGHT PROMPTING WITH 3 EXAMPLES

Original: Q: The Seahawks played the San Francisco 49ers.
Revised: Q: The Seahawks played the San Francisco 49ers.

Original: In the ﬁrst quarter, the Hawks RB Julius Jones got a 27-yard TD run, along with DT Craig Terrill returning a fumble 9 yards for a touchdown.
Revised: In the ﬁrst quarter, the Hawks RB Julius Jones got a 27-yard TD run, along with DT Craig Terrill returning a fumble 9 yards for a touchdown.

Original: In the third quarter, the 49ers almost rallied as RB H. J. Torres made a 12-yard TD pass to Lucas Nelly, along with Mare kicking a 32-yard ﬁeld goal.
Revised: In the third quarter, the 49ers almost rallied as RB H. J. Torres made a 12-yard TD pass to Lucas Nelly, along with Mare kicking a 32-yard ﬁeld goal.

Original: In the ﬁnal quarter, Julius Jones got another 11-yard
Revised: In the ﬁnal quarter, Julius Jones got another 11-yard

Original: TD.
Revised: TD.

Original: How many yards do the shortest touchdown run and the longest touchdown pass combine for?
Revised: How many yards do the shortest touchdown run and the longest touchdown pass combine for?

Original: A: All the touchdown runs are: a 27-yard touchdown run, a 9-yard touchdown run, a 11-yard touchdown run.
Revised: A: All the touchdown runs are: a 27-yard touchdown run, a 9-yard touchdown run, a 11-yard touchdown run.

Original: The smallest number among 27, 9, 11 is 9.
Revised: The smallest number among 27, 9, 11 is 9.

Original: So the shortest touchdown run was 9 yards.
Revised: So the shortest touchdown run was 9 yards.

Original: All the touchdown passes are: a 12-yard touchdown pass.
Revised: All the touchdown passes are: a 12-yard touchdown pass.

Original: So the longest touchdown pass wasyards.
Revised: So the longest touchdown pass wasyards.

Original: So the shortest touchdown run and the longest touchdown pass combine for 9 + 12 =yards.
Revised: So the shortest touchdown run and the longest touchdown pass combine for 9 + 12 =yards.

Original: So the answer is 21 yards.
Revised: So the answer is 21 yards.

Original: Q: The Steelers went home for a duel with the Baltimore Ravens.
Revised: Q: The Steelers went home for a duel with the Baltimore Ravens.

Original: Pittsburgh would deliver the opening punch in the ﬁrst quarter with a 1-yard touchdown from running back Rashard Mendenhall.The Ravens
Revised: Pittsburgh would deliver the opening punch in the ﬁrst quarter with a 1-yard touchdown from running back Rashard Mendenhall.

Original:  would make it even as running back Willis McGahee got a 9-yard TD.
Revised: The Ravens would make it even as running back Willis McGahee got a 9-yard TD.

Original: The Ravens kicker Billy Cundiff got a 45-yard ﬁeld goal in the second quarter, concluding the ﬁrst half with a 10-7 lead.
Revised: The Ravens kicker Billy Cundiff got a 45-yard ﬁeld goal in the second quarter, concluding the ﬁrst half with a 10-7 lead.

Original: The Steelers brought the game into overtime with a 38-yard ﬁeld goal by Andrew Foster.
Revised: The Steelers brought the game into overtime with a 38-yard ﬁeld goal by Andrew Foster.

Original: The Ravens Billy Cundiff pulled off a winning 33-yard ﬁeld goal in overtime.
Revised: The Ravens Billy Cundiff pulled off a winning 33-yard ﬁeld goal in overtime.

Original: How many points did the Ravens have at halftime?
Revised: How many points did the Ravens have at halftime?

Original: A: The Ravens kicker Billy Cundiff got a 45-yard ﬁeld goal in the second quarter, concluding the ﬁrst half with a 10-7 lead.
Revised: A: The Ravens kicker Billy Cundiff got a 45-yard ﬁeld goal in the second quarter, concluding the ﬁrst half with a 10-7 lead.

Original: So the Ravens had 10 points at halftime.
Revised: So the Ravens had 10 points at halftime.

Original: So the answer is 10 points.
Revised: So the answer is 10 points.

Original: Q: The Vikings ﬂew to Bank of America Stadium to face the Carolina Panthers.
Revised: Q: The Vikings ﬂew to Bank of America Stadium to face the Carolina Panthers.

Original: After a scoreless ﬁrst quarter, Carolina got on the board with quarterback Matt Moore ﬁnding fullback Brad Hoover on a 1-yard TD pass.
Revised: After a scoreless ﬁrst quarter, Carolina got on the board with quarterback Matt Moore ﬁnding fullback Brad Hoover on a 1-yard TD pass.

Original: After yet another scoreless quarter, Carolina sealed the game as Matt Moore completed a 42-yard touchdown pass to wide receiver Steve Smith.
Revised: After yet another scoreless quarter, Carolina sealed the game as Matt Moore completed a 42-yard touchdown pass to wide receiver Steve Smith.

Original: How many scoreless quarters were there?
Revised: How many scoreless quarters were there?

Original: A: The ﬁrst and third quarters were the scoreless quarters.
Revised: A: The ﬁrst and third quarters were the scoreless quarters.

Original: So there are 2 scoreless quarters.
Revised: So there are 2 scoreless quarters.

Original: So the answer is 2.
Revised: So the answer is 2.

Original: C.3.  L EAST - TO - MOST PROMPTING I: PROBLEM REDUCING (6 EXAMPLES )
Revised: C.3.  L EAST - TO - MOST PROMPTING I: PROBLEM REDUCING (6 EXAMPLES )

Original: Q: The Seahawks played the San Francisco 49ers.
Revised: Q: The Seahawks played the San Francisco 49ers.

Original: In the ﬁrst quarter, the ’Hawks RB Julius Jones got a 27-yard TD run, along with DT Craig Terrill returning a fumble 9 yards for a touchdown.
Revised: In the ﬁrst quarter, the ’Hawks RB Julius Jones got a 27-yard TD run, along with DT Craig Terrill returning a fumble 9 yards for a touchdown.

Original: In the third quarter, the 49ers almost rallied as RB T. J. Duckett made a 12-yard TD pass to Lucas
Revised: In the third quarter, the 49ers almost rallied as RB T. J. Duckett made a 12-yard TD pass to Lucas

Original: Nelly, along with Mare kicking a 32-yard ﬁeld goal.
Revised: Nelly, along with Mare kicking a 32-yard ﬁeld goal.

Original: How many yards do the shortest touchdown run and the longest touchdown pass combine for?
Revised: How many yards do the shortest touchdown run and the longest touchdown pass combine for?

Original: A: To answer the question “How many yards do the shortest touchdown run and the longest touchdown pass combine for?”
Revised: A: To answer the question “How many yards do the shortest touchdown run and the longest touchdown pass combine for?”

Original: , we need to know: “How many yards was the shortest touchdown run?”, “How many yards was the longest touchdown pass?”.
Revised: , we need to know: “How many yards was the shortest touchdown run?”, “How many yards was the longest touchdown pass?”.

Original: Q: The Steelers went home for an AFC North duel with the Baltimore Ravens.
Revised: Q: The Steelers went home for an AFC North duel with the Baltimore Ravens.

Original: Pittsburgh would deliver the opening punch in the ﬁrst quarter with a 1-yard touchdown run from running back Rashard
Revised: Pittsburgh would deliver the opening punch in the ﬁrst quarter with a 1-yard touchdown run from running back Rashard

Original: Mendenhall.
Revised: Mendenhall.

Original: The Ravens would make it even in the second quarter as running back Willis McGahee got a 9-yard touchdown run.
Revised: The Ravens would make it even in the second quarter as running back Willis McGahee got a 9-yard touchdown run.

Original: The Ravens kicker Billy Cundiff got a 45-yard ﬁeld goal in the second quarter and a 33-yard ﬁeld goal in the third quarter.
Revised: The Ravens kicker Billy Cundiff got a 45-yard ﬁeld goal in the second quarter and a 33-yard ﬁeld goal in the third quarter.

Original: Game ended with a scoreless fourth quarter.
Revised: Game ended with a scoreless fourth quarter.

Original: How many points did the Ravens have at halftime?
Revised: How many points did the Ravens have at halftime?

Original: A: To answer the question “How many points did the Ravens have at halftime?”
Revised: A: To answer the question “How many points did the Ravens have at halftime?”

Original: , we need to know: “What were all the scores the Ravens had at halftime?”.
Revised: , we need to know: “What were all the scores the Ravens had at halftime?”.

Original: Q: In 1995, the Kings overcame a 3-4 start to win eight of their ﬁnal nine games and ﬁnished with a record, the second-best in the AFC.
Revised: Q: In 1995, the Kings overcame a 3-4 start to win eight of their ﬁnal nine games and ﬁnished with a record, the second-best in the AFC.

Original: Quarterback Neil ODonnell, who completed 246 out ofpasses for 2,970 yards and 17 touchdowns, with only seven interceptions led their offense.
Revised: Quarterback Neil ODonnell, who completed 246 out ofpasses for 2,970 yards and 17 touchdowns, with only seven interceptions led their offense.

Original: The
Revised: The

Original: Kings ﬁnished their 1995 season having lost how many games difference to the number of games they had won?
Revised: Kings ﬁnished their 1995 season having lost how many games difference to the number of games they had won?

Original: A: To answer the question “The Kings ﬁnished their 1995 season having lost how many games difference to the number of games they had won?”, we need to know: “How many games the Kings had lost in their 1995 season?”, “How many games the Kings had won in their 1995 season?”.
Revised: A: To answer the question “The Kings ﬁnished their 1995 season having lost how many games difference to the number of games they had won?”, we need to know: “How many games the Kings had lost in their 1995 season?”, “How many games the Kings had won in their 1995 season?”.

Original: Q: The Broncos traveled to Sun Life Stadium to face the Miami Dolphins.
Revised: Q: The Broncos traveled to Sun Life Stadium to face the Miami Dolphins.

Original: The Dolphins grabbed the lead in the second quarter, with ﬁeld goals of 35 and 45 yards by kicker Dan Carpenter.
Revised: The Dolphins grabbed the lead in the second quarter, with ﬁeld goals of 35 and 45 yards by kicker Dan Carpenter.

Original: In the ﬁnal quarter, the Dolphins quarterback Matt Moore threw a 16-yard touchdown pass to tight end Anthony
Revised: In the ﬁnal quarter, the Dolphins quarterback Matt Moore threw a 16-yard touchdown pass to tight end Anthony

Original: Fasano (with a failed two-point conversion attempt), followed by a 35-yard ﬁeld goal by Carpenter, giving Miami a 15-0 lead.
Revised: Fasano (with a failed two-point conversion attempt), followed by a 35-yard ﬁeld goal by Carpenter, giving Miami a 15-0 lead.

Original: Finally, the Broncos answered with Alex Jake kicking a 48-yard ﬁeld goal, ending the game with 15-3.
Revised: Finally, the Broncos answered with Alex Jake kicking a 48-yard ﬁeld goal, ending the game with 15-3.

Original: How many yards was Dan Carpenter’s longest ﬁeld goal?
Revised: How many yards was Dan Carpenter’s longest ﬁeld goal?

Original: A: To answer the question “How many yards was Dan Carpenter’s longest ﬁeld goal?”, we need to know: “What were all Dan Carpenter’s ﬁeld goals?”.
Revised: A: To answer the question “How many yards was Dan Carpenter’s longest ﬁeld goal?”, we need to know: “What were all Dan Carpenter’s ﬁeld goals?”.

Original: Q: The Cardinals ﬂew to Gillette Stadium for a duel with the New England Patriots.
Revised: Q: The Cardinals ﬂew to Gillette Stadium for a duel with the New England Patriots.

Original: In the fourth quarter, New England ended the boring game with Gostkowski’s 30-yard ﬁeld goal.
Revised: In the fourth quarter, New England ended the boring game with Gostkowski’s 30-yard ﬁeld goal.

Original: Original starting quarterback Kurt Warner (6/18 for 30 yards) was pulled in the third quarter via coach’s decision.
Revised: Original starting quarterback Kurt Warner (6/18 for 30 yards) was pulled in the third quarter via coach’s decision.

Original: How many incomplete passes did Kurt Warner have?
Revised: How many incomplete passes did Kurt Warner have?

Original: A: To answer the question “How many incomplete passes did Kurt Warner have?”, we need to know: “How many passes did Kurt Warner attempt?”, “How many passes did Kurt Warner complete?”.
Revised: A: To answer the question “How many incomplete passes did Kurt Warner have?”, we need to know: “How many passes did Kurt Warner attempt?”, “How many passes did Kurt Warner complete?”.

Original: Q: The Vikings ﬂew to Bank of America Stadium to face the Carolina Panthers.
Revised: Q: The Vikings ﬂew to Bank of America Stadium to face the Carolina Panthers.

Original: After a scoreless ﬁrst quarter, Carolina got on the board with quarterback Matt Moore ﬁnding fullback Brad Hoover on a 1-yard touchdown pass.
Revised: After a scoreless ﬁrst quarter, Carolina got on the board with quarterback Matt Moore ﬁnding fullback Brad Hoover on a 1-yard touchdown pass.

Original: After yet another scoreless quarter, Carolina sealed the game as Matt
Revised: After yet another scoreless quarter, Carolina sealed the game as Matt

Original: Moore completed a 42-yard TD pass to wide receiver Steve Smith.
Revised: Moore completed a 42-yard TD pass to wide receiver Steve Smith.

Original: How many scoreless quarters were there?
Revised: How many scoreless quarters were there?

Original: A: To answer the question “How many scoreless quarters were there?”
Revised: A: To answer the question “How many scoreless quarters were there?”

Original: , we need to know: “What were all the scoreless quarters?”.
Revised: , we need to know: “What were all the scoreless quarters?”.

Original: C.3. L EAST - TO - MOST PROMPTING II: PROBLEM SOLVING (3 EXAMPLES )
Revised: C.3. L EAST - TO - MOST PROMPTING II: PROBLEM SOLVING (3 EXAMPLES )

Original: The Seahawks played the San Francisco 49ers.
Revised: The Seahawks played the San Francisco 49ers.

Original: In the ﬁrst quarter, the ’Hawks RB Julius Jones got a 27-yard TD run, along with DT Craig Terrill returning a fumble 9 yards for a touchdown.
Revised: In the ﬁrst quarter, the ’Hawks RB Julius Jones got a 27-yard TD run, along with DT Craig Terrill returning a fumble 9 yards for a touchdown.

Original: In the third quarter, the 49ers almost rallied as RB H. J. Torres made a 12-yard TD pass to Lucas Nelly, along with Mare kicking a 32-yard ﬁeld goal.
Revised: In the third quarter, the 49ers almost rallied as RB H. J. Torres made a 12-yard TD pass to Lucas Nelly, along with Mare kicking a 32-yard ﬁeld goal.

Original: In the ﬁnal quarter, Julius Jones got another 11-yard
Revised: In the ﬁnal quarter, Julius Jones got another 11-yard

Original: TD.
Revised: TD.

Original: Q:How many yards was the shortest touchdown run?
Revised: Q: How many yards was the shortest touchdown run?

Original: A: All the touchdown runs are: a 27-yard touchdown run, a 9-yard touchdown run, a 11-yard touchdown run.
Revised: A: All the touchdown runs are: a 27-yard touchdown run, a 9-yard touchdown run, a 11-yard touchdown run.

Original: The smallest number between 27, 9, 11 is 9.
Revised: The smallest number between 27, 9, 11 is 9.

Original: The shortest touchdown run was 9 yards.
Revised: The shortest touchdown run was 9 yards.

Original: The answer is 
Revised: The answer is 9. Q

Original: Q: How many yards was the longest touchdown pass?
Revised: : How many yards was the longest touchdown pass

Original: A: All the touchdown passes are: a 12-yard touchdown pass.
Revised: A: All the touchdown passes are: a 12-yard touchdown pass.

Original: The longest touchdown pass wasyards.
Revised: The longest touchdown pass wasyards.

Original: The answer is 12.
Revised: The answer is 12.

Original: Q: How many yards do the shortest touchdown run and the longest touchdown pass combine for?
Revised: Q: How many yards do the shortest touchdown run and the longest touchdown pass combine for?

Original: A: We know that the shortest touchdown run was 9-yard.
Revised: A: We know that the shortest touchdown run was 9-yard.

Original: We also know that the longest touchdown pass was 12-yard.
Revised: We also know that the longest touchdown pass was 12-yard.

Original: 9 + 12 = 21.
Revised: 9 + 12 = 21.

Original: The answer is 21.
Revised: The answer is 21.

Original: The Steelers went home for a duel with the Baltimore Ravens.
Revised: The Steelers went home for a duel with the Baltimore Ravens.

Original: Pittsburgh would deliver the opening punch in the ﬁrst quarter with a 1-yard touchdown from running back Rashard Mendenhall.
Revised: Pittsburgh would deliver the opening punch in the ﬁrst quarter with a 1-yard touchdown from running back Rashard Mendenhall.

Original: The
Revised: The

Original: Ravens would make it even as running back Willis McGahee got a 9-yard TD.
Revised: Ravens would make it even as running back Willis McGahee got a 9-yard TD.

Original: The Ravens kicker
Revised: The Ravens kicker

Original: Billy Cundiff got a 45-yard ﬁeld goal in the second quarter, concluding the ﬁrst half.
Revised: Billy Cundiff got a 45-yard ﬁeld goal in the second quarter, concluding the ﬁrst half.

Original: The Steelers brought the game into overtime with a 38-yard ﬁeld goal by Andrew Foster.
Revised: The Steelers brought the game into overtime with a 38-yard ﬁeld goal by Andrew Foster.

Original: The Ravens Billy
Revised: The Ravens Billy

Original: Cundiff pulled off a winning 33-yard ﬁeld goal in overtime. 
Revised: Cundiff pulled off a winning 33-yard ﬁeld goal in overtime. Q:

Original: Q: What were all the scores the Ravens had athalftime?
Revised: What were all the scores the Ravens had at

Revised: halftime

Original: A:
Revised: A:

Original: The answer is: a 9-yard touchdown run, a 45-yard ﬁeld goal.
Revised: The answer is a 9-yard touchdown run, a -yard ﬁeld goal.

Original: Q: How many points did the Ravens have at halftime?
Revised: Q: How many points did the Ravens have at halftime?

Original: A:
Revised: A

Original:  We know that the Ravens had a touchdown and a ﬁeld goal at halftime.
Revised: : We know that the Ravens had a touchdown and a ﬁeld goal at halftime.

Original: We also know that a touchdown is worth 7 points and a ﬁeld goal is worth 3 points.
Revised: We also know that a touchdown is worth 7 points and a ﬁeld goal is worth 3 points.

Original: 7 + 3 = 10.
Revised: 7 + 3 = 10.

Original: The answer is 10.
Revised: The answer is 10.

Original: The Vikings ﬂew to Bank of America Stadium to face the Carolina Panthers.
Revised: The Vikings ﬂew to Bank of America Stadium to face the Carolina Panthers.

Original: After a scoreless ﬁrst quarter, Carolina got on the board with quarterback Matt Moore ﬁnding fullback Brad Hoover on a 1-yard TD pass.
Revised: After a scoreless ﬁrst quarter, Carolina got on the board with quarterback Matt Moore ﬁnding fullback Brad Hoover on a 1-yard TD pass.

Original: After yet another scoreless quarter, Carolina sealed the game as Matt Moore completed a 42-yard touchdown pass to wide receiver Steve Smith.
Revised: After yet another scoreless quarter, Carolina sealed the game as Matt Moore completed a 42-yard touchdown pass to wide receiver Steve Smith.

Original: Q: What were all the scoreless quarters?
Revised: Q: What were all the scoreless quarters?

Original: A:
Revised: A:

Original: The answer is: the ﬁrst quarter and the third quarter.
Revised: The answer is: the ﬁrst quarter and the third quarter.

Original: Q: How many scoreless quarters were there?
Revised: Q: How many scoreless quarters were there?

Original: A: We know that the ﬁrst and third quarters were all the scoreless quarters.
Revised: A: We know that the ﬁrst and third quarters were all the scoreless quarters.

Original: The answer is 2.
Revised: The answer is 2.

Original: C. E XAMPLES WHERE LEAST - TO - MOST SUCCEEDED BUT CHAIN - OF - THOUGHT FAILED C.4. C ASE
Revised: C. E XAMPLES WHERE LEAST - TO - MOST SUCCEEDED BUT CHAIN - OF - THOUGHT FAILED C.4. C ASE

Original: Q: The ﬁrst two Buick automobiles were made in 1899 and 1900 at the “Buick Auto-Vim and Power
Revised: Q: The ﬁrst two Buick automobiles were made in 1899 and 1900 at the “Buick Auto-Vim and Power

Original: Company” by chief-engineer Walter Lorenzo Marr, but company owner David Dunbar Buick was reluctant to begin making automobiles, being satisﬁed with stationary and marine engine production, so Marr left Buick in 1901 to found Marr (automobile) under his own name.
Revised: Company” by chief-engineer Walter Lorenzo Marr, but company owner David Dunbar Buick was reluctant to begin making automobiles, being satisﬁed with stationary and marine engine production, so Marr left Buick in 1901 to found Marr (automobile) under his own name.

Original: His replacement was
Revised: His replacement was

Original: Eugene Richard, who applied for a patent in 1902 for Marrs valve-in-head (Overhead valve engine) engine, which patent, number 771,095, was awarded to Richard in the name of Buick in 1904.
Revised: Eugene Richard, who applied for a patent in 1902 for Marrs valve-in-head (Overhead valve engine) engine, which patent, number 771,095, was awarded to Richard in the name of Buick in 1904.

Original: In 1903, the third Buick automobile was made, this time by Richard, but in 1904 Buick, whose company was now called “Buick Motor Company,” moved to Flint, Michigan, and Richard stayed behind.
Revised: In 1903, the third Buick automobile was made, this time by Richard, but in 1904 Buick, whose company was now called “Buick Motor Company,” moved to Flint, Michigan, and Richard stayed behind.

Original: Marr was rehired in Flint as chief engineer, to begin making automobiles in production.
Revised: Marr was rehired in Flint as chief engineer, to begin making automobiles in production.