Original: L EAST - TO -M OST P ROMPTING E NABLES C OMPLEX R EASONING IN L ARGE L ANGUAGE M ODELS
Revised: L EAST - TO -M OST P ROMPTING E NABLES C OMPLEX R EASONING IN L ARGE L ANGUAGE M ODELS

Original: A BSTRACT
Revised: A BSTRACT

Original: Chain-of-thought prompting has demonstrated remarkable performance on various natural language reasoning tasks.
Revised: Chain-of-thought prompting has demonstrated remarkable performance on various natural language reasoning tasks.

Original: However, it tends to perform poorly on tasks which requires solving problems harder than the exemplars shown in the prompts.
Revised: However, it tends to perform poorly on tasks which requires solving problems harder than the exemplars shown in the prompts.

Original: To overcome this challenge of easy-to-hard generalization, we propose a novel prompting strategy, least-to-most prompting .
Revised: To overcome this challenge of easy-to-hard generalization, we propose a novel prompting strategy, least-to-most prompting .

Original: The key idea in this strategy is to break down a complex problem into a series of simpler subproblems and then solve them in sequence.
Revised: The key idea in this strategy is to break down a complex problem into a series of simpler subproblems and then solve them in sequence.

Original: Solving each subproblem is facilitated by the answers to previously solved subproblems.
Revised: Solving each subproblem is facilitated by the answers to previously solved subproblems.

Original: Our experimental results on tasks related to symbolic manipulation, compositional generalization and math reasoning reveal that least-to-most prompting is capable of generalizing to more difﬁcult problems than those seen in the prompts.
Revised: Our experimental results on tasks related to symbolic manipulation, compositional generalization, and math reasoning reveal that least-to-most prompting is capable of generalizing to more difﬁcult problems than those seen in the prompts.

Original: A notable ﬁnding is that when the GPTcode-davinci-002 model is used with least-to-most prompting, it can solve the compositional generalization benchmark SCAN in any split (including length split) with an accuracy of at least 99% using just 14 exemplars, compared to only 16% accuracy with chain-of-thought prompting.
Revised: A notable ﬁnding is that when the GPTcode-davinci-002 model is used with least-to-most prompting, it can solve the compositional generalization benchmark SCAN in any split (including length split) with an accuracy of at least 99% using just 14 exemplars, compared to only 16% accuracy with chain-of-thought prompting.

Original: This is particularly noteworthy because neural-symbolic models in the literature that specialize in solving SCAN are trained on the entire training set containing over ,000 examples
Revised: This is particularly noteworthy because neural-symbolic models in the literature that specialize in solving SCAN are trained on the entire training set containing over 15,000 examples.

Original: We have included prompts for all the tasks in the Appendix.
Revised: We have included prompts for all the tasks in the Appendix.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Despite the great success of deep learning in the past decade, there still remain huge differences between human intelligence and machine learning: (1) Given a new task, humans usually can learn to accomplish it from only a few demonstration examples, while machine learning requires a large amount of labeled data for model training; (2) Humans can clearly explain the underlying rationale for their predictions or decisions, while machine learning is essentially a black box; (3) Humans can solve problems more difﬁcult than any they have seen before, while for machine learning, examples in training and testing are typically at the same level of difﬁculty.
Revised: Despite the great success of deep learning in the past decade, there still remain huge differences between human intelligence and machine learning: (1) Given a new task, humans usually can learn to accomplish it from only a few demonstration examples, while machine learning requires a large amount of labeled data for model training; (2) Humans can clearly explain the underlying rationale for their predictions or decisions, while machine learning is essentially a black box; (3) Humans can solve problems more difﬁcult than any they have seen before, while for machine learning, examples in training and testing are typically at the same level of difﬁculty.

Original: The recently proposed chain-of-thought prompting approach (Wei et al., 2022; Chowdhery et al., 2022) has taken a signiﬁcant step for narrowing the gap between human intelligence and machine intelligence.
Revised: The recently proposed chain-of-thought prompting approach (Wei et al., 2022; Chowdhery et al., 2022) has taken a signiﬁcant step for narrowing the gap between human intelligence and machine intelligence.

Original: It combines the idea of natural language rationales (Ling et al., 2017; Cobbe et al., 2021) with few-shot prompting (Brown et al., 2020).
Revised: It combines the idea of natural language rationales (Ling et al., 2017; Cobbe et al., 2021) with few-shot prompting (Brown et al., 2020).

Original: When further integrated with self-consistency decoding (Wang et al., 2022b) rather than using the typical greedy decoding, few-shot chain-of-thought prompting largely outperforms the state-of-the-art results in the literature on many challenging natural language processing tasks obtained from specially designed neural models trained with hundreds of times more annotated examples, while being fully interpretable.
Revised: When further integrated with self-consistency decoding (Wang et al., 2022b) rather than using the typical greedy decoding, few-shot chain-of-thought prompting largely outperforms the state-of-the-art results in the literature on many challenging natural language processing tasks obtained from specially designed neural models trained with hundreds of times more annotated examples, while being fully interpretable.

Original: However, chain-of-thought prompting has a key limitation—it often performs poorly on tasks that require generalization of solving problems harder than the demonstration examples, such as compositional generalization (Lake & Baroni, 2018; Keysers et al., 2020).
Revised: However, chain-of-thought prompting has a key limitation—it often performs poorly on tasks that require generalization of solving problems harder than the demonstration examples, such as compositional generalization (Lake & Baroni, 2018; Keysers et al., 2020).

Original: To tackle such easy-to-hard generalization issues, we propose least-to-most prompting .
Revised: To tackle such easy-to-hard generalization issues, we propose least-to-most prompting .

Original: It consists of two stages:
Revised: It consists of two stages:

Original: ﬁrst decomposing a complex problem into a list of easier subproblems, and then sequentially solving these subproblems, whereby solving a given subproblem is facilitated by the answers to previously solved subproblems.
Revised: ﬁrst decomposing a complex problem into a list of easier subproblems, and then sequentially solving these subproblems, whereby solving a given subproblem is facilitated by the answers to previously solved subproblems.

Original: Both stages are implemented by few-shot prompting, so that there is no training or ﬁnetuning in either stage.
Revised: Both stages are implemented by few-shot prompting, so that there is no training or ﬁnetuning in either stage.

Original: An example usage of least-to-most prompting is illustrated in Figure 1.
Revised: An example usage of least-to-most prompting is illustrated in Figure 1.

Original: The term least-to-most prompting is borrowed from educational psychology (Libby et al., 2008), where it is used to denote the technique of using a progressive sequence of prompts to help a student to learn a new skill.
Revised: The term least-to-most prompting is borrowed from educational psychology (Libby et al., 2008), where it is used to denote the technique of using a progressive sequence of prompts to help a student to learn a new skill.

Original: Here we apply this technique for teaching humans to teach language models.
Revised: Here we apply this technique for teaching humans to teach language models.

Original: Empirical results on symbolic manipulation, compositional generalization, and math reasoning show that least-to-most prompting can indeed generalize to problems harder than those demonstrated.
Revised: Empirical results on symbolic manipulation, compositional generalization, and math reasoning show that least-to-most prompting can indeed generalize to problems harder than those demonstrated.

Original:  L EAST - TO -M OST P ROMPTING
Revised:  L EAST - TO -M OST P ROMPTING

Original: Least-to-most prompting teaches language models how to solve a complex problem by decomposing it to a series of simpler subproblems.
Revised: Least-to-most prompting teaches language models how to solve a complex problem by decomposing it to a series of simpler subproblems.

Original: It consists of two sequential stages:
Revised: It consists of two sequential stages:

Original: 1. Decomposition .
Revised: 1. Decomposition .

Original: The prompt in this stage contains constant examples that demonstrate the decomposition, followed by the speciﬁc question to be decomposed.
Revised: The prompt in this stage contains constant examples that demonstrate the decomposition, followed by the speciﬁc question to be decomposed.

Original: Subproblem solving .
Revised: Subproblem solving .

Original: The prompt in this stage consists of three parts: (1) constant examples demonstrating how subproblems are solved; (2) a potentially empty list of previously answered subquestions and generated solutions, and (3) the question to be answered next.
Revised: The prompt in this stage consists of three parts: (1) constant examples demonstrating how subproblems are solved; (2) a potentially empty list of previously answered subquestions and generated solutions, and (3) the question to be answered next.

Original: In the example shown in Figure 1, the language model is ﬁrst asked to decompose the original problem into subproblems.
Revised: In the example shown in Figure 1, the language model is ﬁrst asked to decompose the original problem into subproblems.

Original: The prompt that is passed to the model consists of examples that illustrate how to decompose complex problems (which are not shown in the ﬁgure), followed by the speciﬁc problem to be decomposed (as shown in the ﬁgure).
Revised: The prompt that is passed to the model consists of examples that illustrate how to decompose complex problems (which are not shown in the ﬁgure), followed by the speciﬁc problem to be decomposed (as shown in the ﬁgure).

Original: The language model ﬁgures out that the original problem can be solved via solving an intermediate problem “How long does each trip take?”.
Revised: The language model ﬁgures out that the original problem can be solved via solving an intermediate problem “How long does each trip take?”.

Original: In the next phase, we ask the language model to sequentially solve the subproblems from the problem decomposition stage.
Revised: In the next phase, we ask the language model to sequentially solve the subproblems from the problem decomposition stage.

Original: The original problem is appended as the ﬁnal subproblem.
Revised: The original problem is appended as the ﬁnal subproblem.

Original: The solving starts from passing to the language model a prompt that consists of examples that illustrate how problems are solved (not shown in the ﬁgure), followed by the ﬁrst subproblem “How long does each trip take?”.
Revised: The solving starts from passing to the language model a prompt that consists of examples that illustrate how problems are solved (not shown in the ﬁgure), followed by the ﬁrst subproblem “How long does each trip take?”.

Original: We then take the answer generated by the language model (“... each trip takes 5 minutes.”) and construct the next prompt by appending the generated answer to the previous prompt, followed by the next subproblem, which happens to be the original problem in this example.
Revised: We then take the answer generated by the language model (“... each trip takes 5 minutes.”) and construct the next prompt by appending the generated answer to the previous prompt, followed by the next subproblem, which happens to be the original problem in this example.

Original: The new prompt is then passed back to the language model, which returns the ﬁnal answer.
Revised: The new prompt is then passed back to the language model, which returns the ﬁnal answer.

Original: Least-to-most prompting can be combined with other prompting techniques like chain-of-thought (Wei et al., 2022) and self-consistency (Wang et al., 2022b), but does not need to be.
Revised: Least-to-most prompting can be combined with other prompting techniques like chain-of-thought (Wei et al., 2022) and self-consistency (Wang et al., 2022b), but does not need to be.

Original: Also, for some tasks, the two stages in least-to-most prompting can be merged to form a single-pass prompt.
Revised: Also, for some tasks, the two stages in least-to-most prompting can be merged to form a single-pass prompt.

Original:  R ESULTS
Revised:  R ESULTS

Original: We present least-to-most prompting results for symbolic manipulation, compositional generalization, and math reasoning tasks.
Revised: We present least-to-most prompting results for symbolic manipulation, compositional generalization, and math reasoning tasks.

Original: We start with a symbolic manipulation task, since it is quite simple, while being able to illustrate the key ideas in least-to-most prompting.
Revised: We start with a symbolic manipulation task, since it is quite simple, while being able to illustrate the key ideas in least-to-most prompting.

Original: We will see that language models seem to understand recursion by only observing a two-step demonstration.
Revised: We will see that language models seem to understand recursion by only observing a two-step demonstration.

Original: 3. S YMBOLIC M ANIPULATION
Revised: 3. S YMBOLIC M ANIPULATION

Original: Consider the last-letter-concatenation task (Wei et al., 2022), where the input is a list of words, and the output is the concatenation of the last letters of the words in the list.
Revised: Consider the last-letter-concatenation task (Wei et al., 2022), where the input is a list of words, and the output is the concatenation of the last letters of the words in the list.

Original: For example, “thinking, machine” outputs “ge”, since the last letter of “thinking” is ”g” and the last letter of “machine” is “e”.
Revised: For example, “thinking, machine” outputs “ge”, since the last letter of “thinking” is ”g” and the last letter of “machine” is “e”.

Original: The technique that we present for solving the last-letter-concatenation task can be directly adapted to solve other symbolic manipulation tasks such as reversing a list of words.
Revised: The technique that we present for solving the last-letter-concatenation task can be directly adapted to solve other symbolic manipulation tasks such as reversing a list of words.

Original: The experiment setup is as follows: (1) only two demonstration examples are provided; (2) the lists used for demonstration contain at most three words, while the test lists contain four or more words.
Revised: The experiment setup is as follows: (1) only two demonstration examples are provided; (2) the lists used for demonstration contain at most three words, while the test lists contain four or more words.

Original: This task is trivial for humans, but it is supposed to be hard even for advanced statistical learning methods: (1) models trained with two examples are not expected to generalize; (2) moreover, the length-based train-test split requires out-of-distribution generalization, while machine learning is essentially built on the assumption of identical distributions (Vapnik, 1999).
Revised: This task is trivial for humans, but it is supposed to be hard even for advanced statistical learning methods: (1) models trained with two examples are not expected to generalize; (2) moreover, the length-based train-test split requires out-of-distribution generalization, while machine learning is essentially built on the assumption of identical distributions (Vapnik, 1999).

Original: Chain-of-thought prompting
Revised: Chain-of-thought prompting

Original: Least
Revised: Least

Original: [Table]
Revised: [Table]

Original: Q: “think, machine”
Revised: Q: “think, machine”

Original: A: The last letter of “think” is “k”.
Revised: A: The last letter of “think” is “k”.

Original: The last letter of “machine” is “e”.
Revised: The last letter of “machine” is “e”.

Original: Concatenating “k”, “e” leads to “ke”.
Revised: Concatenating “k”, “e” leads to “ke”.

Original: So, “think, machine” outputs “ke”.
Revised: So, “think, machine” outputs “ke”.

Original: Q
Revised: Q

Original: [Table]
Revised: [Table]

Original: “learning, reasoning, generalization” A: The last letter of “learning” is “g”.
Revised: “learning, reasoning, generalization” A: The last letter of “learning” is “g”.

Original: The last letter of “reasoning” is “g”.
Revised: The last letter of “reasoning” is “g”.

Original: The last letter of “generalization” is “n”.
Revised: The last letter of “generalization” is “n”.

Original: Concatenating
Revised: Concatenating

Original: “g”, “g”, “n” leads to “ggn”.
Revised: “g”, “g”, “n” leads to “ggn”.

Original: So, “learning, reasoning, generalization” outputs “ggn”.
Revised: So, “learning, reasoning, generalization” outputs “ggn”.

Original: Q: “think, machine”
Revised: Q: “think, machine”

Original: A: The last letter of “think” is “k”.
Revised: A: The last letter of “think” is “k”.

Original: The last letter of “machine” is “e”.
Revised: The last letter of “machine” is “e”.

Original: Concatenating “k”, “e” leads to “ke”.
Revised: Concatenating “k”, “e” leads to “ke”.

Original: So, “think, machine” outputs “ke”.
Revised: So, “think, machine” outputs “ke”.

Original: Q: “think, machine, learning” A: “think, machine” outputs “ke”.
Revised: Q: “think, machine, learning” A: “think, machine” outputs “ke”.

Original: The last letter of “learning” is “g”.
Revised: The last letter of “learning” is “g”.

Original: Concatenating “ke”, “g” leads to “keg”.
Revised: Concatenating “ke”, “g” leads to “keg”.

Original: So, “think, machine, learning” outputs “keg”.
Revised: So, “think, machine, learning” outputs “keg”.

Original: Least-to-most
Revised: Least-to-most

Original: We need to deﬁne the implementations for both the problem decomposi tion and solving stages.
Revised: We need to deﬁne the implementations for both the problem decomposi tion and solving stages.

Original: The subproblems here are simply the preﬁxes of the list.
Revised: The subproblems here are simply the preﬁxes of the list.

Original: For example, given a list “think, machine, learning, reasoning”, the least-to-most subproblems are: “think, machine”, “think, machine, learning”, and “think, machine, learning, reasoning”.
Revised: For example, given a list “think, machine, learning, reasoning”, the least-to-most subproblems are: “think, machine”, “think, machine, learning”, and “think, machine, learning, reasoning”.

Original: Such a simple decomposition can be done by a manually written script or by prompting language models (see Appendix 6.
Revised: Such a simple decomposition can be done by a manually written script or by prompting language models (see Appendix 6.

Original: for details about the prompting-based decomposition).
Revised: for details about the prompting-based decomposition).

Original: The constant prompt context for solving the subproblems is presented in the right column of Table 1.
Revised: The constant prompt context for solving the subproblems is presented in the right column of Table 1.

Original: Essentially, this prompt teaches a language model to build answers to new problems using the answers to previously solved problems: (1) the list in the second example (“think, machine, learning”) is an extension of the list in the ﬁrst example (“think, machine”) rather than an independent example; (2) the answer to the second example copies the output from the ﬁrst one by starting with a sentence that says “think, machine” outputs “ke”.
Revised: Essentially, this prompt teaches a language model to build answers to new problems using the answers to previously solved problems: (1) the list in the second example (“think, machine, learning”) is an extension of the list in the ﬁrst example (“think, machine”) rather than an independent example; (2) the answer to the second example copies the output from the ﬁrst one by starting with a sentence that says “think, machine” outputs “ke”.

Original: Although this copy function is trivial for humans, we found that not all language models (e.g. text-curie-001 in GPT-3) are able to consistently support copying, which may cause least-to-most prompting to fail.
Revised: Although this copy function is trivial for humans, we found that not all language models (e.g. text-curie-001 in GPT-3) are able to consistently support copying, which may cause least-to-most prompting to fail.

Original: Q: “mind, emerge” A:
Revised: Q: “mind, emerge” A:

Original: The last
Revised: The last

Original: letter
Revised: letter

Original: of “mind” is “d”.
Revised: of “mind” is “d”.

Original: The last letter of “emerge” is ”e”.
Revised: The last letter of “emerge” is ”e”.

Original: Concatenating “d”, “e” leads to “de”.
Revised: Concatenating “d”, “e” leads to “de”.

Original: So, “mind, emerge” outputs “de”.
Revised: So, “mind, emerge” outputs “de”.

Original: Q: “mind, emerge, from” A: “mind, emerge” outputs “de”.
Revised: Q: “mind, emerge, from” A: “mind, emerge” outputs “de”.

Original: The
Revised: The

Original: last letter of “from” is “m”.
Revised: last letter of “from” is “m”.

Original: Concatenating “de”, “m” leads to “dem”.
Revised: Concatenating “de”, “m” leads to “dem”.

Original: So, “mind, emerge, from” outputs “dem”.
Revised: So, “mind, emerge, from” outputs “dem”.

Original: Q: “mind, emerge, from, neuron” A: “mind, emerge, from” outputs “dem”.
Revised: Q: “mind, emerge, from, neuron” A: “mind, emerge, from” outputs “dem”.

Original: The last letter of “neuron” is “n”.
Revised: The last letter of “neuron” is “n”.

Original: Concatenating “dem”, “n” leads to “demn”.
Revised: Concatenating “dem”, “n” leads to “demn”.

Original: So, “mind, emerge, from, neuron” outputs “demn”.
Revised: So, “mind, emerge, from, neuron” outputs “demn”.

Original: [Table]
Revised: [Table]

Original: Results .
Revised: Results .

Original: Standard prompting and chain-of-thought prompting are used as the baselines.
Revised: Standard prompting and chain-of-thought prompting are used as the baselines.

Original: The chainof-thought and least-to-most prompts are listed in Table 1.
Revised: The chainof-thought and least-to-most prompts are listed in Table 1.

Original: The prompt for standard prompting is formed by removing the rationales from the chain-of-thought prompt.
Revised: The prompt for standard prompting is formed by removing the rationales from the chain-of-thought prompt.

Original: All the methods are 2shot where one example uses a list of 2 words and the other example uses a list of 3 words.
Revised: All the methods are 2shot where one example uses a list of 2 words and the other example uses a list of 3 words.

Original: We randomly sample words in Wiktionary 1 to construct testing lists with lengths varying from 4 to 12.
Revised: We randomly sample words in Wiktionary 1 to construct testing lists with lengths varying from 4 to 12.

Original: For each given length, 500 lists are constructed.
Revised: For each given length, 500 lists are constructed.

Original: The accuracies of different methods with code-davinci-002 in GPT-3 are shown in Table 3.
Revised: The accuracies of different methods with code-davinci-002 in GPT-3 are shown in Table 3.

Original: Standard prompting completely fails all test cases with an accuracy of 0.
Revised: Standard prompting completely fails all test cases with an accuracy of 0.

Original: Chain-of-thought prompting signiﬁcantly boosts the performance over standard prompting, but it still falls well behind least-to-most prompting, particularly when the lists are long.
Revised: Chain-of-thought prompting signiﬁcantly boosts the performance over standard prompting, but it still falls well behind least-to-most prompting, particularly when the lists are long.

Original: Both chain-of-thought prompting and least-to-most prompting perform worse on longer lists.
Revised: Both chain-of-thought prompting and least-to-most prompting perform worse on longer lists.

Original: However, the performance of chain-of-thought prompting drops much faster as the length increases.
Revised: However, the performance of chain-of-thought prompting drops much faster as the length increases.

Original: In Appendices 6.2 and 6.3, we present additional experiments with different prompt examples and different language models.
Revised: In Appendices 6.2 and 6.3, we present additional experiments with different prompt examples and different language models.

Original: Among others, these experiments show that 2-shot least-to-most prompting also outperforms 4-shot and 8-shot chain-of-thought prompting.
Revised: Among others, these experiments show that 2-shot least-to-most prompting also outperforms 4-shot and 8-shot chain-of-thought prompting.

Original: This demonstrates that the accuracy gain is not caused by the fact that the least-to-most examples are slightly longer.
Revised: This demonstrates that the accuracy gain is not caused by the fact that the least-to-most examples are slightly longer.

Original: Note that in contrast to least-to-most prompting, we use use two independent prompt examples for chainof-thought (see Table 1) because this leads to better performance.
Revised: Note that in contrast to least-to-most prompting, we use use two independent prompt examples for chainof-thought (see Table 1) because this leads to better performance.

Original: For completeness, we also present experiments using the same examples as for least-to-most prompting in the Appendix.
Revised: For completeness, we also present experiments using the same examples as for least-to-most prompting in the Appendix.

Original: 3. C OMPOSITIONAL G ENERALIZATION
Revised: 3. C OMPOSITIONAL G ENERALIZATION

Original: SCAN (Lake & Baroni, 2018) is probably the most popular benchmark for evaluating compositional generalization.
Revised: SCAN (Lake & Baroni, 2018) is probably the most popular benchmark for evaluating compositional generalization.

Original: It requires mapping natural language commands to action sequences (Table 4).
Revised: It requires mapping natural language commands to action sequences (Table 4).

Original: Sequence-to-sequence models perform poorly under length split where the action sequences in the training set (about 80% of the full set with over 20,000 examples) are shorter than the action sequences in the testing set.
Revised: Sequence-to-sequence models perform poorly under length split where the action sequences in the training set (about 80% of the full set with over 20,000 examples) are shorter than the action sequences in the testing set.

Original: Many specialized neural-symbolic models have been proposed to solve
Revised: Many specialized neural-symbolic models have been proposed to solve

Original: SCAN (Chen et al., 2020; Liu et al., 2020; Nye et al., 2020; Shaw et al., 2021; Kim, 2021).
Revised: SCAN (Chen et al., 2020; Liu et al., 2020; Nye et al., 2020; Shaw et al., 2021; Kim, 2021).

Original: Here we show how least-to-most prompting solves SCAN using only a few demonstration examples.
Revised: Here we show how least-to-most prompting solves SCAN using only a few demonstration examples.

Original: Command
Revised: Command

Original: Action Sequence “look thrice after jump”
Revised: Action Sequence “look thrice after jump”

Original: JUMP LOOK LOOK LOOK “run left and walk”
Revised: JUMP LOOK LOOK LOOK “run left and walk”

Original: TURN LEFT RUN WALK “look opposite right”
Revised: TURN LEFT RUN WALK “look opposite right”

Original: TURN RIGHT TURN RIGHT LOOK are intended to fully cover semantics to to “RUN” has higher priority use Python notation to make our prompts concise and meet the input size limit of language models (usually up to 2048 tokens).
Revised: TURN RIGHT TURN RIGHT LOOK are intended to fully cover semantics to to “RUN” has higher priority use Python notation to make our prompts concise and meet the input size limit of language models (usually up to 2048 tokens).

Original: To generate the ﬁnal results, we either run a postprocessing script or prompt language models to expand the Python expressions, which can be done with near-perfect accuracy of 99.7% (see Appendix 7.4 for details).
Revised: To generate the ﬁnal results, we either run a postprocessing script or prompt language models to expand the Python expressions, which can be done with near-perfect accuracy of 99.7% (see Appendix 7.4 for details).

Original: We use the same notation also for the baselines (standard and chain-of-thought prompting) to keep the comparison fair.
Revised: We use the same notation also for the baselines (standard and chain-of-thought prompting) to keep the comparison fair.

Original: Q: “look opposite
Revised: Q: “look opposite

Original: right
Revised: right

Original: thrice after walk”
Revised: thrice after walk”

Original: A: “look opposite right thrice” can be solved by: “look opposite right”, “look opposite right thrice”.
Revised: A: “look opposite right thrice” can be solved by: “look opposite right”, “look opposite right thrice”.

Original: “walk” can be solved by “walk”.
Revised: “walk” can be solved by “walk”.

Original: So, “look opposite right thrice after walk” can be solved by: “look opposite right”, “look opposite right thrice”, “walk”.
Revised: So, “look opposite right thrice after walk” can be solved by: “look opposite right”, “look opposite right thrice”, “walk”.

Original: Q:
Revised: Q:

Original: “jump around right”
Revised: “jump around right”

Original: A:
Revised: A:

Original: “jump around right”
Revised: “jump around right”

Original: can
Revised: can

Original: be solved by: “jump right”, “jump around right”.
Revised: be solved by: “jump right”, “jump around right”.

Original: So
Revised: So

Original: “jump around
Revised: “jump around

Original: right” can be
Revised: right” can be

Original: solved by: “jump right”, “jump around right”
Revised: solved by: “jump right”, “jump around right”

Original: Q
Revised: Q

Original: : “look
Revised: : “look

Original: around right thrice
Revised: around right thrice

Original: and
Revised: and

Original: walk”
Revised: walk”

Original: A:
Revised: A:

Original: “look around right thrice”
Revised: “look around right thrice”

Original: can be solved by
Revised: can be solved by

Original: : “look right”, “look around right”, “look around right
Revised: : “look right”, “look around right”, “look around right

Original: thrice”.
Revised: thrice”.

Original: “walk” can be solved by “walk”.
Revised: “walk” can be solved by “walk”.

Original: So, “look around right thrice and walk” can be solved by: “look right”, “look
Revised: So, “look around right thrice and walk” can be solved by: “look right”, “look

Original: around right”
Revised: around right”

Original: , “look around
Revised: , “look around

Original: right
Revised: right

Original: thrice”,
Revised: thrice”,

Original: “walk”
Revised: “walk”

Original: Results .
Revised: Results .

Original: We compare least-to-most prompting with chain-of-thought prompting and standard prompting.
Revised: We compare least-to-most prompting with chain-of-thought prompting and standard prompting.

Original: All the prompting methods share the same example commands, but the regular prompting directly maps them to action sequences without explanations, while chain-of-thought prompting uses the same command-mapping context as least-to-most prompting but does not use command decomposition.
Revised: All the prompting methods share the same example commands, but the regular prompting directly maps them to action sequences without explanations, while chain-of-thought prompting uses the same command-mapping context as least-to-most prompting but does not use command decomposition.

Original: The accuracies of different prompting methods with different language models are presented in Table 8.
Revised: The accuracies of different prompting methods with different language models are presented in Table 8.

Original: Using code-davinci-002 , least-to-most prompting achieves an accuracy of 99 .
Revised: Using code-davinci-002 , least-to-most prompting achieves an accuracy of 99 .

Original: 7% under length split.
Revised: 7% under length split.

Original: A test case of least-to-most prompting is listed in Table 7.
Revised: A test case of least-to-most prompting is listed in Table 7.

Original: More example outputs can be found in Appendix 7.3.
Revised: More example outputs can be found in Appendix 7.3.

Original: We also test our method on all other splits and the full dataset.
Revised: We also test our method on all other splits and the full dataset.

Original: We ﬁnd that the solving rate remains the same.
Revised: We ﬁnd that the solving rate remains the same.

Original: In addition, it may be interesting to note
Revised: In addition, it may be interesting to note

Original: [Table]
Revised: [Table]

Original: A: The output of “jump left” concatenates: the output of “turn left”, the output of “jump”.
Revised: A: The output of “jump left” concatenates: the output of “turn left”, the output of “jump”.

Original: “turn left” outputs “TURN LEFT”.
Revised: “turn left” outputs “TURN LEFT”.

Original: “jump” outputs “JUMP”.
Revised: “jump” outputs “JUMP”.

Original: So concatenating the output of “turn left” and the output of “jump” leads to “TURN LEFT” + “JUMP”.
Revised: So concatenating the output of “turn left” and the output of “jump” leads to “TURN LEFT” + “JUMP”.

Original: So the output of “jump left” is “TURN LEFT” + “JUMP”.
Revised: So the output of “jump left” is “TURN LEFT” + “JUMP”.

Original: [Table]
Revised: [Table]

Original: “run
Revised: “run

Original: A: The output of “run and look twice” concatenates: the output of “run”, the output of “look twice”.
Revised: A: The output of “run and look twice” concatenates: the output of “run”, the output of “look twice”.

Original: “run” outputs “RUN”.
Revised: “run” outputs “RUN”.

Original: “look twice” outputs “LOOK” * 2.
Revised: “look twice” outputs “LOOK” * 2.

Original: So concatenating the output of “run” and the output of “look twice” leads to “RUN” + “LOOK” * 2.
Revised: So concatenating the output of “run” and the output of “look twice” leads to “RUN” + “LOOK” * 2.

Original: So the output of “run and look twice” is “RUN” + “LOOK” * 2.
Revised: So the output of “run and look twice” is “RUN” + “LOOK” * 2.

Original: Q
Revised: Q

Original: “walk
Revised: “walk

Original: left” A: The output of “walk opposite left” concatenates: the output of “turn opposite left”, the output of “walk”.
Revised: left” A: The output of “walk opposite left” concatenates: the output of “turn opposite left”, the output of “walk”.

Original: “turn opposite left” outputs “TURN LEFT” * 2.
Revised: “turn opposite left” outputs “TURN LEFT” * 2.

Original: “walk” outputs “WALK”.
Revised: “walk” outputs “WALK”.

Original: So concatenating the output of “turn opposite left” and the output of “walk” leads to “TURN
Revised: So concatenating the output of “turn opposite left” and the output of “walk” leads to “TURN

Original: + “WALK”.
Revised: + “WALK”.

Original: So the output of “walk opposite left” is “TURN
Revised: So the output of “walk opposite left” is “TURN

Original: * 2 + ”WALK”.
Revised: * 2 + ”WALK”.

Original: [Table]
Revised: [Table]

Original: The
Revised: The

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: right twice” concatenates: the output of “jump right”, the output of “jump right”.
Revised: right twice” concatenates: the output of “jump right”, the output of “jump right”.

Original: “jump right” outputs “TURN RIGHT” + “JUMP”.
Revised: “jump right” outputs “TURN RIGHT” + “JUMP”.

Original: So repeating the output of “jump right” two times leads to (“TURN RIGHT” + “JUMP”)
Revised: So repeating the output of “jump right” two times leads to (“TURN RIGHT” + “JUMP”)

Original: So the output of “jump right twice” is (“TURN RIGHT” + “JUMP”)
Revised: So the output of “jump right twice” is (“TURN RIGHT” + “JUMP”)

Original: Q: “jump right twice after walk”
Revised: Q: “jump right twice after walk”

Original: A: The output of “jump right twice after walk” concatenates: the output of “walk”, the output of “jump right twice”.
Revised: A: The output of “jump right twice after walk” concatenates: the output of “walk”, the output of “jump right twice”.

Original: “walk” outputs “WALK”.
Revised: “walk” outputs “WALK”.

Original: “jump right twice” outputs (“TURN RIGHT” + “JUMP”)
Revised: “jump right twice” outputs (“TURN RIGHT” + “JUMP”)

Original: So concatenating the output of “walk” and the output of “jump right twice” leads to “WALK” + (“TURN RIGHT” + “JUMP”)
Revised: So concatenating the output of “walk” and the output of “jump right twice” leads to “WALK” + (“TURN RIGHT” + “JUMP”)

Original: So the output of “jump right twice after walk” is “WALK” + (“TURN RIGHT” + “JUMP”)
Revised: So the output of “jump right twice after walk” is “WALK” + (“TURN RIGHT” + “JUMP”)

Original: that code-davinci-002 consistently performs better than text-davinci-002 , regardless of prompting methods.
Revised: that code-davinci-002 consistently performs better than text-davinci-002 , regardless of prompting methods.

Original: [Table]
Revised: [Table]

Original: The results of text-davinci-002 are based on a random subset of 100 commands.
Revised: The results of text-davinci-002 are based on a random subset of 100 commands.

Original: 3. M ATH R EASONING
Revised: 3. M ATH R EASONING

Original: In this section, we apply least-to-most prompting to solve math word problems in GSM8K (Cobbe et al., 2021) and DROP (Dua et al., 2019).
Revised: In this section, we apply least-to-most prompting to solve math word problems in GSM8K (Cobbe et al., 2021) and DROP (Dua et al., 2019).

Original: We are particularly interested at seeing if large language models combined with least-to-most prompting can solve problems more difﬁcult than those seen in the prompts.
Revised: We are particularly interested at seeing if large language models combined with least-to-most prompting can solve problems more difﬁcult than those seen in the prompts.

Original: Here, we simply measure the difﬁculty by the number of solving steps.
Revised: Here, we simply measure the difﬁculty by the number of solving steps.

Original: Q: Elsa has 5 apples.
Revised: Q: Elsa has 5 apples.

Original: Anna has 2 more apples than Elsa.
Revised: Anna has 2 more apples than Elsa.

Original: How many apples do they have together?
Revised: How many apples do they have together?

Original: A: Let’s break down this problem: 1.
Revised: A: Let’s break down this problem: 1.

Original: How many apples does Anna have? 2.
Revised: How many apples does Anna have? 2.

Original: How many apples do they have together?
Revised: How many apples do they have together?

Original: Anna has 2 more apples than Elsa.
Revised: Anna has 2 more apples than Elsa.

Original: So Anna has 2 + 5 = 7 apples.
Revised: So Anna has 2 + 5 = 7 apples.

Original: Elsa and Anna have 5 + 7 = 12 apples together.
Revised: Elsa and Anna have 5 + 7 = 12 apples together.

Original: The answer is: 12.
Revised: The answer is: 12.

Original: Q: Elsa has 5 apples.
Revised: Q: Elsa has 5 apples.

Original: Anna has 2 more apples than Elsa.
Revised: Anna has 2 more apples than Elsa.

Original: How many apples do they have together?
Revised: How many apples do they have together?

Original: A: Anna has 2 more apples than Elsa.
Revised: A: Anna has 2 more apples than Elsa.

Original: So Anna has 2 + 5 = 7 apples.
Revised: So Anna has 2 + 5 = 7 apples.

Original: So Elsa and Anna have 5 += 12 apples together.
Revised: So Elsa and Anna have 5 += 12 apples together.

Original: The answer is: 12.
Revised: The answer is: 12.

Original: The prompt that we design to solve GSM8K is shown in Table 9.
Revised: The prompt that we design to solve GSM8K is shown in Table 9.

Original: The demonstration exemplar consists of two parts.
Revised: The demonstration exemplar consists of two parts.

Original: The ﬁrst part (starting from “Let’s break down this problem ...”) shows how the original problem can be decomposed into simpler subproblems, and the the second part shows how the subproblems are solved in sequence.
Revised: The ﬁrst part (starting from “Let’s break down this problem ...”) shows how the original problem can be decomposed into simpler subproblems, and the the second part shows how the subproblems are solved in sequence.

Original: Note that this prompt combines decomposition and subproblem solving into a single pass.
Revised: Note that this prompt combines decomposition and subproblem solving into a single pass.

Original: One may instead design two different prompts respectively for decomposition and subproblem solving, as the least-to-most prompts in the previous sections, to achieve an even better performance.
Revised: One may instead design two different prompts respectively for decomposition and subproblem solving, as the least-to-most prompts in the previous sections, to achieve an even better performance.

Original: Here, we focus on investigating how this simple least-to-most prompt generalizes from a simple 2-step problem to more complex multi-step problems.
Revised: Here, we focus on investigating how this simple least-to-most prompt generalizes from a simple 2-step problem to more complex multi-step problems.

Original: We also construct a chain-of-thought prompt (Table 10) as our baseline.
Revised: We also construct a chain-of-thought prompt (Table 10) as our baseline.

Original: It is derived from the leastto-most prompt (Table 9) by removing the decomposition part.
Revised: It is derived from the leastto-most prompt (Table 9) by removing the decomposition part.

Original: The results are shown in Table 11.
Revised: The results are shown in Table 11.

Original: Overall, least-to-most prompting only slightly improves chain-of-thought prompting: from 60 .
Revised: Overall, least-to-most prompting only slightly improves chain-of-thought prompting: from 60 .

Original: 97% to 63 .
Revised: 97% to 63 .

Original: However, least-to-most prompting essentially improves chain-of-thought prompting in solving problems which need at least 5 steps to be solved: from 39 .
Revised: However, least-to-most prompting essentially improves chain-of-thought prompting in solving problems which need at least 5 steps to be solved: from 39 .

Original: 07% to 45 .
Revised: 07% to 45 .

Original: 23% (Table 12).
Revised: 23% (Table 12).

Original: We ﬁnd that almost every problem in GSM8K that least-to-most prompting fails to solve can be eventually solved by using a manually crafted decomposition.
Revised: We ﬁnd that almost every problem in GSM8K that least-to-most prompting fails to solve can be eventually solved by using a manually crafted decomposition.

Original: This should not be surprising.
Revised: This should not be surprising.

Original: For our humans, as long as we know how to decompose a complex problem into simpler subproblems, we actually have solved it.
Revised: For our humans, as long as we know how to decompose a complex problem into simpler subproblems, we actually have solved it.

Original: However, we also ﬁnd that it is really difﬁcult to teach language models to effectively decompose problems in general.
Revised: However, we also ﬁnd that it is really difﬁcult to teach language models to effectively decompose problems in general.

Original: For the DROP benchmark, least-to-most prompting outperforms chain-of-thought prompting by a large margin (Table 11).
Revised: For the DROP benchmark, least-to-most prompting outperforms chain-of-thought prompting by a large margin (Table 11).

Original: That is probably because most problems in DROP can be trivially decomposed.
Revised: That is probably because most problems in DROP can be trivially decomposed.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Compositional generalization .
Revised: Compositional generalization .

Original: SCAN (Lake & Baroni, 2018) is a widely used benchmark to evaluate compositional generalization.
Revised: SCAN (Lake & Baroni, 2018) is a widely used benchmark to evaluate compositional generalization.

Original: Among all of its splits, the most challenging is the length split, which requires a model to generalize to test sequences longer than training ones.
Revised: Among all of its splits, the most challenging is the length split, which requires a model to generalize to test sequences longer than training ones.

Original: Prior work with good performance on SCAN mostly proposed neural-symbolic architectures (Chen et al., 2020;
Revised: Prior work with good performance on SCAN mostly proposed neural-symbolic architectures (Chen et al., 2020;

Original: Liu et al., 2020) and grammar induction techniques (Nye et al., 2020; Shaw et al., 2021; Kim, 2021).
Revised: Liu et al., 2020) and grammar induction techniques (Nye et al., 2020; Shaw et al., 2021; Kim, 2021).

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Chen et al.
Revised: Chen et al.

Original: (2020) proposed the neural-symbolic stack machine, which contains a neural network as the controller to generate an execution trace for a given input, and a symbolic stack machine to execute the trace and produce the output.
Revised: (2020) proposed the neural-symbolic stack machine, which contains a neural network as the controller to generate an execution trace for a given input, and a symbolic stack machine to execute the trace and produce the output.

Original: The execution trace consists of domain-speciﬁc primitives for sequence manipulation, which allows the machine to break down the input sentence into different components, translate them separately, and compose them together.
Revised: The execution trace consists of domain-speciﬁc primitives for sequence manipulation, which allows the machine to break down the input sentence into different components, translate them separately, and compose them together.

Original: Liu et al.
Revised: Liu et al.

Original: (2020) proposed a framework that cooperatively learns two neural modules, a composer and a solver, to jointly learn the input structure and the symbolic grammar rules.
Revised: (2020) proposed a framework that cooperatively learns two neural modules, a composer and a solver, to jointly learn the input structure and the symbolic grammar rules.

Original: Both Nye et al.
Revised: Both Nye et al.

Original: (2020) and Shaw et al.
Revised: (2020) and Shaw et al.

Original: (2021) inferred the symbolic grammar rules of SCAN, while Kim (2021) proposed to learn a latent neural grammar.
Revised: (2021) inferred the symbolic grammar rules of SCAN, while Kim (2021) proposed to learn a latent neural grammar.

Original: While approaches with symbolic components are able to achieve 100% accuracy on
Revised: While approaches with symbolic components are able to achieve 100% accuracy on

Original: SCAN (Chen et al., 2020; Liu et al., 2020; Nye et al., 2020; Shaw et al., 2021), they require complicated model training and grammar inference algorithms to search in a large grammar space.
Revised: SCAN (Chen et al., 2020; Liu et al., 2020; Nye et al., 2020; Shaw et al., 2021), they require complicated model training and grammar inference algorithms to search in a large grammar space.

Original: Another line of work on SCAN designs data augmentation schemes (Andreas, 2020; Aky ¨ urek et al., 2021;
Revised: Another line of work on SCAN designs data augmentation schemes (Andreas, 2020; Aky ¨ urek et al., 2021;

Original: Lake, 2019).
Revised: Lake, 2019).

Original: Both Andreas (2020) and Aky ¨ urek et al.
Revised: Both Andreas (2020) and Aky ¨ urek et al.

Original: (2021) construct synthetic training samples by recombining fragments occurring in different training samples, and Aky ¨ urek et al.
Revised: (2021) construct synthetic training samples by recombining fragments occurring in different training samples, and Aky ¨ urek et al.

Original: (2021) further designs a sampling scheme that encourages the recombination model to produce rare samples.
Revised: (2021) further designs a sampling scheme that encourages the recombination model to produce rare samples.

Original: On the other hand, Lake (2019) proposed a meta training algorithm, which requires a meta-grammar space to construct training data, and the format of sampled grammars is similar to the SCAN grammar.
Revised: On the other hand, Lake (2019) proposed a meta training algorithm, which requires a meta-grammar space to construct training data, and the format of sampled grammars is similar to the SCAN grammar.

Original: While these data augmentation techniques improve the performance on several compositional generalization benchmarks, they fail to solve the length split of SCAN.
Revised: While these data augmentation techniques improve the performance on several compositional generalization benchmarks, they fail to solve the length split of SCAN.

Original: Other prior works propose neural network architectures to improve compositional generalization, where they encourage the model to learn the word and span mapping (Russin et al., 2019;
Revised: Other prior works propose neural network architectures to improve compositional generalization, where they encourage the model to learn the word and span mapping (Russin et al., 2019;

Original: Li et al., 2019), the alignment of input and output as span trees (Herzig & Berant, 2021), and the permutation equivariance of input and output words (Gordon et al., 2020).
Revised: Li et al., 2019), the alignment of input and output as span trees (Herzig & Berant, 2021), and the permutation equivariance of input and output words (Gordon et al., 2020).

Original: Still, these end-to-end neural networks without symbolic components do not generalize to longer test inputs.
Revised: Still, these end-to-end neural networks without symbolic components do not generalize to longer test inputs.

Original: Unlike the existing work, we demonstrate that without model architectures and symbolic components specially designed to improve compositional generalization, least-to-most prompting achieves 99 .
Revised: Unlike the existing work, we demonstrate that without model architectures and symbolic components specially designed to improve compositional generalization, least-to-most prompting achieves 99 .

Original: 7% accuracy on any split (including length split) with only a handful of demonstration examples, and it does not require any training or ﬁnetuning.
Revised: 7% accuracy on any split (including length split) with only a handful of demonstration examples, and it does not require any training or ﬁnetuning.

Original: Easy-to-hard generalization .
Revised: Easy-to-hard generalization .

Original: In addition to compositional generalization, there are many other tasks where the test cases require more reasoning steps to solve than the training examples, for example, the last-letter-concatenation task where the test lists are longer than the demonstration examples.
Revised: In addition to compositional generalization, there are many other tasks where the test cases require more reasoning steps to solve than the training examples, for example, the last-letter-concatenation task where the test lists are longer than the demonstration examples.

Original: Dong et al.
Revised: Dong et al.

Original: (2019) propose Neural Logic Machines (NLMs) for both inductive learning and logic reasoning.
Revised: (2019) propose Neural Logic Machines (NLMs) for both inductive learning and logic reasoning.

Original: NLMs trained on small-scale tasks (such as small size block worlds) can perfectly generalize to large-scale tasks (such as larger size block worlds).
Revised: NLMs trained on small-scale tasks (such as small size block worlds) can perfectly generalize to large-scale tasks (such as larger size block worlds).

Original: Schwarzschild et al.
Revised: Schwarzschild et al.

Original: (2021) show that recurrent networks trained to solve simple problems with few recurrent steps (such as small size mazes or chess puzzles) can solve more complex problems (such as larger size mazes or chess puzzles) by performing additional recurrences during inference.
Revised: (2021) show that recurrent networks trained to solve simple problems with few recurrent steps (such as small size mazes or chess puzzles) can solve more complex problems (such as larger size mazes or chess puzzles) by performing additional recurrences during inference.

Original: In our method, we achieve easy-to-hard generalization by decomposing a complex problem into a series of easier problems.
Revised: In our method, we achieve easy-to-hard generalization by decomposing a complex problem into a series of easier problems.

Original: Task decomposition .
Revised: Task decomposition .

Original: Perez et al.
Revised: Perez et al.

Original: (2020) decompose a multi-hop question into a number of independent single-hop subquestions, which are answered by an off-the-shelf question answering (QA) model.
Revised: (2020) decompose a multi-hop question into a number of independent single-hop subquestions, which are answered by an off-the-shelf question answering (QA) model.

Original: Then those answers are aggregated to form the ﬁnal answer.
Revised: Then those answers are aggregated to form the ﬁnal answer.

Original: Both question decomposition and answer aggregation are implemented by trained models.
Revised: Both question decomposition and answer aggregation are implemented by trained models.

Original: Wang et al.
Revised: Wang et al.

Original: (2022a) conducts multi-hop
Revised: (2022a) conducts multi-hop

Original: QA by modeling prompts as continuous virtual tokens and progressively eliciting relevant knowledge from language models via iterative prompting.
Revised: QA by modeling prompts as continuous virtual tokens and progressively eliciting relevant knowledge from language models via iterative prompting.

Original: Unlike these methods, our approach does not involve any training or ﬁnetuning.
Revised: Unlike these methods, our approach does not involve any training or ﬁnetuning.

Original: Moreover, the subquestions generated in least-to-most prompting are usually dependent and have to be sequentially solved in a speciﬁc order so that answers to some subquestions can be used as building blocks to solve other subquestions.
Revised: Moreover, the subquestions generated in least-to-most prompting are usually dependent and have to be sequentially solved in a speciﬁc order so that answers to some subquestions can be used as building blocks to solve other subquestions.

Original: Yang et al.
Revised: Yang et al.

Original: (2022) translate natural language questions to SQL queries by decomposing a question into a sequence of slot-ﬁlling natural language prompts corresponding to SQL clauses via a rule-based system.
Revised: (2022) translate natural language questions to SQL queries by decomposing a question into a sequence of slot-ﬁlling natural language prompts corresponding to SQL clauses via a rule-based system.

Original: Wu et al.
Revised: Wu et al.

Original: (2022) propose chaining large language model steps such that the output of one step becomes the input for the next and develop an interactive system for users to construct and modify chains.
Revised: (2022) propose chaining large language model steps such that the output of one step becomes the input for the next and develop an interactive system for users to construct and modify chains.

Original: Least-to-most prompting chains the processes of problem decomposition and subproblem solving.
Revised: Least-to-most prompting chains the processes of problem decomposition and subproblem solving.

Original:  C ONCLUSION AND D ISCUSSION
Revised:  C ONCLUSION AND D ISCUSSION

Original: We proposed least-to-most prompting to enable language models to solve problems that are harder than those seen in the prompt context.
Revised: We proposed least-to-most prompting to enable language models to solve problems that are harder than those seen in the prompt context.

Original: It involves a top-down problem decomposition process and a bottom-up problem solution process.
Revised: It involves a top-down problem decomposition process and a bottom-up problem solution process.

Original: Empirical results on symbolic manipulation, compositional generalization, and math reasoning show that least-to-most prompting outperforms standard prompting and chain-of-thought prompting by a large margin.
Revised: Empirical results on symbolic manipulation, compositional generalization, and math reasoning show that least-to-most prompting outperforms standard prompting and chain-of-thought prompting by a large margin.

Original: However, not all problems can be solved by least-to-most prompting.
Revised: However, not all problems can be solved by least-to-most prompting.

Original: Some problems may not be decomposable or at least not be easy to decompose.
Revised: Some problems may not be decomposable or at least not be easy to decompose.

Original: For example, a math word problem which can be solved using variables and equations may not be that obvious to decompose to subproblems solvable within one arithmetic operation step.
Revised: For example, a math word problem which can be solved using variables and equations may not be that obvious to decompose to subproblems solvable within one arithmetic operation step.

Original: In this case, a possible workaround would be to ﬁrst ﬁnetune the language model to give it knowledge of variables and equations.
Revised: In this case, a possible workaround would be to ﬁrst ﬁnetune the language model to give it knowledge of variables and equations.

Original: How to effectively combine prompting and ﬁnetuning should be a fascinating direction to explore.
Revised: How to effectively combine prompting and ﬁnetuning should be a fascinating direction to explore.

Original: Another limitation of least-to-most prompting stems from its natural language based solution construction.
Revised: Another limitation of least-to-most prompting stems from its natural language based solution construction.

Original: Natural language is expressive and generic but tends to be less precise than computer languages.
Revised: Natural language is expressive and generic but tends to be less precise than computer languages.

Original: For example, a learned computer program may be able to solve symbolic manipulation tasks at any length with an accuracy of 100%.
Revised: For example, a learned computer program may be able to solve symbolic manipulation tasks at any length with an accuracy of 100%.

Original: So we may combine natural language and computer languages in our prompts to make them more powerful.
Revised: So we may combine natural language and computer languages in our prompts to make them more powerful.

Original: This is just like writing a mathematical proof in which both natural language text and abstract symbol based equations are needed.
Revised: This is just like writing a mathematical proof in which both natural language text and abstract symbol based equations are needed.

Original: In the prompt constructed to solve SCAN, we have actually used both natural language and Python notation.
Revised: In the prompt constructed to solve SCAN, we have actually used both natural language and Python notation.

Original: Finally, prompting may not be the best way to enable large language models to reason.
Revised: Finally, prompting may not be the best way to enable large language models to reason.

Original: We can think of prompting as a form of one-way communication: we teach a language model new skills without taking into account its response.
Revised: We can think of prompting as a form of one-way communication: we teach a language model new skills without taking into account its response.

Original: It should be natural to extend prompting to full bidirectional conversations, which would allow us to provide immediate feedback for language models such that they could learn better or faster.
Revised: It should be natural to extend prompting to full bidirectional conversations, which would allow us to provide immediate feedback for language models such that they could learn better or faster.

Original: Least-to-most prompting can be considered as a step toward teaching language models through bidirectional conversations
Revised: Least-to-most prompting can be considered as a step toward teaching language models through bidirectional conversations. A CKNOWLEDGEMENT

Revised: We would like to thank Xinyun Chen, Xinying Song, Jeff Dean, Zoubin Ghahramani, Fernando

Revised: Pereira, Jacob Devlin and Pete Shaw for helpful discussions and suggestions

Original: Appendix Table of Contents
Revised: Appendix Table of Contents

Original: 6 Last-letter-concatenation 6.1 Prompt context for decomposing a word list into subproblems . . . . . . . . . .
Revised: 6 Last-letter-concatenation 6.1 Prompt context for decomposing a word list into subproblems . . . . . . . . . .

Original: 6.2 Prompt contexts with more and different examples . . . . . . . . . . . . . . . .
Revised: 6.2 Prompt contexts with more and different examples . . . . . . . . . . . . . . . .

Original: Standard prompting, 4-shot . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Standard prompting, 4-shot . . . . . . . . . . . . . . . . . . . . . . . .

Original: Chain-of-thought prompting, 4-shot . . . . . . . . . . . . . . . . . . . .
Revised: Chain-of-thought prompting, 4-shot . . . . . . . . . . . . . . . . . . . .

Original: Chain-of-thought prompting, 8-shot . . . . . . . . . . . . . . . . . . . .
Revised: Chain-of-thought prompting, 8-shot . . . . . . . . . . . . . . . . . . . .

Original: Chain-of-thought prompting, 2-shot, same examples as for least-to-most .
Revised: Chain-of-thought prompting, 2-shot, same examples as for least-to-most .

Original: Least-to-most prompting, 4-shot . . . . . . . . . . . . . . . . . . . . .
Revised: Least-to-most prompting, 4-shot . . . . . . . . . . . . . . . . . . . . .

Original: 6.3 Data Generation and additional results . . . . . . . . . . . . . . . . . . . . . .
Revised: 6.3 Data Generation and additional results . . . . . . . . . . . . . . . . . . . . . .

Original: 6.4 Error analysis: Least-to-most prompting . . . . . . . . . . . . . . . . . . . . .
Revised: 6.4 Error analysis: Least-to-most prompting . . . . . . . . . . . . . . . . . . . . .

Original: 6.5 Example outputs from code-davinci-002 . . . . . . . . . . . . . . . . . . . . .
Revised: 6.5 Example outputs from code-davinci-002 . . . . . . . . . . . . . . . . . . . . .

Original: Standard prompting: Failure . . . . . . . . . . . . . . . . . . . . . . .
Revised: Standard prompting: Failure . . . . . . . . . . . . . . . . . . . . . . .

Original: Chain-of-thought prompting: Success . . . . . . . . . . . . . . . . . . .
Revised: Chain-of-thought prompting: Success . . . . . . . . . . . . . . . . . . .

Original: Chain-of-thought prompting: Failure . . . . . . . . . . . . . . . . . . .
Revised: Chain-of-thought prompting: Failure . . . . . . . . . . . . . . . . . . .

Original: Least-to-most prompting: Success . . . . . . . . . . . . . . . . . . . .
Revised: Least-to-most prompting: Success . . . . . . . . . . . . . . . . . . . .

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: : Success . . . . . . . . . . . . . . . . . . .
Revised: : Success . . . . . . . . . . . . . . . . . . .

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: Non-football Subset . .
Revised: Non-football Subset . .

Original: Zero-shot prompting . .
Revised: Zero-shot prompting . .

Original: Standard prompting with 3 examples . . . . . . . . . . . . . . . . . . .
Revised: Standard prompting with 3 examples . . . . . . . . . . . . . . . . . . .

Original: Chain-of-thought prompting with 3 examples . . . . . . . . . . . . . . .
Revised: Chain-of-thought prompting with 3 examples . . . . . . . . . . . . . . .

Original: Least-to-most prompting I: problem decomposition (5 examples) . . . .
Revised: Least-to-most prompting I: problem decomposition (5 examples) . . . .

Original: Least-to-most prompting II: problem solving (3 examples) . . . . . . . .
Revised: Least-to-most prompting II: problem solving (3 examples) . . . . . . . .

Original: [Table]
Revised: [Table]

Original: Football subset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Football subset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: Zero-shot prompting . .
Revised: Zero-shot prompting . .

Original: Standard prompting with 3 examples . . . . . . . . . . . . . . . . . . .
Revised: Standard prompting with 3 examples . . . . . . . . . . . . . . . . . . .

Original: Chain-of-thought prompting with 3 examples . . . . . . . . . . . . . . .
Revised: Chain-of-thought prompting with 3 examples . . . . . . . . . . . . . . .

Original: Least-to-most prompting I: problem decomposition (6 examples) . . . .
Revised: Least-to-most prompting I: problem decomposition (6 examples) . . . .

Original: Least-to-most prompting II: problem solving (3 examples) . . . . . . . .
Revised: Least-to-most prompting II: problem solving (3 examples) . . . . . . . .

Original: 8.4 Examples where least-to-most succeeded but chain-of-thought failed . . . . . . .
Revised: 8.4 Examples where least-to-most succeeded but chain-of-thought failed . . . . . . .

Original: Case 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Case 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: [Table]
Revised: [Table]

Original: Case 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Case 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original:  L AST - LETTER - CONCATENATION 6. P ROMPT CONTEXT FOR DECOMPOSING A WORD LIST INTO SUBPROBLEMS
Revised:  L AST - LETTER - CONCATENATION 6. P ROMPT CONTEXT FOR DECOMPOSING A WORD LIST INTO SUBPROBLEMS

Original: In Section 3.1 we mentioned that language model prompting can be used to decompose a word list such as “think, machine, learning, reasoning” into a sequence of subproblems “think, machine”, “think, machine, learning”, and “think, machine, learning, reasoning”.
Revised: In Section 3.1 we mentioned that language model prompting can be used to decompose a word list such as “think, machine, learning, reasoning” into a sequence of subproblems “think, machine”, “think, machine, learning”, and “think, machine, learning, reasoning”.

Original: The following prompt context achieves 100% accuracy on this task when using the text-davinci-002 model.
Revised: The following prompt context achieves 100% accuracy on this task when using the text-davinci-002 model.

Original: Note that it achieves perfect accuracy on lists up to size 12 (which is the maximum that we tested for our experiment) even though it only contains one exemplar each for lists of sizes 2 and 3.
Revised: Note that it achieves perfect accuracy on lists up to size 12 (which is the maximum that we tested for our experiment) even though it only contains one exemplar each for lists of sizes 2 and 3.

Original: Q: “machine, learning” A: creating sequential sublists of the list “machine, learning”: “machine”“machine, learning” Q: “machine, learning, artiﬁcial” A: creating sequential sublists of the list “machine, learning, artiﬁcial”: “machine”“machine, learning” “machine, learning, artiﬁcial”
Revised: Q: “machine, learning” A: creating sequential sublists of the list “machine, learning”: “machine”“machine, learning” Q: “machine, learning, artiﬁcial” A: creating sequential sublists of the list “machine, learning, artiﬁcial”: “machine”“machine, learning” “machine, learning, artiﬁcial”

Original: 6. P ROMPT CONTEXTS WITH MORE AND DIFFERENT EXAMPLES
Revised: 6. P ROMPT CONTEXTS WITH MORE AND DIFFERENT EXAMPLES

Original: The last-letter-concatenation experiments presented in Section 3.1 are based on prompt contexts that consists of 2 demonstration examples.
Revised: The last-letter-concatenation experiments presented in Section 3.1 are based on prompt contexts that consists of 2 demonstration examples.

Original: To make sure that the accuracy gain achieved by leastto-most prompting is not caused by the slight increase in example length when compared to chainof-thought, we also performed experiments with more context examples so that we can compare least-to-most vs. chain-of-thought for different prompt sizes.
Revised: To make sure that the accuracy gain achieved by leastto-most prompting is not caused by the slight increase in example length when compared to chainof-thought, we also performed experiments with more context examples so that we can compare least-to-most vs. chain-of-thought for different prompt sizes.

Original: Also, we perform experiments where we use for chain-of-thought prompting the same prompt examples that we use for least-to-most prompting (unlike the situation in Table 1 where we use different examples).
Revised: Also, we perform experiments where we use for chain-of-thought prompting the same prompt examples that we use for least-to-most prompting (unlike the situation in Table 1 where we use different examples).

Original: All these prompts are shown below, and we present and discuss the corresponding accuracies in Section 6.3.
Revised: All these prompts are shown below, and we present and discuss the corresponding accuracies in Section 6.3.

Original: 6.2. S TANDARD PROMPTING , 4- SHOT
Revised: 6.2. S TANDARD PROMPTING , 4- SHOT

Original: Q: “think, machine”
Revised: Q: “think, machine”

Original: A: “ke” Q: “learning, reasoning, generalization” A: “ggn” Q: “artiﬁcial, intelligence”
Revised: A: “ke” Q: “learning, reasoning, generalization” A: “ggn” Q: “artiﬁcial, intelligence”

Original: [Table]
Revised: [Table]

Original: 6.2. C HAIN - OF - THOUGHT PROMPTING , 4- SHOT
Revised: 6.2. C HAIN - OF - THOUGHT PROMPTING , 4- SHOT

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: 6.2. C HAIN - OF - THOUGHT PROMPTING , 8- SHOT
Revised: 6.2. C HAIN - OF - THOUGHT PROMPTING , 8- SHOT

Original: [Table]
Revised: [Table]

Original: So,
Revised: So,

Original: [Table]
Revised: [Table]

Original: , teacher”
Revised: , teacher”

Original: [Table]
Revised: [Table]

Original: “lr”.
Revised: “lr”.

Original: [Table]
Revised: [Table]

Original: 6.2. C HAIN - OF - THOUGHT PROMPTING , 2- SHOT , SAME EXAMPLES AS FOR LEAST - TO - MOST
Revised: 6.2. C HAIN - OF - THOUGHT PROMPTING , 2- SHOT , SAME EXAMPLES AS FOR LEAST - TO - MOST

Original: [Table] [Table]
Revised: [Table] [Table]

Original: last letter of
Revised: last letter of

Original: [Table]
Revised: [Table]

Original: is
Revised: is

Original: [Table] [Table]
Revised: [Table] [Table]

Original: to
Revised: to

Original: [Table]
Revised: [Table]

Original: So,
Revised: So,

Original: [Table]
Revised: [Table]

Original: , machine
Revised: , machine

Original: [Table]
Revised: [Table]

Original: learning”
Revised: learning”

Original: [Table]
Revised: [Table]

Original: 6.2. L EAST - TO - MOST PROMPTING
Revised: 6.2. L EAST - TO - MOST PROMPTING

Original: [Table]
Revised: [Table]

Original: “ke”.
Revised: “ke”.

Original: So, “think, machine” outputs “ke”.
Revised: So, “think, machine” outputs “ke”.

Original: Q: “think, machine, learning” A: “think, machine” outputs “ke”.
Revised: Q: “think, machine, learning” A: “think, machine” outputs “ke”.

Original: The last letter of “learning” is “g”.
Revised: The last letter of “learning” is “g”.

Original: Concatenating “ke”, “g” leads to “keg”.
Revised: Concatenating “ke”, “g” leads to “keg”.

Original: So, “think, machine, learning” outputs “keg”.
Revised: So, “think, machine, learning” outputs “keg”.

Original: Q: “transformer, language” A: The last letter of “transformer” is “r”.
Revised: Q: “transformer, language” A: The last letter of “transformer” is “r”.

Original: The last letter of “language” is “e”.
Revised: The last letter of “language” is “e”.

Original: Concatenating: “r”, “e” leads to “re”.
Revised: Concatenating: “r”, “e” leads to “re”.

Original: So, “transformer, language” outputs “re”.
Revised: So, “transformer, language” outputs “re”.

Original: Q: “transformer, language, vision” A: “transformer, language” outputs “re”.
Revised: Q: “transformer, language, vision” A: “transformer, language” outputs “re”.

Original: The last letter of “vision” is “n”.
Revised: The last letter of “vision” is “n”.

Original: Concatenating: “re”, “n” leads to “ren”
Revised: Concatenating: “re”, “n” leads to “ren”

Original: So, “transformer, language, vision” outputs “ren”.
Revised: So, “transformer, language, vision” outputs “ren”.

Original: 6. D ATA G ENERATION AND ADDITIONAL RESULTS
Revised: 6. D ATA G ENERATION AND ADDITIONAL RESULTS

Original: Data generation.
Revised: Data generation.

Original: The last-letter-concatenation dataset is based on a list of the 10k most common
Revised: The last-letter-concatenation dataset is based on a list of the 10k most common

Original: English words (including proper nouns) used in books that are part of project Gutenberg, as collected in Wiktionary 2 .
Revised: English words (including proper nouns) used in books that are part of project Gutenberg, as collected in Wiktionary 2 .

Original: After eliminating profane words, we ended up with a list of 9694 words (all lowercase).
Revised: After eliminating profane words, we ended up with a list of 9694 words (all lowercase).

Original: For each of the desired list sizes 2, 4, 6, 8, 10, 12, we then generated 500 examples, each of which consists of a random sequence of these words (input) and the corresponding sequence of last letters (output).
Revised: For each of the desired list sizes 2, 4, 6, 8, 10, 12, we then generated 500 examples, each of which consists of a random sequence of these words (input) and the corresponding sequence of last letters (output).

Original: We will release the full dataset upon publication of this paper.
Revised: We will release the full dataset upon publication of this paper.

Original: Below arerandom examples of list size 6:
Revised: Below arerandom examples of list size 6:

Original: • IN: “narrative, celebrate, neighbouring, indebted, stove, calling” OUT: “eegdeg” • IN: “barley, silk, thankful, kiss, logs, silent” OUT: “yklsst” • IN: “knitting, conveyance, receives, represent, cow, shut” OUT: “gestwt” • IN: “olive, dark, limitation, airy, pocket, wondered” OUT: “eknytd” • IN: “apprehensive, exclamation, perspiration, trusting, destiny, tactics”
Revised: • IN: “narrative, celebrate, neighbouring, indebted, stove, calling” OUT: “eegdeg” • IN: “barley, silk, thankful, kiss, logs, silent” OUT: “yklsst” • IN: “knitting, conveyance, receives, represent, cow, shut” OUT: “gestwt” • IN: “olive, dark, limitation, airy, pocket, wondered” OUT: “eknytd” • IN: “apprehensive, exclamation, perspiration, trusting, destiny, tactics”

Original: OUT: “enngys” • IN: “qualiﬁed, envoy, disciple, exert, witnesses, plane” OUT: “dyetse” • IN: “decidedly, dome, france, chris, knowing, peaceful” OUT: “yeesgl” • IN: “deceit, reﬁnement, tips, cord, princes, discovery” OUT: “ttsdsy” • IN: “drops, paste, defective, bohemia, requested, convenient” OUT: “seeadt” • IN: “diverse, christopher, homely, agreeable, fright, suspended” OUT: “eryetd”
Revised: OUT: “enngys” • IN: “qualiﬁed, envoy, disciple, exert, witnesses, plane” OUT: “dyetse” • IN: “decidedly, dome, france, chris, knowing, peaceful” OUT: “yeesgl” • IN: “deceit, reﬁnement, tips, cord, princes, discovery” OUT: “ttsdsy” • IN: “drops, paste, defective, bohemia, requested, convenient” OUT: “seeadt” • IN: “diverse, christopher, homely, agreeable, fright, suspended” OUT: “eryetd”

Original: Complete results.
Revised: Complete results.

Original: Table 13 summarizes all the experiments we performed for the last-letterconcatenation task.
Revised: Table 13 summarizes all the experiments we performed for the last-letterconcatenation task.

Original: In addition to the experiments where prompt contexts contain 2 demonstration examples presented in Section 3.1, this includes experiments where the prompts contain 4 anddemonstration examples (see above).
Revised: In addition to the experiments where prompt contexts contain 2 demonstration examples presented in Section 3.1, this includes experiments where the prompts contain 4 anddemonstration examples (see above).

Original: While more prompt examples have no effect for standard prompting (the accuracy remains at 0), they increase the accuracy across the board for chain-of-thought and least-to-most prompting.
Revised: While more prompt examples have no effect for standard prompting (the accuracy remains at 0), they increase the accuracy across the board for chain-of-thought and least-to-most prompting.

Original: However, least-to-most prompting consistently outperforms chain-of-thought prompting.
Revised: However, least-to-most prompting consistently outperforms chain-of-thought prompting.

Original: In fact, even if we compare 2-shot least-to-most (prompt size 123 GPT3 tokens) to 8-shot chain-of-thought (prompt size 573 GPT3 tokens), the accuracy for least-to-most prompting is much higher than for chainof-thought prompting.
Revised: In fact, even if we compare 2-shot least-to-most (prompt size 123 GPT3 tokens) to 8-shot chain-of-thought (prompt size 573 GPT3 tokens), the accuracy for least-to-most prompting is much higher than for chainof-thought prompting.

Original: The difference is especially pronounced for long sequences (e.g., for L = 12, we have least-to-most at 74.0% vs. chain-of-thought at 38.4%).
Revised: The difference is especially pronounced for long sequences (e.g., for L = 12, we have least-to-most at 74.0% vs. chain-of-thought at 38.4%).

Original: This shows that least-to-most prompting is much more data-efﬁcient than chain-of-thought prompting for this problem.
Revised: This shows that least-to-most prompting is much more data-efﬁcient than chain-of-thought prompting for this problem.

Original: Comparing the ﬁrst two rows for chain-of-thought prompting shows that chain-of-thought achieves higher accuracy if we use two independent examples (see prompt in Table 1) instead of the two dependent examples that we use for least-to-most prompting.
Revised: Comparing the ﬁrst two rows for chain-of-thought prompting shows that chain-of-thought achieves higher accuracy if we use two independent examples (see prompt in Table 1) instead of the two dependent examples that we use for least-to-most prompting.

Original: This demonstrates that the accuracy advantage of least-to-most prompting over chain-of
Revised: This demonstrates that the accuracy advantage of least-to-most prompting over chain-of

Original: [Table]
Revised: [Table]

Original: (rather than the full set of 500 exammples).
Revised: (rather than the full set of 500 exammples).

Original: The table also contains the results from running against two additional GPT-3 models: text-davinci-002 and codex-davinci-001 .
Revised: The table also contains the results from running against two additional GPT-3 models: text-davinci-002 and codex-davinci-001 .

Original: While text-davinci-002 shows similar accuracy to code-davinci-002 on small list sizes, the accuracy drops off much faster when moving to larger list sizes, both for chain-of-thought prompting as well as for least-to-most prompting.
Revised: While text-davinci-002 shows similar accuracy to code-davinci-002 on small list sizes, the accuracy drops off much faster when moving to larger list sizes, both for chain-of-thought prompting as well as for least-to-most prompting.

Original: This indicates that the code-davinci-002 model has an advantage when it comes to dealing with iteration and recursion.
Revised: This indicates that the code-davinci-002 model has an advantage when it comes to dealing with iteration and recursion.

Original: The code-davinci-001 model performs much worse than code-davinci-002 across all dimensions.
Revised: The code-davinci-001 model performs much worse than code-davinci-002 across all dimensions.

Original: Even for the shortest list size (L = 4), the accuracy for least-to-most prompting is only 19.6% compared to 96% for code-davinci-002 .
Revised: Even for the shortest list size (L = 4), the accuracy for least-to-most prompting is only 19.6% compared to 96% for code-davinci-002 .

Original: This indicates that there is a large potential for improvement when using the exact same conﬁguration with new model generations.
Revised: This indicates that there is a large potential for improvement when using the exact same conﬁguration with new model generations.

Original: 6. E RROR ANALYSIS : L EAST - TO - MOST PROMPTING
Revised: 6. E RROR ANALYSIS : L EAST - TO - MOST PROMPTING

Original: Error type 2 examples 4 examples
Revised: Error type 2 examples 4 examples

Original: L =
Revised: L =

Original: L = L =
Revised: L = L =

Original: L =
Revised: L =

Original: Concatenation error    -
Revised: Concatenation error    -

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Wrong
Revised: Wrong

Original: [Table]
Revised: [Table]

Original: Incorrect
Revised: Incorrect

Original: [Table]
Revised: [Table]

Original: letter
Revised: letter

Original: Copy error
Revised: Copy error

Original: For least-to-most prompting, we analyzed 20 random failures of the code-davinci-002 model on list lengths 4 and 12 for prompt contexts consisting of 2 and 4 examples.
Revised: For least-to-most prompting, we analyzed 20 random failures of the code-davinci-002 model on list lengths 4 and 12 for prompt contexts consisting of 2 and 4 examples.

Original: The results are shown in
Revised: The results are shown in

Original: Table 14.
Revised: Table 14.

Original: Concatenation errors may either be due to dropping a letter, adding a letter or outputting the letters in the wrong order.
Revised: Concatenation errors may either be due to dropping a letter, adding a letter or outputting the letters in the wrong order.

Original: Wrong template means that the model used the extension template instead of the base template to concatenate the last letter of the ﬁrst two words of the list.
Revised: Wrong template means that the model used the extension template instead of the base template to concatenate the last letter of the ﬁrst two words of the list.

Original: Incorrect last letter means that the model got the last letter of a word wrong, and copy error means that the error was due to making a mistake when copying an intermediate result.
Revised: Incorrect last letter means that the model got the last letter of a word wrong, and copy error means that the error was due to making a mistake when copying an intermediate result.

Original: We observe that for the prompt consisting of 2 examples, the fraction of concatenation errors increases as we go from length 4 to length 12 while the fraction of wrong template errors go down.
Revised: We observe that for the prompt consisting of 2 examples, the fraction of concatenation errors increases as we go from length 4 to length 12 while the fraction of wrong template errors go down.

Original: This makes sense because the number of concatenations grows with the length of the list, while the number of times the model needs to use the base template stays constant.
Revised: This makes sense because the number of concatenations grows with the length of the list, while the number of times the model needs to use the base template stays constant.

Original: Note that the template errors disappear when we move to the double prompt, which means that adding two more examples helps the model recognize which template to use.
Revised: Note that the template errors disappear when we move to the double prompt, which means that adding two more examples helps the model recognize which template to use.

Original: As a consequence, the double prompt has a similar distribution of errors for both list lengths.
Revised: As a consequence, the double prompt has a similar distribution of errors for both list lengths.

Original: Examples of concatenation errors.
Revised: Examples of concatenation errors.

Original: In the example “gratiﬁed, contract, fortitude, blew”, the model drops the last letter in the concatenation of “dte” and “w”, which means that it predicts the last letter sequence to be “dte” instead of “dtew”.
Revised: In the example “gratiﬁed, contract, fortitude, blew”, the model drops the last letter in the concatenation of “dte” and “w”, which means that it predicts the last letter sequence to be “dte” instead of “dtew”.

Original: In the example “hollow, supplies, function, gorgeous”, the model duplicates the last letter “s” in the concatenation of “wsn” and “s”, which means that it predicts the last letter sequence “wsnss” instead of “wsns”.
Revised: In the example “hollow, supplies, function, gorgeous”, the model duplicates the last letter “s” in the concatenation of “wsn” and “s”, which means that it predicts the last letter sequence “wsnss” instead of “wsns”.

Original: In the example “madly, vengeance, cowardice, monk”, the model drops the last letter “k” in the concatenation of “yee” and ”k” and instead adds the letter “g”.
Revised: In the example “madly, vengeance, cowardice, monk”, the model drops the last letter “k” in the concatenation of “yee” and ”k” and instead adds the letter “g”.

Original: Consequently, the model predicts “yeeg” instead of “yeek”.
Revised: Consequently, the model predicts “yeeg” instead of “yeek”.

Original: In the example “slender, lash, throng, scheme”, the model breaks the order of the letters “h” and “g” in the concatenation of “rh” and “g”, which means that it predicts the last letter sequence “rghe” instead of “rhge”.
Revised: In the example “slender, lash, throng, scheme”, the model breaks the order of the letters “h” and “g” in the concatenation of “rh” and “g”, which means that it predicts the last letter sequence “rghe” instead of “rhge”.

Original: Example of incorrect last letter.
Revised: Example of incorrect last letter.

Original: In the example “modiﬁcation, introducing, schools, lunch”, the model determines the last letter of the word “modiﬁcation” to be “g”.
Revised: In the example “modiﬁcation, introducing, schools, lunch”, the model determines the last letter of the word “modiﬁcation” to be “g”.

Original: Consequently, the predicted last letter sequence is “ggsh” instead of “ngsh”.
Revised: Consequently, the predicted last letter sequence is “ggsh” instead of “ngsh”.

Original: Example of wrong template application.
Revised: Example of wrong template application.

Original: In the example “upper, unexpectedly, speciﬁcally, connection”, the model uses the extension template to determine the output of the ﬁrst two words “upper, unexpectedly”.
Revised: In the example “upper, unexpectedly, speciﬁcally, connection”, the model uses the extension template to determine the output of the ﬁrst two words “upper, unexpectedly”.

Original: I.e., it produces:
Revised: I.e., it produces:

Original: • “upper” outputs “er”.
Revised: • “upper” outputs “er”.

Original: The last letter of “unexpectedly” is “y”.
Revised: The last letter of “unexpectedly” is “y”.

Original: Concatenating “er”, “y” leads to “ery”.
Revised: Concatenating “er”, “y” leads to “ery”.

Original: So, “upper, unexpectedly” outputs “ery”.
Revised: So, “upper, unexpectedly” outputs “ery”.

Original: when it should have produced:
Revised: when it should have produced:

Original: • The last letter of “upper” is “r”.
Revised: • The last letter of “upper” is “r”.

Original: The last letter of “unexpectedly” is “y”.
Revised: The last letter of “unexpectedly” is “y”.

Original: Concatenating “r”, “y” leads to “ry”.
Revised: Concatenating “r”, “y” leads to “ry”.

Original: As a consequence, the model predicts the sequence “eryyn” instead of “ryyn”.
Revised: As a consequence, the model predicts the sequence “eryyn” instead of “ryyn”.

Original: Example of copy error.
Revised: Example of copy error.

Original: In the example “displeased, hawk, healing, anchor”, the model correctly computes the ﬁnal concatenation “dkgr” but then makes a mistake when copying the result to the ﬁnal output and drops the ﬁnal “r”, which leads to the prediction “dkg” instead of “dkgr”:
Revised: In the example “displeased, hawk, healing, anchor”, the model correctly computes the ﬁnal concatenation “dkgr” but then makes a mistake when copying the result to the ﬁnal output and drops the ﬁnal “r”, which leads to the prediction “dkg” instead of “dkgr”:

Original: • “displeased, hawk, healing” outputs “dkg”.
Revised: • “displeased, hawk, healing” outputs “dkg”.

Original: The last letter of “anchor” is “r”.
Revised: The last letter of “anchor” is “r”.

Original: Concatenating: “dkg”, “r” leads to “dkgr”.
Revised: Concatenating: “dkg”, “r” leads to “dkgr”.

Original: So, “displeased, hawk, healing, anchor” outputs “dkg”.
Revised: So, “displeased, hawk, healing, anchor” outputs “dkg”.

Original: 6. E XAMPLE OUTPUTS FROM CODE - DAVINCI - 
Revised: 6. E XAMPLE OUTPUTS FROM CODE - DAVINCI -

Original: In this section, we present 5 examples (one example for lengths 4, 6, 8, 10, 12) of successes and failures of the code-davinci-002 model for standard prompting, chain-of-thought prompting, and least-to-most prompting.
Revised: In this section, we present 5 examples (one example for lengths 4, 6, 8, 10, 12) of successes and failures of the code-davinci-002 model for standard prompting, chain-of-thought prompting, and least-to-most prompting.

Original: They are all based on the prompt contexts containing 4 demonstration examples.
Revised: They are all based on the prompt contexts containing 4 demonstration examples.

Original: 6.5. S TANDARD PROMPTING : F AILURE
Revised: 6.5. S TANDARD PROMPTING : F AILURE

Original: “witnesses, arkansas, remaining, privileged”
Revised: “witnesses, arkansas, remaining, privileged”

Original: Golden
Revised: Golden

Original: “ssgd”
Revised: “ssgd”

Original: [Table]
Revised: [Table]

Original: 6.5. C HAIN - OF - THOUGHT PROMPTING : S UCCESS
Revised: 6.5. C HAIN - OF - THOUGHT PROMPTING : S UCCESS

Original: [Table]
Revised: [Table]

Original: version”
Revised: version”

Original: [Table]
Revised: [Table]

Original: , version”
Revised: , version”

Original: [Table]
Revised: [Table]

Original: : The last letter of “geese” is “e”.
Revised: : The last letter of “geese” is “e”.

Original: The last letter of “rogers” is “s”.
Revised: The last letter of “rogers” is “s”.

Original: The last letter of “benefactor” is “r”.
Revised: The last letter of “benefactor” is “r”.

Original: The last letter of “version” is “n”.
Revised: The last letter of “version” is “n”.

Original: Concatenating “e”, “s”, “r”, “n” leads to “esrn”.
Revised: Concatenating “e”, “s”, “r”, “n” leads to “esrn”.

Original: So, “geese, rogers,
Revised: So, “geese, rogers,

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: leads to
Revised: leads to

Original: [Table]
Revised: [Table]

Original: So, “nigel, alaska, rent, hospital, newspaper, medical, nation, dave, associated, rumour” outputs “latlrlnedr”.
Revised: So, “nigel, alaska, rent, hospital, newspaper, medical, nation, dave, associated, rumour” outputs “latlrlnedr”.

Original: “brightness, preparatory, intense, accumulated, reluctantly, nets, inquiries, grip, inquisition, alley, innocent, herbert”
Revised: “brightness, preparatory, intense, accumulated, reluctantly, nets, inquiries, grip, inquisition, alley, innocent, herbert”

Original: Golden: “syedysspnytt”
Revised: Golden: “syedysspnytt”

Original: Solution:
Revised: Solution:

Original: Q: “brightness, preparatory, intense, accumulated, reluctantly, nets, inquiries, grip, inquisition, alley, innocent, herbert”
Revised: Q: “brightness, preparatory, intense, accumulated, reluctantly, nets, inquiries, grip, inquisition, alley, innocent, herbert”

Original: A: The last letter of “brightness” is “s”.
Revised: A: The last letter of “brightness” is “s”.

Original: The last letter of “preparatory” is “y”.
Revised: The last letter of “preparatory” is “y”.

Original: The last letter of “intense” is “e”.
Revised: The last letter of “intense” is “e”.

Original: The last letter of “accumulated” is “d”.
Revised: The last letter of “accumulated” is “d”.

Original: The last letter of “reluctantly” is “y”.
Revised: The last letter of “reluctantly” is “y”.

Original: The last letter of “nets” is “s”. The last letter of “inquiries” is “s”.
Revised: The last letter of “nets” is “s”. The last letter of “inquiries” is “s”.

Original: The last letter of “grip” is “p”.
Revised: The last letter of “grip” is “p”.

Original: The last letter of “inquisition” is “n”.
Revised: The last letter of “inquisition” is “n”.

Original: The last letter of “alley” is “y”.
Revised: The last letter of “alley” is “y”.

Original: The last letter of “innocent” is “t”.
Revised: The last letter of “innocent” is “t”.

Original: The last letter of
Revised: The last letter of

Original: “herbert” is “t”.
Revised: “herbert” is “t”.

Original: Concatenating “s”, “y”, “e”, “d”, “y”, “s”, “s”, “p”, “n”, “y”, “t”, “t” leads to “syedysspnytt”.
Revised: Concatenating “s”, “y”, “e”, “d”, “y”, “s”, “s”, “p”, “n”, “y”, “t”, “t” leads to “syedysspnytt”.

Original: So, “brightness, preparatory, intense, accumulated, reluctantly, nets, inquiries, grip, inquisition, alley, innocent, herbert” outputs “syedysspnytt”.
Revised: So, “brightness, preparatory, intense, accumulated, reluctantly, nets, inquiries, grip, inquisition, alley, innocent, herbert” outputs “syedysspnytt”.

Original: 6.5. C HAIN - OF - THOUGHT PROMPTING : F AILURE
Revised: 6.5. C HAIN - OF - THOUGHT PROMPTING : F AILURE

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: “freezing, arduous, pushing, vitality, hue
Revised: “freezing, arduous, pushing, vitality, hue

Original: [Table]
Revised: [Table]

Original: , white, motto, matthew, preferred, hounds, breast”
Revised: , white, motto, matthew, preferred, hounds, breast”

Original: Golden: “gsgyeyeowdst”
Revised: Golden: “gsgyeyeowdst”

Original: Solution:
Revised: Solution:

Original: Q: “freezing, arduous, pushing, vitality, hue, generosity, white, motto, matthew, preferred, hounds, breast”A:The last letter of “freezing” is “g”.
Revised: Q: “freezing, arduous, pushing, vitality, hue, generosity, white, motto, matthew, preferred, hounds, breast”A:The last letter of “freezing” is “g”.

Original: The last letter of “arduous” is “s”.
Revised: The last letter of “arduous” is “s”.

Original: The last letter of “pushing” is “g”.
Revised: The last letter of “pushing” is “g”.

Original: The last letter of “vitality” is “y”.
Revised: The last letter of “vitality” is “y”.

Original: The last letter of “hue” is “e”.
Revised: The last letter of “hue” is “e”.

Original: The last letter of “generosity” is “y”.
Revised: The last letter of “generosity” is “y”.

Original: The last letter of “white” is “e”.
Revised: The last letter of “white” is “e”.

Original: The last letter of “motto” is “o”.
Revised: The last letter of “motto” is “o”.

Original: The last letter of “matthew” is “w”.
Revised: The last letter of “matthew” is “w”.

Original: The last letter of “preferred” is “d”.
Revised: The last letter of “preferred” is “d”.

Original: The last letter of “hounds” is “s”.
Revised: The last letter of “hounds” is “s”.

Original: The last letter of “breast” is “t”.
Revised: The last letter of “breast” is “t”.

Original: Concatenating “g”, “s”, “g”, “y”, “e”, “y”, “e”, “o”, “w”, “d”, “s”, “t” leads to “gsgyeyowdst”.
Revised: Concatenating “g”, “s”, “g”, “y”, “e”, “y”, “e”, “o”, “w”, “d”, “s”, “t” leads to “gsgyeyowdst”.

Original: So, “freezing, arduous, pushing, vitality, hue, generosity, white, motto, matthew, preferred, hounds, breast” outputs “gsgyeyowdst”.
Revised: So, “freezing, arduous, pushing, vitality, hue, generosity, white, motto, matthew, preferred, hounds, breast” outputs “gsgyeyowdst”.

Original: 6.5. L EAST - TO - MOST PROMPTING : S UCCESS
Revised: 6.5. L EAST - TO - MOST PROMPTING : S UCCESS

Original: “worm
Revised: “worm

Original: [Equation]
Revised: [Equation]

Original: jackson
Revised: jackson

Original: [Equation]
Revised: [Equation]

Original: Golden: “mnwr”
Revised: Golden: “mnwr”

Original: Solution: Q: “worm, jackson”
Revised: Solution: Q: “worm, jackson”

Original: A: The last letter of “worm” is “m”.
Revised: A: The last letter of “worm” is “m”.

Original: The last letter of “jackson” is “n”.
Revised: The last letter of “jackson” is “n”.

Original: Concatenating: “m”, “n” leads to “mn”.
Revised: Concatenating: “m”, “n” leads to “mn”.

Original: So, “worm, jackson” outputs “mn”.
Revised: So, “worm, jackson” outputs “mn”.

Original: Q: “worm, jackson, widow” A: “worm, jackson” outputs “mn”.
Revised: Q: “worm, jackson, widow” A: “worm, jackson” outputs “mn”.

Original: The last letter of “widow” is “w”.
Revised: The last letter of “widow” is “w”.

Original: Concatenating: “mn”, “w” leads to “mnw”.
Revised: Concatenating: “mn”, “w” leads to “mnw”.

Original: So, “worm, jackson, widow”
Revised: So, “worm, jackson, widow”

Original: [Table] [Table]
Revised: [Table] [Table]

Original: The last letter of
Revised: The last letter of

Original: [Table]
Revised: [Table]

Original: is “o”.
Revised: is “o”.

Original: The last letter of “observations” is “s”.
Revised: The last letter of “observations” is “s”.

Original: Concatenating
Revised: Concatenating

Original: [Table]
Revised: [Table]

Original: “o”, “s” leads
Revised: “o”, “s” leads

Original: [Table]
Revised: [Table]

Original: “os”.
Revised: “os”.

Original: So, “leo, observations” outputs “os”.
Revised: So, “leo, observations” outputs “os”.

Original: [Table]
Revised: [Table]

Original: “leo, observations, power”
Revised: “leo, observations, power”

Original: [Table]
Revised: [Table]

Original: “leo, observations” outputs “os”.
Revised: “leo, observations” outputs “os”.

Original: The last letter of “power” is “r”.
Revised: The last letter of “power” is “r”.

Original: Concatenating: “os”, “r” leads to “osr”.
Revised: Concatenating: “os”, “r” leads to “osr”.

Original: So, “leo
Revised: So, “leo

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: “osrutysge”
Revised: “osrutysge”

Original: [Table]
Revised: [Table]

Original: So
Revised: So

Original: [Table]
Revised: [Table]

Original: Solution:
Revised: Solution:

Original: [Table]
Revised: [Table]

Original: greedy”
Revised: greedy”

Original: A: The last letter of “niece” is “e”.
Revised: A: The last letter of “niece” is “e”.

Original: The last letter of “greedy” is “y”.
Revised: The last letter of “greedy” is “y”.

Original: Concatenating: “e”, “y” leads to “ey”.
Revised: Concatenating: “e”, “y” leads to “ey”.

Original: So, “niece, greedy” outputs “ey”.
Revised: So, “niece, greedy” outputs “ey”.

Original: Q: “niece, greedy, anything” A: “niece, greedy” outputs “ey”.
Revised: Q: “niece, greedy, anything” A: “niece, greedy” outputs “ey”.

Original: The last letter of “anything” is “g”.
Revised: The last letter of “anything” is “g”.

Original: Concatenating: “ey”, “g” leads to “eyg”.
Revised: Concatenating: “ey”, “g” leads to “eyg”.

Original: So, “niece, greedy, anything” outputs “eyg”.
Revised: So, “niece, greedy, anything” outputs “eyg”.

Original: Q: “niece, greedy, anything, retain” A: “niece, greedy, anything” outputs “eyg”.
Revised: Q: “niece, greedy, anything, retain” A: “niece, greedy, anything” outputs “eyg”.

Original: The last letter of “retain” is “n”.
Revised: The last letter of “retain” is “n”.

Original: Concatenating: “eyg”, “n”
Revised: Concatenating: “eyg”, “n”

Original: leads to
Revised: leads to

Original: “eygn”.
Revised: “eygn”.

Original: So, “niece, greedy, anything, retain” outputs “eygn”.
Revised: So, “niece, greedy, anything, retain” outputs “eygn”.

Original: Q: “niece, greedy, anything, retain, moisture” A: “niece, greedy, anything, retain” outputs “eygn”.
Revised: Q: “niece, greedy, anything, retain, moisture” A: “niece, greedy, anything, retain” outputs “eygn”.

Original: The last letter of “moisture” is “e”.
Revised: The last letter of “moisture” is “e”.

Original: Concatenating
Revised: Concatenating

Original: “eygn”, “e” leads to “eygne”.
Revised: “eygn”, “e” leads to “eygne”.

Original: So, “niece, greedy, anything, retain, moisture”
Revised: So, “niece, greedy, anything, retain, moisture”

Original: [Table]
Revised: [Table]

Original: anything, retain, moisture, jest, adrian, attic, hugo, accurate” outputs “eygnetncoe”.
Revised: anything, retain, moisture, jest, adrian, attic, hugo, accurate” outputs “eygnetncoe”.

Original: The last letter of “wet” is “t”.
Revised: The last letter of “wet” is “t”.

Original: Concatenating: “eygnetncoe”, “t” leads to “eygnetncoet”.
Revised: Concatenating: “eygnetncoe”, “t” leads to “eygnetncoet”.

Original: So, “niece, greedy, anything, retain, moisture, jest, adrian, attic, hugo, accurate, wet” outputs “eygnetncoet”.
Revised: So, “niece, greedy, anything, retain, moisture, jest, adrian, attic, hugo, accurate, wet” outputs “eygnetncoet”.

Original: Q: “niece, greedy, anything, retain, moisture, jest, adrian, attic, hugo, accurate, wet, howard”
Revised: Q: “niece, greedy, anything, retain, moisture, jest, adrian, attic, hugo, accurate, wet, howard”

Original: A: “niece, greedy, anything, retain, moisture, jest, adrian, attic, hugo, accurate, wet” outputs “eygnetncoet”.
Revised: A: “niece, greedy, anything, retain, moisture, jest, adrian, attic, hugo, accurate, wet” outputs “eygnetncoet”.

Original: The last letter of “howard” is “d”.
Revised: The last letter of “howard” is “d”.

Original: Concatenating: “eygnetncoet”, “d” leads to “eygnetncoetd”.
Revised: Concatenating: “eygnetncoet”, “d” leads to “eygnetncoetd”.

Original: So, “niece, greedy, anything, retain, moisture, jest, adrian, attic, hugo, accurate, wet, howard” outputs “eygnetncoetd”.
Revised: So, “niece, greedy, anything, retain, moisture, jest, adrian, attic, hugo, accurate, wet, howard” outputs “eygnetncoetd”.

Original: 6.5. L EAST - TO - MOST PROMPTING : F AILURE “supper, procession, region, ruby”
Revised: 6.5. L EAST - TO - MOST PROMPTING : F AILURE “supper, procession, region, ruby”

Original: Golden: “rnny”
Revised: Golden: “rnny”

Original: Solution: Q: “supper, procession”
Revised: Solution: Q: “supper, procession”

Original: [Table]
Revised: [Table]

Original: to
Revised: to

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: “release, appeared
Revised: “release, appeared

Original: [Table] [Table]
Revised: [Table] [Table]

Original: “mentally, uncomfortable, amount”
Revised: “mentally, uncomfortable, amount”

Original: [Table]
Revised: [Table]

Original: “yet”.
Revised: “yet”.

Original: [Table]
Revised: [Table]

Original: “mentally, uncomfortable, amount, straight” A: “mentally, uncomfortable, amount” outputs “yet”.
Revised: “mentally, uncomfortable, amount, straight” A: “mentally, uncomfortable, amount” outputs “yet”.

Original: The last letter of “straight” is “t”.
Revised: The last letter of “straight” is “t”.

Original: Concatenating: “yet”, “t” leads to “yett”.
Revised: Concatenating: “yet”, “t” leads to “yett”.

Original: So, “mentally, uncomfortable, amount, straight” outputs “yett”.
Revised: So, “mentally, uncomfortable, amount, straight” outputs “yett”.

Original: [Table]
Revised: [Table]

Original: : “mentally, uncomfortable, amount, straight, whatsoever” A: “mentally, uncomfortable, amount, straight” outputs “yett”.
Revised: : “mentally, uncomfortable, amount, straight, whatsoever” A: “mentally, uncomfortable, amount, straight” outputs “yett”.

Original: The last letter of “whatsoever” is “r”.
Revised: The last letter of “whatsoever” is “r”.

Original: Concatenating: “yett”, “r” leads to “yettar”.
Revised: Concatenating: “yett”, “r” leads to “yettar”.

Original: So, “mentally, uncomfortable, amount, straight, whatsoever” outputs “yettar”.
Revised: So, “mentally, uncomfortable, amount, straight, whatsoever” outputs “yettar”.

Original: [Table]
Revised: [Table]

Original: : “mentally, uncomfortable, amount, straight, whatsoever, deference” A: “mentally, uncomfortable, amount, straight, whatsoever” outputs “yettar”.
Revised: : “mentally, uncomfortable, amount, straight, whatsoever, deference” A: “mentally, uncomfortable, amount, straight, whatsoever” outputs “yettar”.

Original: The last letter of “deference” is “e”.
Revised: The last letter of “deference” is “e”.

Original: Concatenating: “yettar”, “e” leads to “yettare”.
Revised: Concatenating: “yettar”, “e” leads to “yettare”.

Original: So, “mentally, uncomfortable, amount, straight, whatsoever, deference” outputs “yettare”.
Revised: So, “mentally, uncomfortable, amount, straight, whatsoever, deference” outputs “yettare”.

Original: [Table]
Revised: [Table]

Original: : “mentally, uncomfortable, amount, straight, whatsoever, deference, channel” A: “mentally, uncomfortable, amount, straight, whatsoever, deference” outputs “yettare”.
Revised: : “mentally, uncomfortable, amount, straight, whatsoever, deference, channel” A: “mentally, uncomfortable, amount, straight, whatsoever, deference” outputs “yettare”.

Original: The last letter of “channel” is “l”.
Revised: The last letter of “channel” is “l”.

Original: Concatenating: “yettare”, “l” leads to “yettarel”.
Revised: Concatenating: “yettare”, “l” leads to “yettarel”.

Original: So, “mentally, uncomfortable, amount, straight, whatsoever, deference, channel” outputs “yettarel”.
Revised: So, “mentally, uncomfortable, amount, straight, whatsoever, deference, channel” outputs “yettarel”.

Original: Q: “mentally, uncomfortable, amount, straight, whatsoever, deference, channel, integrity” A: “mentally, uncomfortable, amount, straight, whatsoever, deference, channel” outputs “yettarel”.
Revised: Q: “mentally, uncomfortable, amount, straight, whatsoever, deference, channel, integrity” A: “mentally, uncomfortable, amount, straight, whatsoever, deference, channel” outputs “yettarel”.

Original: The last letter of “integrity” is “y”.
Revised: The last letter of “integrity” is “y”.

Original: [Table]
Revised: [Table]

Original: “yettarel”, “y” leads to “yettarely”.
Revised: “yettarel”, “y” leads to “yettarely”.

Original: So, “mentally, uncomfortable, amount, straight, whatsoever, deference, channel, integrity” outputs “yettarely”.
Revised: So, “mentally, uncomfortable, amount, straight, whatsoever, deference, channel, integrity” outputs “yettarely”.

Original: “complexion, criminal, assume, seven, driver, hero, horse, purpose, zeus, joint”
Revised: “complexion, criminal, assume, seven, driver, hero, horse, purpose, zeus, joint”

Original: [Table] [Table]
Revised: [Table] [Table]

Original: leads to “darl”.
Revised: leads to “darl”.

Original: So, “provoked, columbia, sir, principal” outputs “darl”.
Revised: So, “provoked, columbia, sir, principal” outputs “darl”.

Original: Q: “provoked, columbia, sir, principal, seven”
Revised: Q: “provoked, columbia, sir, principal, seven”

Original: A: “provoked, columbia, sir, principal” outputs “darl”.
Revised: A: “provoked, columbia, sir, principal” outputs “darl”.

Original: The last letter of “seven” is “n”.
Revised: The last letter of “seven” is “n”.

Original: Concatenating: “darl”, “n” leads to “darln”.
Revised: Concatenating: “darl”, “n” leads to “darln”.

Original: So, “provoked, columbia, sir, principal, seven” outputs “darln”.
Revised: So, “provoked, columbia, sir, principal, seven” outputs “darln”.

Original: Q: “provoked, columbia, sir, principal, seven, extreme”
Revised: Q: “provoked, columbia, sir, principal, seven, extreme”

Original: A: “provoked, columbia, sir, principal, seven” outputs “darln”.
Revised: A: “provoked, columbia, sir, principal, seven” outputs “darln”.

Original: The last letter of “extreme” is “e”.
Revised: The last letter of “extreme” is “e”.

Original: Concatenating: “darln”, “e” leads to “darle”.
Revised: Concatenating: “darln”, “e” leads to “darle”.

Original: So, “provoked, columbia, sir, principal, seven, extreme” outputs “darle”.
Revised: So, “provoked, columbia, sir, principal, seven, extreme” outputs “darle”.

Original: Q: “provoked, columbia, sir, principal, seven, extreme, timothy”
Revised: Q: “provoked, columbia, sir, principal, seven, extreme, timothy”

Original: A: “provoked, columbia, sir, principal, seven, extreme” outputs “darle”.
Revised: A: “provoked, columbia, sir, principal, seven, extreme” outputs “darle”.

Original: The last letter of “timothy” is “y”.
Revised: The last letter of “timothy” is “y”.

Original: Concatenating: “darle”, “y” leads to “darley”.
Revised: Concatenating: “darle”, “y” leads to “darley”.

Original: So, “provoked, columbia, sir, principal, seven, extreme, timothy” outputs “darley”.
Revised: So, “provoked, columbia, sir, principal, seven, extreme, timothy” outputs “darley”.

Original: Q: “provoked, columbia, sir, principal, seven, extreme, timothy, submitted”
Revised: Q: “provoked, columbia, sir, principal, seven, extreme, timothy, submitted”

Original: A: “provoked, columbia, sir, principal, seven, extreme, timothy” outputs “darley”.
Revised: A: “provoked, columbia, sir, principal, seven, extreme, timothy” outputs “darley”.

Original: The last letter of “submitted” is “d”.
Revised: The last letter of “submitted” is “d”.

Original: Concatenating: “darley”, “d” leads to “darleyd”.
Revised: Concatenating: “darley”, “d” leads to “darleyd”.

Original: So, “provoked, columbia, sir, principal, seven, extreme, timothy, submitted” outputs “darleyd”.
Revised: So, “provoked, columbia, sir, principal, seven, extreme, timothy, submitted” outputs “darleyd”.

Original: Q: “provoked, columbia, sir, principal, seven, extreme, timothy, submitted, considered”
Revised: Q: “provoked, columbia, sir, principal, seven, extreme, timothy, submitted, considered”

Original: A: “provoked, columbia, sir, principal, seven, extreme, timothy, submitted” outputs “darleyd”.
Revised: A: “provoked, columbia, sir, principal, seven, extreme, timothy, submitted” outputs “darleyd”.

Original: The last letter of “considered” is “d”.
Revised: The last letter of “considered” is “d”.

Original: Concatenating: “darleyd”, “d” leads to “darleydd”.
Revised: Concatenating: “darleyd”, “d” leads to “darleydd”.

Original: So, “provoked, columbia, sir, principal, seven, extreme, timothy, submitted, considered” outputs “darleydd”.
Revised: So, “provoked, columbia, sir, principal, seven, extreme, timothy, submitted, considered” outputs “darleydd”.

Original: Q: “provoked, columbia, sir, principal, seven, extreme, timothy, submitted, considered, spies”
Revised: Q: “provoked, columbia, sir, principal, seven, extreme, timothy, submitted, considered, spies”

Original: A: “provoked, columbia, sir, principal, seven, extreme, timothy, submitted, considered” outputs “darleydd”.
Revised: A: “provoked, columbia, sir, principal, seven, extreme, timothy, submitted, considered” outputs “darleydd”.

Original: The last letter of “spies” is “s”.
Revised: The last letter of “spies” is “s”.

Original: Concatenating: “darleydd”, “s” leads to “darleydds”.
Revised: Concatenating: “darleydd”, “s” leads to “darleydds”.

Original: So, “provoked, columbia, sir, principal, seven, extreme, timothy, submitted, considered, spies” outputs “darleydds”.
Revised: So, “provoked, columbia, sir, principal, seven, extreme, timothy, submitted, considered, spies” outputs “darleydds”.

Original: Q: “provoked, columbia, sir, principal, seven, extreme, timothy, submitted, considered, spies, dost”A:“provoked, columbia, sir, principal,
Revised: Q: “provoked, columbia, sir, principal, seven, extreme, timothy, submitted, considered, spies, dost”A:“provoked, columbia, sir, principal,

Original: seven
Revised: seven

Original: , extreme, timothy,
Revised: , extreme, timothy,

Original: submitted
Revised: submitted

Original: , considered, spies” outputs “darleydds”.
Revised: , considered, spies” outputs “darleydds”.

Original: The last letter of “dost” is “t”.
Revised: The last letter of “dost” is “t”.

Original: Concatenating: “darleydds”, “t” leads to “darleyddst”.
Revised: Concatenating: “darleydds”, “t” leads to “darleyddst”.

Original: So, “provoked, columbia, sir, principal, seven, extreme, timothy, submitted, considered, spies, dost” outputs “darleyddst”.
Revised: So, “provoked, columbia, sir, principal, seven, extreme, timothy, submitted, considered, spies, dost” outputs “darleyddst”.

Original: Q: “provoked, columbia, sir, principal, seven,
Revised: Q: “provoked, columbia, sir, principal, seven,

Original: extreme
Revised: extreme

Original: , timothy, submitted, considered, spies, dost,
Revised: , timothy, submitted, considered, spies, dost,

Original: treasury”
Revised: treasury”

Original: A: “provoked,
Revised: A: “provoked,

Original: columbia
Revised: columbia

Original: sir
Revised: sir

Original: principal, seven
Revised: principal, seven

Original: extreme
Revised: extreme

Original: timothy
Revised: timothy

Original: submitted
Revised: submitted

Original: , considered, spies, dost” outputs “darleyddst”.
Revised: , considered, spies, dost” outputs “darleyddst”.

Original: The last letter of “treasury” is “y”.
Revised: The last letter of “treasury” is “y”.

Original: Concatenating: “darleyddst”, “y” leads to “darleyddsty”.
Revised: Concatenating: “darleyddst”, “y” leads to “darleyddsty”.

Original: So, “provoked,
Revised: So, “provoked,

Original: columbia
Revised: columbia

Original: , sir, principal,
Revised: , sir, principal,

Original: seven
Revised: seven

Original: extreme
Revised: extreme

Original: , timothy,
Revised: , timothy,

Original: submitted
Revised: submitted

Original: , considered, spies,
Revised: , considered, spies,

Original: dost
Revised: dost

Original: , treasury” outputs “darleyddsty”.
Revised: , treasury” outputs “darleyddsty”.

Original:  SCAN 7. P ROMPT CONTEXTS
Revised:  SCAN 7. P ROMPT CONTEXTS

Original: In this section we present the prompt contexts used for the SCAN benchmark in Section 3.2.
Revised: In this section we present the prompt contexts used for the SCAN benchmark in Section 3.2.

Original: It includes one context for each of standard prompting, least-to-most prompting, and chain-of-thought prompting.
Revised: It includes one context for each of standard prompting, least-to-most prompting, and chain-of-thought prompting.

Original: 7.1. S TANDARD PROMPTING
Revised: 7.1. S TANDARD PROMPTING

Original: The context for standard prompting consist of a set of commands together with the corresponding action sequences.
Revised: The context for standard prompting consist of a set of commands together with the corresponding action sequences.

Original: Q: “turn left”
Revised: Q: “turn left”

Original: A: “TURN LEFT”
Revised: A: “TURN LEFT”

Original: Q: “turn right”
Revised: Q: “turn right”

Original: A: “TURN RIGHT”
Revised: A: “TURN RIGHT”

Original: Q: “jump left”
Revised: Q: “jump left”

Original: A: “TURN LEFT” + “JUMP”
Revised: A: “TURN LEFT” + “JUMP”

Original: Q: “run right”
Revised: Q: “run right”

Original: A: “TURN RIGHT” + “RUN”
Revised: A: “TURN RIGHT” + “RUN”

Original: Q: “look twice”
Revised: Q: “look twice”

Original: A: “LOOK” *
Revised: A: “LOOK” *

Original: Q: “run and look twice”
Revised: Q: “run and look twice”

Original: A: “RUN” + “LOOK” *
Revised: A: “RUN” + “LOOK” *

Original: Q: “jump right thrice”
Revised: Q: “jump right thrice”

Original: A: (“TURN RIGHT” + “JUMP”)
Revised: A: (“TURN RIGHT” + “JUMP”)

Original: Q: “walk after run”
Revised: Q: “walk after run”

Original: A: “RUN” + “WALK”
Revised: A: “RUN” + “WALK”

Original: Q: “turn opposite left”
Revised: Q: “turn opposite left”

Original: A: “TURN LEFT” *
Revised: A: “TURN LEFT” *

Original: Q: “turn around left”
Revised: Q: “turn around left”

Original: A: “TURN LEFT” *
Revised: A: “TURN LEFT” *

Original: Q: “turn opposite right”
Revised: Q: “turn opposite right”

Original: A: “TURN RIGHT” *
Revised: A: “TURN RIGHT” *

Original: Q: “turn around right”
Revised: Q: “turn around right”

Original: A: “TURN RIGHT” *
Revised: A: “TURN RIGHT” *

Original: Q: “walk opposite left”
Revised: Q: “walk opposite left”

Original: A: “TURN LEFT” * 2 + “WALK”
Revised: A: “TURN LEFT” * 2 + “WALK”

Original: Q: “walk around left”
Revised: Q: “walk around left”

Original: A: (“TURN LEFT” + “WALK”) *
Revised: A: (“TURN LEFT” + “WALK”) *

Original: 7.1. L EAST - TO - MOST PROMPTING
Revised: 7.1. L EAST - TO - MOST PROMPTING

Original: For least-to-most prompting, we need 2 different prompt contexts: one to decompose commands into a sequence of components and one to map commands to actions.
Revised: For least-to-most prompting, we need 2 different prompt contexts: one to decompose commands into a sequence of components and one to map commands to actions.

Original: Command mapping.
Revised: Command mapping.

Original: The command mapping prompt context consists of a set of commands together with the corresponding action sequences and explanations of how they were obtained.
Revised: The command mapping prompt context consists of a set of commands together with the corresponding action sequences and explanations of how they were obtained.

Original: Q: “turn left”
Revised: Q: “turn left”

Original: A: “turn left” outputs “TURN LEFT”.
Revised: A: “turn left” outputs “TURN LEFT”.

Original: Q: “turn right”
Revised: Q: “turn right”

Original: A: “turn right” outputs “TURN RIGHT”.
Revised: A: “turn right” outputs “TURN RIGHT”.

Original: So
Revised: So

Original: So
Revised: So

Original: Command decomposition.
Revised: Command decomposition.

Original: The command decomposition prompt context consists of a set of commands together with the corresponding sequence of decomposed commands.
Revised: The command decomposition prompt context consists of a set of commands together with the corresponding sequence of decomposed commands.

Original: Q: “look
Revised: Q: “look

Original: A: “look right
Revised: A: “look right

Original: be solved
Revised: be solved

Original: “look right”
Revised: “look right”

Original: Q
Revised: Q

Original: A
Revised: A

Original: “jump
Revised: “jump

Original: right”
Revised: right”

Original: right
Revised: right

Original: “walk”
Revised: “walk”

Original: A
Revised: A

Original: “run
Revised: “run

Original: So
Revised: So

Original: solved by
Revised: solved by

Original: Q
Revised: Q

Original: “run
Revised: “run

Original: right” A: “run opposite right”
Revised: right” A: “run opposite right”

Original: solved
Revised: solved

Original: opposite right”.
Revised: opposite right”.

Original: Q: “look opposite right
Revised: Q: “look opposite right

Original: A: “look opposite right thrice” can be solved by
Revised: A: “look opposite right thrice” can be solved by

Original: “look opposite right”, “look opposite right thrice”.
Revised: “look opposite right”, “look opposite right thrice”.

Original: “walk” can be solved by “walk”.
Revised: “walk” can be solved by “walk”.

Original: So, “look opposite right thrice after walk” can be solved by
Revised: So, “look opposite right thrice after walk” can be solved by

Original: “look opposite right”, “look opposite right thrice”, “walk”.
Revised: “look opposite right”, “look opposite right thrice”, “walk”.

Original: Q
Revised: Q

Original: “jump around right”
Revised: “jump around right”

Original: 7.1. C HAIN - OF - THOUGHT PROMPTING
Revised: 7.1. C HAIN - OF - THOUGHT PROMPTING

Original: We reuse the command mapping prompt context from least-to-most prompting shown above.
Revised: We reuse the command mapping prompt context from least-to-most prompting shown above.

Original: 7. E RROR ANALYSIS : L EAST - TO - MOST PROMPTING
Revised: 7. E RROR ANALYSIS : L EAST - TO - MOST PROMPTING

Original: For least-to-most prompting, we analyzed random failures for the models code-davinci-001 and text-davinci-002 , and we analyzed all 13 errors for the model code-davinci-002 .
Revised: For least-to-most prompting, we analyzed random failures for the models code-davinci-001 and text-davinci-002 , and we analyzed all 13 errors for the model code-davinci-002 .

Original: The results are shown in Table 15.
Revised: The results are shown in Table 15.

Original: Errors may either occur during command decomposition or during command translation.
Revised: Errors may either occur during command decomposition or during command translation.

Original: The translation errors are further split into the following types.
Revised: The translation errors are further split into the following types.

Original: Incorrect interpretation of “twice” and “thrice” means that the model made an error when applying “twice” and “thrice” to an expression.
Revised: Incorrect interpretation of “twice” and “thrice” means that the model made an error when applying “twice” and “thrice” to an expression.

Original: “After” interpreted as “and” means that the model translated an expression containing “after” as if it instead contained “and”.
Revised: “After” interpreted as “and” means that the model translated an expression containing “after” as if it instead contained “and”.

Original: Copy error means that the model made a mistake when copying an intermediate result.
Revised: Copy error means that the model made a mistake when copying an intermediate result.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: We observe that the best model text-davinci-002 made only 2 types of mistakes: it sometimes makes a mistake when applying “twice” and “thrice” to an expression containing “around”, and it sometimes interprets “after” as “and”.
Revised: We observe that the best model text-davinci-002 made only 2 types of mistakes: it sometimes makes a mistake when applying “twice” and “thrice” to an expression containing “around”, and it sometimes interprets “after” as “and”.

Original: For text-davinci-001 , which is the older version of the same model, the errors are spread across all types.
Revised: For text-davinci-001 , which is the older version of the same model, the errors are spread across all types.

Original: In particular, it’s worth noting that text-davinci-001 makes a signiﬁcant number of decomposition errors and copy errors that were completely eliminated by its successor.
Revised: In particular, it’s worth noting that text-davinci-001 makes a signiﬁcant number of decomposition errors and copy errors that were completely eliminated by its successor.

Original: The model text-davinci-002 made most of its mistakes when interpreting “twice” and “thrice” following “around”.
Revised: The model text-davinci-002 made most of its mistakes when interpreting “twice” and “thrice” following “around”.

Original: In addition, it sometimes made a mistake when translating “left” and “right”, which is something we did not observe with the other models.
Revised: In addition, it sometimes made a mistake when translating “left” and “right”, which is something we did not observe with the other models.

Original: In some cases, it dropped the command entirely, whereas in other cases it invented a new action such as “LOOK LEFT” (see examples below).
Revised: In some cases, it dropped the command entirely, whereas in other cases it invented a new action such as “LOOK LEFT” (see examples below).

Original: Examples of decomposition errors.
Revised: Examples of decomposition errors.

Original: In the example “run around left twice after jump around right thrice”, the code-davinci-001 model does not properly decompose the sub-expression “run around left twice”.
Revised: In the example “run around left twice after jump around right thrice”, the code-davinci-001 model does not properly decompose the sub-expression “run around left twice”.

Original: Instead of decomposing it to the sequence [“run left”, “run around left”, “run around left twice”], it skips “run around left” and decomposes it to [“run left”, “run around left twice”].
Revised: Instead of decomposing it to the sequence [“run left”, “run around left”, “run around left twice”], it skips “run around left” and decomposes it to [“run left”, “run around left twice”].

Original: Consequently, the model translates this sub-expression to (“TURN LEFT” + “RUN”) *instead of (“TURN LEFT” + “RUN”)
Revised: Consequently, the model translates this sub-expression to (“TURN LEFT” + “RUN”) *instead of (“TURN LEFT” + “RUN”)

Original: In the example “look around right twice after jump around left twice”, the code-davincimodel does not properly decompose the sub-expression “jump around left twice”.
Revised: In the example “look around right twice after jump around left twice”, the code-davincimodel does not properly decompose the sub-expression “jump around left twice”.

Original: Instead of decomposing it to the sequence [“jump left”, “jump around left”, “jump around left twice”], it skips “jump around left” and decomposes it to [“jump left”, “jump around left twice”].
Revised: Instead of decomposing it to the sequence [“jump left”, “jump around left”, “jump around left twice”], it skips “jump around left” and decomposes it to [“jump left”, “jump around left twice”].

Original: Interestingly, the model is able to recover from this mistake and correctly translates the sub-expression to (“TURN LEFT” + “JUMP”)
Revised: Interestingly, the model is able to recover from this mistake and correctly translates the sub-expression to (“TURN LEFT” + “JUMP”)

Original: * 4 * 2, but still produces the wrong ﬁnal action sequence because it interprets “after” like “and”.
Revised: * 4 * 2, but still produces the wrong ﬁnal action sequence because it interprets “after” like “and”.

Original: Examples of incorrect interpretation of “twice” and “thrice”.
Revised: Examples of incorrect interpretation of “twice” and “thrice”.

Original: In the example “walk opposite right twice after run around right thrice”, the code-davinci-002 model correctly translates the expression “run around right” to (“TURN RIGHT” + “RUN”) * 4.
Revised: In the example “walk opposite right twice after run around right thrice”, the code-davinci-002 model correctly translates the expression “run around right” to (“TURN RIGHT” + “RUN”) * 4.

Original: Then it makes a mistake when applying “thrice” to this expression and produces (“TURN RIGHT” + “RUN”)
Revised: Then it makes a mistake when applying “thrice” to this expression and produces (“TURN RIGHT” + “RUN”)

Original: * 9 instead of (“TURN RIGHT” + “RUN”) * 4 * 3 or (“TURN RIGHT” + “RUN”)
Revised: * 9 instead of (“TURN RIGHT” + “RUN”) * 4 * 3 or (“TURN RIGHT” + “RUN”)

Original: In the example “jump opposite right twice and jump around right thrice”, the code-davinci-002 model correctly translates the expression “jump around right” to (“TURN RIGHT” + “JUMP”)
Revised: In the example “jump opposite right twice and jump around right thrice”, the code-davinci-002 model correctly translates the expression “jump around right” to (“TURN RIGHT” + “JUMP”)

Original: Then it makes a mistake when applying “thrice” to this expression and produces (“TURN RIGHT” + “JUMP”)
Revised: Then it makes a mistake when applying “thrice” to this expression and produces (“TURN RIGHT” + “JUMP”)

Original: * 8 instead of (“TURN RIGHT” + “JUMP”)
Revised: * 8 instead of (“TURN RIGHT” + “JUMP”)

Original: ** 3 or (“TURN RIGHT” + “JUMP”)
Revised: ** 3 or (“TURN RIGHT” + “JUMP”)

Original: In the example “walk around left thrice after run opposite left thrice”, the code-davincimodel correctly translates the expression “run opposite left”
Revised: In the example “walk around left thrice after run opposite left thrice”, the code-davincimodel correctly translates the expression “run opposite left”

Original: In the example “walk around left thrice after look right twice”, the code-davinci-001 model correctly translates the expression “look right” to “TURN RIGHT” + “LOOK”.
Revised: In the example “walk around left thrice after look right twice”, the code-davinci-001 model correctly translates the expression “look right” to “TURN RIGHT” + “LOOK”.

Original: Then it makes a mistake when applying “twice” to this expression and produces “TURN RIGHT” + “LOOK” *rather than (“TURN RIGHT” + “LOOK”) * 2.
Revised: Then it makes a mistake when applying “twice” to this expression and produces “TURN RIGHT” + “LOOK” *rather than (“TURN RIGHT” + “LOOK”) * 2.

Original: In the example “walk left and run around right thrice”, the code-davinci-001 model interprets “thrice” as ‘twice”.
Revised: In the example “walk left and run around right thrice”, the code-davinci-001 model interprets “thrice” as ‘twice”.

Original: This means that it produces “TURN LEFT” + “WALK” + (“TURN RIGHT” + “RUN”) * 4 * 2 instead of “TURN LEFT” + “WALK” + (“TURN RIGHT” + “RUN”) * 4 * 3.
Revised: This means that it produces “TURN LEFT” + “WALK” + (“TURN RIGHT” + “RUN”) * 4 * 2 instead of “TURN LEFT” + “WALK” + (“TURN RIGHT” + “RUN”) * 4 * 3.

Original: In the example “jump right twice and look around left thrice”, the text-davinci-002 model correctly translates the sub-expression “look around left” to (“TURN LEFT” + “LOOK”) * 4.
Revised: In the example “jump right twice and look around left thrice”, the text-davinci-002 model correctly translates the sub-expression “look around left” to (“TURN LEFT” + “LOOK”) * 4.

Original: But when applying “thrice”, it produces the incorrect translation (“TURN LEFT” + “LOOK”)
Revised: But when applying “thrice”, it produces the incorrect translation (“TURN LEFT” + “LOOK”)

Original: * 3 instead of (“TURN LEFT” + “LOOK”) * 4 * 3.
Revised: * 3 instead of (“TURN LEFT” + “LOOK”) * 4 * 3.

Original: Example of interpreting “after” as “and”.
Revised: Example of interpreting “after” as “and”.

Original: In the example “run opposite left thrice after run around left twice”, the code-davinci-002 model produces the correct translations for both sub-expressions that are connected by “after”, but it combines them as if they were connected by “and”.
Revised: In the example “run opposite left thrice after run around left twice”, the code-davinci-002 model produces the correct translations for both sub-expressions that are connected by “after”, but it combines them as if they were connected by “and”.

Original: This means that the model produces (“TURN LEFT” * 2 + “RUN”)
Revised: This means that the model produces (“TURN LEFT” * 2 + “RUN”)

Original: * 3 + (“TURN LEFT” + “RUN”) * 4 * 2 instead of (“TURN LEFT” + “RUN”) * 4 * 2 + (“TURN LEFT” * 2 + “RUN”)
Revised: * 3 + (“TURN LEFT” + “RUN”) * 4 * 2 instead of (“TURN LEFT” + “RUN”) * 4 * 2 + (“TURN LEFT” * 2 + “RUN”)

Original: In the example “walk around left thrice after walk twice”, the code-davinci-002 model produces the correct translations for both sub-expressions that are connected by “after”, but it combines them as if they were connected by “and”.
Revised: In the example “walk around left thrice after walk twice”, the code-davinci-002 model produces the correct translations for both sub-expressions that are connected by “after”, but it combines them as if they were connected by “and”.

Original: This means that the model produces (“TURN LEFT” + “WALK”)
Revised: This means that the model produces (“TURN LEFT” + “WALK”)

Original: * 4 * 3 + “WALK” * 2 instead of “WALK” * 2 + (“TURN LEFT” + “WALK”)
Revised: * 4 * 3 + “WALK” * 2 instead of “WALK” * 2 + (“TURN LEFT” + “WALK”)

Original: In the example “look around right twice after jump around left twice”, the code-davincimodel produces the correct translations for both sub-expressions that are connected by “after”, but it combines them as if they were connected by “and”.
Revised: In the example “look around right twice after jump around left twice”, the code-davincimodel produces the correct translations for both sub-expressions that are connected by “after”, but it combines them as if they were connected by “and”.

Original: This means that the model produces (“TURN RIGHT” + “LOOK”)
Revised: This means that the model produces (“TURN RIGHT” + “LOOK”)

Original: * 4 * 2 + (“TURN LEFT” + “JUMP”)
Revised: * 4 * 2 + (“TURN LEFT” + “JUMP”)

Original: * 4 * 2 instead of (“TURN LEFT” + “JUMP”)
Revised: * 4 * 2 instead of (“TURN LEFT” + “JUMP”)

Original: * 4 * 2 + (“TURN RIGHT”
Revised: * 4 * 2 + (“TURN RIGHT”

Original: + “LOOK”)
Revised: + “LOOK”)

Original: Examples of incorrect interpretation of “left” and “right”.
Revised: Examples of incorrect interpretation of “left” and “right”.

Original: In the example “look opposite right thrice after look around left thrice”, the text-davinci-002 model translates the component “look left” to “LOOK” instead of “TURN LEFT LOOK”.
Revised: In the example “look opposite right thrice after look around left thrice”, the text-davinci-002 model translates the component “look left” to “LOOK” instead of “TURN LEFT LOOK”.

Original: As a consequence, the whole command is translated to “LOOK” * 4 * 3 + (“TURN RIGHT” * 2 + “LOOK”)
Revised: As a consequence, the whole command is translated to “LOOK” * 4 * 3 + (“TURN RIGHT” * 2 + “LOOK”)

Original: * 3 instead of (“TURN LEFT” + “LOOK”) * 4 * 3 + (“TURN RIGHT” * 2 + “LOOK”) * 3.
Revised: * 3 instead of (“TURN LEFT” + “LOOK”) * 4 * 3 + (“TURN RIGHT” * 2 + “LOOK”) * 3.

Original: In the example “turn around right thrice after look around left twice”, the text-davincimodel makes up the new action “LOOK LEFT” as the translation of the component “look left”.
Revised: In the example “turn around right thrice after look around left twice”, the text-davincimodel makes up the new action “LOOK LEFT” as the translation of the component “look left”.

Original: As a consequence, it translates the whole command to (“LOOK LEFT” * 4) * 2 + (“TURN RIGHT” * 4) * 3 instead of (“TURN LEFT” + “LOOK”)
Revised: As a consequence, it translates the whole command to (“LOOK LEFT” * 4) * 2 + (“TURN RIGHT” * 4) * 3 instead of (“TURN LEFT” + “LOOK”)

Original: * 4 * 2 + (“TURN RIGHT” * 4) * 3.
Revised: * 4 * 2 + (“TURN RIGHT” * 4) * 3.

Original: Example of copy error.
Revised: Example of copy error.

Original: In the example “walk opposite right twice after look around left thrice”, the code-davinci-001 model produces the correct translations for both sub-expressions that are connected with “after”.
Revised: In the example “walk opposite right twice after look around left thrice”, the code-davinci-001 model produces the correct translations for both sub-expressions that are connected with “after”.

Original: In particular, it correctly translates “walk opposite right twice” to (“TURN RIGHT” * 2 + “WALK”)
Revised: In particular, it correctly translates “walk opposite right twice” to (“TURN RIGHT” * 2 + “WALK”)

Original: Then it makes two mistakes when combining them: it treats “after” like an “and”, and it omits the parenthesis when copying the expression (“TURN RIGHT” * 2 + “WALK”)
Revised: Then it makes two mistakes when combining them: it treats “after” like an “and”, and it omits the parenthesis when copying the expression (“TURN RIGHT” * 2 + “WALK”)

Original: This means that the model produces the translation “TURN RIGHT” * 2 + “WALK” * 2 + (“TURN LEFT” + “LOOK”)
Revised: This means that the model produces the translation “TURN RIGHT” * 2 + “WALK” * 2 + (“TURN LEFT” + “LOOK”)

Original: * 4 * 3” instead of (“TURN LEFT” + “LOOK”)
Revised: * 4 * 3” instead of (“TURN LEFT” + “LOOK”)

Original: * 4 * 3 + (“TURN RIGHT” * 2 + “WALK”)
Revised: * 4 * 3 + (“TURN RIGHT” * 2 + “WALK”)

Original: 7. E XAMPLE OUTPUTS FROM CODE - DAVINCI - 
Revised: 7. E XAMPLE OUTPUTS FROM CODE - DAVINCI -

Original: In this section, we present 5 examples of successes and failures of the code-davinci-002 model for chain-of-thought prompting and least-to-most prompting.
Revised: In this section, we present 5 examples of successes and failures of the code-davinci-002 model for chain-of-thought prompting and least-to-most prompting.

Original: [Table]
Revised: [Table]

Original: HAIN - OF - THOUGHT
Revised: HAIN - OF - THOUGHT

Original: [Table]
Revised: [Table]

Original: : S
Revised: : S

Original: [Table]
Revised: [Table]

Original: WALK
Revised: WALK

Original: [Table]
Revised: [Table]

Original: LEFT
Revised: LEFT

Original: [Table]
Revised: [Table]

Original: LEFT WALK
Revised: LEFT WALK

Original: [Table]
Revised: [Table]

Original: LEFT
Revised: LEFT

Original: [Table]
Revised: [Table]

Original: LEFT
Revised: LEFT

Original: [Table]
Revised: [Table]

Original: RIGHT
Revised: RIGHT

Original: TURN RIGHT TURN RIGHT
Revised: TURN RIGHT TURN RIGHT

Original: TURN RIGHT TURN RIGHT
Revised: TURN RIGHT TURN RIGHT

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: TURN LEFT RUN TURN LEFT RUN TURN LEFT RUN TURN LEFT RUN TURN LEFT
Revised: TURN LEFT RUN TURN LEFT RUN TURN LEFT RUN TURN LEFT RUN TURN LEFT

Original: TURN LEFT RUN TURN LEFT RUN TURN LEFT RUN TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK
Revised: TURN LEFT RUN TURN LEFT RUN TURN LEFT RUN TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK

Original: TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK
Revised: TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK

Original: TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT JUMP TURN LEFT
Revised: TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT JUMP TURN LEFT

Original: JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT
Revised: JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT

Original: JUMP
Revised: JUMP

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: TURN LEFT
Revised: TURN LEFT

Original: TURN LEFT
Revised: TURN LEFT

Original: TURN LEFT RUN TURN LEFT RUN TURN LEFT RUN
Revised: TURN LEFT RUN TURN LEFT RUN TURN LEFT RUN

Original: TURN LEFT RUN TURN LEFT RUN TURN LEFT RUN TURN LEFT RUN TURN LEFT RUN TURN LEFT RUN TURN LEFT RUN TURN LEFT RUN TURN LEFT RUN
Revised: TURN LEFT RUN TURN LEFT RUN TURN LEFT RUN TURN LEFT RUN TURN LEFT RUN TURN LEFT RUN TURN LEFT RUN TURN LEFT RUN TURN LEFT RUN

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Golden: WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT
Revised: Golden: WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT

Original: WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK
Revised: WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK

Original: [Table]
Revised: [Table]

Original: TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN RIGHT JUMP TURN RIGHT JUMP
Revised: TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN RIGHT JUMP TURN RIGHT JUMP

Original: TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT JUMP
Revised: TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT JUMP

Original: TURN RIGHT JUMP
Revised: TURN RIGHT JUMP

Original: [Table]
Revised: [Table]

Original: TURN RIGHT WALK TURN RIGHT
Revised: TURN RIGHT WALK TURN RIGHT

Original: WALK TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT WALK
Revised: WALK TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT WALK

Original: TURN RIGHT WALK TURN RIGHT
Revised: TURN RIGHT WALK TURN RIGHT

Original: WALK TURN RIGHT WALK TURN RIGHT TURN RIGHT RUN
Revised: WALK TURN RIGHT WALK TURN RIGHT TURN RIGHT RUN

Original: Solution: Q: “jump left”
Revised: Solution: Q: “jump left”

Original: [Table]
Revised: [Table]

Original: TURN RIGHT RUN TURN RIGHT RUN TURN RIGHT RUN TURN RIGHT RUN TURN RIGHT RUN
Revised: TURN RIGHT RUN TURN RIGHT RUN TURN RIGHT RUN TURN RIGHT RUN TURN RIGHT RUN

Original: TURN RIGHT RUN TURN RIGHT RUN TURN RIGHT RUN TURN RIGHT RUN TURN RIGHT RUN TURN RIGHT RUN TURN RIGHT RUN TURN LEFT TURN LEFT TURN LEFT TURN LEFT
Revised: TURN RIGHT RUN TURN RIGHT RUN TURN RIGHT RUN TURN RIGHT RUN TURN RIGHT RUN TURN RIGHT RUN TURN RIGHT RUN TURN LEFT TURN LEFT TURN LEFT TURN LEFT

Original: TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT
Revised: TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Golden: TURN LEFT LOOK TURN LEFT LOOK TURN LEFT LOOK TURN LEFT LOOK TURN LEFT LOOK
Revised: Golden: TURN LEFT LOOK TURN LEFT LOOK TURN LEFT LOOK TURN LEFT LOOK TURN LEFT LOOK

Original: TURN LEFT LOOK TURN LEFT LOOK TURN LEFT LOOK TURN LEFT LOOK TURN LEFT LOOK TURN LEFT LOOK TURN LEFT LOOK TURN LEFT
Revised: TURN LEFT LOOK TURN LEFT LOOK TURN LEFT LOOK TURN LEFT LOOK TURN LEFT LOOK TURN LEFT LOOK TURN LEFT LOOK TURN LEFT

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT
Revised: TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT

Original: TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT
Revised: TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT

Original: JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP
Revised: JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: left”,
Revised: left”,

Original: [Table]
Revised: [Table]

Original: around left”,
Revised: around left”,

Original: [Table]
Revised: [Table]

Original: around left thrice”.
Revised: around left thrice”.

Original: “jump around left thrice” can be solved by: “jump left”, “jump around left”, “jump around left thrice”.
Revised: “jump around left thrice” can be solved by: “jump left”, “jump around left”, “jump around left thrice”.

Original: So, “turn around left thrice and jump around left thrice” can be solved by: “turn left”, “turn around left”, “turn around left thrice”, “jump left”, “jump around left”, “jump around left thrice”.
Revised: So, “turn around left thrice and jump around left thrice” can be solved by: “turn left”, “turn around left”, “turn around left thrice”, “jump left”, “jump around left”, “jump around left thrice”.

Original: Solution: Q: “turn around left”
Revised: Solution: Q: “turn around left”

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: Q: “turn around left thrice and jump around left thrice” A: The output of “turn around left thrice and jump around left thrice” concatenates: the output of “turn around left thrice”, the output of “jump around left thrice”.
Revised: Q: “turn around left thrice and jump around left thrice” A: The output of “turn around left thrice and jump around left thrice” concatenates: the output of “turn around left thrice”, the output of “jump around left thrice”.

Original: “turn around left thrice” outputs (“TURN LEFT” * 4) * 3.
Revised: “turn around left thrice” outputs (“TURN LEFT” * 4) * 3.

Original: “jump around left thrice” outputs ((“TURN LEFT” + “JUMP”)
Revised: “jump around left thrice” outputs ((“TURN LEFT” + “JUMP”)

Original: So concatenating the output of “turn around left thrice” and the output of “jump around left thrice” leads to (“TURN LEFT” * 4) * 

Revised: So concatenating the output of “turn around left thrice” and the output of “jump around left thrice” leads to (“TURN LEFT” * 4) *

Original: + ((“TURN LEFT” + “JUMP”) * 4) * 3.
Revised: + ((“TURN LEFT” + “JUMP”) * 4) * 3.

Original: So the output of “turn around left thrice and jump around left thrice” is (“TURN LEFT” * 4) * 

Revised: So the output of “turn around left thrice and jump around left thrice” is (“TURN LEFT” * 4) *

Original: + ((“TURN LEFT” + “JUMP”) * 4) * 3.
Revised: + ((“TURN LEFT” + “JUMP”) * 4) * 3.

Original: “walk around right twice after walk around right twice”
Revised: “walk around right twice after walk around right twice”

Original: Golden:
Revised: Golden:

Original: TURN RIGHT WALK TURN RIGHT
Revised: TURN RIGHT WALK TURN RIGHT

Original: WALK TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT WALK
Revised: WALK TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT WALK

Original: TURN RIGHT WALK TURN RIGHT
Revised: TURN RIGHT WALK TURN RIGHT

Original: WALK TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT WALK
Revised: WALK TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT WALK

Original: Decomposition:
Revised: Decomposition:

Original: Q: “walk around right twice after walk around right twice”
Revised: Q: “walk around right twice after walk around right twice”

Original: A: “walk around right twice” can be solved by: “walk right”, “walk around right”, “walk around right twice”.
Revised: A: “walk around right twice” can be solved by: “walk right”, “walk around right”, “walk around right twice”.

Original: “walk around right twice” can be solved by: “walk right”, “walk around right”, “walk around right twice”.
Revised: “walk around right twice” can be solved by: “walk right”, “walk around right”, “walk around right twice”.

Original: So, “walk around right twice after walk around right twice” can be solved by: “walk right”, “walk around right”, “walk around right twice”, “walk right”, “walk around right”, “walk around right twice”.
Revised: So, “walk around right twice after walk around right twice” can be solved by: “walk right”, “walk around right”, “walk around right twice”, “walk right”, “walk around right”, “walk around right twice”.

Original: Solution:
Revised: Solution:

Original: Q: “walk right”
Revised: Q: “walk right”

Original: A: The output of “walk right” concatenates: the output of “turn right”, the output of “walk”.
Revised: A: The output of “walk right” concatenates: the output of “turn right”, the output of “walk”.

Original: “turn right” outputs “TURN RIGHT”.
Revised: “turn right” outputs “TURN RIGHT”.

Original: “walk” outputs “WALK”.
Revised: “walk” outputs “WALK”.

Original: So concatenating the output of “turn right” and the output of “walk” leads to “TURN RIGHT” + “WALK”
Revised: So concatenating the output of “turn right” and the output of “walk” leads to “TURN RIGHT” + “WALK”

Original: [Table]
Revised: [Table]

Original: RIGHT
Revised: RIGHT

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Golden:
Revised: Golden:

Original: TURN RIGHT WALK TURN RIGHT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT
Revised: TURN RIGHT WALK TURN RIGHT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT

Original: WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK
Revised: WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: Solution:
Revised: Solution:

Original: 7. E XPANDING P YTHON EXPRESSIONS USING PROMPTING
Revised: 7. E XPANDING P YTHON EXPRESSIONS USING PROMPTING

Original: In Section 3.2, we mention that expanding the Python expressions that we use as an intermediate representation can be done either with a simple postprocessing script or by prompting a language model.
Revised: In Section 3.2, we mention that expanding the Python expressions that we use as an intermediate representation can be done either with a simple postprocessing script or by prompting a language model.

Original: In the following, we present a prompt that achieves 99.7% accuracy on a random sample of 1000 Python expressions that are outputted by our solution (using the code-davincimodel).
Revised: In the following, we present a prompt that achieves 99.7% accuracy on a random sample of 1000 Python expressions that are outputted by our solution (using the code-davincimodel).

Original: This demonstrates that the L2M method can solve SCAN with a combined accuracy more than 99% (99.7% accuracy for generating the intermediate Python expressions and 99.7% for expanding these expressions), even if we do not use the Python executor and instead perform the expansion of the intermediate representation via prompting.
Revised: This demonstrates that the L2M method can solve SCAN with a combined accuracy more than 99% (99.7% accuracy for generating the intermediate Python expressions and 99.7% for expanding these expressions), even if we do not use the Python executor and instead perform the expansion of the intermediate representation via prompting.

Original: Q: “JUMP” *
Revised: Q: “JUMP” *

Original: Rewrite: “JUMP” *
Revised: Rewrite: “JUMP” *

Original: A: 1 JUMP 2 JUMP 3 JUMP
Revised: A: 1 JUMP 2 JUMP 3 JUMP

Original: Q: “RUN” * 4 *
Revised: Q: “RUN” * 4 *

Original: Rewrite: “RUN” *
Revised: Rewrite: “RUN” *

Original: A: 1 RUN 2 RUN 3 RUN 4 RUN 5 RUN
Revised: A: 1 RUN 2 RUN 3 RUN 4 RUN 5 RUN

Original: [Table]
Revised: [Table]

Original: Q: (“TURN LEFT” + “LOOK”)
Revised: Q: (“TURN LEFT” + “LOOK”)

Original: * 2 + “TURN LEFT” + “LOOK”
Revised: * 2 + “TURN LEFT” + “LOOK”

Original: Rewrite: (“TURN LEFT” + “LOOK”) * 2 + “TURN LEFT” + “LOOK”
Revised: Rewrite: (“TURN LEFT” + “LOOK”) * 2 + “TURN LEFT” + “LOOK”

Original: A: 1 (TURN LEFT LOOK) 2 (TURN LEFT LOOK) TURN LEFT LOOK
Revised: A: 1 (TURN LEFT LOOK) 2 (TURN LEFT LOOK) TURN LEFT LOOK

Original: Q: (“TURN RIGHT” * 2 + “JUMP”) *
Revised: Q: (“TURN RIGHT” * 2 + “JUMP”) *

Original: Rewrite: (“TURN RIGHT” * 2 + “JUMP”) *
Revised: Rewrite: (“TURN RIGHT” * 2 + “JUMP”) *

Original: A: 1 (1 TURN RIGHT 2 TURN RIGHT JUMP)
Revised: A: 1 (1 TURN RIGHT 2 TURN RIGHT JUMP)

Original: 2 (1 TURN RIGHT 2 TURN RIGHT JUMP) 3 (
Revised: 2 (1 TURN RIGHT 2 TURN RIGHT JUMP) 3 (

Original: TURN RIGHT 2 TURN RIGHT JUMP)
Revised: TURN RIGHT 2 TURN RIGHT JUMP)

Original: 4 (1 TURN RIGHT 2 TURN RIGHT JUMP)
Revised: 4 (1 TURN RIGHT 2 TURN RIGHT JUMP)

Original: Q: “TURN LEFT” * 2 + (“TURN RIGHT” + “WALK”)
Revised: Q: “TURN LEFT” * 2 + (“TURN RIGHT” + “WALK”)

Original: Rewrite: “TURN LEFT” * 2 + (“TURN RIGHT” + “WALK”) *
Revised: Rewrite: “TURN LEFT” * 2 + (“TURN RIGHT” + “WALK”) *

Original: A: 1 TURN LEFT 2 TURN LEFT 1 (TURN RIGHT WALK) 2 (TURN RIGHT WALK) 3 (TURN RIGHT
Revised: A: 1 TURN LEFT 2 TURN LEFT 1 (TURN RIGHT WALK) 2 (TURN RIGHT WALK) 3 (TURN RIGHT

Original: WALK) 4 (TURN RIGHT WALK) 5 (TURN RIGHT WALK) 6 (TURN RIGHT WALK) 7 (TURN RIGHT
Revised: WALK) 4 (TURN RIGHT WALK) 5 (TURN RIGHT WALK) 6 (TURN RIGHT WALK) 7 (TURN RIGHT

Original: WALK) 8 (TURN RIGHT WALK)
Revised: WALK) 8 (TURN RIGHT WALK)

Original: Discussion.
Revised: Discussion.

Original: The prompt consists of 6 examples, each of which illustrates part of the knowledge needed for this task.
Revised: The prompt consists of 6 examples, each of which illustrates part of the knowledge needed for this task.

Original: Note that we add numbers and parentheses when we unfold multiplication to make it easier for the model to keep track of the repetitions.
Revised: Note that we add numbers and parentheses when we unfold multiplication to make it easier for the model to keep track of the repetitions.

Original: Multiplication 2.
Revised: Multiplication 2.

Original: Sequential multiplication 3.
Revised: Sequential multiplication 3.

Original: Addition 4.
Revised: Addition 4.

Original: Avoid commutativity / associativity in addition 5.
Revised: Avoid commutativity / associativity in addition 5.

Original: Nested multiplication 6.
Revised: Nested multiplication 6.

Original: Addition of two multiplications
Revised: Addition of two multiplications

Original:  DROP 8. R ESULTS WITH TEXT - DAVINCI -002 AND LM-540B
Revised:  DROP 8. R ESULTS WITH TEXT - DAVINCI -002 AND LM-540B

Original: We reported the results using code-davinci-002 .
Revised: We reported the results using code-davinci-002 .

Original: Here, we report results using the text-davinci-002 model and a language model with 540 billion parameters ( LM-540B ).
Revised: Here, we report results using the text-davinci-002 model and a language model with 540 billion parameters ( LM-540B ).

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: 8. N ON - FOOTBALL S UBSET 8.2. Z ERO - SHOT PROMPTING
Revised: 8. N ON - FOOTBALL S UBSET 8.2. Z ERO - SHOT PROMPTING

Original: For zero-shot, the prompt format is as follows:
Revised: For zero-shot, the prompt format is as follows:

Original: Q: { question } A:
Revised: Q: { question } A:

Original: The answer is
Revised: The answer is

Original: Notice that we add “The answer is” at the beginning of the answer section.
Revised: Notice that we add “The answer is” at the beginning of the answer section.

Original: [Table]
Revised: [Table]

Original: 8.2. S TANDARD PROMPTING WITH 3 EXAMPLES
Revised: 8.2. S TANDARD PROMPTING WITH 3 EXAMPLES

Original: Q: Since the 1970s, U.S. governments have negotiated managed-trade agreements, such as the North
Revised: Q: Since the 1970s, U.S. governments have negotiated managed-trade agreements, such as the North

Original: American Free Trade Agreement in the 1990s, the Dominican Republic-Central America Free Trade
Revised: American Free Trade Agreement in the 1990s, the Dominican Republic-Central America Free Trade

Original: Agreement in 2006, and a number of bilateral agreements.
Revised: Agreement in 2006, and a number of bilateral agreements.

Original: In Europe, six countries formed the
Revised: In Europe, six countries formed the

Original: European Coal and Steel Community in 1951 which became the European Economic Community in 1958.
Revised: European Coal and Steel Community in 1951 which became the European Economic Community in 1958.

Original: Two core objectives of the EEC were the development of a common market, subsequently renamed the single market, and establishing a customs union between its member states.
Revised: Two core objectives of the EEC were the development of a common market, subsequently renamed the single market, and establishing a customs union between its member states.

Original: How many years did the European Coal and Steel Community exist?
Revised: How many years did the European Coal and Steel Community exist?

Original: A:
Revised: A:

Original: The answer is 7.
Revised: The answer is 7.

Original: Q: In the county, the population was spread out with 23.50% under the age of 18, 8.70% from 18 to 24, 29.70% from 25 to 44, 24.70% from 45 to 64, and 13.30% who were 65 years of age or older.
Revised: Q: In the county, the population was spread out with 23.50% under the age of 18, 8.70% from 18 to 24, 29.70% from 25 to 44, 24.70% from 45 to 64, and 13.30% who were 65 years of age or older.

Original: How many more percent are under the age of 18 compared to the 18 to 24 group?
Revised: How many more percent are under the age of 18 compared to the 18 to 24 group?

Original: A:
Revised: A:

Original: The answer is 14.8.
Revised: The answer is 14.8.

Original: Q: Playing in their second straight Thanksgiving game, the Eagles struggled especially on defense, where they were unable to stop the much-hyped Lions offense.
Revised: Q: Playing in their second straight Thanksgiving game, the Eagles struggled especially on defense, where they were unable to stop the much-hyped Lions offense.

Original: The worst of it all was how unproven rookie Eric Rowe was tasked with covering wide receiver Calvin Johnson, leading to Johnson catching 3 touchdowns.
Revised: The worst of it all was how unproven rookie Eric Rowe was tasked with covering wide receiver Calvin Johnson, leading to Johnson catching 3 touchdowns.

Original: Stafford’s ﬁve passing touchdowns, including three of them to Johnson was too much for the Eagles to overcome and for the second consecutive time this season, the Eagles gave up 45 points in a game.
Revised: Stafford’s ﬁve passing touchdowns, including three of them to Johnson was too much for the Eagles to overcome and for the second consecutive time this season, the Eagles gave up 45 points in a game.

Original: With the loss, the Eagles drop to 4-7 on the season and 6-1 when playing on
Revised: With the loss, the Eagles drop to 4-7 on the season and 6-1 when playing on

Original: Thanksgiving.
Revised: Thanksgiving.

Original: How many TD passes did Stafford throw other than to Johnson?
Revised: How many TD passes did Stafford throw other than to Johnson?

Original: A:
Revised: A:

Original: The answer is 2.
Revised: The answer is 2.

Original: 8.2. C HAIN - OF - THOUGHT PROMPTING WITH 3 EXAMPLES
Revised: 8.2. C HAIN - OF - THOUGHT PROMPTING WITH 3 EXAMPLES

Original: Q: Since the 1970s, U.S. governments have negotiated managed-trade agreements, such as the North
Revised: Q: Since the 1970s, U.S. governments have negotiated managed-trade agreements, such as the North

Original: American Free Trade Agreement in the 1990s, the Dominican Republic-Central America Free Trade
Revised: American Free Trade Agreement in the 1990s, the Dominican Republic-Central America Free Trade

Original: Agreement in 2006, and a number of bilateral agreements.
Revised: Agreement in 2006, and a number of bilateral agreements.

Original: In Europe, six countries formed the
Revised: In Europe, six countries formed the

Original: European Coal and Steel Community in 1951 which became the European Economic Community in 1958.
Revised: European Coal and Steel Community in 1951 which became the European Economic Community in 1958.

Original: Two core objectives of the EEC were the development of a common market, subsequently renamed the single market, and establishing a customs union between its member states.
Revised: Two core objectives of the EEC were the development of a common market, subsequently renamed the single market, and establishing a customs union between its member states.

Original: How many years did the European Coal and Steel Community exist?
Revised: How many years did the European Coal and Steel Community exist?

Original: A:
Revised: A:

Original: According to the passage, the European Coal and Steel Community was established in 1951 and became the EEC in 1958.
Revised: According to the passage, the European Coal and Steel Community was established in 1951 and became the EEC in 1958.

Original: 1958 - 1951 = 7.
Revised: 1958 - 1951 = 7.

Original: So the answer is 7.
Revised: So the answer is 7.

Original: Q: In the county, the population was spread out with 23.50% under the age of 18, 8.70% from 18 to 24, 29.70% from 25 to 44, 24.70% from 45 to 64, and 13.30% who were 65 years of age or older.
Revised: Q: In the county, the population was spread out with 23.50% under the age of 18, 8.70% from 18 to 24, 29.70% from 25 to 44, 24.70% from 45 to 64, and 13.30% who were 65 years of age or older.

Original: How many more percent are under the age of 18 compared to the 18 to 24 group?
Revised: How many more percent are under the age of 18 compared to the 18 to 24 group?

Original: A:
Revised: A:

Original: According to the passage, 23.5% are under the age of 18, and 8.7% are from ages 18 to 24.
Revised: According to the passage, 23.5% are under the age of 18, and 8.7% are from ages 18 to 24.

Original: 23.5% - 8.7% = 14.8%.
Revised: 23.5% - 8.7% = 14.8%.

Original: So the answer is 14.8.
Revised: So the answer is 14.8.

Original: Q: Playing in their second straight Thanksgiving game, the Eagles struggled especially on defense, where they were unable to stop the much-hyped Lions offense.
Revised: Q: Playing in their second straight Thanksgiving game, the Eagles struggled especially on defense, where they were unable to stop the much-hyped Lions offense.

Original: The worst of it all was how unproven rookie Eric Rowe was tasked with covering wide receiver Calvin Johnson, leading to Johnson catching 3 touchdowns.
Revised: The worst of it all was how unproven rookie Eric Rowe was tasked with covering wide receiver Calvin Johnson, leading to Johnson catching 3 touchdowns.

Original: Stafford’s ﬁve passing touchdowns, including three of them to Johnson was too much for the Eagles to overcome and for the second consecutive time this season, the Eagles gave up 45 points in a game.
Revised: Stafford’s ﬁve passing touchdowns, including three of them to Johnson was too much for the Eagles to overcome and for the second consecutive time this season, the Eagles gave up 45 points in a game.

Original: With the loss, the Eagles drop to 4-7 on the season and 6-1 when playing on
Revised: With the loss, the Eagles drop to 4-7 on the season and 6-1 when playing on

Original: Thanksgiving.
Revised: Thanksgiving.

Original: How many TD passes did Stafford throw other than to Jhonson?
Revised: How many TD passes did Stafford throw other than to Jhonson?

Original: A:
Revised: A:

Original: According to the passage, Stafford threw 5 TD passes, 3 of which were to Johnson.
Revised: According to the passage, Stafford threw 5 TD passes, 3 of which were to Johnson.

Original: 5 - 3 = 2.
Revised: 5 - 3 = 2.

Original: So the answer is 2.
Revised: So the answer is 2.

Original: 8.2. L EAST - TO - MOST PROMPTING I: PROBLEM DECOMPOSITION (5 EXAMPLES )
Revised: 8.2. L EAST - TO - MOST PROMPTING I: PROBLEM DECOMPOSITION (5 EXAMPLES )

Original: Q: The median age in the city was 22.1 years.
Revised: Q: The median age in the city was 22.1 years.

Original: 10.1% of residents were under the age of 18; 56.2% were between the ages of 18 and 24; 16.1% were from 25 to 44; 10.5% were from 45 to 64; and 7% were 65 years of age or older.
Revised: 10.1% of residents were under the age of 18; 56.2% were between the ages of 18 and 24; 16.1% were from 25 to 44; 10.5% were from 45 to 64; and 7% were 65 years of age or older.

Original: Which age group is larger: under the age of 18 or 18 and 24?
Revised: Which age group is larger: under the age of 18 or 18 and 24?

Original: A: To answer the question “Which age group is larger: under the age of 18 or 18 and 24?”, we need to know: “How many percent were under the age of 18?”, “How many percent were between the ages of 18 and 24?”.
Revised: A: To answer the question “Which age group is larger: under the age of 18 or 18 and 24?”, we need to know: “How many percent were under the age of 18?”, “How many percent were between the ages of 18 and 24?”.

Original: Q: Old age pensions were raised by 300 francs per month to 1,700 francs for a single person and to 3,700 francs for a couple, while health insurance beneﬁts were made more widely available to unemployed persons and part-time employees.
Revised: Q: Old age pensions were raised by 300 francs per month to 1,700 francs for a single person and to 3,700 francs for a couple, while health insurance beneﬁts were made more widely available to unemployed persons and part-time employees.

Original: How many francs were the old age pensions for a single person before they were raised?
Revised: How many francs were the old age pensions for a single person before they were raised?

Original: A: To answer the question “How many francs were the old age pensions for a single person before they were raised?”
Revised: A: To answer the question “How many francs were the old age pensions for a single person before they were raised?”

Original: , we need to know: “How many francs were the old age pensions for a single person?”, “How many francs were old age pensions raised for a single person?”.
Revised: , we need to know: “How many francs were the old age pensions for a single person?”, “How many francs were old age pensions raised for a single person?”.

Original: Q: In April 2011, the ECB raised interest rates for the ﬁrst time since 2008 from 1% to 1.25%, with a further increase to 1.50% in July 2011.
Revised: Q: In April 2011, the ECB raised interest rates for the ﬁrst time since 2008 from 1% to 1.25%, with a further increase to 1.50% in July 2011.

Original: However, in 2012-2013 the ECB lowered interest rates to encourage economic growth, reaching the historically low 0.25% in November 2013.
Revised: However, in 2012-2013 the ECB lowered interest rates to encourage economic growth, reaching the historically low 0.25% in November 2013.

Original: Soon after the rates were cut to 0.15%, then on 4 September 2014 the central bank reduced the rates from 0.15% to 0.05%, the lowest rates on record.
Revised: Soon after the rates were cut to 0.15%, then on 4 September 2014 the central bank reduced the rates from 0.15% to 0.05%, the lowest rates on record.

Original: How many percentage points did interest rates drop between
Revised: How many percentage points did interest rates drop between

Original: April 2011 and September 2014?
Revised: April 2011 and September 2014?

Original: A: To answer the question “How many percentage points did interest rates drop between Apriland September 2014?”
Revised: A: To answer the question “How many percentage points did interest rates drop between Apriland September 2014?”

Original: , we need to know: “What was the interest rate in April 2011?”, “What was the interest rate in September 2014?”.
Revised: , we need to know: “What was the interest rate in April 2011?”, “What was the interest rate in September 2014?”.

Original: Q: Non-nationals make up more than half of the population of Bahrain.
Revised: Q: Non-nationals make up more than half of the population of Bahrain.

Original: According to government statistics dated between 2005-2009 roughly 290,000 Indians, 125,000 Bangladeshis, 45,000 Pakistanis, 45,000 Filipinos, and 8,000 Indonesians.
Revised: According to government statistics dated between 2005-2009 roughly 290,000 Indians, 125,000 Bangladeshis, 45,000 Pakistanis, 45,000 Filipinos, and 8,000 Indonesians.

Original: How many Pakistanis and Indonesians are in Bahrain?
Revised: How many Pakistanis and Indonesians are in Bahrain?

Original: A: To answer the question “How many Pakistanis and Indonesians are in Bahrain?”, we need to know: “How many Pakistanis are in Bahrain?”, “How many Indonesians are in Bahrain?”.
Revised: A: To answer the question “How many Pakistanis and Indonesians are in Bahrain?”, we need to know: “How many Pakistanis are in Bahrain?”, “How many Indonesians are in Bahrain?”.

Original: Q: The Census Bureaus 2006-2010 American Community Survey showed that (in 2010 inﬂation adjustment dollars) median household income was $52,056 and the median family income was $58,942.
Revised: Q: The Census Bureaus 2006-2010 American Community Survey showed that (in 2010 inﬂation adjustment dollars) median household income was $52,056 and the median family income was $58,942.

Original: How many years did the Census Bureaus American Community Survey last?
Revised: How many years did the Census Bureaus American Community Survey last?

Original: A: To answer the question “How many years did the Census Bureaus American Community Survey last?”, we need to know: “When did the Census Bureaus American Community Survey start?”, “When did the Census Bureaus American Community Survey end?”.
Revised: A: To answer the question “How many years did the Census Bureaus American Community Survey last?”, we need to know: “When did the Census Bureaus American Community Survey start?”, “When did the Census Bureaus American Community Survey end?”.

Original: 8.2. L EAST - TO - MOST PROMPTING II: PROBLEM SOLVING (3 EXAMPLES )
Revised: 8.2. L EAST - TO - MOST PROMPTING II: PROBLEM SOLVING (3 EXAMPLES )

Original: The city of Bangkok has a population of 8,280,925.
Revised: The city of Bangkok has a population of 8,280,925.

Original: The census showed that it is home to 81,
Revised: The census showed that it is home to 81,

Original: Japanese and 55,893 Chinese nationals, as well as 117,071 expatriates from other Asian countries, 48,341 from Europe, 23,418 from the Americas.
Revised: Japanese and 55,893 Chinese nationals, as well as 117,071 expatriates from other Asian countries, 48,341 from Europe, 23,418 from the Americas.

Original: Q: How many Chinese nationals are there?
Revised: Q: How many Chinese nationals are there?

Original: A: “The census showed that it is home to 81,570 Japanese and 55,893 Chinese nationals”.
Revised: A: “The census showed that it is home to 81,570 Japanese and 55,893 Chinese nationals”.

Original: So the answer is 55,893 Chinese nationals.
Revised: So the answer is 55,893 Chinese nationals.

Original: Q: How many European nationals are there?
Revised: Q: How many European nationals are there?

Original: A: “48,341 from Europe”.
Revised: A: “48,341 from Europe”.

Original: So the answer is 48,341.
Revised: So the answer is 48,341.

Original: Q: How many more Chinese nationals are there than European nationals?
Revised: Q: How many more Chinese nationals are there than European nationals?

Original: A: We know that there are 55,893 Chinese nationals.
Revised: A: We know that there are 55,893 Chinese nationals.

Original: We also know that there are 48,341 European nationals.
Revised: We also know that there are 48,341 European nationals.

Original: So there are 55,893 - 48,341 = 7552 more Chinese nationals than European nationals.
Revised: So there are 55,893 - 48,341 = 7552 more Chinese nationals than European nationals.

Original: So the answer is 7552.
Revised: So the answer is 7552.

Original: The gender distribution of the population was 50.2% male and 49.8% female.
Revised: The gender distribution of the population was 50.2% male and 49.8% female.

Original: Of the adult population, 29 people or 14.6% of the population are between 20 and 29 years old.
Revised: Of the adult population, 29 people or 14.6% of the population are between 20 and 29 years old.

Original: 28 people or 14.1% are
Revised: 28 people or 14.1% are

Original: 30 to 39, 36 people or 18.2% are 40 to 49, and 31 people or 15.7% are 50 to 59.
Revised: 30 to 39, 36 people or 18.2% are 40 to 49, and 31 people or 15.7% are 50 to 59.

Original: Q: What is the percentage of people between 40 to 49?.
Revised: Q: What is the percentage of people between 40 to 49?.

Original: A: “36 people or 18.2% are 40 to 49”.
Revised: A: “36 people or 18.2% are 40 to 49”.

Original: So the answer is 18.
Revised: So the answer is 18.

Original: Q: How many percent of people were not 40 to 49?
Revised: Q: How many percent of people were not 40 to 49?

Original: A:
Revised: A:

Original: We know that 18.2% are 40 to 49.
Revised: We know that 18.2% are 40 to 49.

Original: So 100% - 18.2% = 81.8% are not 40 to 49.
Revised: So 100% - 18.2% = 81.8% are not 40 to 49.

Original: So the answer is 81.8%.
Revised: So the answer is 81.8%.

Original: During the 1970s the oil shocks (1973 oil crisis and 1979 energy crisis) hit Iceland hard.
Revised: During the 1970s the oil shocks (1973 oil crisis and 1979 energy crisis) hit Iceland hard.

Original: Inﬂation rose to 43% in 1974 and 59% in 1980, falling to 15% in 1987 but rising to 30% in 1988.
Revised: Inﬂation rose to 43% in 1974 and 59% in 1980, falling to 15% in 1987 but rising to 30% in 1988.

Original: Iceland experienced moderately strong GDP growth (3% on average) from 1995 to 2004.
Revised: Iceland experienced moderately strong GDP growth (3% on average) from 1995 to 2004.

Original: Growth slowed between 2000 and 2002, but the economy expanded by 4.3% in 2003 and grew by 6.2% in 2004.
Revised: Growth slowed between 2000 and 2002, but the economy expanded by 4.3% in 2003 and grew by 6.2% in 2004.

Original: Growth in 2005 exceeded 6%.
Revised: Growth in 2005 exceeded 6%.

Original: Q: When did the oil crisis occur?
Revised: Q: When did the oil crisis occur?

Original: A: “1973 oil crisis and 1979 energy crisis”.
Revised: A: “1973 oil crisis and 1979 energy crisis”.

Original: So the answer is 1973.
Revised: So the answer is 1973.

Original: Q: When did the energy crisis occur?
Revised: Q: When did the energy crisis occur?

Original: A: “1979 energy crisis”.
Revised: A: “1979 energy crisis”.

Original: So the answer is 1979.
Revised: So the answer is 1979.

Original: Q: How many years passed between the oil and energy crises?
Revised: Q: How many years passed between the oil and energy crises?

Original: A:
Revised: A:

Original: We know that the oil crisis occurred in 1973.
Revised: We know that the oil crisis occurred in 1973.

Original: We also know that the energy crisis occurred in 1979.
Revised: We also know that the energy crisis occurred in 1979.

Original: So 1979 - 1973 = 6 years passed between the oil and energy crises.
Revised: So 1979 - 1973 = 6 years passed between the oil and energy crises.

Original: So the answer is 6.
Revised: So the answer is 6.

Original: 8. F OOTBALL SUBSET 8.3. Z ERO - SHOT PROMPTING
Revised: 8. F OOTBALL SUBSET 8.3. Z ERO - SHOT PROMPTING

Original: For zero-shot, the prompt format is as follows:
Revised: For zero-shot, the prompt format is as follows:

Original: Q: { question } A:
Revised: Q: { question } A:

Original: The answer is
Revised: The answer is

Original: Notice that we add “The answer is” at the beginning of the answer section.
Revised: Notice that we add “The answer is” at the beginning of the answer section.

Original: 8.3. S TANDARD PROMPTING WITH 3 EXAMPLES
Revised: 8.3. S TANDARD PROMPTING WITH 3 EXAMPLES

Original: Q: The Seahawks played the San Francisco 49ers.
Revised: Q: The Seahawks played the San Francisco 49ers.

Original: In the ﬁrst quarter, the Hawks RB Julius Jones got a 27-yard TD run, along with DT Craig Terrill returning a fumble 9 yards for a touchdown.
Revised: In the ﬁrst quarter, the Hawks RB Julius Jones got a 27-yard TD run, along with DT Craig Terrill returning a fumble 9 yards for a touchdown.

Original: In the third quarter, the 49ers almost rallied as RB H. J. Torres made a 12-yard TD pass to Lucas Nelly, along with Mare kicking a 32-yard ﬁeld goal.
Revised: In the third quarter, the 49ers almost rallied as RB H. J. Torres made a 12-yard TD pass to Lucas Nelly, along with Mare kicking a 32-yard ﬁeld goal.

Original: In the ﬁnal quarter, Julius Jones got another 11-yard
Revised: In the ﬁnal quarter, Julius Jones got another 11-yard

Original: TD.
Revised: TD.

Original: How many yards do the shortest touchdown run and the longest touchdown pass combine for?
Revised: How many yards do the shortest touchdown run and the longest touchdown pass combine for?

Original: A:
Revised: A:

Original: The answer is 21.
Revised: The answer is 21.

Original: Q: The Steelers went home for a duel with the Baltimore Ravens.
Revised: Q: The Steelers went home for a duel with the Baltimore Ravens.

Original: Pittsburgh would deliver the opening punch in the ﬁrst quarter with a 1-yard touchdown from running back Rashard Mendenhall.
Revised: Pittsburgh would deliver the opening punch in the ﬁrst quarter with a 1-yard touchdown from running back Rashard Mendenhall.

Original: The Ravens would make it even as running back Willis McGahee got a 9-yard TD.
Revised: The Ravens would make it even as running back Willis McGahee got a 9-yard TD.

Original: The Ravens kicker Billy Cundiff got a 45-yard ﬁeld goal in the second quarter, concluding the ﬁrst half with a 10-7 lead.
Revised: The Ravens kicker Billy Cundiff got a 45-yard ﬁeld goal in the second quarter, concluding the ﬁrst half with a 10-7 lead.

Original: The Steelers brought the game into overtime with a 38-yard ﬁeld goal by Andrew Foster.
Revised: The Steelers brought the game into overtime with a 38-yard ﬁeld goal by Andrew Foster.

Original: The Ravens Billy Cundiff pulled off a winning 33-yard ﬁeld goal in overtime.
Revised: The Ravens Billy Cundiff pulled off a winning 33-yard ﬁeld goal in overtime.

Original: How many points did the Ravens have at halftime?
Revised: How many points did the Ravens have at halftime?

Original: A:
Revised: A:

Original: The answer is 10.
Revised: The answer is 10.

Original: Q: The Vikings ﬂew to Bank of America Stadium to face the Carolina Panthers.
Revised: Q: The Vikings ﬂew to Bank of America Stadium to face the Carolina Panthers.

Original: After a scoreless ﬁrst quarter, Carolina got on the board with quarterback Matt Moore ﬁnding fullback Brad Hoover on a 1-yard TD pass.
Revised: After a scoreless ﬁrst quarter, Carolina got on the board with quarterback Matt Moore ﬁnding fullback Brad Hoover on a 1-yard TD pass.

Original: After yet another scoreless quarter, Carolina sealed the game as Matt Moore completed a 42-yard touchdown pass to wide receiver Steve Smith.
Revised: After yet another scoreless quarter, Carolina sealed the game as Matt Moore completed a 42-yard touchdown pass to wide receiver Steve Smith.

Original: How many scoreless quarters were there?
Revised: How many scoreless quarters were there?

Original: A:
Revised: A:

Original: The answer is 2.
Revised: The answer is 2.

Original: 8.3. C HAIN - OF - THOUGHT PROMPTING WITH 3 EXAMPLES
Revised: 8.3. C HAIN - OF - THOUGHT PROMPTING WITH 3 EXAMPLES

Original: Q: The Seahawks played the San Francisco 49ers.
Revised: Q: The Seahawks played the San Francisco 49ers.

Original: In the ﬁrst quarter, the Hawks RB Julius Jones got a 27-yard TD run, along with DT Craig Terrill returning a fumble 9 yards for a touchdown.
Revised: In the ﬁrst quarter, the Hawks RB Julius Jones got a 27-yard TD run, along with DT Craig Terrill returning a fumble 9 yards for a touchdown.

Original: In the third quarter, the 49ers almost rallied as RB H. J. Torres made a 12-yard TD pass to Lucas Nelly, along with Mare kicking a 32-yard ﬁeld goal.
Revised: In the third quarter, the 49ers almost rallied as RB H. J. Torres made a 12-yard TD pass to Lucas Nelly, along with Mare kicking a 32-yard ﬁeld goal.

Original: In the ﬁnal quarter, Julius Jones got another 11-yard
Revised: In the ﬁnal quarter, Julius Jones got another 11-yard

Original: TD.
Revised: TD.

Original: How many yards do the shortest touchdown run and the longest touchdown pass combine for?
Revised: How many yards do the shortest touchdown run and the longest touchdown pass combine for?

Original: A: All the touchdown runs are: a 27-yard touchdown run, a 9-yard touchdown run, a 11-yard touchdown run.
Revised: A: All the touchdown runs are: a 27-yard touchdown run, a 9-yard touchdown run, a 11-yard touchdown run.

Original: The smallest number among 27, 9, 11 is 9.
Revised: The smallest number among 27, 9, 11 is 9.

Original: So the shortest touchdown run was 9 yards.
Revised: So the shortest touchdown run was 9 yards.

Original: All the touchdown passes are: a 12-yard touchdown pass.
Revised: All the touchdown passes are: a 12-yard touchdown pass.

Original: So the longest touchdown pass wasyards.
Revised: So the longest touchdown pass wasyards.

Original: So the shortest touchdown run and the longest touchdown pass combine for 9 + 12 =yards.
Revised: So the shortest touchdown run and the longest touchdown pass combine for 9 + 12 =yards.

Original: So the answer is 21 yards.
Revised: So the answer is 21 yards.

Original: Q: The Steelers went home for a duel with the Baltimore Ravens.
Revised: Q: The Steelers went home for a duel with the Baltimore Ravens.

Original: Pittsburgh would deliver the opening punch in the ﬁrst quarter with a 1-yard touchdown from running back Rashard Mendenhall.
Revised: Pittsburgh would deliver the opening punch in the ﬁrst quarter with a 1-yard touchdown from running back Rashard Mendenhall.

Original: The Ravens would make it even as running back Willis McGahee got a 9-yard TD.
Revised: The Ravens would make it even as running back Willis McGahee got a 9-yard TD.

Original: The Ravens kicker Billy Cundiff got a 45-yard ﬁeld goal in the second quarter, concluding the ﬁrst half with a 10-7 lead.
Revised: The Ravens kicker Billy Cundiff got a 45-yard ﬁeld goal in the second quarter, concluding the ﬁrst half with a 10-7 lead.

Original: The Steelers brought the game into overtime with a 38-yard ﬁeld goal by Andrew Foster.
Revised: The Steelers brought the game into overtime with a 38-yard ﬁeld goal by Andrew Foster.

Original: The Ravens Billy Cundiff pulled off a winning 33-yard ﬁeld goal in overtime.
Revised: The Ravens Billy Cundiff pulled off a winning 33-yard ﬁeld goal in overtime.

Original: How many points did the Ravens have at halftime?
Revised: How many points did the Ravens have at halftime?

Original: A: The Ravens kicker Billy Cundiff got a 45-yard ﬁeld goal in the second quarter, concluding the ﬁrst half with a 10-7 lead.
Revised: A: The Ravens kicker Billy Cundiff got a 45-yard ﬁeld goal in the second quarter, concluding the ﬁrst half with a 10-7 lead.

Original: So the Ravens had 10 points at halftime.
Revised: So the Ravens had 10 points at halftime.

Original: So the answer is 10 points.
Revised: So the answer is 10 points.

Original: Q: The Vikings ﬂew to Bank of America Stadium to face the Carolina Panthers.
Revised: Q: The Vikings ﬂew to Bank of America Stadium to face the Carolina Panthers.

Original: After a scoreless ﬁrst quarter, Carolina got on the board with quarterback Matt Moore ﬁnding fullback Brad Hoover on a 1-yard TD pass.
Revised: After a scoreless ﬁrst quarter, Carolina got on the board with quarterback Matt Moore ﬁnding fullback Brad Hoover on a 1-yard TD pass.

Original: After yet another scoreless quarter, Carolina sealed the game as Matt Moore completed a 42-yard touchdown pass to wide receiver Steve Smith.
Revised: After yet another scoreless quarter, Carolina sealed the game as Matt Moore completed a 42-yard touchdown pass to wide receiver Steve Smith.

Original: How many scoreless quarters were there?
Revised: How many scoreless quarters were there?

Original: A: The ﬁrst and third quarters were the scoreless quarters.
Revised: A: The ﬁrst and third quarters were the scoreless quarters.

Original: So there are 2 scoreless quarters.
Revised: So there are 2 scoreless quarters.

Original: So the answer is 2.
Revised: So the answer is 2.

Original: 8.3. L EAST - TO - MOST PROMPTING I: PROBLEM DECOMPOSITION (6 EXAMPLES )
Revised: 8.3. L EAST - TO - MOST PROMPTING I: PROBLEM DECOMPOSITION (6 EXAMPLES )

Original: Q: The Seahawks played the San Francisco 49ers.
Revised: Q: The Seahawks played the San Francisco 49ers.

Original: In the ﬁrst quarter, the ’Hawks RB Julius Jones got a 27-yard TD run, along with DT Craig Terrill returning a fumble 9 yards for a touchdown.
Revised: In the ﬁrst quarter, the ’Hawks RB Julius Jones got a 27-yard TD run, along with DT Craig Terrill returning a fumble 9 yards for a touchdown.

Original: In the third quarter, the 49ers almost rallied as RB T. J. Duckett made a 12-yard TD pass to Lucas
Revised: In the third quarter, the 49ers almost rallied as RB T. J. Duckett made a 12-yard TD pass to Lucas

Original: Nelly, along with Mare kicking a 32-yard ﬁeld goal.
Revised: Nelly, along with Mare kicking a 32-yard ﬁeld goal.

Original: How many yards do the shortest touchdown run and the longest touchdown pass combine for?
Revised: How many yards do the shortest touchdown run and the longest touchdown pass combine for?

Original: A: To answer the question “How many yards do the shortest touchdown run and the longest touchdown pass combine for?”
Revised: A: To answer the question “How many yards do the shortest touchdown run and the longest touchdown pass combine for?”

Original: , we need to know: “How many yards was the shortest touchdown run?”, “How many yards was the longest touchdown pass?”.
Revised: , we need to know: “How many yards was the shortest touchdown run?”, “How many yards was the longest touchdown pass?”.

Original: Q: The Steelers went home for an AFC North duel with the Baltimore Ravens.
Revised: Q: The Steelers went home for an AFC North duel with the Baltimore Ravens.

Original: Pittsburgh would deliver the opening punch in the ﬁrst quarter with a 1-yard touchdown run from running back Rashard
Revised: Pittsburgh would deliver the opening punch in the ﬁrst quarter with a 1-yard touchdown run from running back Rashard

Original: Mendenhall.
Revised: Mendenhall.

Original: The Ravens would make it even in the second quarter as running back Willis McGahee got a 9-yard touchdown run.
Revised: The Ravens would make it even in the second quarter as running back Willis McGahee got a 9-yard touchdown run.

Original: The Ravens kicker Billy Cundiff got a 45-yard ﬁeld goal in the second quarter and a 33-yard ﬁeld goal in the third quarter.
Revised: The Ravens kicker Billy Cundiff got a 45-yard ﬁeld goal in the second quarter and a 33-yard ﬁeld goal in the third quarter.

Original: Game ended with a scoreless fourth quarter.
Revised: Game ended with a scoreless fourth quarter.

Original: How many points did the Ravens have at halftime?
Revised: How many points did the Ravens have at halftime?

Original: A: To answer the question “How many points did the Ravens have at halftime?”
Revised: A: To answer the question “How many points did the Ravens have at halftime?”

Original: , we need to know: “What were all the scores the Ravens had at halftime?”.
Revised: , we need to know: “What were all the scores the Ravens had at halftime?”.

Original: Q: In 1995, the Kings overcame a 3-4 start to win eight of their ﬁnal nine games and ﬁnished with a record, the second-best in the AFC.
Revised: Q: In 1995, the Kings overcame a 3-4 start to win eight of their ﬁnal nine games and ﬁnished with a record, the second-best in the AFC.

Original: Quarterback Neil ODonnell, who completed 246 out ofpasses for 2,970 yards and 17 touchdowns, with only seven interceptions led their offense.
Revised: Quarterback Neil ODonnell, who completed 246 out ofpasses for 2,970 yards and 17 touchdowns, with only seven interceptions led their offense.

Original: The
Revised: The

Original: Kings ﬁnished their 1995 season having lost how many games difference to the number of games they had won?
Revised: Kings ﬁnished their 1995 season having lost how many games difference to the number of games they had won?

Original: A: To answer the question “The Kings ﬁnished their 1995 season having lost how many games difference to the number of games they had won?”, we need to know: “How many games the Kings had lost in their 1995 season?”, “How many games the Kings had won in their 1995 season?”.
Revised: A: To answer the question “The Kings ﬁnished their 1995 season having lost how many games difference to the number of games they had won?”, we need to know: “How many games the Kings had lost in their 1995 season?”, “How many games the Kings had won in their 1995 season?”.

Original: Q: The Broncos traveled to Sun Life Stadium to face the Miami Dolphins.
Revised: Q: The Broncos traveled to Sun Life Stadium to face the Miami Dolphins.

Original: The Dolphins grabbed the lead in the second quarter, with ﬁeld goals of 35 and 45 yards by kicker Dan Carpenter.
Revised: The Dolphins grabbed the lead in the second quarter, with ﬁeld goals of 35 and 45 yards by kicker Dan Carpenter.

Original: In the ﬁnal quarter, the Dolphins quarterback Matt Moore threw a 16-yard touchdown pass to tight end Anthony
Revised: In the ﬁnal quarter, the Dolphins quarterback Matt Moore threw a 16-yard touchdown pass to tight end Anthony

Original: Fasano (with a failed two-point conversion attempt), followed by a 35-yard ﬁeld goal by Carpenter, giving Miami a 15-0 lead.
Revised: Fasano (with a failed two-point conversion attempt), followed by a 35-yard ﬁeld goal by Carpenter, giving Miami a 15-0 lead.

Original: Finally, the Broncos answered with Alex Jake kicking a 48-yard ﬁeld goal, ending the game with 15-3.
Revised: Finally, the Broncos answered with Alex Jake kicking a 48-yard ﬁeld goal, ending the game with 15-3.

Original: How many yards was Dan Carpenter’s longest ﬁeld goal?
Revised: How many yards was Dan Carpenter’s longest ﬁeld goal?

Original: A: To answer the question “How many yards was Dan Carpenter’s longest ﬁeld goal?”, we need to know: “What were all Dan Carpenter’s ﬁeld goals?”.
Revised: A: To answer the question “How many yards was Dan Carpenter’s longest ﬁeld goal?”, we need to know: “What were all Dan Carpenter’s ﬁeld goals?”.

Original: Q: The Cardinals ﬂew to Gillette Stadium for a duel with the New England Patriots.
Revised: Q: The Cardinals ﬂew to Gillette Stadium for a duel with the New England Patriots.

Original: In the fourth quarter, New England ended the boring game with Gostkowski’s 30-yard ﬁeld goal.
Revised: In the fourth quarter, New England ended the boring game with Gostkowski’s 30-yard ﬁeld goal.

Original: Original starting quarterback Kurt Warner (6/18 for 30 yards) was pulled in the third quarter via coach’s decision.
Revised: Original starting quarterback Kurt Warner (6/18 for 30 yards) was pulled in the third quarter via coach’s decision.

Original: How many incomplete passes did Kurt Warner have?
Revised: How many incomplete passes did Kurt Warner have?

Original: A: To answer the question “How many incomplete passes did Kurt Warner have?”, we need to know: “How many passes did Kurt Warner attempt?”, “How many passes did Kurt Warner complete?”.
Revised: A: To answer the question “How many incomplete passes did Kurt Warner have?”, we need to know: “How many passes did Kurt Warner attempt?”, “How many passes did Kurt Warner complete?”.

Original: Q: The Vikings ﬂew to Bank of America Stadium to face the Carolina Panthers.
Revised: Q: The Vikings ﬂew to Bank of America Stadium to face the Carolina Panthers.

Original: After a scoreless ﬁrst quarter, Carolina got on the board with quarterback Matt Moore ﬁnding fullback Brad Hoover on a 1-yard touchdown pass.
Revised: After a scoreless ﬁrst quarter, Carolina got on the board with quarterback Matt Moore ﬁnding fullback Brad Hoover on a 1-yard touchdown pass.

Original: After yet another scoreless quarter, Carolina sealed the game as Matt
Revised: After yet another scoreless quarter, Carolina sealed the game as Matt

Original: Moore completed a 42-yard TD pass to wide receiver Steve Smith.
Revised: Moore completed a 42-yard TD pass to wide receiver Steve Smith.

Original: How many scoreless quarters were there?
Revised: How many scoreless quarters were there?

Original: A: To answer the question “How many scoreless quarters were there?”
Revised: A: To answer the question “How many scoreless quarters were there?”

Original: , we need to know: “What were all the scoreless quarters?”.
Revised: , we need to know: “What were all the scoreless quarters?”.

Original: 8.3. L EAST - TO - MOST PROMPTING II: PROBLEM SOLVING (3 EXAMPLES )
Revised: 8.3. L EAST - TO - MOST PROMPTING II: PROBLEM SOLVING (3 EXAMPLES )

Original: The Seahawks played the San Francisco 49ers.
Revised: The Seahawks played the San Francisco 49ers.

Original: In the ﬁrst quarter, the ’Hawks RB Julius Jones got a 27-yard TD run, along with DT Craig Terrill returning a fumble 9 yards for a touchdown.
Revised: In the ﬁrst quarter, the ’Hawks RB Julius Jones got a 27-yard TD run, along with DT Craig Terrill returning a fumble 9 yards for a touchdown.

Original: In the third quarter, the 49ers almost rallied as RB H. J. Torres made a 12-yard TD pass to Lucas Nelly, along with Mare kicking a 32-yard ﬁeld goal.
Revised: In the third quarter, the 49ers almost rallied as RB H. J. Torres made a 12-yard TD pass to Lucas Nelly, along with Mare kicking a 32-yard ﬁeld goal.

Original: In the ﬁnal quarter, Julius Jones got another 11-yard
Revised: In the ﬁnal quarter, Julius Jones got another 11-yard

Original: TD.
Revised: TD.

Original: Q: How many yards was the shortest touchdown run?
Revised: Q: How many yards was the shortest touchdown run?

Original: A: All the touchdown runs are: a 27-yard touchdown run, a 9-yard touchdown run, a 11-yard touchdown run.
Revised: A: All the touchdown runs are: a 27-yard touchdown run, a 9-yard touchdown run, a 11-yard touchdown run.

Original: The smallest number between 27, 9, 11 is 9.
Revised: The smallest number between 27, 9, 11 is 9.

Original: The shortest touchdown run was 9 yards.
Revised: The shortest touchdown run was 9 yards.

Original: The answer is 9.
Revised: The answer is 9.

Original: Q: How many yards was the longest touchdown pass?
Revised: Q: How many yards was the longest touchdown pass?

Original: A:
Revised: A:

Original: All the touchdown passes are: a 12-yard touchdown pass.
Revised: All the touchdown passes are: a 12-yard touchdown pass.

Original: The longest touchdown pass wasyards.
Revised: The longest touchdown pass wasyards.

Original: The
Revised: The

Original: answer
Revised: answer

Original: is 12.
Revised: is 12.

Original: Q: How many yards do the
Revised: Q: How many yards do the

Original: shortest
Revised: shortest

Original: touchdown run and the
Revised: touchdown run and the

Original: longest
Revised: longest

Original: touchdown pass combine for?
Revised: touchdown pass combine for?

Original: A:
Revised: A:

Original: We know that the shortest touchdown run was yard.
Revised: We know that the shortest touchdown run was yard.

Original: We also know that the longest touchdown pass was 12-yard
Revised: We also know that the longest touchdown pass was 12-yard

Original: +
Revised: +

Original: =
Revised: =

Original: The
Revised: The

Original: answer is
Revised: answer is

Original: The Steelers went home for a duel with the Baltimore Ravens.
Revised: The Steelers went home for a duel with the Baltimore Ravens.

Original: Pittsburgh would deliver the opening punch in the ﬁrst quarter with a 1-yard touchdown from running back Rashard Mendenhall.
Revised: Pittsburgh would deliver the opening punch in the ﬁrst quarter with a 1-yard touchdown from running back Rashard Mendenhall.

Original: The
Revised: The

Original: Ravens would make it even as running back Willis McGahee got a 9-yard TD.
Revised: Ravens would make it even as running back Willis McGahee got a 9-yard TD.

Original: The Ravens kicker
Revised: The Ravens kicker

Original: Billy Cundiff got a 45-yard ﬁeld goal in the second quarter, concluding the ﬁrst half.
Revised: Billy Cundiff got a 45-yard ﬁeld goal in the second quarter, concluding the ﬁrst half.

Original: The Steelers brought the game into overtime with a 38-yard ﬁeld goal by Andrew Foster.
Revised: The Steelers brought the game into overtime with a 38-yard ﬁeld goal by Andrew Foster.

Original: The Ravens Billy
Revised: The Ravens Billy

Original: Cundiff pulled off a winning 33-yard ﬁeld goal in overtime.
Revised: Cundiff pulled off a winning 33-yard ﬁeld goal in overtime.

Original: Q: What were all the scores the Ravens had at halftime?
Revised: Q: What were all the scores the Ravens had at halftime?

Original: A:
Revised: A:

Original: The answer is: a 9-yard touchdown run, a 45-yard ﬁeld goal.
Revised: The answer is: a 9-yard touchdown run, a 45-yard ﬁeld goal.

Original: Q: How many points did the Ravens have at halftime?
Revised: Q: How many points did the Ravens have at halftime?

Original: A:
Revised: A: