Original: Large Language Models are Zero-Shot Reasoners
Revised: Large Language Models are Zero-Shot Reasoners

Original: Abstract
Revised: Abstract

Original: Chain of thought (CoT) prompting, a recent technique for eliciting multi-step reasoning through step-by-step answer examples, achieved the stateof-the-art performances in arithmetics and symbolic reasoning.
Revised: Chain of thought (CoT) prompting, a recent technique for eliciting multi-step reasoning through step-by-step answer examples, achieved the stateof-the-art performances in arithmetics and symbolic reasoning.

Original: While these successes are often attributed to LLMs’ ability for few-shot learning, we show that LLMs are decent zero-shot reasoners by simply adding “Let’s think step by step” before each answer.
Revised: While these successes are often attributed to LLMs’ ability for few-shot learning, we show that LLMs are decent zero-shot reasoners by simply adding “Let’s think step by step” before each answer.

Original: Experimental results demonstrate that our Zero-shot-CoT, using the same single prompt template, significantly outperforms zeroshot LLM performances on diverse benchmark reasoning tasks including arithmetics (MultiArith,
Revised: Experimental results demonstrate that our Zero-shot-CoT, using the same single prompt template, significantly outperforms zeroshot LLM performances on diverse benchmark reasoning tasks including arithmetics (MultiArith,

Original: GSM8K, AQUA-RAT, SVAMP), symbolic reasoning (Last Letter, Coin Flip), and other logical reasoning tasks (Date Understanding, Tracking Shuffled Objects), without any hand-crafted fewshot examples, e.g. increasing the accuracy on
Revised: GSM8K, AQUA-RAT, SVAMP), symbolic reasoning (Last Letter, Coin Flip), and other logical reasoning tasks, without any hand-crafted fewshot examples, e.g. increasing the accuracy on

Original: MultiArith from 17.7% to 78.7% and GSM8K from 10.4% to 40.7% with an 175B parameterInstruct-GPT.
Revised: MultiArith from 17.7% to 78.7% and GSM8K from 10.4% to 40.7% with an 175B parameter

Revised: Instruct-GPT, as well as similar magnitudes of improvements with 540B parameter PaLM.

Original: The versatility of this single prompt across very diverse reasoning tasks hints at untapped and understudied fundamental zeroshot capabilities of LLMs, suggesting high-level, multi-task broad cognitive capabilities may be extracted through simple prompting.
Revised: The versatility of this single prompt across very diverse reasoning tasks hints at untapped and understudied fundamental zero-shot capabilities of LLMs, suggesting high-level, multi-task broad cognitive capabilities may be extracted through simple prompting.

Original: We hope our work not only serves as the minimal strongest zero-shot baseline for the challenging reasoning benchmarks, but also highlights the importance of carefully exploring and analyzing the enormous zero-shot knowledge hidden inside LLMs.
Revised: We hope our work not only serves as the minimal strongest zero-shot baseline for the challenging reasoning benchmarks, but also highlights the importance of carefully exploring and analyzing the enormous zero-shot knowledge hidden inside LLMs.

Original: 1. Introduction
Revised: 1. Introduction

Original: Scaling up the size of language models has been key ingredients of recent revolutions in natural language processing(NLP) (Vaswani et al., 2017;
Revised: Scaling up the size of language models has been key ingredients of recent revolutions in natural language processing

Original:  Devlin et al., 2019; Raffel et al., 2020; Brown et al., 2020; Thoppilan et al., 2022;
Revised: (NLP) (Vaswani et al., 2017; Devlin et al., 2019; Raffel et al., 2020; Brown et al., 2020; Thoppilan et al., 2022;

Original: Rae et al., 2021; Chowdhery et al., 2022).
Revised: Rae et al., 2021; Chowdhery et al., 2022).

Original: The success of large language models (LLMs) is often attributed to (incontext) few-shot or zero-shot learning.
Revised: The success of large language models (LLMs) is often attributed to (incontext) few-shot or zero-shot learning.

Original: It can solve various tasks by simply conditioning the models on a few examples ( Few-shot ) or instructions describing the task ( Zero-shot ) (See Fig.1-(a) and 1-(c)).
Revised: It can solve various tasks by simply conditioning the models on a few examples ( Few-shot ) or instructions describing the task ( Zero-shot ) (See Fig.1-(a) and 1-(c)).

Original: The method of conditioning the language model is called “prompting” (Liu et al., 2021b), and designing prompts either manually (Schick & Sch ¨ utze, 2021; Reynolds & McDonell, 2021) or automatically (Gao et al., 2021; Shin et al., 2020) becomes a hot topic in NLP.
Revised: The method of conditioning the language model is called “prompting” (Liu et al., 2021b), and designing prompts either manually (Schick & Sch ¨ utze, 2021; Reynolds & McDonell, 2021) or automatically (Gao et al., 2021; Shin et al., 2020) becomes a hot topic in NLP.

Original: In contrast to the excellent performance of LLMs in intuitive and single-step system-1 (Stanovich & West, 2000) tasks with task-specific Few-shot or Zero-shot prompting (Liu et al., 2021b), even language models at the scale of 100B or more parameters had struggled on system-2 tasks requiring slow and multi-step reasoning (Rae et al., 2021).
Revised: In contrast to the excellent performance of LLMs in intuitive and single-step system-1 (Stanovich & West, 2000) tasks with task-specific Few-shot or Zero-shot prompting (Liu et al., 2021b), even language models at the scale of 100B or more parameters had struggled on system-2 tasks requiring slow and multi-step reasoning (Rae et al., 2021).

Original: To address this shortcoming, Wei et al.
Revised: To address this shortcoming, Wei et al.

Original: ; Wang et al.
Revised: ; Wang et al.

Original: (2022) have proposed chain of thought prompting (CoT), which feed LLMs with the step-by-step reasoning examples rather than standard question and answer examples (We call this approach as Few-shot-CoT .
Revised: (2022) have proposed chain of thought prompting (CoT), which feed LLMs with the step-by-step reasoning examples rather than standard question and answer examples (We call this approach as Few-shot-CoT .

Original: see Fig.1-(b)).
Revised: see Fig.1-(b)).

Original: Such chain of thought demonstrations facilitate models to generate a reasoning path that decomposes the complex reasoning into multiple easier steps.
Revised: Such chain of thought demonstrations facilitate models to generate a reasoning path that decomposes the complex reasoning into multiple easier steps.

Original: Notably with CoT, the reasoning performance then satisfies the scaling laws better and jumps up with the size of the language models.
Revised: Notably with CoT, the reasoning performance then satisfies the scaling laws better and jumps up with the size of the language models.

Original: While the successes of CoT prompting (Wei et al., 2022), along those of many other task-specific prompting work (Gao et al., 2021; Schick & Sch¨utze, 2021; Liu et al., 2021b), are often attributed to LLMs’ ability for few-shot learning (Brown et al., 2020), we show that LLMs are decent zero-shot reasoners by adding a simple prompt, Let’s think step by step , to facilitate step-by-step thinking before answering each question (see Fig.1-(d)).
Revised: While the successes of CoT prompting (Wei et al., 2022), along those of many other task-specific prompting work (Gao et al., 2021; Schick & Sch¨utze, 2021; Liu et al., 2021b), are often attributed to LLMs’ ability for few-shot learning (Brown et al., 2020), we show that LLMs are decent zero-shot reasoners by adding a simple prompt, Let’s think step by step , to facilitate step-by-step thinking before answering each question (see Fig.1-(d)).

Original: Despite the simplicity, our Zero-shot-CoT successfully generates a plausible reasoning path in a zero-shot manner and reaches the correct answer in a problem where the standard zero-shot approach fails.
Revised: Despite the simplicity, our Zero-shot-CoT successfully generates a plausible reasoning path in a zero-shot manner and reaches the correct answer in a problem where the standard zero-shot approach fails.

Original: Importantly, our Zero-shot-CoT is versatile and task-agnostic , unlike most prior task-specific prompt engineering in the forms of examples (few-shot) or templates (zero-shot) (Liu et al., 2021b): it can facilitate step-by-step answers across various reasoning tasks, including arithmetic(MultiArith (Roy & Roth, 2015), GSM8K (Cobbe et al., [Table] [Table]
Revised: Importantly, our Zero-shot-CoT is versatile and task-agnostic , unlike most prior task-specific prompt engineering in the forms of examples (few-shot) or templates (zero-shot) (Liu et al., 2021b): it can facilitate step-by-step answers across various reasoning tasks, including arithmetic

Original: [Figure]
Revised: [Figure]

Original: , AQUA-RAT (Ling et al., 2017), and SVAMP (Patel et al., 2021)), symbolic (Last letter and Coin flip), commonsense reasoning (CommonSenseQA (Talmor et al., 2019) and Strategy QA (Geva et al., 2021)), and 
Revised: and al Shuffled Objects (big, 2021)) without modifying the prompt per task.

Original: We empirically evaluate Zero-shot-CoT against other prompting baselines in Figure 1.
Revised: We empirically evaluate Zero-shot-CoT against other prompting baselines in Figure 1.

Original: While our Zero-shotCoT underperforms Few-shot-CoT with carefully-crafted and task-specific step-by-step examples, Zero-shot-CoT achieves enormous score gains compared to the zero-shot baseline, e.g. from 17.7% to 78.7% on MultiArith and from 10.4% to 40.7% on GSM8K. Importantly, with our single fixed prompt, zero-shot LLMs have a significantly better scaling curve comparable to that of the few-shot CoT baseline.
Revised: While our Zero-shotCoT underperforms Few-shot-CoT with carefully-crafted and task-specific step-by-step examples, Zero-shot-CoT achieves enormous score gains compared to the zero-shot baseline, e.g. from 17.7% to 78.7% on MultiArith and from 10.4% to 40.7% on GSM8K. Importantly, with our single fixed prompt, zero-shot LLMs have a significantly better scaling curve comparable to that of the few-shot CoT baseline.

Original: We also show that besides Few-shot-CoT requiring human engineering of multi-step reasoning prompts, their performance deteriorates if prompt example question types and task question type are unmatched, suggesting high sensitivity to per-task prompt designs.
Revised: We also show that besides Few-shot-CoT requiring human engineering of multi-step reasoning prompts, their performance deteriorates if prompt example question types and task question type are unmatched, suggesting high sensitivity to per-task prompt designs.

Original: In contrast, the versatility of this single prompt across diverse reasoning tasks hints at untapped and understudied zero-shot fundamental capabilities of LLMs, such as higher-level broad cognitive capabilities like generic logical reasoning (Chollet, 2019).
Revised: In contrast, the versatility of this single prompt across diverse reasoning tasks hints at untapped and understudied zero-shot fundamental capabilities of LLMs, such as higher-level broad cognitive capabilities like generic logical reasoning (Chollet, 2019).

Original: While the vibrant field of LLMs started from the premise of excellent few-shot learners (Brown et al., 2020), we hope our work encourages more research into uncovering high-level and multi-task zero-shot capabilities hidden inside those models.
Revised: While the vibrant field of LLMs started from the premise of excellent few-shot learners (Brown et al., 2020), we hope our work encourages more research into uncovering high-level and multi-task zero-shot capabilities hidden inside those models.

Original: 2. Zero-shot Chain of Thought
Revised: 2. Zero-shot Chain of Thought

Original: We propose Zero-shot-CoT, a zero-shot template-based prompting for chain of thought reasoning.
Revised: We propose Zero-shot-CoT, a zero-shot template-based prompting for chain of thought reasoning.

Original: The core ideaof our method is simple, asdescribed in Figure 1: add Let’s think step by step , or aa similar text (see Table 2), to extract step-by-step reasoning.
Revised: The core idea is simple as in Figure 1: add Let’s think step by step , or a similar text (see Table 2), to extract step-by-step reasoning.

Original: 2.1. Two-stage prompting
Revised: 2.1. Two-stage prompting

Original: 1st prompt: reasoning extraction
Revised: 1st prompt: reasoning extraction

Original: In this step we first modify the input question x into a prompt using a simple template “Q: [X] .
Revised: In this step we first modify the input question x into a prompt using a simple template “Q: [X] .

Original: A: [Z] ”, where [X] is an input slot for x and [T] is an slot for hand-crafted trigger sentence t that would extract chain of though to answer the question x .
Revised: A: [T] ”, where [X] is an input slot for x and [T] is an slot for hand-crafted trigger sentence t that would extract chain of though to answer the question x .

Original: For example, if we use “Let’s think step by step” as a trigger sentence, the prompt would be “Q: [X] .
Revised: For example, if we use “Let’s think step by step” as a trigger sentence, the prompt would be “Q: [X] .

Original: A: Let’s think step by step.”.
Revised: A: Let’s think step by step.”.

Original: Prompted text x ′ is then fed into a language model and generate subsequent sentence z .
Revised: Prompted text x ′ is then fed into a language model and generate subsequent sentence z .

Original: We can use any decoding strategy, but we used greedy decoding throughout the paper for the simplicity.
Revised: We can use any decoding strategy, but we used greedy decoding throughout the paper for the simplicity.

Original: 2nd prompt: answer extraction
Revised: 2nd prompt: answer extraction

Original: In the second step, we use generated sentence z along with prompted sentence x ′ to extract the final answer from the language model.
Revised: In the second step, we use generated sentence z along with prompted sentence x ′ to extract the final answer from the language model.

Original: To be concrete, we simply concatenate three elements as with “ [X ′ ]
Revised: To be concrete, we simply concatenate three elements as with “ [X ′ ]

Original: [Z] [A] ”:
Revised: [Z] [A] ”:

Original: [X ′ ] for 1st prompt x ′ , [Z] for sentence generated at the first step, and [A] for a trigger sentence toextract answer.
Revised: [X ′ ] for 1st prompt x ′ , [Z] for sentence generated at the first step, and [A] for a trigger sentence to

Original: The prompt for this step is self-augmented ,

Original: [Table]
Revised: [Table]

Original: [Table] [Table] since the prompt contains the sentence z generated by the same language model.
Revised: extract answer. The prompt for this step is self-augmented , since the prompt contains the sentence z generated by the same language model.

Original: In experiment, we use slightly different answer trigger depends on the answer format.
Revised: In experiment, we use slightly different answer trigger depends on the answer format.

Original: For example, we use “Therefore, among A through E, the answer is” for multi-choice QA, and “Therefore, the answer (arabic numerals) is” for math problem requiring numerical answer.
Revised: For example, we use “Therefore, among A through E, the answer is” for multi-choice QA, and “Therefore, the answer (arabic numerals) is” for math problem requiring numerical answer.

Original: See Appendix D.5 for the details.
Revised: See Appendix D.5 for the details.

Original: Finally, the language model is fed the prompted text as input to generate sentences ˆy and parse the final answer.
Revised: Finally, the language model is fed the prompted text as input to generate sentences ˆy and parse the final answer.

Original: 3. Experiment
Revised: 3. Experiment

Original: Tasks and datasets We evaluate our proposal ondatasets from four categories of reasoning tasks: arithmetic, commonsense, symbolic, and other logical reasoning tasks.
Revised: Tasks and datasets We evaluate our proposal ondatasets from four categories of reasoning tasks: arithmetic, commonsense, symbolic, and other logical reasoning tasks.

Original: See Appendix D.2 for the description of each dataset.
Revised: See Appendix D.2 for the description of each dataset.

Original: Models We experiment with 13 models in total: InstructGPT3 (Ouyang et al., 2022) and GPT3 with four different model sizes (ada, babbage, curie, and davinci) 1 , GPT-2,
Revised: Models We experiment with 13 models in total: InstructGPT3 (Ouyang et al., 2022) and GPT3 with four different model sizes (ada, babbage, curie, and davinci) 1 , GPT-2,

Original: GPT-Neo, GPT-J, T0 (Sanh et al., 2022), and OPT (Zhang et al., 2022).
Revised: GPT-Neo, GPT-J, T0 (Sanh et al., 2022), and OPT (Zhang et al., 2022).

Original: The size of LMs ranges from 0.3B to 175B. We include both standard (e.g. GPT-3 and OPT), and instruction following variants (e.g. Instruct-GPT3 and T0).
Revised: The size of LMs ranges from 0.3B to 175B. We include both standard (e.g. GPT-3 and OPT), and instruction following variants (e.g. Instruct-GPT3 and T0).

Original: See
Revised: See

Original: Appendix D.3 for details.
Revised: Appendix D.3 for details.

Original: Without mentioning otherwise, we use Text-davinci-002 throughout the experiments.
Revised: Without mentioning otherwise, we use Text-davinci-002 throughout the experiments.

Original: Baselines We compare our Zero-shot-CoT mainly to standard Zero-shot prompting to verify the effectiveness of its chain of thought reasoning.
Revised: Baselines We compare our Zero-shot-CoT mainly to standard Zero-shot prompting to verify the effectiveness of its chain of thought reasoning.

Original: To better evaluate the zero-shot ability of LLMs on reasoning tasks, we also compare our method to Few-shot and Few-shot-CoT baselines from (Wei et al., 2022), using the same in-context examples.
Revised: To better evaluate the zero-shot ability of LLMs on reasoning tasks, we also compare our method to Few-shot and Few-shot-CoT baselines from (Wei et al., 2022), using the same in-context examples.

Original: Throughout the experiments, we use greedy decoding across all the methods.
Revised: Throughout the experiments, we use greedy decoding across all the methods.

Original: For the zero-shot approaches, the results are there- fore deterministic.
Revised: For the zero-shot approaches, the results are therefore deterministic.

Original: For the few-shot approaches, the order of in-context examples could affect the results (Lu et al., 2022), we run each experiment only once with a fixed seed across all methods.
Revised: For the few-shot approaches, the order of in-context examples could affect the results (Lu et al., 2022), we run each experiment only once with a fixed seed across all methods.

Original: Wei et al.
Revised: Wei et al.

Original: (2022) showed that the order of examples did not cause large variance in CoT experiments.
Revised: (2022) showed that the order of examples did not cause large variance in CoT experiments.

Original: 3.1. Results Zero-shot-CoT vs. Zero-shot
Revised: 3.1. Results Zero-shot-CoT vs. Zero-shot

Original: Table 1 summarize accuracy of our method (Zero-shot-CoT) and standard zero-shot prompting (Zero-shot) for each dataset.
Revised: Table 1 summarize accuracy of our method (Zero-shot-CoT) and standard zero-shot prompting (Zero-shot) for each dataset.

Original: Zero-shot-CoT substantially outperforms four out of six arithmetic reasoning tasks (MultiArith, GSM8K, AQUA, SVAMP), all symbolic reasoning, and all other logical reasoning tasks (from BIG-bench (big, 2021)).
Revised: Zero-shot-CoT substantially outperforms four out of six arithmetic reasoning tasks (MultiArith, GSM8K, AQUA, SVAMP), all symbolic reasoning, and all other logical reasoning tasks (from BIG-bench (big, 2021)).

Original: Our method gives on-par performances for the remaining two arithmetic reasoning tasks (SingleEq and AddSub), which is expected since they do not require multi-step reasoning.
Revised: Our method gives on-par performances for the remaining two arithmetic reasoning tasks (SingleEq and AddSub), which is expected since they do not require multi-step reasoning.

Original: In commonsense reasoning tasks, Zero-shot-CoT does not provide performance gains.
Revised: In commonsense reasoning tasks, Zero-shot-CoT does not provide performance gains.

Original: It is expected as Wei et al.
Revised: It is expected as Wei et al.

Original: (2022) also reports that even Fewshot-CoT does not provide performance gains on Lambda (135B), but does improve StrategyQA when combined with substantially larger PaLM (540B) model, which may also apply for ours.
Revised: (2022) also reports that even Fewshot-CoT does not provide performance gains on Lambda (135B), but does improve StrategyQA when combined with substantially larger PaLM (540B) model, which may also apply for ours.

Original: More importantly, we observe that many generated chain of thought themselves are surprisingly logically correct or only contains human-understandable mistakes , suggesting that Zero-shot-CoT does elicit for better commonsense reasoning even when the task metrics do not directly reflect it.
Revised: More importantly, we observe that many generated chain of thought themselves are surprisingly logically correct or only contains human-understandable mistakes , suggesting that Zero-shot-CoT does elicit for better commonsense reasoning even when the task metrics do not directly reflect it.

Original: (See Appendix E and Appendix G for more detail analysis.)
Revised: (See Appendix E and Appendix G for more detail analysis.)

Original: We provide samples generated by
Revised: We provide samples generated by

Original: Zero-shot-CoT for each dataset in Appendix F.
Revised: Zero-shot-CoT for each dataset in Appendix F.

Original: Comparison with other baselines Table 9 in Appendix E indicates that while Zero-shot-CoT naturally underperforms Few-shot-CoT , it substantially outperforms standard Fewshot prompting with even 8 examples per task.
Revised: Comparison with other baselines Table 10 in Appendix E indicates that while Zero-shot-CoT naturally underperforms Few-shot-CoT, it substantially outperforms standard Few-shot prompting with even 8 examples per task.

Original: For GSM8K, Zero-shot-CoT with Instruct GPT-3 (175B)also outperforms finetuned GPT-3 and standard few-shot prompting with large models (PaLM, 540B), reported in Wei et al.
Revised: Major improvements are confirmed on both Instruct GPT- (175B) and PaLM (540B) models.

Original: [Table]
Revised: variant.

Revised: This template is used in (Ahn et al.,

Original:  where a language model is prompted to generate step-bystep actions given a high-level instruction for controlling robotic actions.
Revised: ) where a language model is prompted to generate step-bystep actions given a high-level instruction for controlling robotic actions.

Original: This template is used in (Reynolds &
Revised: This template is used in (Reynolds &

Original: McDonell, 2021) but is not quantitatively evaluated.
Revised: McDonell, 2021) but is not quantitatively evaluated.

Original: [Table]
Revised: [Table]

Original: Does model size matter for zero-shot reasoning?
Revised: Does model size matter for zero-shot reasoning?

Original: Table 10 in Appendix E compares performance of various language models.
Revised: Table 11 in Appendix E compares performance of various language models.

Original: The performance drastically increases with chain of thought reasoning, as the model size gets bigger.
Revised: The performance drastically increases with chain of thought reasoning, as the model size gets bigger.

Original: When the model size is smaller, chain of thought reasoning is not effective.
Revised: When the model size is smaller, chain of thought reasoning is not effective.

Original: How does prompt selection affect Zero-shot-CoT?
Revised: How does prompt selection affect Zero-shot-CoT?

Original: Table 2 summarizes performance using eight different templates.
Revised: Table 2 summarizes performance using eight different templates.

Original: The results indicate that the performance is improved if the text is written in a way that encourages chain of thought reasoning.
Revised: The results indicate that the performance is improved if the text is written in a way that encourages chain of thought reasoning.

Original: However, the difference in accuracy is significant depending on the sentence.
Revised: However, the difference in accuracy is significant depending on the sentence.

Original: (see Appendix F for example outputs by each template).
Revised: (see Appendix F for example outputs by each template).

Original: How does prompt selection affect Few-shot-CoT?
Revised: How does prompt selection affect Few-shot-CoT?

Original: Table 11 in Appendix E shows the performance of Few-shotCoT when using examples from different datasets.
Revised: Table 9 in Appendix E shows the performance of Few-shotCoT when using examples from different datasets.

Original: The results are worse than Zero-shot-CoT, affirming the importance of task-specific sample engineering in Few-shot-CoT.
Revised: The results are worse than Zero-shot-CoT, affirming the importance of task-specific sample engineering in Few-shot-CoT.

Original: 4. Discussion and Related Work
Revised: 4. Discussion and Related Work

Original: Reasoning Ability of LLMs Several studies have shown that pre-trained models usually are not good at reasoning (Brown et al., 2020; Smith et al., 2022;
Revised: Reasoning Ability of LLMs Several studies have shown that pre-trained models usually are not good at reasoning (Brown et al., 2020; Smith et al., 2022;

Original: Rae et al., 2021), but its ability can be substantially increased by making them produce step-by-step reasoning, either by fine-tuning (Rajani et al., 2019; Cobbe et al., 2021; Zelikman et al., 2022;
Revised: Rae et al., 2021), but its ability can be substantially increased by making them produce step-by-step reasoning, either by fine-tuning (Rajani et al., 2019; Cobbe et al., 2021; Zelikman et al., 2022;

Original: Nye et al., 2022) or few-shot prompting (Wei et al., 2022;
Revised: Nye et al., 2022) or few-shot prompting (Wei et al., 2022;

Original: Wang et al., 2022; Chowdhery et al., 2022) (Table 3).
Revised: Wang et al., 2022; Chowdhery et al., 2022) (Table 3).

Original: Unlike most prior work, we focus on zero-shot prompting and show that a single fixed trigger prompt substantially increases the zero-shot reasoning ability of LLMs across a variety of tasks requiring complex multi-hop thinking (Table 1), especially when the model is scaled up (Table 10).
Revised: Unlike most prior work, we focus on zero-shot prompting and show that a single fixed trigger prompt substantially increases the zero-shot reasoning ability of LLMs across a variety of tasks requiring complex multi-hop thinking (Table 1), especially when the model is scaled up (Table 11).

Original: It also generates reasonable and understandable chain of thought across diverse tasks (Appendix F), even when the final prediction is wrong (Appendix G).
Revised: It also generates reasonable and understandable chain of thought across diverse tasks (Appendix F), even when the final prediction is wrong (Appendix G).

Original: Similar to our work, Reynolds & McDonell (2021) demonstrate a prompt, “Let’s solve this problem by splitting it into steps”, would facilitate the multi-step reasoning in a simple arithmetic problem.
Revised: Similar to our work, Reynolds & McDonell (2021) demonstrate a prompt, “Let’s solve this problem by splitting it into steps”, would facilitate the multi-step reasoning in a simple arithmetic problem.

Original: However, they treated it as a task-specific example and did not evaluate quantitatively on diverse reasoning tasks against baselines.
Revised: However, they treated it as a task-specific example and did not evaluate quantitatively on diverse reasoning tasks against baselines.

Original: Shwartz et al.
Revised: Shwartz et al.

Original: (2020) propose to decompose a commonsense question into a series of information seeking question, such as “what is the definition of [X] ”.
Revised: (2020) propose to decompose a commonsense question into a series of information seeking question, such as “what is the definition of [X] ”.

Original: It does not require demonstrations but requires substantial manual prompt engineering per each reasoning task.
Revised: It does not require demonstrations but requires substantial manual prompt engineering per each reasoning task.

Original: Our results strongly suggest that LLMs are decent zero-shot reasoners, while prior work (Wei et al., 2022) often emphasize only few-shot learning and task-specific in-context learning, e.g. no zeroshot baselines were reported.
Revised: Our results strongly suggest that LLMs are decent zero-shot reasoners, while prior work (Wei et al., 2022) often emphasize only few-shot learning and task-specific in-context learning, e.g. no zeroshot baselines were reported.

Original: Our method does not require time-consuming fine-tuning or expensive sample engineering, and can be combined with any pre-trained LLM, serving as the strongest zero-shot baseline for all reasoning tasks.
Revised: Our method does not require time-consuming fine-tuning or expensive sample engineering, and can be combined with any pre-trained LLM, serving as the strongest zero-shot baseline for all reasoning tasks.

Original: Zero-shot Abilities of LLMs
Revised: Zero-shot Abilities of LLMs

Original: Radford et al.
Revised: Radford et al.

Original: (2019) show that LLMs have excellent zero-shot abilities in many system1 tasks, including reading comprehension, translation, and summarization.
Revised: (2019) show that LLMs have excellent zero-shot abilities in many system1 tasks, including reading comprehension, translation, and summarization.

Original: Sanh et al.
Revised: Sanh et al.

Original: ; Ouyang et al.
Revised: ; Ouyang et al.

Original: (2022) show that such zero-shot abilities of LLMs can be increased by explicitly fine-tuning models to follow instructions.
Revised: (2022) show that such zero-shot abilities of LLMs can be increased by explicitly fine-tuning models to follow instructions.

Original: Although these work focus on the zero-shot performances of LLMs, we focus on many system-2 tasks beyond systemtasks, considered a grand challenge for LLMs given flat scaling curves.
Revised: Although these work focus on the zero-shot performances of LLMs, we focus on many system-2 tasks beyond systemtasks, considered a grand challenge for LLMs given flat scaling curves.

Original: In addition, Zero-shot-CoT is orthogonal to instruction tuning; it increases zero-shot performance for both Instruct GPT3 and vanilla GPT3 (Table 10).
Revised: In addition, Zero-shot-CoT is orthogonal to instruction tuning; it increases zero-shot performance for both Instruct GPT3 and vanilla GPT3 (Table 11).

Original: From Narrow (task-specific) to Broad (multi-task) 
Revised: From Narrow (task-specific) to Broad (multi-task) Prompting

Original: Prompting Most prompts are task-specific.
Revised: Most prompts are task-specific.

Original: While fewshot prompts are naturally so due to task-specific in-context samples (Brown et al., 2020; Wei et al., 2022), majority of zero-shot prompts have also focused on per-task engineering (of templates)
Revised: While fewshot prompts are naturally so due to task-specific in-context samples (Brown et al., 2020; Wei et al., 2022), majority of zero-shot prompts have also focused on per-task engineering (of templates)

Original: (Liu et al., 2021b; Reynolds & McDonell, 2021).
Revised: (Liu et al., 2021b; Reynolds & McDonell, 2021).

Original: Borrowing terminologies from Chollet (2019) which builds on hierarchical models of intelligence (McGrew, 2005; Johnson & Bouchard Jr, 2005), these prompts are arguably eliciting “narrow generalization” or task-specific skills from LLMs.
Revised: Borrowing terminologies from Chollet (2019) which builds on hierarchical models of intelligence (McGrew, 2005; Johnson & Bouchard Jr, 2005), these prompts are arguably eliciting “narrow generalization” or task-specific skills from LLMs.

Original: On the other hand, our method is a multitask prompt and elicits “broad generalization” or broad cognitive abilities in LLMs, such as logical reasoning or system-2 itself.
Revised: On the other hand, our method is a multitask prompt and elicits “broad generalization” or broad cognitive abilities in LLMs, such as logical reasoning or system-2 itself.

Original: We hope our work can serve as a reference for accelerating not just logical reasoning research with
Revised: We hope our work can serve as a reference for accelerating not just logical reasoning research with

Original: LLMs, but also discovery of other broad cognitive capabilities within LLMs.
Revised: LLMs, but also discovery of other broad cognitive capabilities within LLMs.

Original: (More discussion in Appendix H)
Revised: (More discussion in Appendix H)

Original: A. Background
Revised: A. Background

Original: We briefly review the two core preliminary concepts that form the basis of this work: the advent of large language models (LLMs) and prompting, and chain of thought (CoT) prompting for multi-step reasoning.
Revised: We briefly review the two core preliminary concepts that form the basis of this work: the advent of large language models (LLMs) and prompting, and chain of thought (CoT) prompting for multi-step reasoning.

Original: Large language models and prompting A language model (LM), is a model that looks to estimate the probability distribution over text.
Revised: Large language models and prompting A language model (LM), is a model that looks to estimate the probability distribution over text.

Original: Recently, scaling improvements through larger model sizes (from a few million (Merity et al., 2016) to hundreds of millions (Devlin et al., 2019) to hundreds of billions (Brown et al., 2020) parameters) and larger data (e.g. webtext corpora (Gao et al., 2020)) have enabled pre-trained large language models (LLMs) to be incredibly adept at many downstream NLP tasks.
Revised: Recently, scaling improvements through larger model sizes (from a few million (Merity et al., 2016) to hundreds of millions (Devlin et al., 2019) to hundreds of billions (Brown et al., 2020) parameters) and larger data (e.g. webtext corpora (Gao et al., 2020)) have enabled pre-trained large language models (LLMs) to be incredibly adept at many downstream NLP tasks.

Original: Besides the classic “pre-train and fine-tune” paradigm (Liu et al., 2021b), models scaled to 100B+ parameters exhibit properties conducive to few-shot learning (Brown et al., 2020), by way of in context learning, where one can use a text or template known as a prompt to strongly guide the generation to output answers for desired tasks, thus beginning an era of “pre-train and prompt” (Liu et al., 2021a).
Revised: Besides the classic “pre-train and fine-tune” paradigm (Liu et al., 2021b), models scaled to 100B+ parameters exhibit properties conducive to few-shot learning (Brown et al., 2020), by way of in context learning, where one can use a text or template known as a prompt to strongly guide the generation to output answers for desired tasks, thus beginning an era of “pre-train and prompt” (Liu et al., 2021a).

Original: In work, we call such prompts with explicit conditioning on few task examples as few-shot prompts, and other template-only prompts as zero-shot prompts.
Revised: In work, we call such prompts with explicit conditioning on few task examples as few-shot prompts, and other template-only prompts as zero-shot prompts.

Original: Chain of thought prompting Multi-step arithmetic and logical reasoning benchmarks have particularly challenged the scaling laws of large language models (Rae et al., 2021).
Revised: Chain of thought prompting Multi-step arithmetic and logical reasoning benchmarks have particularly challenged the scaling laws of large language models (Rae et al., 2021).

Original: Chain of thought (CoT) prompting (Wei et al., 2022), an instance of few-shot prompting, proposed a simple solution by modifying the answers in few-shot examples to step-by-step answers, and achieved significant boosts in performance across these difficult benchmarks, especially when combined with very large language models like PaLM (Chowdhery et al., 2022).
Revised: Chain of thought (CoT) prompting (Wei et al., 2022), an instance of few-shot prompting, proposed a simple solution by modifying the answers in few-shot examples to step-by-step answers, and achieved significant boosts in performance across these difficult benchmarks, especially when combined with very large language models like PaLM (Chowdhery et al., 2022).

Original: The top row of Figure 1 shows standard few-shot prompting against (few-shot) CoT prompting.
Revised: The top row of Figure 1 shows standard few-shot prompting against (few-shot) CoT prompting.

Original: Notably, few-shot learning was taken as a given for tackling such difficult tasks, and the zero-shot baseline performances were not even reported in the original work (Wei et al., 2022).
Revised: Notably, few-shot learning was taken as a given for tackling such difficult tasks, and the zero-shot baseline performances were not even reported in the original work (Wei et al., 2022).

Original: To differentiate it from our method, we call Wei et al.
Revised: To differentiate it from our method, we call Wei et al.

Original: as Few-shot-CoT in this work.
Revised: as Few-shot-CoT in this work.

Original: B. Overview of Our Proposal
Revised: B. Overview of Our Proposal

Original: Q: On average Joe throws 25 punches per minute.
Revised: Q: On average Joe throws 25 punches per minute.

Original: A fight lasts 5 rounds of 3 minutes.
Revised: A fight lasts 5 rounds of 3 minutes.

Original: How many punches did he throw?
Revised: How many punches did he throw?

Original: A: Let's think step by step.
Revised: A: Let's think step by step.

Original: In one minute, Joe throws 25 punches.
Revised: In one minute, Joe throws 25 punches.

Original: In three minutes, Joe throws 3 * 25 = 75 punches.
Revised: In three minutes, Joe throws 3 * 25 = 75 punches.

Original: In five rounds, Joe throws 5 * 75 = 375 punches.
Revised: In five rounds, Joe throws 5 * 75 = 375 punches.

Original: Q: On average Joe throws 25 punches per minute.
Revised: Q: On average Joe throws 25 punches per minute.

Original: A fight lasts 5 rounds of 3 ・・・
Revised: A fight lasts 5 rounds of 3 ・・・

Original: A: Let's think step by step.
Revised: A: Let's think step by step.

Original: In one minute, Joe throws 25 punches.
Revised: In one minute, Joe throws 25 punches.

Original: In five rounds, Joe throws 5 * 75 = 375 punches. .
Revised: In five rounds, Joe throws 5 * 75 = 375 punches. .

Original: Therefore, the answer (arabic numerals) is 375.
Revised: Therefore, the answer (arabic numerals) is 375.

Original: [Figure]
Revised: [Figure]

Original: We propose Zero-shot-CoT, a zero-shot template-based prompting for chain of thought reasoning.
Revised: We propose Zero-shot-CoT, a zero-shot template-based prompting for chain of thought reasoning.

Original: It differs from the original chain of thought prompting (Wei et al., 2022) as it does not require step-by-step few-shot examples, and it differs from most of the prior template prompting (Liu et al., 2021b) as it is inherently task-agnostic and elicits multi-hop reasoning across a wide range of tasks with a single template.
Revised: It differs from the original chain of thought prompting (Wei et al., 2022) as it does not require step-by-step few-shot examples, and it differs from most of the prior template prompting (Liu et al., 2021b) as it is inherently task-agnostic and elicits multi-hop reasoning across a wide range of tasks with a single template.

Original: The core idea of our method is simple, as described in Figure 1: add Let’s think step by step , or a a similar text (see Table 2), to extract step-by-step reasoning.
Revised: The core idea of our method is simple, as described in Figure 1: add Let’s think step by step , or a a similar text (see Table 2), to extract step-by-step reasoning.

Original: While Zero-shot-CoT is conceptually simple, its subtlety is that it uses prompting twice, as explained in Figure 2.
Revised: While Zero-shot-CoT is conceptually simple, its subtlety is that it uses prompting twice, as explained in Figure 2.

Original: This is due to the fact that the zero-shot baseline (see the bottom-left in Figure 1) already uses prompting in the form of “The answer is”, to extract the answers in correct formats.
Revised: This is due to the fact that the zero-shot baseline (see the bottom-left in Figure 1) already uses prompting in the form of “The answer is”, to extract the answers in correct formats.

Original: Few-shot prompting, standard or CoT, avoids needing such answer-extraction prompting by explicitly designing the few-shot example answers to end in such formats (see the top-right in Figure 1).
Revised: Few-shot prompting, standard or CoT, avoids needing such answer-extraction prompting by explicitly designing the few-shot example answers to end in such formats (see the top-right in Figure 1).

Original: In summary, Few-shot-CoT (Wei et al., 2022) requires careful human engineering of a few prompt examples with specific answer formats per task, while Zero-shot-CoT does not require such engineering but requires prompting LLMs twice.
Revised: In summary, Few-shot-CoT (Wei et al., 2022) requires careful human engineering of a few prompt examples with specific answer formats per task, while Zero-shot-CoT does not require such engineering but requires prompting LLMs twice.

Original: C. Related Work
Revised: C. Related Work

Original: [Table]
Revised: [Table]

Original: Model

Original: [Table] et al., 2019)
Revised: (Rajani et al., 2019)

Original: [Table]
Revised: [Table]

Original: ✓

Original: CR GPT [Table] et al., 2021)
Revised: (Cobbe et al., 2021)

Original: [Table]
Revised: [Table]

Original: ✓
Revised: ✓

Original: AR
Revised: AR

Original: GPT-

Original: [Table]
Revised: [Table]

Original: Zelikman et al., [Table]
Revised: - (Zelikman et al., 2022)

Original: [Table]
Revised: [Table]

Original: ✓
Revised: ✓

Original: AR,CR
Revised: AR,CR

Original: GPT-3, etc [Table] Nye et al., 2022)
Revised: GPT-3, etc (Nye et al., 2022)

Original: [Table]
Revised: [Table]

Original: ✓
Revised: ✓

Original: AR
Revised: AR

Original: Transformer
Revised: Transformer

Original: [Table]
Revised: [Table]

Original: ) (Brown et al., 2020)
Revised: (Brown et al., 2020)

Original: Few

Original: [Table]
Revised: [Table]

Original: Shot

Original: CR
Revised: CR

Original: GPT- ( [Table] et al., 2022)
Revised: GPT- (Smith et al., 2022)

Original: Few

Original: [Table]
Revised: [Table]

Original: Shot

Original: AR,CR
Revised: AR,CR

Original: MT-NLG (
Revised: MT-NLG (Rae et al., 2021)

Original: [Table]
Revised: [Table]

Original: et al., 2021) Few
Revised: AR

Original: [Table]
Revised: [Table]

Original: Shot

Original: AR,CR

Original: Gopher (Wei et al., 2022)
Revised: Gopher (Wei et al., 2022)

Original: Few

Original: [Table]
Revised: [Table]

Original: Shot ✓
Revised: ✓

Original: AR,CR
Revised: AR,CR

Original: PaLM, LaMBDA, GPT- (Wang et al., 2022)
Revised: PaLM, LaMBDA, GPT- (Wang et al., 2022)

Original: Few

Original: [Table]
Revised: [Table]

Original: ✓ AR,CR
Revised: AR [Table]

Original: PaLM, etc (Chowdhery et al., 2022)
Revised: PaLM [Table] etc (Chowdhery et al., 2022)

Original: Few-Shot ✓
Revised: [Table] ✓

Original: AR,CR
Revised: AR,CR

Original: PaLM (Shwartz et al., 2020)
Revised: PaLM (Shwartz et al., 2020)

Original: Zero-Shot ✓CR
Revised: [Table] ✓

Original: GPT-2, etc (Reynolds & McDonell, 2021)
Revised: [Table] (Reynolds & McDonell, 2021)

Original: Zero-Shot ✓
Revised: [Table] ✓

Original: AR
Revised: AR

Original: GPTZero-shot-CoT (Ours)
Revised: GPT- [Table] CoT (Ours)

Original: Zero-Shot ✓
Revised: [Table] ✓

Original: AR,CR
Revised: AR,CR

Original: GPT-
Revised: GPT-

Original: D. Details of Experimental SetupD.1. Code
Revised: D. Details of Experimental Setup

Original: Code is available at https://anonymous.4open.science/r/zero shot cot-1B2D/
Revised: [Table]

Original:  D.2. Datasets D.2.1. D ATASET D ESCRIPTION
Revised: -1B2D/ D.2. Datasets D.2.1. D ATASET D ESCRIPTION

Original: For arithmetic reasoning, we consider the following six datasets: (1) SingleEq (Koncel-Kedziorski et al., 2015), (2) AddSub (Hosseini et al., 2014), (3)
Revised: For arithmetic reasoning, we consider the following six datasets: (1) SingleEq (Koncel-Kedziorski et al., 2015), (2) AddSub (Hosseini et al., 2014), (3)

Original: MultiArith (Roy & Roth, 2015), (4) AQUA-RAT (Ling et al., 2017), (5) GSM8K (Cobbe et al., 2021), and (6) SVAMP (Patel et al., 2021).
Revised: MultiArith (Roy & Roth, 2015), (4) AQUA-RAT (Ling et al., 2017), (5) GSM8K (Cobbe et al., 2021), and (6) SVAMP (Patel et al., 2021).

Original: The first three are from the classic Math World Problem Repository (Koncel-Kedziorski et al., 2016), and the last three are from more recent benchmarks.
Revised: The first three are from the classic Math World Problem Repository (Koncel-Kedziorski et al., 2016), and the last three are from more recent benchmarks.

Original: SingleEq and AddSub contain easier problems, which do not require multi-step calculation to solve the tasks.
Revised: SingleEq and AddSub contain easier problems, which do not require multi-step calculation to solve the tasks.

Original: MultiArith, AQUA-RAT, GSM8k, and SVAMP are more challenging datasets that require multi-step reasoning to solve.
Revised: MultiArith, AQUA-RAT, GSM8k, and SVAMP are more challenging datasets that require multi-step reasoning to solve.

Original: For commonsense reasoning, we use CommonsenseQA (Talmor et al., 2019) and StrategyQA (Geva et al., 2021).
Revised: For commonsense reasoning, we use CommonsenseQA (Talmor et al., 2019) and StrategyQA (Geva et al., 2021).

Original: CommonsenseQA asks questions with complex semantics that often require reasoning based on prior knowledge (Talmor et al., 2019).
Revised: CommonsenseQA asks questions with complex semantics that often require reasoning based on prior knowledge (Talmor et al., 2019).

Original: StrategyQA requires models to infer an implicit multi-hop reasoning to answer questions (Geva et al., 2021).
Revised: StrategyQA requires models to infer an implicit multi-hop reasoning to answer questions (Geva et al., 2021).

Original: For symbolic reasoning, we use Last Letter Concatenation and Coin Flip (Wei et al., 2022).
Revised: For symbolic reasoning, we use Last Letter Concatenation and Coin Flip (Wei et al., 2022).

Original: Last letter Concatenation asks the model to concatenate the last letters of each word.
Revised: Last letter Concatenation asks the model to concatenate the last letters of each word.

Original: We used randomly selected four names for each sample.
Revised: We used randomly selected four names for each sample.

Original: Coin flip asks the model to answer whether a coin is still heads up after people either flip or do not flip the coin.
Revised: Coin flip asks the model to answer whether a coin is still heads up after people either flip or do not flip the coin.

Original: We created samples of four times flip or do not flip trials.
Revised: We created samples of four times flip or do not flip trials.

Original: Although these tasks are easy for humans, language models typically exhibit a flat scaling curve.
Revised: Although these tasks are easy for humans, language models typically exhibit a flat scaling curve.

Original: For other logical reasoning tasks, we choose two evaluation sets from the BIG-bench effort (big, 2021):
Revised: For other logical reasoning tasks, we choose two evaluation sets from the BIG-bench effort (big, 2021):

Original: Date Understanding
Revised: Date Understanding

Original: 2 and Tracking Shuffled Objects.
Revised: 2 and Tracking Shuffled Objects.

Original: Date Understanding asks models to infer the date from a context.
Revised: Date Understanding asks models to infer the date from a context.

Original: Tracking Shuffled
Revised: Tracking Shuffled

Original: Objects tests a model’s ability to infer the final state of objects given its initial state and a sequence of object shuffling.
Revised: Objects tests a model’s ability to infer the final state of objects given its initial state and a sequence of object shuffling.

Original: We used a dataset of tracking three shuffled objects for our experiment.
Revised: We used a dataset of tracking three shuffled objects for our experiment.

Original: Table 4 summarizes the description of each dataset used in our experiment.[Table]
Revised: Table 4 summarizes the description of each dataset used in our experiment.

Original: Shuffled Objects

Original: [Table]
Revised: [Table]

Original: D.2.2. D ATASET CREATION
Revised: D.2.2. D ATASET CREATION

Original: Regarding “Last Letter Concatenation” and “Coin Flip”, datasets are not publicly available so we created the datasets following (Wei et al., 2022) with a minor rephrasing of the question template.
Revised: Regarding “Last Letter Concatenation” and “Coin Flip”, datasets are not publicly available so we created the datasets following (Wei et al., 2022) with a minor rephrasing of the question template.

Original: Specifically, as for Last Letter Concatenation, we use the following template.
Revised: Specifically, as for Last Letter Concatenation, we use the following template.

Original: We randomly select human names from names-dataset library ( https://pypi.org/ project/names-dataset/ ) and insert them into { Name1 } through { Name4 } .
Revised: We randomly select human names from names-dataset library ( https://pypi.org/ project/names-dataset/ ) and insert them into { Name1 } through { Name4 } .

Original: • ’Take the last letters of each words in ” { Name1 } { Name2 } { Name3 } { Name4 } ” and concatenate them.
Revised: • ’Take the last letters of each words in ” { Name1 } { Name2 } { Name3 } { Name4 } ” and concatenate them.’

Original: As for Coin Flip, we use the following template.
Revised: As for Coin Flip, we use the following template.

Original: We randomly select human names from names-dataset library and insert them into { Name1 } through { Name4 } .
Revised: We randomly select human names from names-dataset library and insert them into { Name1 } through { Name4 } .

Original: We also randomly pick up “flips” or “does not flip” and insert the phrase into each { flips — does not flip } part, respectively.
Revised: We also randomly pick up “flips” or “does not flip” and insert the phrase into each { flips — does not flip } part, respectively.

Original: • ’A coin is heads up.
Revised: • ’A coin is heads up.

Original: { Name1 } { flips — does not flip } the coin.
Revised: { Name1 } { flips — does not flip } the coin.

Original: { Name2 } { flips — does not flip } the coin.
Revised: { Name2 } { flips — does not flip } the coin.

Original: { Name3 } { flips — does not flip } the coin.
Revised: { Name3 } { flips — does not flip } the coin.

Original: { Name4 } { flips — does not flip } the coin.
Revised: { Name4 } { flips — does not flip } the coin.

Original: Is the coin still heads up?
Revised: Is the coin still heads up?

Original: Note that ”flip” here means ”reverse”.’
Revised: Note that ”flip” here means ”reverse”.’

Original: D.3. Language Models
Revised: D.3. Language Models

Original: Our experiment uses multiple language models as described at Table
Revised: Our experiment uses multiple language models as described at Table

Original: D.4. Implementation details
Revised: D.4. Implementation details

Original: For GPT-3 and Instruct-GPT3, we used OpenAI API.
Revised: For GPT-3 and Instruct-GPT3, we used OpenAI API.

Original: For other models, we used Hugging Face Transformer Library (Wolf et al., 2020).
Revised: For other models, we used Hugging Face Transformer Library (Wolf et al., 2020).

Original: We set max tokens = 128 and temperature = 0 across all the methods and models when requesting the APIs.
Revised: We set max tokens = 128 and temperature = 0 across all the methods and models when requesting the APIs.

Original: “Q:” is set as a customized stop sequence for all the models except for Instruct-GPT3 to stop the models from repeating[Table] [Table] [Table]
Revised: “Q:” is set as a customized stop sequence for all the models except for Instruct-GPT3 to stop the models from repeating

Original: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: questions and answers by themselves.
Revised: questions and answers by themselves.

Original: We run our experiments on cloud V100 instances without GPU for GPT-3 models, on cloud A100x8 GPU(60GB) instances for T0 and OTP, and on cloud A100x1 GPU(60GB) instances for the other models.
Revised: We run our experiments on cloud V100 instances without GPU for GPT-3 models, on cloud A100x8 GPU(60GB) instances for T0 and OTP [Table] and on cloud A100x1 GPU(60GB) instances for the other models.

Original: Our implementation is in PyTorch (Paszke et al., 2019).
Revised: Our implementation is in PyTorch (Paszke et al., 2019) [Table]

Original: D.5. Prompts For Answer Extraction
Revised: D.5. Prompts For Answer Extraction

Original: Table 6 and Table 7 summarizes a list of answer extraction prompts used for the experiments at Table 1.
Revised: Table 6 and Table 7 summarizes a list of answer extraction prompts used for the experiments at Table [Table] The answer is [Table]

Revised: The

Original: We used Zero-shot (left) and Zero-shot-CoT (left) as default prompts for answer extraction across all the experiments.
Revised: [Table] is [Table] The

Revised: [Table]

Revised: is

Revised: [Table]

Revised: The

Revised: [Table]

Revised: is

Revised: [Table]

Revised: The

Revised: [Table]

Revised: is

Original: [Table]
Revised: [Table]

Original: Answer Cleansing
Revised: The answer is [Table] The [Table]

Revised: The

Revised: [Table]

Revised: is

Revised: [Table]

Revised: The

Revised: [Table]

Revised: is

Revised: [Table]

Revised: The

Revised: [Table]

Revised: is

Revised: [Table]

Revised: The

Original:  Answer cleansing After the model outputs a text by answer extraction (see § 2 and Figure 2), our method picks up only the part of the answer text that first satisfies the answer format.
Revised: [Table] is [Table] Answer cleansing After the model outputs a text by answer extraction (see § 2 and Figure 2), our method picks up only the part of the answer text that first satisfies the answer format.

Original: For example, if the answer prompting outputs “probably 375 and 376” on arithmetic tasks, we extract the first number “375” and set it as the model prediction.
Revised: For example, if the answer prompting outputs “probably 375 and 376” on arithmetic tasks, we extract the first number “375” and set it as the model prediction.

Original: In the case of multiple-choice, the first large letter we encounter is set as the prediction.
Revised: In the case of multiple-choice, the first large letter we encounter is set as the prediction.

Original: Standard Zero-shot method follows the same idea.
Revised: Standard Zero-shot method follows the same idea.

Original: For Few-shot and Few-shot-CoT methods, we follow (Wang et al., 2022) and first extract the answer text after ”The answer is ” from the model output, and apply the same answer cleansing to parse the answer text.
Revised: For Few-shot and Few-shot-CoT methods, we follow (Wang et al., 2022) and first extract the answer text after ”The answer is ” from the model output, and apply the same answer cleansing to parse the answer text.

Original: If “The answer is” is not found in the model output, we search from the back of the text and set the first text that satisfies the answer format as the prediction.
Revised: If “The answer is” is not found in the model output, we search from the back of the text and set the first text that satisfies the answer format as the prediction.

Original: Table 8 summarizes a list of answer cleansing approaches used across all the experiments.[Table] [Table]
Revised: Table 8 summarizes a list of answer cleansing approaches used across all the experiments.

Original: [Table]
Revised: [Table]

Original: Therefore, the answer (arabic numerals) is
Revised: Therefore, the answer (arabic numerals) is

Original: Therefore, the answer is
Revised: Therefore, the answer is

Original: [Table]
Revised: [Table]

Original: Therefore, the answer (arabic numerals) is
Revised: Therefore, the answer (arabic numerals) is

Original: Therefore, the answer is
Revised: Therefore, the answer is

Original: [Table]
Revised: [Table]

Original: Therefore, the answer (arabic numerals) is
Revised: Therefore, the answer (arabic numerals) is

Original: Therefore, the answer is
Revised: Therefore, the answer is

Original: [Table]
Revised: [Table]

Original: Therefore, the answer (arabic numerals) is
Revised: Therefore, the answer (arabic numerals) is

Original: Therefore, the answer is
Revised: Therefore, the answer is

Original: [Table]
Revised: [Table]

Original: Therefore, among A through E, the answer is
Revised: Therefore, among A through E, the answer is

Original: Therefore, the answer is
Revised: Therefore, the answer is

Original: [Table]
Revised: [Table]

Original: Therefore, the answer (arabic numerals) is
Revised: Therefore, the answer (arabic numerals) is

Original: Therefore, the answer is
Revised: Therefore, the answer is

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Therefore, among A through E, the answer is
Revised: Therefore, among A through E, the answer is

Original: Therefore, the answer is
Revised: Therefore, the answer is

Original: [Table]
Revised: [Table]

Original: StrategyQA

Original: Therefore, the answer (Yes or No) is
Revised: Therefore, the answer (Yes or No) is

Original: Therefore, the answer is
Revised: Therefore, the answer is

Original: [Table]
Revised: [Table]

Original: Therefore, among A through F, the answer is
Revised: Therefore, among A through F, the answer is

Original: Therefore, the answer is
Revised: Therefore, the answer is

Original: [Table]

Original: S

Original: [Table]
Revised: [Table]

Original: Therefore, among A through C, the answer is
Revised: Therefore, among A through C, the answer is

Original: Therefore, the answer is
Revised: Therefore, the answer is

Original: [Table]
Revised: [Table]

Original: Therefore, the answer is
Revised: Therefore, the answer is

Original: [Table]
Revised: [Table]

Original: Therefore, the answer (Yes or No) is

Original: Therefore, theanswer is
Revised: Therefore, the

Original: [Table]
Revised: [Table]

Original: Just

Original: [Table]
Revised: [Table]

Original: unnecessary let
Revised: or No) is Therefore, the answer is

Original: [Table]
Revised: [Table]

Original: , such as ”
Revised: Free Format Just remove unnecessary letters, such as ”.”.

Original: [Table]
Revised: [Table]

Original: E. Detailed Experiment Results Zero-shot-CoT vs. Zero-shot
Revised: E. Detailed Experiment Results Zero-shot-CoT vs. Zero-shot

Original: Table 1 summarize accuracy of our method (Zero-shot-CoT) and standard zero-shot prompting (Zero-shot) for each dataset.
Revised: Table 1 summarize accuracy of our method (Zero-shot-CoT) and standard zero-shot prompting (Zero-shot) for each dataset.

Original: Zero-shot-CoT substantially outperforms four out of six arithmetic reasoning tasks (MultiArith, GSM8K, AQUA, SVAMP), all symbolic reasoning [Table] and all other logical reasoning tasks (from BIG-bench (big, 2021)).
Revised: Zero-shot-CoT substantially outperforms four out of six arithmetic reasoning tasks (MultiArith, GSM8K, AQUA, SVAMP), all symbolic reasoning, and all other logical reasoning tasks (from BIG-bench (big, 2021)).

Original: For example, Zero-shot-CoT achieves score gains from 17.7% to 78.7% on MultiArith and from 10.4% to 40.7% on
Revised: For example, Zero-shot-CoT achieves score gains from 17.7% to 78.7% on MultiArith and from 10.4% to 40.7% on

Original: GSM8K. Our method gives on-par performances for the remaining two arithmetic reasoning tasks (SingleEq and AddSub), which is expected since they do not require multi-step reasoning.
Revised: GSM8K. Our method gives on-par performances for the remaining two arithmetic reasoning tasks (SingleEq and AddSub), which is expected since they do not require multi-step reasoning.

Original: In commonsense reasoning tasks, Zero-shot-CoT does not provide performance gains.
Revised: In commonsense reasoning tasks, Zero-shot-CoT does not provide performance gains.

Original: It is expected as Wei et al.
Revised: It is expected as Wei et al.

Original: (2022) also reports that even Few-shot-CoT does not provide performance gains on Lambda (135B), but does improve StrategyQA when combined with substantially larger PaLM (540B) model, which may also apply for ours.
Revised: (2022) also reports that even Few-shot-CoT does not provide performance gains on Lambda (135B), but does improve StrategyQA when combined with substantially larger PaLM (540B) model, which may also apply for ours.

Original: More importantly, we observe that many generated chain of thought themselves are surprisingly logically correct or only contains human-understandable mistakes (see Table 12), suggesting that Zero-shot-CoT does elicit for better commonsense reasoning even when the task metrics donot directly reflect it.
Revised: More importantly, we observe that many generated chain of thought themselves are surprisingly logically correct or only contains human-understandable mistakes (see Table 12), suggesting that Zero-shot-CoT does elicit for better commonsense reasoning even when the task metrics do

Original: We provide samples generated by Zero-shot-CoT for each dataset in Appendix F.
Revised: [Table]

Original: Comparison with other baselines Table 9 compares the performances on two arithmetic reasoning benchmarks (MultiArith and GSM8K) across Zero-shot-CoT and baselines.
Revised: Comparison with other baselines Table 10 compares the performances on two arithmetic reasoning benchmarks (MultiArith and GSM8K) across Zero-shot-CoT and baselines.

Original: The large gap between standard prompting (top block) and chain of thought prompting (middle block) suggests that these tasks are difficult without eliciting multi-step reasoning.
Revised: The large gap between standard prompting (1st block) and chain of thought prompting (2nd block) suggests that these tasks are difficult without eliciting multi-step reasoning.

Original: While
Revised: Major improvements are confirmed on both Instruct GPT-3 (175B) and PaLM (540B) models (4th block).

Original:  Zero-shot-CoT naturally underperforms Few-shot-CoT, it substantially outperforms standard Few-shot prompting with even 8 examples pertask.
Revised: While Zero-shot-CoT naturally underperforms Few-shot-CoT, it substantially outperforms standard Few-shot prompting with even 8 examples per

Original: For GSM8K, Zero-shot-CoT with Instruct GPT-3 (175B) also outperforms finetuned GPT-3 and standard few-shot prompting with large models (PaLM, 540B), reported in Wei et al.
Revised: For GSM8K, Zero-shot-CoT with Instruct GPT-3 (175B) also outperforms finetuned GPT-3 and standard few-shot prompting with large models (PaLM, 540B), reported in Wei al and

Original: (2022) (bottom block).

Original: Error Analysis To better understand the behavior of Zero-shot-CoT, we manually investigated randomly selected examples generated by Instruct-GPT3 with Zero-shot-CoT prompting.
Revised: Error Analysis To better understand the behavior of Zero-shot-CoT, we manually investigated randomly selected examples generated by Instruct-GPT3 with Zero-shot-CoT prompting.

Original: See Appendix G for examples, where some of the observations include: (1) In commonsense reasoning (CommonsenseQA), Zero-shot-CoT often produces flexible and reasonable chain of thought even when the final prediction is not correct.
Revised: See Appendix G for examples, where some of the observations include: (1) In commonsense reasoning (CommonsenseQA), Zero-shot-CoT often produces flexible and reasonable chain of thought even when the final prediction is not correct.

Original: Zero-shot-CoT often output multiple answer choices when the model find it is difficult to narrow it down to one (see Table 12 for examples).
Revised: Zero-shot-CoT often output multiple answer choices when the model find it is difficult to narrow it down to one (see Table 12 for examples).

Original: In arithmetic reasoning (MultiArith), Zero-shotCoT and Few-shot-CoT show substantial differences regarding the error patterns.
Revised: In arithmetic reasoning (MultiArith), Zero-shotCoT and Few-shot-CoT show substantial differences regarding the error patterns.

Original: First, it tends to output unnecessary steps of reasoning after getting the correct prediction, which results in changing the prediction to incorrect one.
Revised: First, it tends to output unnecessary steps of reasoning after getting the correct prediction, which results in changing the prediction to incorrect one.

Original: Zero-shot-CoT also sometimes does not start reasoning, just rephrasing the input question.
Revised: Zero-shot-CoT also sometimes does not start reasoning, just rephrasing the input question.

Original: In contrast, Few-shot-CoT tend to fail when generated chain of thought include ternary operation, e.g. (3 + 2) ∗ 4 .
Revised: In contrast, Few-shot-CoT tend to fail when generated chain of thought include ternary operation, e.g. (3 + 2) ∗ 4 .

Original: Does model size matter for zero-shot reasoning?
Revised: Does model size matter for zero-shot reasoning?

Original: Table 10 compares performance of various language models on
Revised: Table 11 compares performance of various language models on

Original: MultiArith dataset.
Revised: MultiArith dataset.

Original: Without chain of thought reasoning, the performance does not increase or increases slowly as the model scale is increased, i.e., the curve is mostly flat.
Revised: Without chain of thought reasoning, the performance does not increase or increases slowly as the model scale is increased, i.e., the curve is mostly flat.

Original: In contrast, the performance drastically increases with chain of thought reasoning, as the model size gets bigger.
Revised: In contrast, the performance drastically increases with chain of thought reasoning, as the model size gets bigger.

Original: When the model size is smaller, chain of thought reasoning is not effective.
Revised: When the model size is smaller, chain of thought reasoning is not effective.

Original: This result aligns with the few-shot experiment results in (Wei et al., 2022).
Revised: This result aligns with the few-shot experiment results in (Wei et al., 2022).

Original: We also manually investigated the quality of generated chain of thought, and large-scale models clearly demonstrate better reasoning (see Appendix F for the sampled outputs for each model).
Revised: We also manually investigated the quality of generated chain of thought, and large-scale models clearly demonstrate better reasoning (see Appendix F for the sampled outputs for each model).

Original: How does prompt selection affect Zero-shot-CoT?
Revised: How does prompt selection affect Zero-shot-CoT?

Original: We validate the robustness of Zero-shot-CoT against input prompts.
Revised: We validate the robustness of Zero-shot-CoT against input prompts.

Original: Table 2 summarizes performance using eight different templates.
Revised: Table 2 summarizes performance using eight different templates.

Original: The results indicate that the performance is improved if the text is written in a way that encourages chain of thought reasoning.
Revised: The results indicate that the performance is improved if the text is written in a way that encourages chain of thought reasoning.

Original: However, the difference in accuracy is significant depending on the sentence.
Revised: However, the difference in accuracy is significant depending on the sentence.

Original: In this experiment, ”Let’s think step by step.”
Revised: In this experiment, ”Let’s think step by step.”

Original: achieves the best results.
Revised: achieves the best results.

Original: Interestingly, it is found that different templates encourage the model to express reasoning quite differently (see Appendix F for example outputs by each template).
Revised: Interestingly, it is found that different templates encourage the model to express reasoning quite differently (see Appendix F for example outputs by each template).

Original: How does prompt selection affect Few-shot-CoT?
Revised: How does prompt selection affect Few-shot-CoT?

Original: Table 11 shows the performance of Few-shot-CoT when using examples from different datasets: CommonsenseQA to AQUA-RAT and CommonsenseQA to MultiArith.
Revised: Table 9 shows the performance of Few-shot-CoT when using examples from different datasets: CommonsenseQA to AQUA-RAT and CommonsenseQA to MultiArith.

Original: The domains are different in both cases, but the answer format is the same in the former.
Revised: The domains are different in both cases, but the answer format is the same in the former.

Original: Surprisingly, the chain of thought examples from different domains (common sense to arithmetic) but with the same answer (multiple-choice) format provide substantial performance gain over Zero-shot (to AQUA-RAT), measured relative to the possible improvements from Zero-shot-CoT or
Revised: Surprisingly, the chain of thought examples from different domains (common sense to arithmetic) but with the same answer (multiple-choice) format provide substantial performance gain over Zero-shot (to AQUA-RAT), measured relative to the possible improvements from Zero-shot-CoT or

Original: Few-shot-CoT.
Revised: Few-shot-CoT.

Original: In contrast, the performance gain becomes much less when using examples with different answer types (to
Revised: In contrast, the performance gain becomes much less when using examples with different answer types (to

Original: MultiArith), confirming prior work (Min et al., 2022) that suggests LLMs mostly leverage the few-shot examples to infer the repeated format rather than the task itself in-context.
Revised: MultiArith), confirming prior work (Min et al., 2022) that suggests LLMs mostly leverage the few-shot examples to infer the repeated format rather than the task itself in-context.

Original: Nevertheless, for both cases the results are worse than Zero-shot-CoT, affirming the importance of task-specific sample engineering in Few-shot-CoT.
Revised: Nevertheless, for both cases the results are worse than Zero-shot-CoT, affirming the importance of task-specific sample engineering in Few-shot-CoT.

Revised: [Table]

Revised: that CoT is effective when the model is larger, such as Instruct GPT-3 (175B parameters; text-davinci-001 and text-davinci-002) and Original GPT-3 (175B parameters; davinci).

Revised: In this experiment, the order of performance (ascending) is Zero-shot, Few-shot (8samples), Zero-shot-CoT, and Few-shot-CoT (8samples) for davinci text

Revised: [Table]

Original: F. Generated Text Samples
Revised: [Table] Exercise regularly 2. Eat a healthy diet 3. [Table]

Revised: enough

Revised: [Table]

Revised: F. Generated Text Samples

Original: This section summarizes more example texts generated in our experiments.
Revised: This section summarizes more example texts generated in our experiments.

Original: Note that for readability all texts are modified from the original ones by omitting or inserting some linebreaks.
Revised: Note that for readability all texts are modified from the original ones by omitting or inserting some linebreaks.

Original: • Table 13 lists example texts generated by Zero-shot-CoT for each dataset (See Table 1).
Revised: • Table 13 lists example texts generated by Zero-shot-CoT for each dataset (See Table 1).

Original: • Table 14 lists example texts generated by Zero-shot-CoT for each reasoning extraction template (See Table 2).
Revised: • Table 14 lists example texts generated by Zero-shot-CoT for each reasoning extraction template (See Table 2).

Original: • Table 15 and Table 16 lists example texts generated by Zero-shot-CoT for each langugage model (See Table 10).
Revised: • Table 15 and Table 16 lists example texts generated by Zero-shot-CoT for each langugage model (See Table 11).

Original: • Table 17 has an example text generated by Few-shot.[Table] [Table]
Revised: • Table 17 has an example text generated by Few-shot.

Original: [Table]

Original: • Table 18 has an example text generated by Few-shot-CoT. • Table 19 has an example text generated by Few-shot-CoT with exemplars from a different task (Exemplars from CommonsenseQA, and a task is from MultiArith).• Table 20 has an [Table] [Table]
Revised: • Table 18 has an example text generated by Few-shot-CoT. • Table 19 has an example text generated by Few-shot-CoT with exemplars from a different task (Exemplars from CommonsenseQA, and a task is from MultiArith).

Original: [Table]

Original: [Table]

Original: things: [Table] Get enough sleep If you can do all three of those things, then you will be well on your way to getting in shape! Therefore, among A through E, the answer is B, C, and D. (GT : B, Zero-shot: B)
Revised: • Table 20 has an example text generated by Zero-Plus-Few-Shot-CoT.

Original: [Table]
Revised: [Table]

Original: SingleEq
Revised: SingleEq

Original: Q: A spaceship traveled 0.5 of a lightyear from Earth to Planet X and 0.1 of a light-year from Planet X to Planet
Revised: Q: A spaceship traveled 0.5 of a lightyear from Earth to Planet X and 0.1 of a light-year from Planet X to Planet

Original: Y. Then it traveled 0.1 of a light-year from Planet Y back to Earth.
Revised: Y. Then it traveled 0.1 of a light-year from Planet Y back to Earth.

Original: How many light-years did the spaceship travel in all?
Revised: How many light-years did the spaceship travel in all?

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: The spaceship traveled 0.
Revised: The spaceship traveled 0.

Original: light-years from Earth to Planet X. Then it traveled 0.
Revised: light-years from Earth to Planet X. Then it traveled 0.

Original: light-years from Planet X to
Revised: light-years from Planet X to

Original: Planet Y. Finally, it traveled 0.1 light-years from Planet Y back to Earth.
Revised: Planet Y. Finally, it traveled 0.1 light-years from Planet Y back to Earth.

Original: So the total distance the spaceship traveled is 0.5 + 0.

Revised: So the total distance the spaceship traveled is 0.5 + 0.

Original: + 0.1 = 0.
Revised: + 0.1 = 0.

Original: light-years.
Revised: light-years.

Original: Therefore, the answer (arabic numerals) is: 0.7 light-years ✓
Revised: Therefore, the answer (arabic numerals) is: 0.7 light-years ✓

Original: Q: While making desserts for a bake sale, Victor used 0.625 of a scoop of brown sugar as well as 0.25 of a scoop of white sugar.
Revised: Q: While making desserts for a bake sale, Victor used 0.625 of a scoop of brown sugar as well as 0.25 of a scoop of white sugar.

Original: How much more brown sugar did Victor use?
Revised: How much more brown sugar did Victor use?

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: 0.625 of a scoop of brown sugar is the same as 0.25 of a scoop of white sugar.
Revised: 0.625 of a scoop of brown sugar is the same as 0.25 of a scoop of white sugar.

Original: This means that Victor used the same amount of brown sugar as he did white sugar.
Revised: This means that Victor used the same amount of brown sugar as he did white sugar.

Original: Therefore, the answer (arabic numerals) is 0.
Revised: Therefore, the answer (arabic numerals) is 0.

Original: ✗ (GT : 0.375)
Revised: ✗ (GT : 0.375)

Original: AddSub
Revised: AddSub

Original: Sandy grew 6 carrots .
Revised: Sandy grew 6 carrots .

Original: Sam grewcarrots .
Revised: Sam grewcarrots .

Original: How many carrots did they grow in total ?
Revised: How many carrots did they grow in total ?

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: Sandy grew 6 carrots.
Revised: Sandy grew 6 carrots.

Original: Sam grew 3 carrots.
Revised: Sam grew 3 carrots.

Original: That means they grew a total of 9 carrots.
Revised: That means they grew a total of 9 carrots.

Original: Therefore, the answer (arabic numerals) is 9.
Revised: Therefore, the answer (arabic numerals) is 9.

Original: ✓
Revised: ✓

Original: Q: Melanie bought a Batman game for $ 6.95 , a strategy game for $ 7.90 , and a Superman game for $ 7.73 .
Revised: Q: Melanie bought a Batman game for $ 6.95 , a strategy game for $ 7.90 , and a Superman game for $ 7.73 .

Original: Melanie already owns 4 games .
Revised: Melanie already owns 4 games .

Original: How much did Melanie spend on video games ?
Revised: How much did Melanie spend on video games ?

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: Melanie bought a Batman game for $ 6.95.
Revised: Melanie bought a Batman game for $ 6.95.

Original: So she spent $ 6.95 on video games.
Revised: So she spent $ 6.95 on video games.

Original: Then she bought a strategy game for $ 7.90.
Revised: Then she bought a strategy game for $ 7.90.

Original: So she spent $ 14.85 on video games.
Revised: So she spent $ 14.85 on video games.

Original: Then she bought a Superman game for $ 7.73.
Revised: Then she bought a Superman game for $ 7.73.

Original: So she spent $ 22.58 on video games.
Revised: So she spent $ 22.58 on video games.

Original: Melanie already owns 4 games.
Revised: Melanie already owns 4 games.

Original: So she spent $ 18.
Revised: So she spent $ 18.

Original: on video games.
Revised: on video games.

Original: Therefore, Melanie spent a total of $ 18.58 on video games.
Revised: Therefore, Melanie spent a total of $ 18.58 on video games.

Original: Therefore, the answer (arabic numerals) is: 18.58 ✗ (GT : 22.58)
Revised: Therefore, the answer (arabic numerals) is: 18.58 ✗ (GT : 22.58)

Original: Continued on next page
Revised: Continued on next page

Original: Dataset
Revised: [Table]

Original: Question
Revised: Question

Original: Answer
Revised: Answer

Original: MultiArith
Revised: [Table]

Original: Q: Adam bought 2 boxes of chocolate candy and 5 boxes of caramel candy.
Revised: Q: Adam bought 2 boxes of chocolate candy and 5 boxes of caramel candy.

Original: If each box has 4 pieces inside it, how much candy did he have total?
Revised: If each box has 4 pieces inside it, how much candy did he have total?

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: Adam bought 2 boxes of chocolate candy and 5 boxes of caramel candy.
Revised: Adam bought 2 boxes of chocolate candy and 5 boxes of caramel candy.

Original: We know that each box has 4 pieces inside it.
Revised: We know that each box has 4 pieces inside it.

Original: So, we can multiply 2 by 4 to find out how many pieces of chocolate candy Adam bought.
Revised: So, we can multiply 2 by 4 to find out how many pieces of chocolate candy Adam bought.

Original: This will give us 8.
Revised: This will give us 8.

Original: We can also multiply 5 by 4 to find out how many pieces of caramel candy Adam bought.
Revised: We can also multiply 5 by 4 to find out how many pieces of caramel candy Adam bought.

Original: This will give us 20.
Revised: This will give us 20.

Original: Now, we can add 8 and 20 together to find out how much candy Adam bought in total.
Revised: Now, we can add 8 and 20 together to find out how much candy Adam bought in total.

Original: 8 + 20 = 

Revised: 8 + 20 =

Original: Adam bought 28 pieces of candy in Therefore, the answer (arabic numerals) is 28.
Revised: Adam bought 28 pieces of candy in Therefore, the answer (arabic numerals) is 28.

Original: ✓
Revised: ✓

Original: Q: Billy was organizing his baseball cards in a binder with 5 on each page.
Revised: Q: Billy was organizing his baseball cards in a binder with 5 on each page.

Original: If he had 3 new cards and 42 old cards to put in the binder, how many pages would he use?
Revised: If he had 3 new cards and 42 old cards to put in the binder, how many pages would he use?

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: Billy has 42 old cards.
Revised: Billy has 42 old cards.

Original: He can put 5 old cards on each page, which means he will use 9 pages for old cards.
Revised: He can put 5 old cards on each page, which means he will use 9 pages for old cards.

Original: Billy has 3 new cards.
Revised: Billy has 3 new cards.

Original: He can put 5 new cards on each page, which means he will use 1 page for new cards.
Revised: He can put 5 new cards on each page, which means he will use 1 page for new cards.

Original: So in total, Billy will usepages.
Revised: So in total, Billy will usepages.

Original: Therefore, the answer (arabic numerals) is 10.
Revised: Therefore, the answer (arabic numerals) is 10.

Original: ✗ (GT : 9)
Revised: ✗ (GT : 9)

Original: GSM8K
Revised: GSM8K

Original: Q: James buys a plane.
Revised: Q: James buys a plane.

Original: The plane cost $150,000.
Revised: The plane cost $150,000.

Original: He pays $5000 a month to rent a hanger to keep it in.
Revised: He pays $5000 a month to rent a hanger to keep it in.

Original: He also spends twice as much as that on fuel per month.
Revised: He also spends twice as much as that on fuel per month.

Original: How much did it cost him to get and maintain the plane for the first year?
Revised: How much did it cost him to get and maintain the plane for the first year?

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: James buys a plane.
Revised: James buys a plane.

Original: This costs him $150,000.
Revised: This costs him $150,000.

Original: He pays $5000 a month to rent a hanger.
Revised: He pays $5000 a month to rent a hanger.

Original: This costs him $60,000 for the year.
Revised: This costs him $60,000 for the year.

Original: He spends twice as much as that on fuel per month.
Revised: He spends twice as much as that on fuel per month.

Original: This costs him $120,000 for the year.
Revised: This costs him $120,000 for the year.

Original: So the total cost for the first year is $150,000 + $60,000 + $120,000 = $330,000.
Revised: So the total cost for the first year is $150,000 + $60,000 + $120,000 = $330,000.

Original: Therefore, the answer (arabic numerals) is 330,000.
Revised: Therefore, the answer (arabic numerals) is 330,000.

Original: ✓
Revised: ✓

Original: Q: James is counting his Pokemon cards.
Revised: Q: James is counting his Pokemon cards.

Original: He has 30 fire type, 20 grass type, and 40 water type.
Revised: He has 30 fire type, 20 grass type, and 40 water type.

Original: If he losesof the water type and buys 14 grass type, what’s the percentage chance (rounded to the nearest integer) that a randomly picked card will be a water type?
Revised: If he losesof the water type and buys 14 grass type, what’s the percentage chance (rounded to the nearest integer) that a randomly picked card will be a water type?

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: There are 30 fire type,grass type, and 40 water type.
Revised: There are 30 fire type,grass type, and 40 water type.

Original: If he loses 8 of the water type, then he has 32 cards in total.
Revised: If he loses 8 of the water type, then he has 32 cards in total.

Original: If he buys 14 grass type, then he has 46 cards in total, and the number of water type cards becomes 32.
Revised: If he buys 14 grass type, then he has 46 cards in total, and the number of water type cards becomes 32.

Original: The percentage chance that a randomly picked card will be a water type is $\frac{32}{46} \times 100\% = \fbox{70}\%$ .
Revised: The percentage chance that a randomly picked card will be a water type is $\frac{32}{46} \times 100\% = \fbox{70}\%$ .

Original: Therefore, the answer (arabic numerals) is 70.
Revised: Therefore, the answer (arabic numerals) is 70.

Original: ✗ (GT : 33)
Revised: ✗ (GT : 33)

Original: AQUA-RAT
Revised: AQUA-RAT

Original: Q: The capacity of a tank of dimensions (8 m × 6 m × 2.5 m) is Answer Choices: (A) 120 litres (B)litres (C) 12000 litres (D)litres (E)
Revised: Q: The capacity of a tank of dimensions (8 m × 6 m × 2.5 m) is Answer Choices: (A) 120 litres (B)litres (C) 12000 litres (D)litres (E)

Original: None of these
Revised: None of these

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: The first thing we need to do is calculate the volume of the tank.
Revised: The first thing we need to do is calculate the volume of the tank.

Original: The volume of the tank is 8m x 6m x 2.5m.
Revised: The volume of the tank is 8m x 6m x 2.5m.

Original: 8m x 6m x 2.5m = 120m ˆ
Revised: 8m x 6m x 2.5m = 120m ˆ

Original: Now that we have the volume of the tank, we need to convert it to litres.
Revised: Now that we have the volume of the tank, we need to convert it to litres.

Original: 1m ˆ 3 = 1000 litres 120m ˆ 3 = 120,000 litres
Revised: [Equation]

Original: The capacity of the tank is 120, litres
Revised: The capacity of the tank is 120,000 litres.

Original: Therefore, among A through E, the answer is D. ✓
Revised: Therefore, among A through E, the answer is D. ✓

Original: Continued on [Figure]
Revised: Continued on next page

Original: Dataset
Revised: Dataset

Original: Question
Revised: [Table]

Original: Answer
Revised: Answer

Original: Q: In a row of children Neha is 12th from left end and Radha is 6th from right end.
Revised: Q: In a row of children Neha is 12th from left end and Radha is 6th from right end.

Original: When Radha is shifted to left by 2 places and Neha is shifted to right by 2 places there 6 children between Radha and Neha.
Revised: When Radha is shifted to left by 2 places and Neha is shifted to right by 2 places there 6 children between Radha and Neha.

Original: How many children are there in the row?
Revised: How many children are there in the row?

Original: Answer
Revised: Answer

Original: Choices: (A) 23 (B) 27 (C) 26 (D)(E)
Revised: Choices: (A) 23 (B) 27 (C) 26 (D)(E)

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: There are 12 children to the left of Neha, so there must be at least 12 children in the row.
Revised: There are 12 children to the left of Neha, so there must be at least 12 children in the row.

Original: There are 6 children to the right of Radha, so there must be at least 18 children in the row.
Revised: There are 6 children to the right of Radha, so there must be at least 18 children in the row.

Original: There are 6 children between Radha and Neha, so there must be at least 24 children in the row.
Revised: There are 6 children between Radha and Neha, so there must be at least 24 children in the row.

Original: The answer must be between 24 and 27, and the only answer that satisfies this is 26.
Revised: The answer must be between 24 and 27, and the only answer that satisfies this is 26.

Original: Therefore, the answer is 26.
Revised: Therefore, the answer is 26.

Original: Therefore, among A through E, the answer is C. ✗ (GT : D)
Revised: Therefore, among A through E, the answer is C. ✗ (GT : D)

Original: SVAMP
Revised: SVAMP

Original: Q: There are 7 baskets of peaches.
Revised: Q: There are 7 baskets of peaches.

Original: Each basket has 10 red peaches and 2 green peaches.
Revised: Each basket has 10 red peaches and 2 green peaches.

Original: How many green peaches are in the baskets altogether?
Revised: How many green peaches are in the baskets altogether?

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: There are 7 baskets.
Revised: There are 7 baskets.

Original: Each basket has 10 red peaches and 2 green peaches.
Revised: Each basket has 10 red peaches and 2 green peaches.

Original: So, there are 2 green peaches in each basket.
Revised: So, there are 2 green peaches in each basket.

Original: Now, we need to multiply because there are 7 baskets.
Revised: Now, we need to multiply because there are 7 baskets.

Original: 2 multiplied by 7 is 14.
Revised: 2 multiplied by 7 is 14.

Original: So, there are 14 green peaches in the baskets altogether.
Revised: So, there are 14 green peaches in the baskets altogether.

Original: Therefore, the answer (arabic numerals) is 14.
Revised: Therefore, the answer (arabic numerals) is 14.

Original: ✓
Revised: ✓

Original: Q: Mary is baking a cake.
Revised: Q: Mary is baking a cake.

Original: The recipe calls for 12 cups of flour 14 cups of sugar and 7 cups of salt.
Revised: The recipe calls for 12 cups of flour 14 cups of sugar and 7 cups of salt.

Original: She already put in 2 cups of flour.
Revised: She already put in 2 cups of flour.

Original: How many more cups of flour than cups of salt does she need to add now?
Revised: How many more cups of flour than cups of salt does she need to add now?

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: Mary has already put incups of flour.
Revised: Mary has already put incups of flour.

Original: The recipe calls for a total of 12 cups of flour.
Revised: The recipe calls for a total of 12 cups of flour.

Original: That means Mary needs to add 10 more cups of flour.
Revised: That means Mary needs to add 10 more cups of flour.

Original: The recipe calls for a total of 7 cups of salt.
Revised: The recipe calls for a total of 7 cups of salt.

Original: That means Mary needs to add 5 more cups of salt.
Revised: That means Mary needs to add 5 more cups of salt.

Original: So, Mary needs to add 10 more cups of flour than she does salt.
Revised: So, Mary needs to add 10 more cups of flour than she does salt.

Original: Therefore, the answer (arabic numerals) is 10.
Revised: Therefore, the answer (arabic numerals) is 10.

Original: ✗ (GT : 3)
Revised: ✗ (GT : 3)

Original: CommonsenseQA
Revised: CommonsenseQA

Original: Q: If there is a pond with trees around it, where is it likely located?
Revised: Q: If there is a pond with trees around it, where is it likely located?

Original: Answer
Revised: Answer

Original: Choices: (A) ground (B) bathroom (C) forest (D) countryside (E) rural area
Revised: Choices: (A) ground (B) bathroom (C) forest (D) countryside (E) rural area

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: A pond is likely to be located near trees because trees need water to survive.
Revised: A pond is likely to be located near trees because trees need water to survive.

Original: Therefore, the most likely location for a pond with trees around it is in a forest.
Revised: Therefore, the most likely location for a pond with trees around it is in a forest.

Original: Therefore, among A through E, the answer is C. ✓
Revised: Therefore, among A through E, the answer is C. ✓

Original: Q: David watched some nesting birds using his binoculars while on vacation.
Revised: Q: David watched some nesting birds using his binoculars while on vacation.

Original: Where might David be?.
Revised: Where might David be?.

Original: Answer Choices:

Original: (A
Revised: An-

Original:  sky (B) vaccation (C) forest (D) countryside (E) roof
Revised: swer Choices: (A) sky (B) vaccation (C) forest (D) countryside (E) roof

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: First, we need to understand what the word ”binoculars” means.
Revised: First, we need to understand what the word ”binoculars” means.

Original: Binoculars are two telescopes that are joined together and allow a person to see things that are far away.
Revised: Binoculars are two telescopes that are joined together and allow a person to see things that are far away.

Original: Now,we need to think about where a person might use binoculars.
Revised: Now,we need to think about where a person might use binoculars.

Original: A person might use binoculars to see things that are far away and in the sky, or to see things that are far away and on the ground.
Revised: A person might use binoculars to see things that are far away and in the sky, or to see things that are far away and on the ground.

Original: So, the answer could be either (A) sky or (D) countryside.
Revised: So, the answer could be either (A) sky or (D) countryside.

Original: Therefore, among A through E, the answer is (A) or (D).
Revised: Therefore, among A through E, the answer is (A) or (D).

Original: ✗ (GT : C)
Revised: ✗ [Equation]

Original: Continued on next page
Revised: Continued on next page

Original: Dataset Question Answer
Revised: [Table]

Original: StrategyQA
Revised: StrategyQA

Original: Q: Would JPEG be a good format for saving an image of Da Vinci’s Vitruvian Man?
Revised: Q: Would JPEG be a good format for saving an image of Da Vinci’s Vitruvian Man?

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: The JPEG format is a lossy compression format, which means that some of the image data will be lost when the image is compressed.
Revised: The JPEG format is a lossy compression format, which means that some of the image data will be lost when the image is compressed.

Original: The
Revised: The

Original: Vitruvian Man is a very detailed image, with lots of small details.
Revised: Vitruvian Man is a very detailed image, with lots of small details.

Original: So, when we compress the image using
Revised: So, when we compress the image using

Original: JPEG, some of those small details will be lost, and the image will not be as sharp and clear as the original.
Revised: JPEG, some of those small details will be lost, and the image will not be as sharp and clear as the original.

Original: So, in conclusion, JPEG is not the best format for saving an image of the Vitruvian Man.
Revised: So, in conclusion, JPEG is not the best format for saving an image of the Vitruvian Man.

Original: Therefore, the answer (Yes or No) is No.
Revised: Therefore, the answer (Yes or No) is No.

Original: ✓
Revised: ✓

Original: Q: Is Mickey Mouse hypothetically unlikely to make a purchase at Zazzle?
Revised: Q: Is Mickey Mouse hypothetically unlikely to make a purchase at Zazzle?

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: First, we need to consider what Zazzle is.
Revised: First, we need to consider what Zazzle is.

Original: Zazzle is a website where people can create and sell custom products.
Revised: Zazzle is a website where people can create and sell custom products.

Original: This includes things like t-shirts, mugs, phone cases, and more.
Revised: This includes things like t-shirts, mugs, phone cases, and more.

Original: Next, we need to think about what Mickey Mouse is.
Revised: Next, we need to think about what Mickey Mouse is.

Original: Mickey
Revised: Mickey

Original: Mouse is a cartoon character created by Walt Disney.
Revised: Mouse is a cartoon character created by Walt Disney.

Original: He is a mouse who wears red shorts, large yellow shoes, and white gloves.
Revised: He is a mouse who wears red shorts, large yellow shoes, and white gloves.

Original: Now, we need to think about whether or not Mickey Mouse is likely to make a purchase at
Revised: Now, we need to think about whether or not Mickey Mouse is likely to make a purchase at

Original: Zazzle.
Revised: Zazzle.

Original: It is unlikely that Mickey Mouse would make a purchase at Z Therefore, the answer (Yes or No) is No. ✗ (GT : Yes)
Revised: It is unlikely that Mickey Mouse would make a purchase at Z Therefore, the answer (Yes or No) is No. [Table]

Original: Date
Revised: Date

Original: Understand.
Revised: Understand.

Original: Q: Today is 3/5, and it is Jane’s second time in the year 1973 to see a meteor shower.
Revised: Q: Today is 3/5, and it is Jane’s second time in the year 1973 to see a meteor shower.

Original: What is the date a month ago in MM/DD/YYYY?
Revised: What is the date a month ago in MM/DD/YYYY?

Original: Answer Choices: (A) 03/05/ (B) 01/02/1973 (C)
Revised: Answer Choices: (A) 03/05/ (B) 01/02/1973 (C)

Original: 03/08/ (D) 03/05/1973 (E) 02/05/1973 (F) 04/05/
Revised: 03/08/ (D) 03/05/1973 (E) 02/05/1973 (F) 04/05/

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: The question is asking for the date a month ago from 03/05/1973.
Revised: The question is asking for the date a month ago from 03/05/1973.

Original: A month ago would be 02/05/1973.
Revised: A month ago would be 02/05/1973.

Original: Therefore, among A through F, the answer is E. ✓
Revised: Therefore, among A through F, the answer is E. ✓

Original: Q: The deadline is Jun 1, 2021, which is 2 days away from now.
Revised: Q: The deadline is Jun 1, 2021, which is 2 days away from now.

Original: What is the date 24 hours later in
Revised: What is the date 24 hours later in

Original: MM/DD/YYYY?
Revised: MM/DD/YYYY?

Original: Answer Choices: (A) 06/02/

Revised: Answer Choices: (A) 06/02/

Original: (B) 05/31/ (C) 06/16/

Revised: (B) 05/31/ (C) 06/16/

Original: (D) 05/31/1966 (E) 08/18/2021 (F) 05/31/
Revised: (D) 05/31/1966 (E) 08/18/2021 (F

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: The deadline is Jun 1, 2021, which is 2 days away from now.
Revised: The deadline is Jun 1, 2021, which is 2 days away from now.

Original: In 24 hours, it will be
Revised: In 24 hours, it will be

Original: Jun 2, 2021.
Revised: Jun 2, 2021.

Original: Therefore, among A through F, the answer is A. ✗ (GT : B)
Revised: Therefore, among A through F, the answer is A. ✗ (GT : B)

Original: Continued on next page
Revised: Continued on next page

Original: Dataset
Revised: Dataset

Original: Question
Revised: Question

Original: Answer
Revised: Answer

Original: ShuffledObjects.
Revised: [Table]

Original: Q: Alice, Bob, and Claire are playing a game.
Revised: Q: Alice, Bob, and Claire are playing a game.

Original: At the start of the game, they are each holding a ball: Alice has a green ball, Bob has a purple ball, and
Revised: At the start of the game, they are each holding a ball: Alice has a green ball, Bob has a purple ball, and

Original: Claire has a blue ball.
Revised: Claire has a blue ball.

Original: As the game progresses, pairs of players trade balls.
Revised: As the game progresses, pairs of players trade balls.

Original: First, Bob and Alice swap balls.
Revised: First, Bob and Alice swap balls.

Original: Then, Claire and Bob swap balls.
Revised: Then, Claire and Bob swap balls.

Original: Finally, Bob and Alice swap balls.
Revised: Finally, Bob and Alice swap balls.

Original: At the end of the game,
Revised: At the end of the game,

Original: Bob has the Which choice is true ?
Revised: Bob has the Which choice is true ?

Original: Answer Choices: (A) green ball.
Revised: Answer Choices: (A) green ball.

Original: (B) purple ball.
Revised: (B) purple ball.

Original: (C) blue ball.
Revised: (C) blue ball.

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: First, Bob and Alice swap balls.
Revised: First, Bob and Alice swap balls.

Original: This means that Alice now has the purple ball and Bob has the green ball.
Revised: This means that Alice now has the purple ball and Bob has the green ball.

Original: Then, Claire and Bob swap balls.
Revised: Then, Claire and Bob swap balls.

Original: This means that Claire now has the green ball and Bob has the blue ball.
Revised: This means that Claire now has the green ball and Bob has the blue ball.

Original: Finally, Bob and Alice swap balls.
Revised: Finally, Bob and Alice swap balls.

Original: This means that
Revised: This means that

Original: Alice now has the blue ball and Bob has the purple ball.
Revised: Alice now has the blue ball and Bob has the purple ball.

Original: At the end of the game, Bob has the purple ball.
Revised: At the end of the game, Bob has the purple ball.

Original: Therefore, among A through C, the answer is B. ✓
Revised: Therefore, among A through C, the answer is B. ✓

Original: Q: Alice, Bob, and Claire are dancers at a square dance.
Revised: Q: Alice, Bob, and Claire are dancers at a square dance.

Original: At the start of a song, they each have a partner:
Revised: At the start of a song, they each have a partner:

Original: Alice is dancing with Sam, Bob is dancing with Helga, and Claire is dancing with Karl.
Revised: Alice is dancing with Sam, Bob is dancing with Helga, and Claire is dancing with Karl.

Original: Throughout the song, the dancers often trade partners.
Revised: Throughout the song, the dancers often trade partners.

Original: First, Claire and Alice switch partners.
Revised: First, Claire and Alice switch partners.

Original: Then, Bob and Alice switch partners.
Revised: Then, Bob and Alice switch partners.

Original: Finally, Claire and Bob switch partners.
Revised: Finally, Claire and Bob switch partners.

Original: At the end of the dance, Alice is dancing with Which choice is true ?
Revised: At the end of the dance, Alice is dancing with Which choice is true ?

Original: Answer Choices: (A) Sam.
Revised: Answer Choices: (A) Sam.

Original: (B) Helga.
Revised: (B) Helga.

Original: (C) Karl.
Revised: (C) Karl.

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: First, Claire and Alice switch partners.
Revised: First, Claire and Alice switch partners.

Original: This means that at the end of this step, Alice is dancing with Karl and
Revised: This means that at the end of this step, Alice is dancing with Karl and

Original: Claire is dancing with Sam.
Revised: Claire is dancing with Sam.

Original: Then, Bob and Alice switch partners.
Revised: Then, Bob and Alice switch partners.

Original: This means that at the end of this step, Alice is dancing with Helga and
Revised: This means that at the end of this step, Alice is dancing with Helga and

Original: Bob is dancing with Karl.
Revised: Bob is dancing with Karl.

Original: Finally, Claire and Bob switch partners.
Revised: Finally, Claire and Bob switch partners.

Original: This means that at the end of this step, Claire is dancing with Helga and
Revised: This means that at the end of this step, Claire is dancing with Helga and

Original: Bob is dancing with Sam.
Revised: Bob is dancing with Sam.

Original: At the end of the dance, Alice is dancing with Sam.
Revised: At the end of the dance, Alice is dancing with Sam.

Original: Therefore, among A through C, the answer is A. ✗ (GT : B)
Revised: Therefore, among A through C, the answer is A. ✗ (GT : B)

Original: Last Letters
Revised: [Table]

Original: Q: Take the last letters of each words in ”Vinny Landon Miguel Caitlyn” and concatenate them.
Revised: Q: Take the last letters of each words in ”Vinny Landon Miguel Caitlyn” and concatenate them

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: The last letter of ”Vinny” is ”y”.
Revised: The last letter of ”Vinny” is ”y”.

Original: The last letter of ”Landon” is ”n”.
Revised: The last letter of ”Landon” is ”n”.

Original: The last letter of ”Miguel” is ”l”.
Revised: The last letter of ”Miguel” is ”l”.

Original: The last letter of ”Caitlyn” is ”n”.
Revised: The last letter of ”Caitlyn” is ”n”.

Original: So, the final answer would be ”ynln”.
Revised: So, the final answer would be ”ynln”.

Original: Therefore, the answer is ”ynln”.
Revised: Therefore, the answer is ”ynln”.

Original: ✓
Revised: ✓

Original: Q: Take the last letters of each words in ”Lamont Frederick Lance Esperanza” and concatenate them.
Revised: Q : Take the last letters of each words in ”Lamont Frederick Lance Esperanza” and concatenate them.

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: The last letter of ”Lamont” is ”t”.
Revised: The last letter of ”Lamont” is ”t”.

Original: The last letter of ”Frederick” is ”k”.
Revised: The last letter of ”Frederick” is ”k”.

Original: The last letter of ”Lance” is ”e”.
Revised: The last letter of ”Lance” is ”e”.

Original: The last letter of ”Esperanza” is ”a”.
Revised: The last letter of ”Esperanza” is ”a”.

Original: So the final answer would be ”tkaea”.
Revised: So the final answer would be ”tkaea”.

Original: Therefore, the answer is ”tkaea”.
Revised: Therefore, the answer is ”tkaea”.

Original: ✗ (GT : tkea)
Revised: ✗ (GT : tkea)

Original: Continued on next page
Revised: Continued on next page

Original: Dataset Question Answer Coin Flip
Revised: [Table]

Original: Q: A coin is heads up.
Revised: Q: A coin is [Table] up.

Original: Jeff does not flip the coin.
Revised: Jeff does not flip the coin.

Original: Jen flips the coin.
Revised: Jen flips the coin.

Original: Giselle flips the coin.
Revised: Giselle flips the coin.

Original: Noel does not flip the coin.
Revised: Noel does not flip the coin.

Original: Is the coin still heads up?
Revised: Is the coin still heads up?

Original: Note that ”flip” here means ”reverse”.
Revised: Note that ”flip” here means ”reverse”.

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: Jeff does not flip the coin, so the coin remains heads up.
Revised: Jeff does not flip the coin, so the coin remains heads up.

Original: Jen flips the coin, so the coin is now tails up.
Revised: Jen flips the coin, so the coin is now tails up.

Original: Giselle flips the coin, so the coin is now heads up again.
Revised: Giselle flips the coin, so the coin is now heads up again.

Original: Noel does not flip the coin, so the coin remains heads up.
Revised: Noel does not flip the coin, so the coin remains heads up.

Original: Therefore, the final answer is that the coin is heads up.
Revised: Therefore, the final answer is that the coin is heads up.

Original: Therefore, the answer (Yes or No) is Yes.
Revised: Therefore, the answer (Yes or No) is Yes.

Original: ✓
Revised: ✓

Original: Q: A coin is heads up.
Revised: [Table] A coin is heads up.

Original: Caroline flips the coin.
Revised: Caroline flips the coin.

Original: Demetrius flips the coin.
Revised: Demetrius flips the coin.

Original: Fidel does not flip the coin.
Revised: Fidel does not [Table] the coin.

Original: Solomon flips the coin.
Revised: Solomon flips the coin.

Original: Is the coin still heads up?
Revised: Is the coin still heads up?

Original: Note that ”flip” here means ”reverse”.
Revised: Note that ”flip” here means ”reverse”.

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: Caroline flips the coin.
Revised: Caroline flips the coin.

Original: The coin is now tails up.
Revised: The coin is now tails up.

Original: Demetrius flips the coin.
Revised: Demetrius flips the coin.

Original: The coin is now heads up.
Revised: The coin is now heads up.

Original: Fidel does not flip the coin.
Revised: Fidel does not flip the coin.

Original: The coin remains tails up.
Revised: The coin remains tails up.

Original: Solomon flips the coin.
Revised: Solomon flips the coin.

Original: The coin is now heads up.
Revised: The coin is now heads up.

Original: So, the answer is yes, the coin is still heads up.
Revised: So, the answer is yes, the coin is still heads up.

Original: Therefore, the answer (Yes or No) is: Yes ✗ (GT : No) 
Revised: Therefore, the answer (Yes or No) is: Yes ✗ (GT : No) [Table] Q [Table]

Revised: For

Original: [Table]
Revised: [Table]

Original: For Halloween Megan received 11 pieces of candy from neighbors and 5 pieces from her older sister.
Revised: received [Table] of [Table]

Revised: [Table]

Original: If she only ate 8 pieces a day, how long would the candy last her?
Revised: If she only ate 8 pieces a day, how long would the candy last her?

Original: [Table]
Revised: [Table]

Revised: A

Original:  Let’s think step by step.
Revised: : Let’s think step by step.

Original: Megan received 11 pieces of candy from neighbors and 5 pieces from her older sister.
Revised: Megan received 11 pieces of candy from neighbors and 5 pieces from her older sister.

Original: That means she has a total of 16 pieces of candy.
Revised: That means she has a total of 16 pieces of candy.

Original: If she only ate 8 pieces a day, she would have to divide her candy into two days.
Revised: If she only ate 8 pieces a day, she would have to divide her candy into two days.

Original: That means she would have 8 pieces of candy for Day 1 and 8 pieces of candy for Day2.
Revised: That means she would have 8 pieces of candy for Day 1 and 8 pieces of candy for Day

Original: Therefore, the answer (arabic numerals) is 2.
Revised: Therefore, the answer (arabic numerals) is 2.

Original: ✓
Revised: ✓

Original: [Table]
Revised: [Table]

Revised: A:

Revised: First,

Revised: we

Revised: need

Original: A: First, we need to find the total number of pieces of candy Megan has.
Revised: to find the total number of pieces of candy Megan has

Original: She has11 pieces from neighbors and 5 pieces from her older sister, so the total number of pieces is 16.
Revised: She has pieces from neighbors and 5 pieces from her older sister, so the total number of pieces is 16.

Original: If she only atepieces a day, it would take her 2 days to finish the candy.
Revised: If she only atepieces a day, it would take her 2 days to finish the candy.

Original: Therefore, the answer (arabic numerals) is 2.
Revised: Therefore , the answer (arabic numerals is 2.

Original: ✓
Revised: ✓

Original: [Table]
Revised: [Table]

Original: : Let’s think about this logically.
Revised: A: Let’s think about this logically.

Original: There are 11 + 5 = 16 pieces of candy.
Revised: There are 11 + 5 = 16 pieces of candy.

Original: 8 pieces a day means that it would last her 2 days.
Revised: 8 pieces a day means that it would last her 2 days.

Original: 16 / 8 =
Revised: / 8 =

Original: Therefore, the answer (arabic numerals) is 2.
Revised: Therefore, the answer (arabic numerals) is 2.

Original: ✓
Revised: ✓

Original: [Table]
Revised: [Table]

Original: ) A: Let’s solve this problem by splitting it into steps.
Revised: A: Let’s solve this problem by splitting it into steps.

Original: Step 1: Megan received 11 pieces of candy from neighbors.
Revised: Step 1: Megan received 11 pieces of candy from neighbors.

Original: Step 2: Megan received 5 pieces of candy from her older sister.
Revised: Step 2: Megan received 5 pieces of candy from her older sister.

Original: Step 3: If she only ate 8 pieces a day, how long would the candy last her?
Revised: Step 3: If she only ate 8 pieces a day, how long would the candy last her?

Original: To answer the question in Step 3, we need to add the number of pieces of candy Megan received from her neighbors and her sister.
Revised: To answer the question in Step 3, we need to add the number of pieces of candy Megan received from her neighbors and her sister.

Original: This is called finding the sum.
Revised: This is called finding the sum.

Original: The sum of 11 and 5 is 16.
Revised: The sum of 11 and 5 is 16.

Original: So, Megan has a total of 16 pieces of candy.
Revised: So, Megan has a total of 16 pieces of candy.

Original: If Megan only eats 8 pieces of candy a day, Therefore, the answer (arabic numerals) is 2 days.
Revised: If Megan only eats 8 pieces of candy a day , Therefore, the answer (arabic numerals) is 2 days.

Original: ✓
Revised: ✓

Original: [Table]
Revised: [Table]

Original: ) A: Let’s be realistic and think step by step.
Revised: A: Let’s be realistic and think step by step.

Original: Megan received 11 pieces of candy from neighbors and 5 pieces from her older sister.
Revised: Megan received 11 pieces of candy from neighbors and 5 pieces from her older sister.

Original: That’s a total of 16 pieces of candy.
Revised: That’s a total of 16 pieces of candy.

Original: 8 pieces a day would mean that it would last her 2 days.
Revised: 8 pieces a day would mean that it would last her 2 days.

Original: So the candy would last her 2 days.
Revised: So the candy would last her 2 days.

Original: Therefore, the answer (arabic numerals) is 2.
Revised: Therefore, the answer (arabic numerals) is 2.

Original: ✓
Revised: ✓

Original: [Table]
Revised: [Table]

Original: A: Let’s think like a detective step by step.
Revised: A: Let’s think like a detective step by step.

Original: Megan received 11 pieces of candy from neighbors and 5 pieces from her older sister.
Revised: Megan received 11 pieces of candy from neighbors and 5 pieces from her older sister.

Original: That’s a total of 16 pieces of candy.
Revised: That’s a total of 16 pieces of candy.

Original: If she only ate 8 pieces a day, that would mean that she would have to divide her candy into 2 days, and she would have 8 pieces of candy left over.
Revised: If she only ate 8 pieces a day, that would mean that she would have to divide her candy into 2 days, and she would have 8 pieces of candy left over.

Original: So the candy would last her 2 days.
Revised: So the candy would last her 2 days.

Original: Therefore, the answer (arabic numerals) is 2.
Revised: Therefore, the answer (arabic numerals) is 2.

Original: ✓
Revised: ✓

Original: [Table]
Revised: [Table]

Original: A: Let’s think about this in terms of days.
Revised: A: Let’s think about this in terms of days.

Original: Megan would receive candy for 16 days.
Revised: Megan would receive candy for 16 days.

Original: Therefore, the answer (arabic numerals) is 16.
Revised: Therefore, the answer (arabic numerals) is 16.

Original: ✗
Revised: ✗

Original: [Table]
Revised: [Table]

Original: A: Before we dive into the answer, we need to think about what the question is asking.
Revised: A: Before we dive into the answer, we need to think about what the question is asking.

Original: The question is asking how long it would take Megan to eat all of her candy if she ate 8 pieces a day.
Revised: The question is asking how long it would take Megan to eat all of her candy if she ate 8 pieces a day.

Original: There are a total of 16 pieces of candy.
Revised: There are a total of 16 pieces of candy.

Original: If Megan ate 8 pieces a day, it would take her 2 days to eat all of her candy.
Revised: If Megan ate 8 pieces a day, it would take her 2 days to eat all of her candy.

Original: Therefore, the answer (arabic numerals) is 2.
Revised: Therefore, the answer (arabic numerals) is 2.

Original: ✓
Revised: ✓

Original: [Table]
Revised: [Table]

Original: -shot-CoT (9)

Original: A:
Revised: A:

Original: The answer is after the proof.
Revised: The answer is after the proof.

Original: 11 + 5 =16 ÷ 8 =2 days
Revised: 11 + 5 =16 ÷ 8 =2 days

Original: Therefore, the answer (arabic numerals) is 2.
Revised: Therefore, the answer (arabic numerals) is 2.

Original: ✓
Revised: ✓

Original: Question
Revised: Question

Original: Q: Adam bought 2 boxes of chocolate candy and 5 boxes of caramel candy.
Revised: Q: Adam bought 2 boxes of chocolate candy and 5 boxes of caramel candy.

Original: If each box has 4 pieces inside it, how much candy did he have total?
Revised: If each box has 4 pieces inside it, how much candy did he have total?

Original: (GT : 28)
Revised: (GT : 28)

Original: Instruct GPT- (0.3B)
Revised: Instruct GPT- (0.3B)

Original: (Text-ada-001)
Revised: (Text-ada-001)

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: If each box has 4 pieces inside it, Adam would have 4 pieces of candy in it.
Revised: If each box has 4 pieces inside it, Adam would have 4 pieces of candy in it.

Original: Therefore, the answer (arabic numerals) is :5 ✗
Revised: Therefore, the answer (arabic numerals) is :5 ✗

Original: Instruct GPT- (1.3B)
Revised: Instruct GPT- (1.3B)

Original: (Text-babbage-001)
Revised: (Text-babbage-001)

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: Adam bought 2 boxes of candy and 5 boxes of caramel candy.
Revised: Adam bought 2 boxes of candy and 5 boxes of caramel candy.

Original: Each box has 4 pieces inside it.
Revised: Each box has 4 pieces inside it.

Original: So he would have had 18 pieces of candy.
Revised: So he would have had 18 pieces of candy.

Original: Therefore, the answer (arabic numerals) is 18.
Revised: Therefore, the answer (arabic numerals) is 18.

Original: ✗
Revised: ✗

Original: Instruct GPT- (6.7B) (Text-curie-001)
Revised: Instruct GPT- (6.7B) (Text-curie-001)

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: Adam bought 2 boxes of chocolate candy and 5 boxes of caramel candy.
Revised: Adam bought 2 boxes of chocolate candy and 5 boxes of caramel candy.

Original: Each box of candy has 4 pieces inside it.
Revised: Each box of candy has 4 pieces inside it.

Original: So, Adam bought 10 pieces of candy.
Revised: So, Adam bought 10 pieces of candy.

Original: Therefore, the answer (arabic numerals) is 60.
Revised: Therefore, the answer (arabic numerals) is 60.

Original: ✗
Revised: ✗

Original: Instruct GPT- (175B) (Text-davinci-002)
Revised: Instruct GPT- (175B) (Text-davinci-002)

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: Adam bought 2 boxes of chocolate candy and 5 boxes of caramel candy.
Revised: Adam bought 2 boxes of chocolate candy and 5 boxes of caramel candy.

Original: We know that each box has 4 pieces inside it.
Revised: We know that each box has 4 pieces inside it.

Original: So, we can multiplyby 4 to find out how many pieces of chocolate candy Adam bought.
Revised: So, we can multiplyby 4 to find out how many pieces of chocolate candy Adam bought.

Original: This will give us 8.
Revised: This will give us 8.

Original: We can also multiply 5 by 4 to find out how many pieces of caramel candy Adam bought.
Revised: We can also multiply 5 by 4 to find out how many pieces of caramel candy Adam bought.

Original: This will give us 20.
Revised: This will give us 20.

Original: Now, we can add 8 and 20 together to find out how much candy Adam bought in total.
Revised: Now, we can add 8 and 20 together to find out how much candy Adam bought in total.

Original: 8 + 20 = 

Revised: 8 + 20 =

Original: Adam bought 28 pieces of candy in
Revised: Adam bought 28 pieces of candy in

Original: Therefore, the answer (arabic numerals) is 28.
Revised: Therefore, the answer (arabic numerals) is 28.

Original: ✓
Revised: ✓

Original: Original GPT- (0.3B) (ada)
Revised: Original GPT- (0.3B) (ada)

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: Adam bought 2 boxes of chocolate candy and 5 boxes of caramel candy.
Revised: Adam bought 2 boxes of chocolate candy and 5 boxes of caramel candy.

Original: If each box has 4 pieces inside it, how much candy did he have total? 2.
Revised: If each box has 4 pieces inside it, how much candy did he have total? 2.

Original: Adam bought 2 boxes of chocolate candy and 5 boxes of caramel candy.
Revised: Adam bought 2 boxes of chocolate candy and 5 boxes of caramel candy.

Original: If each box has 4 pieces inside it, how much candy did he have total? 3.
Revised: If each box has 4 pieces inside it, how much candy did he have total? 3.

Original: Adam bought 2 boxes of chocolate candy and 5 boxes of caramel candy.
Revised: Adam bought 2 boxes of chocolate candy and 5 boxes of caramel candy.

Original: If each box has 4 pieces inside it, how much candy did he have total? 4.
Revised: If each box has 4 pieces inside it, how much candy did he have total? 4.

Original: Adam bought 2 boxes of chocolate candy and 5 boxes of caramel candy.
Revised: Adam bought 2 boxes of chocolate candy and 5 boxes of caramel candy.

Original: If each box has 4 pieces inside it,
Revised: If each box has 4 pieces inside it,

Original: Therefore, the answer (arabic numerals) is : 5.
Revised: Therefore, the answer (arabic numerals) is : 5.

Original: Adam bought 2 boxes of chocolate candy and 5 boxes of caramel candy.
Revised: Adam bought 2 boxes of chocolate candy and 5 boxes of caramel candy.

Original: If each box has 4 pieces inside it, how much candy did he ✗
Revised: If each box has 4 pieces inside it, how much candy did he ✗

Original: Original GPT- (1.3B) (babbage)
Revised: Original GPT- (1.3B) (babbage)

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: Therefore, the answer (arabic numerals) is : ✗
Revised: Therefore, the answer (arabic numerals) is : ✗

Original: Original GPT- (6.7B) (curie)
Revised: Original GPT- (6.7B) (curie)

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: First, we need to find the total number of pieces in the boxes.
Revised: First, we need to find the total number of pieces in the boxes.

Original: Let’s start with the chocolate candy.
Revised: Let’s start with the chocolate candy.

Original: There are 4 pieces in each box.
Revised: There are 4 pieces in each box.

Original: So, the total number of pieces in the chocolate candy is 4 x 4 = 16.
Revised: So, the total number of pieces in the chocolate candy is 4 x 4 = 16.

Original: Now, let’s find the total number of pieces in the caramel candy.
Revised: Now, let’s find the total number of pieces in the caramel candy.

Original: There are 5 pieces in each box.
Revised: There are 5 pieces in each box.

Original: So, the total number of pieces in the caramel candy is 5 x 5 = 25.
Revised: So, the total number of pieces in the caramel candy is 5 x 5 = 25.

Original: Now, we can find the total number of pieces in the candy.
Revised: Now, we can find the total number of pieces in the candy.

Original: The total number of pieces in the chocolate candy is 16 + 25 =
Revised: The total number of pieces in the chocolate candy is 16 + 25 =

Original: Therefore, the answer (arabic numerals) is 41.
Revised: Therefore, the answer (arabic numerals) is 41.

Original: ✗
Revised: ✗

Original: Original GPT- (175B) (davinci)
Revised: Original GPT- (175B) (davinci)

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: First, we need to find out how many pieces of candy
Revised: First, we need to find out how many pieces of candy

Original: Adam has.
Revised: Adam has.

Original: 2 boxes of chocolate candy + 5 boxes of caramel candy = 7 boxes of candy 7 boxes of candy = 7 × 4 pieces of candy 7 × 4 = 28 pieces of candy
Revised: 2 boxes of chocolate candy + 5 boxes of caramel candy = 7 boxes of candy 7 boxes of candy = 7 × 4 pieces of candy 7 × 4 = 28 pieces of candy

Original: So, Adam haspieces of candy.
Revised: So, Adam haspieces of candy.

Original: Therefore, the answer (arabic numerals) is 28.
Revised: Therefore, the answer (arabic numerals) is 28.

Original: ✓
Revised: ✓

Original: [Table]
Revised: Question

Original:  Megan baked 68 cupcakes for her school’s bake sale.
Revised: Q: Megan baked 68 cupcakes for her school’s bake sale.

Original: If her brother, Todd, ate 32 of them how many packages could she make if she put 6 cupcake in each package
Revised: If her brother, Todd, ate 32 of them how many packages could she make if she put 6 cupcake in each package? (GT : 6)

Original: [Table]
Revised: [Table]

Original: (1.5B)
Revised: [Table]

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: Todd’s brother, Todd, eats 32 cupcakes.
Revised: Todd’s brother, Todd, eats 32 cupcakes.

Original: Todd’s brother, Todd, eats 32 cupcakes.Todd’s brother, Todd, eats 32 cupcakes. Todd’s brother, Todd, eats 32 cupcakes. Todd’s brother, Todd, eats 32 cupcakes.
Revised: Todd’s brother, Todd, eats 32 cupcakes.

Original: Todd’s brother, Todd, eats  cupcakes.
Revised: Todd’s brother, Todd, eats 32 cupcakes.

Original: Todd’s brother, Todd, eats  cupcakes.
Revised: Todd’s brother, Todd, eats 32 cupcakes.

Original: Todd’s brother
Revised: Todd’s brother,

Original: Todd, eats cupcakes
Revised: Todd, eats 32 cupcakes.

Original: Todd’s brother , Todd, eats cupcakes
Revised: Todd’s brother, Todd, eats 32 cupcakes.

Original: Todd’s brother, Todd eats cupcakes
Revised: Todd’s brother, Todd, eats 32 cupcakes.

Original: Todd’s brother , Todd, eats  cupcakes.
Revised: Todd’s brother, Todd, eats 32 cupcakes.

Original: Todd’s brother, Todd, eats Therefore, the answer (arabic numerals) is: Therefore, the answer (arabic numerals) is :
Revised: Todd’s brother, Todd, eats 32 cupcakes. Todd’s brother, Todd, eats 32 cupcakes.

Original: [Table] [Table] [Table] [Table]
Revised: Todd’s brother, Todd, eatscupcakes.

Original: [Table] [Table] [Table] [Table]
Revised: Todd’s brother, Todd, eats Therefore, the answer (arabic numerals) is:

Original: [Table] [Table] [Table] [Table]
Revised: Therefore, the answer (arabic numerals) is : 6,8,12,16,20,24,28,32,36,40,44,48,52,56,60,64 ✗

Original: [Table]
Revised: [Table]

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: Step 1: She baked 68 cupcakes.
Revised: Step 1: She baked 68 cupcakes.

Original: Step 2: She put 32 of them in a box.
Revised: Step 2: She put 32 of them in a box.

Original: Step 3: She put cupcakes in each box.
Revised: Step 3: She putcupcakes in each box.

Original: Step 4: She put the box in the freezer.
Revised: Step 4: She put the box in the freezer.

Original: Step 5: She took the box out of the freezer. Step 6: She took the box out of the freezer.
Revised: Step 5: She took the box out of the freezer. Step 6: She took the box out of the freezer.

Original: Step 7: She took the box out of the freezer. Step 8: She took the box out of the freezer.
Revised: Step 7: She took the box out of the freezer. Step 8: She took the box out of the freezer.

Original: Step 9: She took the box out of the freezer.
Revised: Step 9: She took the box out of the freezer.

Original: Step 10: She took the box out of the freezer.
Revised: Step 10: She took the box out of the freezer.

Original: Step
Revised: Step

Original: Therefore, the answer (arabic numerals) is :
Revised: Therefore, the answer (arabic numerals) is :

Original: Step : She baked cupcakes
Revised: Step 1: She baked 68 cupcakes.

Original: Step
Revised: Step

Original: : She put
Revised: : She put

Original:  of them in a box
Revised: 32 of them in a box.

Original: Step 3: She putcup ✗GPT
Revised: Step 3: She put cup ✗

Original: -J (6B
Revised: GPT-J (6B)

Original: ) A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: Step 1: Step 2: Step 3: Step 4: Step 5: Step 6: Step 7: Step 8: Step 9: Step 10: Step 11: Step 12: Step 13: Step 14: Step 15: Step 16: Step 17: Step 18: Step 19: Step 20: Step 21: Step 22: Step 23: Step 24: Step 25: Step
Revised: Step 1: Step 2: Step 3: Step 4: Step 5: Step 6: Step 7: Step 8: Step 9: Step 10: Step 11: Step 12: Step 13: Step 14: Step 15: Step 16: Step 17: Step 18: Step 19: Step 20: Step 21: Step 22: Step 23: Step 24: Step 25: Step

Original: Therefore, the answer (arabic numerals) is :
Revised: Therefore, the answer (arabic numerals) is :

Original: A:
Revised: A:

Original: The answer is: 68 × 6 = 408 Step 1: The first step is to count ✗
Revised: The answer is: 68 × 6 = 408 Step 1: The first step is to count ✗

Original: T (11B)
Revised: T (11B)

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: Therefore, the answer (arabic numerals) is12 ✗
Revised: Therefore, the answer arabic numerals) is

Revised: ✗

Original: OPT (13B)
Revised: OPT (13B)

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: Megan baked 68 cupcakes
Revised: Megan baked 68 cupcakes.

Original: She divided  by 6 to get She divided
Revised: She divided 68 by 6 to get 17.

Original:  by  to get 
Revised: She divided 17 by 32 to get 6.

Original: Megan can make 6 packages of cupcakes if she puts 6 cupcakes in each package
Revised: Megan can make 6 packages of cupcakes if she puts 6 cupcakes in each package.

Original: Therefore, the answer (arabic numerals) is : ✗
Revised: Therefore, the answer (arabic numerals) is : ✗

Original: [Table]

Original: [Table]
Revised: [Table]

Original: Q: Jason had 20 lollipops.
Revised: Q: Jason had 20 lollipops.

Original: He gave Denny some lollipops.
Revised: He gave Denny some lollipops.

Original: Now Jason has 12 lollipops.
Revised: Now Jason has 12 lollipops.

Original: How many lollipops did Jason give to Denny?
Revised: How many lollipops did Jason give to Denny?

Original: A:
Revised: A:

Original: The answer is 8.
Revised: The answer is 8.

Original: Q: Michael had 58 golf balls.
Revised: Q: Michael had 58 golf balls.

Original: On tuesday, he lost 23 golf balls.
Revised: On tuesday, he lost 23 golf balls.

Original: On wednesday, he lost 2 more.
Revised: On wednesday, he lost 2 more.

Original: How many golf balls did he have at the end of wednesday?
Revised: How many golf balls did he have at the end of wednesday?

Original: A:
Revised: A:

Original: The answer is 33.
Revised: The answer is 33.

Original: Q: If there are 3 cars in the parking lot and 2 more cars arrive, how many cars are in the parking lot?
Revised: Q: If there are 3 cars in the parking lot and 2 more cars arrive, how many cars are in the parking lot?

Original: A:
Revised: A:

Original: The answer is 5.
Revised: The answer is 5.

Original: Q: There were nine computers in the server room.
Revised: Q: There were nine computers in the server room.

Original: Five more computers were installed each day, from monday to thursday.
Revised: Five more computers were installed each day, from monday to thursday.

Original: How many computers are now in the server room?
Revised: How many computers are now in the server room?

Original: A:
Revised: A:

Original: The answer is 29.
Revised: The answer is 29.

Original: Q: Olivia has 23 .Sheboughtfivebagelsfor 3 each.
Revised: Q: Olivia has 23 .Sheboughtfivebagelsfor 3 each.

Original: How much money does she have left?
Revised: How much money does she have left?

Original: A:
Revised: A:

Original: The answer is 8.
Revised: The answer is 8.

Original: Q: There are 15 trees in the grove.
Revised: Q: There are 15 trees in the grove.

Original: Grove workers will plant trees in the grove today.
Revised: Grove workers will plant trees in the grove today.

Original: After they are done, there will be 21 trees.
Revised: After they are done, there will be 21 trees.

Original: How many trees did the grove workers plant today?
Revised: How many trees did the grove workers plant today?

Original: A:
Revised: A:

Original: The answer is 6.
Revised: The answer is 6.

Original: Q: Shawn has five toys.
Revised: Q: Shawn has five toys.

Original: For Christmas, he got two toys each from his mom and dad.
Revised: For Christmas, he got two toys each from his mom and dad.

Original: How many toys does he have now?
Revised: How many toys does he have now?

Original: A:
Revised: A:

Original: The answer is 9.
Revised: The answer is 9.

Original: Q: Leah had 32 chocolates and her sister had 42.
Revised: Q: Leah had 32 chocolates and her sister had 42.

Original: If they ate 35, how many pieces do they have left in total?
Revised: If they ate 35, how many pieces do they have left in total?

Original: A:
Revised: A:

Original: The answer is 39.
Revised: The answer is 39.

Original: [Table]

Original: End In-Context Examples

Original: [Table]
Revised: [Table]

Original: Q: For Halloween Megan received 11 pieces of candy from neighbors and 5 pieces from her older sister.
Revised: Q: For Halloween Megan received 11 pieces of candy from neighbors and 5 pieces from her older sister.

Original: If she only ate 8 pieces a day, how long would the candy last her?
Revised: If she only ate 8 pieces a day, how long would the candy last her?

Original: A:
Revised: A:

Original: The answer is 3 days.
Revised: The answer is 3 days.

Original: ✗ (GT : 2)
Revised: ✗ (GT : 2)

Original: [Table]
Revised: [Table]

Original: Q: Jason had 20 lollipops.
Revised: Q: Jason had 20 lollipops.

Original: He gave Denny some lollipops.
Revised: He gave Denny some lollipops.

Original: Now Jason has 12 lollipops.
Revised: Now Jason has 12 lollipops.

Original: How many lollipops did Jason give to Denny?
Revised: How many lollipops did Jason give to Denny?

Original: A:
Revised: A:

Original: Jason started with 20 lollipops.
Revised: Jason started with 20 lollipops.

Original: Then he had 12 after giving some to Denny.
Revised: Then he had 12 after giving some to Denny.

Original: So he gave Denny 20 - 12 = 8.
Revised: So he gave Denny 20 - 12 = 8.

Original: The answer is 8.
Revised: The answer is 8.

Original: Q: Michael had 58 golf balls.
Revised: Q: Michael had 58 golf balls.

Original: On tuesday, he lost 23 golf balls.
Revised: On tuesday, he lost 23 golf balls.

Original: On wednesday, he lost 2 more.
Revised: On wednesday, he lost 2 more.

Original: How many golf balls did he have at the end of wednesday?
Revised: How many golf balls did he have at the end of wednesday?

Original: A: Michael started with 58 golf balls.
Revised: A: Michael started with 58 golf balls.

Original: After losing 23 on tuesday, he had 58 - 23 = 35.
Revised: After losing 23 on tuesday, he had 58 - 23 = 35.

Original: After losing 2 more, he had 35 - 2 = 33 golf balls.
Revised: After losing 2 more, he had 35 - 2 = 33 golf balls.

Original: The answer is 33.
Revised: The answer is 33.

Original: Q: If there are 3 cars in the parking lot and 2 more cars arrive, how many cars are in the parking lot?
Revised: Q: If there are 3 cars in the parking lot and 2 more cars arrive, how many cars are in the parking lot?

Original: A:
Revised: A:

Original: There are originally 3 cars.
Revised: There are originally 3 cars.

Original: 2 more cars arrive.
Revised: 2 more cars arrive.

Original: 3 + 2 = 5.
Revised: 3 + 2 = 5.

Original: The answer is 5.
Revised: The answer is 5.

Original: Q: There were nine computers in the server room.
Revised: Q: There were nine computers in the server room.

Original: Five more computers were installed each day, from monday to thursday.
Revised: Five more computers were installed each day, from monday to thursday.

Original: How many computers are now in the server room?
Revised: How many computers are now in the server room?

Original: A:
Revised: A:

Original: There were originally 9 computers.
Revised: There were originally 9 computers.

Original: For each of 4 days, 5 more computers were added.
Revised: For each of 4 days, 5 more computers were added.

Original: So 5 * 4 = 20 computers were added.
Revised: So 5 * 4 = 20 computers were added.

Original: 9 + 20 is 29.
Revised: 9 + 20 is 29.

Original: The answer is 29.
Revised: The answer is 29.

Original: Q: Olivia has 23 .Sheboughtfivebagelsfor 3 each.
Revised: Q: Olivia has 23 .Sheboughtfivebagelsfor 3 each.

Original: How much money does she have left?
Revised: How much money does she have left?

Original: A: Olivia had 23 dollars.
Revised: A: Olivia had 23 dollars.

Original: 5 bagels for 3 dollars each will be 5 x 3 = 15 dollars.
Revised: 5 bagels for 3 dollars each will be 5 x 3 = 15 dollars.

Original: So she has 23 - 15 dollars left.
Revised: So she has 23 - 15 dollars left.

Original: 23 - 15 is 8.
Revised: 23 - 15 is 8.

Original: The answer is 8.
Revised: The answer is 8.

Original: Q: There are 15 trees in the grove.
Revised: Q: There are 15 trees in the grove.

Original: Grove workers will plant trees in the grove today.
Revised: Grove workers will plant trees in the grove today.

Original: After they are done, there will be 21 trees.
Revised: After they are done, there will be 21 trees.

Original: How many trees did the grove workers plant today?
Revised: How many trees did the grove workers plant today?

Original: A:
Revised: A:

Original: There are 15 trees originally.
Revised: There are 15 trees originally.

Original: Then there were 21 trees after some more were planted.
Revised: Then there were 21 trees after some more were planted.

Original: So there must have been 21 -15 =6.
Revised: So there must have been 21 - =

Original: The answer is6.
Revised: The answer is

Original: Q: Shawn has five toys.
Revised: Q: Shawn has five toys.

Original: For Christmas, he got two toys each from his mom and dad.
Revised: For Christmas, he got two toys each from his mom and dad.

Original: How many toys does he have now?
Revised: How many toys does he have now

Original: A:
Revised: A:

Original: Shawn started with 5 toys.
Revised: Shawn started with 5 toys.

Original: If he got 2 toys each from his mom and dad, then that is 4 more toys.
Revised: If he got 2 toys each from his mom and dad, then that is 4 more toys.

Original: 5 + 4 = 9.
Revised: 5 + 4 = 9.

Original: The answer is 9.
Revised: The answer is 9.

Original: Q: Leah had 32 chocolates and her sister had 42.
Revised: Q: Leah had 32 chocolates and her sister had 42.

Original: If they ate 35, how many pieces do they have left in total?
Revised: If they ate 35, how many pieces do they have left in total?

Original: A: Originally, Leah had 32 chocolates.
Revised: A: Originally, Leah had 32 chocolates.

Original: Her sister had 42.
Revised: Her sister had 42.

Original: So in total they had 32 += 74.
Revised: So in total they had 32 += 74.

Original: After eating 35, they had 74 - 35 = 39.
Revised: After eating 35, they had 74 - 35 = 39.

Original: The answer is 39.
Revised: The answer is 39.

Original: ***** End In-Context Examples ***** Q: For Halloween Megan received 11 pieces of candy from neighbors and 5 pieces from her older sister.
Revised: [Table] Q: For Halloween Megan received 11 pieces of candy from neighbors and 5 pieces from her older sister.

Original: If she only ate 8 pieces a day, how long would the candy last her?
Revised: If she only ate 8 pieces a day, how long would the candy last her?

Original: A: Megan received 11 pieces of candy from neighbors and 5 pieces from her older sister.
Revised: A: Megan received 11 pieces of candy from neighbors and 5 pieces from her older sister.

Original: So she had 11 + 5 = 16 pieces of candy.
Revised: So she had 11 + 5 = 16 pieces of candy.

Original: If she ate 8 pieces a day, then she would have 16 / 8 = 2 days worth of candy.
Revised: If she ate 8 pieces a day, then she would have 16 / 8 = 2 days worth of candy.

Original: The answer is 2.
Revised: The answer is 2.

Original: ✓
Revised: ✓

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: Q: Sammy wanted to go to where the people were. Where might he go? Answer Choices: (A) populated areas (B) race track (C) desert (D) apartment (E) roadblock A:

Original: Maps and other highway and street GPS services
Revised: The answer must be a place with a lot of people. Of the above choices, only populated areas have a lot of people. The answer is A. Q: Before getting a divorce, what did the wife feel who was doing all the work? Answer Choices: (A) harder (B) anguish (C) bitterness (D) tears (E) sadness

Original: [Table]
Revised: A:

Original: replaced what
Revised: The answer should be the feeling of someone getting divorced who was doing all the work.

Original: [Table]
Revised: Of the above choices, the closest feeling is bitterness The answer is C. Q: Google Maps and other highway and street GPS services have replaced what? Answer Choices: (A) united states (B) mexico (C) countryside (D) atlas

Original:  The answer must be something that used to do what Google Maps and GPS services do, which is to give directions.
Revised: A: The answer must be something that used to do what Google Maps and GPS services do, which is to give directions.

Original: Of the above choices, only atlases are used to give directions.
Revised: Of the above choices, only atlases are used to give directions.

Original: The answer is D [Table] The fox walked from the city into the forest, what was it looking for?
Revised: The answer is D. Q: The fox walked from the city into the forest, what was it looking for?

Original: Answer Choices: (A) pretty flowers (B) hen house (C) natural habitat (D) storybook A: The answer must be something in the forest.
Revised: Answer Choices: (A) pretty flowers (B) hen house (C) natural habitat (D) storybook A: The answer must be something in the forest.

Original: Of the above choices, only natural habitat is in the forest.
Revised: Of the above choices, only natural habitat is in the forest.

Original: The answer is B. Q: What do people use to absorb extra ink from a fountainpen?
Revised: The answer is B. Q: What do people use to absorb extra ink from a fountainpen?

Original: Answer Choices: (A) shirt pocket (B) calligrapher’s hand (C)
Revised: Answer Choices: (A) shirt pocket (B) calligrapher’s hand (C)

Original: inkwell (D) desk drawer (E) blotter
Revised: inkwell (D) desk drawer (E) blotter

Original: A:
Revised: A:

Original: The answer must be an item that can absorb ink.
Revised: The answer must be an item that can absorb ink.

Original: Of the above choices, only blotters are usedto absorb ink.
Revised: Of the above choices only blotters are used

Revised: to absorb

Original:  The answer is E. Q: Where do you put your grapes just before checking out?
Revised: ink. The answer is E. Q: Where do you put your grapes just before checking out?

Original: Answer Choices: (A) mouth (B) grocery cart (C)super market (D) fruit basket (E) fruit market
Revised: Answer Choices: (A) mouth (B) grocery cart (C)super market (D) fruit basket (E) fruit market

Original: A:
Revised: A:

Original: The answer should be the place where grocery items are placed before checking out.
Revised: The answer should be the place where grocery items are placed before checking out.

Original: Of the above choices, grocery cart makes the most sense for holding grocery items.
Revised: Of the above choices , grocery cart makes the most sense for holding grocery items.

Original: The answer is B. Q: What home entertainment equipment requires cable?
Revised: The answer is B. Q: What home entertainment equipment requires cable?

Original: Answer Choices: (A) radio shack (B) substation (C) television (D) cabinet A:
Revised: Answer Choices: (A) radio shack (B) substation (C) television (D) cabinet A:

Original: The answer must require cable.
Revised: The answer must require cable.

Original: Of the above choices, only television requires cable.
Revised: Of the above choices, only television requires cable.

Original: The answer is C.
Revised: The answer is C.

Original: [Table]
Revised: [Table]

Original: Q : Paige and her friends were recycling paper for their class.
Revised: Q: Paige and her friends were recycling paper for their class.

Original: For every 4 pounds they recycled they earned one point.
Revised: For every 4 pounds they recycled they earned one point.

Original: If Paige recycled 14 pounds and her friends recycled pounds how many points did they earn
Revised: If Paige recycled 14 pounds and her friends recycledpounds, how many points did they earn?

Original: A: Paige and her friends would have earned 3 points for recycling paper.
Revised: A: Paige and her friends would have earned 3 points for recycling paper.

Original: ✗ (GT : 4)
Revised: ✗ (GT : 4)

Original: [Table]
Revised: [Table]

Original: Q: Jason had 20 lollipops.
Revised: Q [Table] Jason had 20 lollipops.

Original: He gave Denny some lollipops.
Revised: He gave Denny some lollipops.

Original: Now Jason has 12 lollipops.
Revised: Now Jason has 12 lollipops.

Original: How many lollipops did Jason give to Denny?
Revised: How many lollipops did Jason give to Denny?

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: Jason started with 20 lollipops.
Revised: Jason started with 20 lollipops.

Original: Then he had 12 after giving some to Denny.
Revised: Then he had 12 after giving some to Denny.

Original: So he gave Denny 20 - 12 = 8.
Revised: So he gave Denny 20 - 12 = 8.

Original: The answer is 8.
Revised: The answer is 8.

Original: Q: Michael had 58 golf balls.
Revised: Q: Michael had 58 golf balls.

Original: On tuesday, he lost 23 golf balls.
Revised: On tuesday, he lost 23 golf balls.

Original: On wednesday, he lost 2 more.
Revised: On wednesday, he lost 2 more.

Original: How many golf balls did he have at the end of wednesday?
Revised: How many golf balls did he have at the end of wednesday?

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: Michael started with 58 golf balls.
Revised: Michael started with 58 golf balls.

Original: After losing 23 on tuesday, he had 58 - 23 = 35.
Revised: After losing 23 on tuesday, he had 58 - 23 = 35.

Original: After losing 2 more, he had 35 - 2 = 33 golf balls.
Revised: After losing 2 more, he had 35 - 2 = 33 golf balls.

Original: The answer is 33.
Revised: The answer is 33.

Original: Q: If there are 3 cars in the parking lot and 2 more cars arrive, how many cars are in the parking lot?
Revised: Q: If there are 3 cars in the parking lot and 2 more cars arrive, how many cars are in the parking lot?

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: There are originally 3 cars.
Revised: There are originally 3 cars.

Original: 2 more cars arrive.
Revised: 2 more cars arrive.

Original: 3 + 2 = 5.
Revised: 3 + 2 = 5.

Original: The answer is 5.
Revised: The answer is 5.

Original: Q: There were nine computers in the server room.
Revised: Q: There were nine computers in the server room.

Original: Five more computers were installed each day, from monday to thursday.
Revised: Five more computers were installed each day, from monday to thursday.

Original: How many computers are now in the server room?
Revised: How many computers are now in the server room?

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: There were originally 9 computers.
Revised: There were originally 9 computers.

Original: For each of 4 days,more computers were added.
Revised: For each of 4 days,more computers were added.

Original: So 5 * 4 = 20 computers were added.
Revised: So 5 * 4 = 20 computers were added.

Original: 9 + 20 is 29.
Revised: 9 + 20 is 29.

Original: The answer is 29.
Revised: The answer is 29.

Original: Q: Olivia has 23 .Sheboughtfivebagelsfor 3 each.
Revised: Q: Olivia has 23 .Sheboughtfivebagelsfor 3 each.

Original: How much money does she have left?
Revised: How much money does she have left?

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: Olivia had 23 dollars.
Revised: Olivia had 23 dollars.

Original: 5 bagels for 3 dollars each will bex 3 = 15 dollars.
Revised: 5 bagels for 3 dollars each will bex 3 = 15 dollars.

Original: So she has 23 - 15 dollars left.
Revised: So she has 23 - 15 dollars left.

Original: 23 - 15 is 8.
Revised: 23 - 15 is 8.

Original: The answer is 8.
Revised: The answer is 8.

Original: Q: There are 15 trees in the grove.
Revised: Q: There are 15 trees in the grove.

Original: Grove workers will plant trees in the grove today.
Revised: Grove workers will plant trees in the grove today.

Original: After they are done, there will be 21 trees.
Revised: After they are done, there will be 21 trees.

Original: How many trees did the grove workers plant today?
Revised: How many trees did the grove workers plant today

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: There are 15 trees originally.
Revised: There are 15 trees originally.

Original: Then there were 21 trees after some more were planted.
Revised: Then there were 21 trees after some more were planted.

Original: So there must have been 21 - 15 = 6.
Revised: So there must have been 21 - 15 = 6.

Original: The answer is 6.
Revised: The answer is 6.

Original: Q: Shawn has five toys.
Revised: Q: Shawn has five toys.

Original: For Christmas, he got two toys each from his mom and dad.
Revised: For Christmas, he got two toys each from his mom and dad.

Original: How many toys does he have now?
Revised: How many toys does he have now?

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: Shawn started with 5 toys.
Revised: Shawn started with 5 toys.

Original: If he got 2 toys each from his mom and dad, then that is 4 more toys.
Revised: If he got 2 toys each from his mom and dad, then that is 4 more toys.

Original: 5 + 4 = 9.
Revised: 5 + 4 = 9.

Original: The answer is 9.
Revised: The answer is 9.

Original: Q: Leah had 32 chocolates and her sister had 42.
Revised: Q: Leah had 32 chocolates and her sister had 42.

Original: If they ate 35, how many pieces do they have left in total?
Revised: If they ate 35, how many pieces do they have left in total?

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: Originally, Leah had 32 chocolates.
Revised: Originally, Leah had 32 chocolates.

Original: Her sister had 42.
Revised: Her sister had 42.

Original: So in total they had 32 + 42 = 74.
Revised: So in total they had 32 + 42 = 74.

Original: After eating 35, they had 74 - 35 = 39.
Revised: After eating 35, they had 74 - 35 = 39.

Original: The answer is 39.
Revised: The answer is 39.

Original: ***** End In-Context Examples ***** Q: For Halloween Megan received 11 pieces of candy from neighbors and 5 pieces from her older sister.
Revised: ***** End In [Table] Examples ***** Q: For Halloween Megan received 11 pieces of candy from neighbors and 5 pieces from her older sister.

Original: If she only ate 8 pieces a day, how long would the candy last her?
Revised: If she only ate 8 pieces a day, how long would the candy last her?

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: Megan received 11 pieces of candy from neighbors andpieces from her older sister.
Revised: Megan received 11 pieces of candy from neighbors andpieces from her older sister.

Original: So in total she had 11 + 5 = 16 pieces of candy.
Revised: So in total she had 11 + 5 = 16 pieces of candy.

Original: If she ate 8 pieces a day, it would last her 16 / 8 = 2 days.
Revised: If she ate 8 pieces a day, it would last her 16 / 8 = 2 days.

Original: The answer is 2.
Revised: The answer is 2.

Original: ✓
Revised: ✓

Original: [Table]

Original: [Table]

Original: [Table]

Original: G. Sample Study
Revised: G. Sample Study

Original: To validate the correctness of chain of thought, we analyze texts generated by Zero-shot-CoT for CommonsenseQA and
Revised: To validate the correctness of chain of thought, we analyze texts generated by Zero-shot-CoT for CommonsenseQA and

Original: MultiArith datasets.
Revised: MultiArith datasets.

Original: G.1. CommonsenseQA
Revised: G.1. CommonsenseQA

Original: [Table]
Revised: [Table]

Original: Table 21 summarizes the categorization results of texts generated by Zero-shot-CoT for CommonsenseQA.
Revised: Table 21 summarizes the categorization results of texts generated by Zero-shot-CoT for CommonsenseQA.

Original: We randomly picked up 50 samples whose prediction results were correct and 50 samples whose prediction results were incorrect.
Revised: We randomly picked up 50 samples whose prediction results were correct and 50 samples whose prediction results were incorrect.

Original: We categorized those samples by CoT types.
Revised: We categorized those samples by CoT types.

Original: Some picked-up samples from each category are found Table 22.
Revised: Some picked-up samples from each category are found Table 22.

Original: First, it is found that the correct samples contain a certain amount of incorrect chain of thought.
Revised: First, it is found that the correct samples contain a certain amount of incorrect chain of thought.

Original: The main tendency is that
Revised: The main tendency is that

Original: Zero-shot-CoT cannot narrow down the prediction to one from multiple answer choices, and produce multiple predictions as answers but fortunately the first output answer was correct.
Revised: Zero-shot-CoT cannot narrow down the prediction to one from multiple answer choices, and produce multiple predictions as answers but fortunately the first output answer was correct.

Original: See “Correct - CoT is INCORRECT ” rows in Table
Revised: See “Correct - CoT is INCORRECT ” rows in Table

Original: Second, as for incorrect samples, commonsense mistake is the most frequent error type.
Revised: Second, as for incorrect samples, commonsense mistake is the most frequent error type.

Original: By observing the produced chain of thought texts, it is found that Zero-shot-CoT often produces a flexible and reasonable chain of thought (logically correct but lacks common sense) even when the final prediction is not correct.
Revised: By observing the produced chain of thought texts, it is found that Zero-shot-CoT often produces a flexible and reasonable chain of thought (logically correct but lacks common sense) even when the final prediction is not correct.

Original: See “CommonSense Mistake” rows in Tableshot-CoT for CommonsenseQA. Category
Revised: See “CommonSense Mistake” rows in Table

Original: [Table]
Revised: [Table]

Original: Correct - CoT is correct Q: If there is a pond with trees around it, where it it likely located?
Revised: correct Q: If there is a pond with trees around it, where it it likely located?

Original: Answer Choices: (A) ground (B) bathroom (C) forest (D) countryside (E) rural area
Revised: Answer Choices: (A) ground (B) bathroom (C) forest (D) countryside (E) rural area

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: A pond is likely to be located near trees because trees need water to survive.
Revised: A pond is likely to be located near trees because trees need water to survive.

Original: Therefore, the most likely location for a pond with trees around it is in a forest.
Revised: Therefore, the most likely location for a pond with trees around it is in a forest.

Original: Therefore, among A through E, the answer is C. ✓ Correct - CoT is correct
Revised: Therefore, among A through E, the answer is C. ✓ Correct - CoT is correct

Original: Q: Human beings learn about current events from what print item?
Revised: Q: Human beings learn about current events from what print item?

Original: Answer Choices: (A) question authority (B) melt (C) read newspapers (D) act (E) dictionary
Revised: Answer Choices: (A) question authority (B) melt (C) read newspapers (D) act (E) dictionary

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: How do human beings learn?
Revised: How do human beings learn?

Original: They learn by observing and listening to others, and they also learn by reading.
Revised: They learn by observing and listening to others, and they also learn by reading.

Original: So it’s most likely that human beings learn about current events from reading newspapers.
Revised: So it’s most likely that human beings learn about current events from reading newspapers.

Original: Therefore, among A through E, the answer is C. ✓
Revised: Therefore, among A through E, the answer is C. ✓

Original: Continued on next page
Revised: Continued on next page

Original: [Table]
Revised: [Table]

Original:  artist was sitting quietly pondering, then suddenly he began to paint when what struck him [Table] in [Table]
Revised: : The artist was sitting quietly pondering, then suddenly he began to paint when what struck him?

Original: spiration

Original: [Table] D [Table] E
Revised: Answer Choices:

Original: [Table]

Original: in

Original: [Table]
Revised: [Table]

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: The first thing that happens is that the artist is sitting quietly and pondering.
Revised: The first thing that happens is that the artist is sitting quietly and pondering.

Original: This means he’s thinking deeply about something, probably trying to come up with an idea.
Revised: This means he’s thinking deeply about something, probably trying to come up with an idea.

Original: Then, suddenly, he begins to paint.
Revised: Then, suddenly, he begins to paint.

Original: This means he was inspired by something he thought of.
Revised: This means he was inspired by something he thought of.

Original: The most likely explanation is that he had an insight, or a sudden realization, that led him to start painting.
Revised: The most likely explanation is that he had an insight, or a sudden realization, that led him to start painting.

Original: Therefore, among A through E, the answer is most likely (C), inspiration.
Revised: Therefore, among A through E, the answer is most likely (C), inspiration.

Original: ✓
Revised: ✓

Original: [Table]
Revised: [Table]

Original: INCORRECT
Revised: INCORRECT

Original: Q: What is likely to satisfy [Table]
Revised: Q: What is likely to satisfy someone’s curiosity?

Original: Answer
Revised: Answer

Original: [Table]

Original:  hear [Table] read book (C) see favorite show [Table]
Revised: Choices: (A) hear news (B) read book (C) see favorite show (D) comedy show (E) go somewhere

Original: D

Original: [Table]

Original: comedy show

Original: [Table]

Original: E) go somewhere

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: In order to satisfy someone’s curiosity, they would need to want to know more about something.
Revised: In order to satisfy someone’s curiosity, they would need to want to know more about something.

Original: So, (A) hearing news and (B) reading a book are both good answers because they provide information that the person may be curious about.
Revised: So, (A) hearing news and (B) reading a book are both good answers because they provide information that the person may be curious about.

Original: (C)
Revised: (C)

Original: Seeing a favorite show could also work because it might be something the person is curious about and wants to know more about.
Revised: Seeing a favorite show could also work because it might be something the person is curious about and wants to know more about.

Original: (D)
Revised: (D)

Original: A comedy show could potentially satisfy someone’s curiosity if the person is curious about the comedian’s life or what they are like in person.
Revised: A comedy show could potentially satisfy someone’s curiosity if the person is curious about the comedian’s life or what they are like in person.

Original: (E) Going somewhere could also work if the person is curious about a certain place and wants to know more Therefore, among A through E, the answer is A, B, C, D, or E. ✓ (Model cannot choose only one.)
Revised: (E) Going somewhere could also work if the person is curious about a certain place and wants to know more Therefore, among A through E, the answer is A, B, C, D, or E. ✓ (Model cannot choose only one.)

Original: (GT : (A), Zero-shot: (E))
Revised: (GT : (A), Zero-shot: (E))

Original: Correct - CoT is
Revised: [Table]

Original: INCORRECT
Revised: INCORRECT

Original: Q: What would be necessary for getting in shape?
Revised: Q: What would be necessary for getting in shape?

Original: Answer
Revised: Answer

Original: Choices: (A) good health (B) exercise (C) muscle tone (D) sweat (E) feel better
Revised: Choices: (A) good health (B) exercise (C) muscle tone (D) sweat (E) feel better

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: In order to get in shape, you need to be able to do three things: 1.
Revised: In order to get in shape, you need to be able to do three things: 1.

Original: Exercise regularly 2.
Revised: Exercise regularly 2.

Original: Eat a healthy diet 3.
Revised: Eat a healthy diet 3.

Original: Get enough sleep If you can do all three of those things, then you will be well on your way to getting in shape!
Revised: Get enough sleep If you can do all three of those things, then you will be well on your way to getting in shape!

Original: Therefore, among A through E, the answer is B, C, and
Revised: Therefore, among A through E, the answer is B, C, and

Original: D. ✓ (Model cannot choose only one.)
Revised: D. ✓ (Model cannot choose only one.)

Original: (GT : (B), Zero-shot: (B))
Revised: (GT : (B), Zero-shot: (B))

Original: Correct - CoT is
Revised: Correct - CoT is

Original: INCORRECT
Revised: INCORRECT

Original: Q: The man had a fear of illness, so he never visited friends who were a what?
Revised: Q: The man had a fear of illness, so he never visited friends who were a what?

Original: Answer
Revised: Answer

Original: Choices: (A) sick person (B) hospital (C) elderly person (D) graveyard (E) doctor’s office
Revised: Choices: (A) sick person (B) hospital (C) elderly person (D) graveyard (E) doctor’s office

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: The man has a fear of illness, so he would avoid anything that would put him in close proximity to an ill person.
Revised: The man has a fear of illness, so he would avoid anything that would put him in close proximity to an ill person.

Original: This would include a hospital, where sick people are treated, and a doctor’s office, where sick people go to receive care.
Revised: This would include a hospital, where sick people are treated, and a doctor’s office, where sick people go to receive care.

Original: An elderly person is more likely to be ill than a young person, so the man would avoid them as well.
Revised: An elderly person is more likely to be ill than a young person, so the man would avoid them as well.

Original: A graveyard is where people who have died from illness are buried, so the man would avoid that as well.
Revised: A graveyard is where people who have died from illness are buried, so the man would avoid that as well.

Original: The answer is A, sick person.
Revised: The answer is A, sick person.

Original: Therefore, among A through E, the answer is A. ✓ (Model cannot narrow down choice to one in CoT.)
Revised: Therefore, among A through E, the answer is A. ✓ (Model cannot narrow down choice to one in CoT.)

Original: (GT : (A), Zero-shot: (A))
Revised: (GT : (A), Zero-shot: (A))

Original: CommonSenseMistake
Revised: CommonSenseMistake

Revised: Q: Where is

Revised: a

Original: Q: Where is a well used toy car likely to be found?
Revised: well used toy car likely to be found

Original: Answer
Revised: Answer

Original: Choices: ( A) child’s room (B) boy’s bedroom (C) own home
Revised: Choices: (A) child’s room (B) boy’s bedroom (C) own home

Original: (D) toy store (E) house
Revised: (D) toy store (E) house

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: A toy car is likely to be found in a child’s room.
Revised: A toy car is likely to be found in a child’s room.

Original: A child’s room is likely to be found in a house.
Revised: A child’s room is likely to be found in a house.

Original: Therefore, a toy car is likely to be found in a house.
Revised: Therefore, a toy car is likely to be found in a house.

Original: Therefore, among A through E, the answer is E. ✗ (GT : (A), Zero-shot: (A))
Revised: Therefore, among A through E, the answer is E. ✗ (GT : (A), Zero-shot: (A))

Original: Continued on next page
Revised: Continued on next page

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: A

Original: [Table]
Revised: [Table]

Original:  Let’s think step by step.
Revised: A: Let’s think step by step.

Original: If you don’t have a piano, you can’t use your own.
Revised: If you don’t have a piano, you can’t use your own.

Original: You could go to a music school, but that seems like it would be for a lesson rather than just to use a piano.
Revised: You could go to a music school, but that seems like it would be for a lesson rather than just to use a piano.

Original: A music store might have a piano that you could use, but you would probably have to buy something.
Revised: A music store might have a piano that you could use, but you would probably have to buy something.

Original: You could go to a neighbor’s house, but that seems like a lot to ask.
Revised: You could go to a neighbor’s house, but that seems like a lot to ask.

Original: You could go to a lunch, but that doesn’t make sense.
Revised: You could go to a lunch, but that doesn’t make sense.

Original: You could go to a drawing room, but that also doesn’t make sense.
Revised: You could go to a drawing room, but that also doesn’t make sense.

Original: The answer must be (B) music store.
Revised: The answer must be (B) music store.

Original: Therefore, among A through
Revised: Therefore, among A through

Original: E, 
Revised: E, the answer is B. ✗ (GT : (C), Zero-shot: (C)) [Table] : David watched some nesting birds using his binoculars while on vacation. Where might David be?.

Revised: [Table]

Revised: Choices:

Original: [Table]
Revised: [Table]

Revised: vaccation

Original: [Table]
Revised: [Table]

Original:  Let’s think step by step.
Revised: A: Let’s think step by step.

Original: First, we need to understand what the word ”binoculars” means.
Revised: First, we need to understand what the word ”binoculars” means.

Original: Binoculars are two telescopes that are joined together and allow a person to see things that are far away.
Revised: Binoculars are two telescopes that are joined together and allow a person to see things that are far away.

Original: Now, we need to think about where a person might use binoculars.
Revised: Now, we need to think about where a person might use binoculars.

Original: A person might use binoculars to see things that are far away and in the sky, or to see things that are far away and on the ground.
Revised: A person might use binoculars to see things that are far away and in the sky, or to see things that are far away and on the ground.

Original: So, the answer could be either (A) sky or (D) countryside.
Revised: So, the answer could be either (A) sky or (D) countryside.

Original: Therefore, among A through E, the answer is (A) or (D).
Revised: Therefore, among A through E, the answer is (A) or (D).

Original: ✗ (GT : (C), Zero-shot: (D))
Revised: ✗ (GT : (C), Zero-shot: (D))

Original: CommonSense
Revised: [Table]

Original: Mistake Q: When you get up in the morning before you begin work you should do what?
Revised: Q: When you get up in the morning before you begin work you should do what?

Original: Answer Choices: (A) apply for job (B) sleep (C) concentrate (D) shower (E) just do
Revised: Answer Choices: (A) apply for job (B) sleep (C) concentrate (D) shower (E) just do

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: The first thing you should do when you wake up is probably to stretch and yawn to get your body moving.
Revised: The first thing you should do when you wake up is probably to stretch and yawn to get your body moving.

Original: Then, you should probably brush your teeth and wash your face to get ready for the day.
Revised: Then, you should probably brush your teeth and wash your face to get ready for the day.

Original: After that, you might want to eat breakfast to give yourself some energy for the day.
Revised: After that, you might want to eat breakfast to give yourself some energy for the day.

Original: Once you’re all set, you can start your work for the day.
Revised: Once you’re all set, you can start your work for the day.

Original: Therefore, among A through E, the answer is C. ✗ (GT : (D), Zero-shot: (C))
Revised: Therefore, among A through E, the answer is C. ✗ (GT : (D), Zero-shot: (C))

Original: CommonSense
Revised: CommonSense

Original: Mistake
Revised: Mistake

Original: Q: What is someone doing if he or she is sitting quietly and his or her eyes are moving?
Revised: Q: What is someone doing if he or she is sitting quietly and his or her eyes are moving?

Original: Answer Choices: (A) reading (B) meditate (C) fall asleep (D) bunk (E) think
Revised: Answer Choices: (A) reading (B) meditate (C) fall asleep (D) bunk (E) think

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: If someone is sitting quietly, that means they are not moving around.
Revised: If someone is sitting quietly, that means they are not moving around.

Original: And if their eyes are moving, they are probably not asleep.
Revised: And if their eyes are moving, they are probably not asleep.

Original: So it’s probably not (C) or (D).
Revised: So it’s probably not (C) or (D).

Original: If they are reading, they would be looking at the book, not moving their eyes around, so it’s probably not (A), which leaves us with (B) and (E).
Revised: If they are reading, they would be looking at the book, not moving their eyes around, so it’s probably not (A), which leaves us with (B) and (E).

Original: It’s more likely that they are thinking, so (E) is the best answer.
Revised: It’s more likely that they are thinking, so (E) is the best answer.

Original: Therefore, among A through E, the answer is E. ✗ (GT : (A), Zero-shot: (E))
Revised: Therefore, among A through E, the answer is E. ✗ (GT : (A), Zero-shot: (E))

Original: LogicalMistake
Revised: LogicalMistake

Original: Q: If you really wanted a grape, where would you go to get it?
Revised: Q: If you really wanted a grape, where would you go to get it?

Original: Answer Choices: (A) winery (B) fruit stand (C) field (D) [Table]
Revised: Answer Choices: (A) winery (B) fruit stand (C field (D) kitchen (E) food

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: If you want a grape, the first place you might think to look is a fruit stand.
Revised: If you want a grape, the first place you might think to look is a fruit stand.

Original: If there are no grapes at the fruit stand, your next best bet would be a grocery store.
Revised: If there are no grapes at the fruit stand, your next best bet would be a grocery store.

Original: If there are no grapes at the grocery store, you might try a vineyard or winery.
Revised: If there are no grapes at the grocery store, you might try a vineyard or winery.

Original: If there are no grapes at the vineyard or winery, your last resort would be to grow your own grapes.
Revised: If there are no grapes at the vineyard or winery, your last resort would be to grow your own grapes.

Original: Therefore, among A through E, the answer is E. ✗ (GT : (B),
Revised: Therefore, among A through E, the answer is E. ✗ (GT : (B),

Original: Zero-shot: (B)) [Table] [Table]
Revised: Zero-

Original: [Table]
Revised: [Table]

Original: G.2. MultiArith
Revised: G.2. MultiArith

Original: Table 23 summarizes the categorization results of texts generated by Zero-shot-CoT and Few-shot-CoT for MultiArith.
Revised: Table 23 summarizes the categorization results of texts generated by Zero-shot-CoT and Few-shot-CoT for MultiArith.

Original: We compared Zero-shot-CoT and Few-shot-CoT to contrast the difference of chain of thought produced by these two methods.
Revised: We compared Zero-shot-CoT and Few-shot-CoT to contrast the difference of chain of thought produced by these two methods.

Original: Specifically, we randomly picked up correct 50 samples and incorrect 50 samples produced by each method and categorized them by types.
Revised: Specifically, we randomly picked up correct 50 samples and incorrect 50 samples produced by each method and categorized them by types.

Original: As an exception, the maximum number of incorrect samples from Few-shot-CoT for MultiArith was 42.
Revised: As an exception, the maximum number of incorrect samples from Few-shot-CoT for MultiArith was 42.

Original: As for correct samples, we examined if the produced chain of thought is logical and consistent with the correct prediction.
Revised: As for correct samples, we examined if the produced chain of thought is logical and consistent with the correct prediction.

Original: The result shows that almost all the chain of thought is correct, with slightly more reasoning mistakes found in Zero-shot-CoT than Few-shot-CoT.
Revised: The result shows that almost all the chain of thought is correct, with slightly more reasoning mistakes found in Zero-shot-CoT than Few-shot-CoT.

Original: As for incorrect samples, it is found that Zero-shot-CoT tends to output unnecessary steps of reasoning after getting the correct prediction, which results in changing the prediction to incorrect one.
Revised: As for incorrect samples, it is found that Zero-shot-CoT tends to output unnecessary steps of reasoning after getting the correct prediction, which results in changing the prediction to incorrect one.

Original: Zero-shot-CoT also sometimes does not start reasoning, just rephrasing the input question.
Revised: Zero-shot-CoT also sometimes does not start reasoning, just rephrasing the input question.

Original: In contrast, Few-shot-CoT tends to fail when generated chain of thought include ternary operation, e.g. (3 + 2) ∗ 4 .
Revised: In contrast, Few-shot-CoT tends to fail when generated chain of thought include ternary operation, e.g. (3 + 2) ∗ 4 .

Original: Another finding is that Zero-shot-CoT and Few-shot-CoT have a certain amount of common sense mistakes to interpret a question.
Revised: Another finding is that Zero-shot-CoT and Few-shot-CoT have a certain amount of common sense mistakes to interpret a question.

Original: Some examples are found at Table 24.
Revised: Some examples are found at Table 24.

Revised: Question

Original: [Table]
Revised: [Table]

Original: Q:
Revised: Q:

Original: At the fair Adam boughttickets.
Revised: At the fair Adam boughttickets.

Original: After riding the ferris wheel he had 4 tickets left.
Revised: After riding the ferris wheel he had 4 tickets left.

Original: If each ticket cost 9 dollars, how much money did Adam spend riding the ferris wheel?
Revised: If each ticket cost 9 dollars, how much money did Adam spend riding the ferris wheel?

Original: A (ONE UNNECESSARY STEP ERROR) :
Revised: A (ONE UNNECESSARY STEP ERROR) :

Original: Let’s think step by step.
Revised: Let’s think step by step.

Original: Adam bought 13 tickets.
Revised: Adam bought 13 tickets.

Original: That means he spent 13 * 9 = 117 dollars on tickets.
Revised: That means he spent 13 * 9 = 117 dollars on tickets.

Original: After riding the ferris wheel, he had 4 tickets left.
Revised: After riding the ferris wheel, he had 4 tickets left.

Original: That means he used 9 tickets to ride the ferris wheel.
Revised: That means he used 9 tickets to ride the ferris wheel.

Original: That means he spent 9 * 9 = 81 dollars on riding the ferris wheel.
Revised: That means he spent 9 * 9 = 81 dollars on riding the ferris wheel.

Original: So the answer is 117 - 81 = 36 dollars.
Revised: So the answer is 117 - 81 = 36 dollars.

Original: Therefore, the answer (arabic numerals) is 36.
Revised: Therefore, the answer (arabic numerals) is 36.

Original: ✗
Revised: ✗

Original: A: Adam started with 13 tickets.
Revised: A: Adam started with 13 tickets.

Original: He had 4 left after riding the ferris wheel.
Revised: He had 4 left after riding the ferris wheel.

Original: So he must have spent 13 - 4 = 9 tickets.
Revised: So he must have spent 13 - 4 = 9 tickets.

Original: 9 tickets times 9 dollars per ticket is 9 * 9 = 81.
Revised: 9 tickets times 9 dollars per ticket is 9 * 9 = 81.

Original: The answer is 81.
Revised: The answer is 81.

Original: ✓
Revised: ✓

Original: Continued on next page
Revised: Continued on next page

Original: Question
Revised: Question

Original: Zero-shot-CoT
Revised: Zero-shot-CoT

Original: Few-shot-CoT
Revised: Few-shot-CoT

Original: Q: At a restaurant each adult meal costs $5 and kids eat free.
Revised: Q: At a restaurant each adult meal costs $5 and kids eat free.

Original: If a group of 15 people came in and 8 were kids, how much would it cost for the group to eat?
Revised: If a group of 15 people came in and 8 were kids, how much would it cost for the group to eat?

Original: A (ONE UNNECESSARY STEP ERROR) :
Revised: A (ONE UNNECESSARY STEP ERROR) :

Original: Let’s think step by step.
Revised: Let’s think step by step.

Original: There are 8 kids, so that means there are 7 adults.
Revised: There are 8 kids, so that means there are 7 adults.

Original: Each adult meal costs $5, so that means that the 7 adults will cost $5*
Revised: Each adult meal costs $5, so that means that the 7 adults will cost $5*

Original: 7=$35 Now we just need to add the two numbers together.
Revised: 7=$35 Now we just need to add the two numbers together.

Original: $35+8=43$
Revised: $35+8=43$

Original: So it would cost $ for the group to eat.
Revised: So it would cost $ for the group to eat.

Original: Therefore, the answer (arabic numerals) is 43.
Revised: Therefore, the answer (arabic numerals) is 43.

Original: ✗
Revised: ✗

Original: A: If 8 were kids, then that means there were 15 - 8 =adults.
Revised: A: If 8 were kids, then that means there were 15 - 8 =adults.

Original: Each adult meal costs $5.
Revised: Each adult meal costs $5.

Original: So the total cost would be 7 * 5 = 35.
Revised: So the total cost would be 7 * 5 = 35.

Original: The answer is 35.
Revised: The answer is 35.

Original: ✓
Revised: ✓

Original: Q: April’s discount flowers was having a sale where each flower was 6 dollars.
Revised: Q: April’s discount flowers was having a sale where each flower was 6 dollars.

Original: If Katie bought 5 roses and 5 daisies, how much did she spend?
Revised: If Katie bought 5 roses and 5 daisies, how much did she spend?

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: First, we need to calculate how much each type of flower costs.
Revised: First, we need to calculate how much each type of flower costs.

Original: There are 5 roses, and each rose costs 6 dollars.
Revised: There are 5 roses, and each rose costs 6 dollars.

Original: So, the cost of the roses is 5*6=30 dollars.
Revised: So, the cost of the roses is 5*6=30 dollars.

Original: There are 5 daisies, and each daisy costs 6 dollars.
Revised: There are 5 daisies, and each daisy costs 6 dollars.

Original: So, the cost of the daisies is 5*6=30 dollars.
Revised: So, the cost of the daisies is 5*6=30 dollars.

Original: Then, we need to calculate the total cost.
Revised: Then, we need to calculate the total cost.

Original: The total cost is the cost of the roses plus the cost of the daisies.
Revised: The total cost is the cost of the roses plus the cost of the daisies.

Original: So, the total cost is 30+30=60 dollars.
Revised: So, the total cost is 30+30=60 dollars.

Original: Therefore, the answer (arabic numerals) is 60.
Revised: Therefore, the answer (arabic numerals) is 60.

Original: ✓
Revised: ✓

Original: A (CALCULATOR
Revised: A (CALCULATOR

Original: ERROR) :
Revised: ERROR) :

Original: Each flower wasdollars.
Revised: Each flower wasdollars.

Original: Katie bought 5 roses and 5 daisies.
Revised: Katie bought 5 roses and 5 daisies.

Original: So she spent 5 * 6 + 5 * 6 = 60 + 30 = 90.
Revised: So she spent 5 * 6 + 5 * 6 = 60 + 30 = 90.

Original: The answer is 90.
Revised: The answer is 90.

Original: ✗
Revised: ✗

Original: Q: While shopping for music online, Faye bought 2 country albums and 3 pop albums.
Revised: Q: While shopping for music online, Faye bought 2 country albums and 3 pop albums.

Original: Each album came with a lyric sheet and had 6 songs.
Revised: Each album came with a lyric sheet and had 6 songs.

Original: How many songs did Faye buy total?
Revised: How many songs did Faye buy total?

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: Faye bought 2 country albums.
Revised: Faye bought 2 country albums.

Original: Each album has 6 songs.
Revised: Each album has 6 songs.

Original: So she bought 2 * 6 = 12 songs from the country albums.
Revised: So she bought 2 * 6 = 12 songs from the country albums.

Original: Faye bought 3 pop albums.
Revised: Faye bought 3 pop albums.

Original: Each album has 6 songs.
Revised: Each album has 6 songs.

Original: So she bought 3 * 6 = 18 songs from the pop albums.
Revised: So she bought 3 * 6 = 18 songs from the pop albums.

Original: In total, Faye bought 12 + 18 = 30 songs.
Revised: In total, Faye bought 12 + 18 = 30 songs.

Original: Therefore, the answer (arabic numerals) is 30.
Revised: Therefore, the answer (arabic numerals) is 30.

Original: ✓
Revised: ✓

Original: A (CALCULATOR
Revised: A (CALCULATOR

Original: ERROR) :
Revised: ERROR) :

Original: Faye bought 2 country albums and 3 pop albums.
Revised: Faye bought 2 country albums and 3 pop albums.

Original: Each album had 6 songs.
Revised: Each album had 6 songs.

Original: So she bought 2 * 6 + 3 * 6 =songs.
Revised: So she bought 2 * 6 + 3 * 6 =songs.

Original: The answer is 24.
Revised: The answer is 24.

Original: ✗
Revised: ✗

Original: Q: Jerry was helping the cafeteria workers pick up lunch trays, but he could only carry 8 trays at a time.
Revised: Q: Jerry was helping the cafeteria workers pick up lunch trays, but he could only carry 8 trays at a time.

Original: If he had to pick uptrays from one table and 7 trays from another, how many trips will he make?
Revised: If he had to pick uptrays from one table and 7 trays from another, how many trips will he make?

Original: A (COMMONSENSE MISTAKE) : Let’s think step by step.
Revised: A (COMMONSENSE MISTAKE) : Let’s think step by step.

Original: Jerry can carry 8 trays at a time.
Revised: Jerry can carry 8 trays at a time.

Original: There are 9 trays on the first table.
Revised: There are 9 trays on the first table.

Original: That means
Revised: That means

Original: Jerry will need to make 2 trips to the first table.
Revised: Jerry will need to make 2 trips to the first table.

Original: There are 7 trays on the second table.
Revised: There are 7 trays on the second table.

Original: That means
Revised: That means

Original: Jerry will need to make 1 trip to the second table.
Revised: Jerry will need to make 1 trip to the second table.

Original: In total, Jerry will make 3 trips.
Revised: In total, Jerry will make 3 trips.

Original: Therefore, the answer (arabic numerals) is 3.
Revised: Therefore, the answer (arabic numerals) is 3.

Original: ✗
Revised: ✗

Original: A: Jerry can carry 8 trays at a time.
Revised: A: Jerry can carry 8 trays at a time.

Original: If he has to pick uptrays from one table and 7 trays from another, he will have to make 2 trips.
Revised: If he has to pick uptrays from one table and 7 trays from another, he will have to make 2 trips.

Original: The answer is 2.
Revised: The answer is 2.

Original: ✓
Revised: ✓

Original: Q: Kaleb bought 14 boxes of chocolate candy and gaveto his little brother.
Revised: Q: Kaleb bought 14 boxes of chocolate candy and gaveto his little brother.

Original: If each box has 6 pieces inside it, how many pieces did Kaleb still have?
Revised: If each box has 6 pieces inside it, how many pieces did Kaleb still have?

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: Kaleb bought 14 boxes of chocolate candy.
Revised: Kaleb bought 14 boxes of chocolate candy.

Original: This means that he hadpieces of chocolate candy.
Revised: This means that he hadpieces of chocolate candy.

Original: He gave 5 boxes to his little brother, which means that he gave his little brother 30 pieces of chocolate candy.
Revised: He gave 5 boxes to his little brother, which means that he gave his little brother 30 pieces of chocolate candy.

Original: Kaleb still had 54 pieces of chocolate candy.
Revised: Kaleb still had 54 pieces of chocolate candy.

Original: Therefore, the answer (arabic numerals) is 54.
Revised: Therefore, the answer (arabic numerals) is 54.

Original: ✓
Revised: ✓

Original: A (COMMONSENSE MISTAKE) : Kaleb boughtboxes.
Revised: A (COMMONSENSE MISTAKE) : Kaleb boughtboxes.

Original: Each box has 6 pieces.
Revised: Each box has 6 pieces.

Original: So 14 * 6 = 84 pieces.
Revised: So 14 * 6 = 84 pieces.

Original: He gave 5 to his brother.
Revised: He gave 5 to his brother.

Original: So he has-
Revised: So he has-

Original: 5 = 79 pieces.
Revised: 5 = 79 pieces.

Original: The answer is 79.
Revised: The answer is 79.

Original: ✗
Revised: ✗

Original: Q:
Revised: Q:

Original: At the fair there werepeople in line for the bumper cars.
Revised: At the fair there werepeople in line for the bumper cars.

Original: If 10 of them got tired of waiting and left and 15 more got in line, how many people would be in line?
Revised: If 10 of them got tired of waiting and left and 15 more got in line, how many people would be in line?

Original: A (OTHERS;
Revised: A (OTHERS;

Original: REASONING
Revised: REASONING

Original: IS
Revised: IS

Original: NOT
Revised: NOT

Original: STARTED) : Let’s think step by step.
Revised: STARTED) : Let’s think step by step.

Original: There were 12 people in line for the bumper cars.
Revised: There were 12 people in line for the bumper cars.

Original: 10 of them got tired of waiting and left.
Revised: 10 of them got tired of waiting and left.

Original: 15 more got in line.
Revised: 15 more got in line.

Original: That means that there are now 15 people in line for the bumper cars.
Revised: That means that there are now 15 people in line for the bumper cars.

Original: Therefore, the answer (arabic numerals) is 15.
Revised: Therefore, the answer (arabic numerals) is 15.

Original: ✗
Revised: ✗

Original: A: There were originallypeople in line.
Revised: A: There were originallypeople in line.

Original: 10 of them left, so that left 12 - 10 = 2.
Revised: 10 of them left, so that left 12 - 10 = 2.

Original: Then 15 more got in line, so that is+ 15 = 17.
Revised: Then 15 more got in line, so that is+ 15 = 17.

Original: The answer is 17.
Revised: The answer is 17.

Original: ✓
Revised: ✓

Original: Continued on next page
Revised: Continued on next page

Original: Question
Revised: Question

Original: Zero
Revised: Zero

Revised: [Table]

Revised: shot

Original: [Table]
Revised: [Table]

Revised: CoT

Original: Few
Revised: Few

Original: [Table]
Revised: [Table]

Revised: shot-CoT

Original: Q: Luke made 9 dollars mowing lawns and 18 dollars weed eating.
Revised: Q: Luke made 9 dollars mowing lawns and 18 dollars weed eating.

Original: If he only spent 3 dollar a week, how long would the money last him?
Revised: If he only spent 3 dollar a week, how long would the money last him?

Original: A (OTHERS; QUESTION UNDERSTANDING ERROR) : Let’s think step by step.
Revised: A (OTHERS; QUESTION UNDERSTANDING ERROR) : Let’s think step by step.

Original: How much money does Luke make in a week? How much money does Luke spend in a week?
Revised: How much money does Luke make in a week? How much money does Luke spend in a week?

Original: How much money does Luke have left over at the end of the week?
Revised: How much money does Luke have left over at the end of the week?

Original: How long will it take Luke to save up $54?
Revised: How long will it take Luke to save up $54?

Original: Luke makes $27 a week.
Revised: Luke makes $27 a week.

Original: He spends $ a week.
Revised: He spends $ a week.

Original: He has $24 left over at the end of the week.
Revised: He has $24 left over at the end of the week.

Original: It will take Luke 2 weeks to save up $54.
Revised: It will take Luke 2 weeks to save up $54.

Original: Therefore, the answer (arabic numerals) is 2.
Revised: Therefore, the answer (arabic numerals) is 2.

Original: ✗
Revised: ✗

Original: A: Luke made 9 dollars mowing lawns and 18 dollars weed eating.
Revised: A: Luke made 9 dollars mowing lawns and 18 dollars weed eating.

Original: In total, he made 9 + 18 = 27 dollars.
Revised: In total, he made 9 + 18 = 27 dollars.

Original: If he spendsdollars a week, that is 3 dollars * 4 weeks =
Revised: If he spendsdollars a week, that is 3 dollars * 4 weeks =

Original: 12 dollars a month.
Revised: 12 dollars a month.

Original: So the money would last him 27 / 12 = 2.25 months.
Revised: So the money would last him 27 / 12 = 2.25 months.

Original: The answer is 2.25 months.
Revised: The answer is 2.25 months.

Original: ✗ (GT :weeks)
Revised: ✗ (GT :weeks)

Original: Q: Each chocolate bar in a box cost $3.
Revised: Q: Each chocolate bar in a box cost $3.

Original: If a box had 9 bars total and Wendy sold all butbars, how much money would she have made?
Revised: If a box had 9 bars total and Wendy sold all butbars, how much money would she have made?

Original: A (OTHERS; MAXIMUM LIMIT OF TEXT
Revised: A (OTHERS; MAXIMUM LIMIT OF TEXT

Original: LENGTH) :
Revised: LENGTH) :

Original: Let’s think step by step.
Revised: Let’s think step by step.

Original: Each chocolate bar in a box costs $3.
Revised: Each chocolate bar in a box costs $3.

Original: If a box had 9 bars total, then Wendy sold all but 3 bars.
Revised: If a box had 9 bars total, then Wendy sold all but 3 bars.

Original: How much money would she have made?
Revised: How much money would she have made?

Original: We can solve this problem using algebra.
Revised: We can solve this problem using algebra.

Original: Let’s start by creating a variable to represent the number of chocolate bars
Revised: Let’s start by creating a variable to represent the number of chocolate bars

Original: Wendy sold.
Revised: Wendy sold.

Original: We’ll call this variable ”x”.
Revised: We’ll call this variable ”x”.

Original: If Wendy sold ”x” chocolate bars, then she would have- x chocolate bars left in the box.
Revised: If Wendy sold ”x” chocolate bars, then she would have- x chocolate bars left in the box.

Original: We know that each chocolate bar costs $3, so the total cost of the chocolate bars Wendy sold
Revised: We know that each chocolate bar costs $3, so the total cost of the chocolate bars Wendy sold

Original: Therefore, the answer (arabic numerals) is:
Revised: Therefore, the answer (arabic numerals) is:

Original: Wendy would have made $12.
Revised: Wendy would have made $12.

Original: ✗
Revised: ✗

Original: A: Each chocolate bar costdollars.
Revised: A: Each chocolate bar costdollars.

Original: So if Wendy sold all but 3, she would have sold 93 = 6.
Revised: So if Wendy sold all but 3, she would have sold 93 = 6.

Original: 6 * 3 = 18.
Revised: 6 * 3 = 18.

Original: The answer is 18.
Revised: The answer is 18.

Original: ✓
Revised: ✓

Original: H. Discussion
Revised: H. Discussion

Original: Limitation and Social Impact
Revised: Limitation and Social Impact

Original: Our work is based on prompting methods for large language models.
Revised: Our work is based on prompting methods for large language models.

Original: LLMs have been trained on large corpora from various sources on the web, and have shown to capture and amplify biases found in the training data.
Revised: LLMs have been trained on large corpora from various sources on the web, and have shown to capture and amplify biases found in the training data.

Original: Prompting is a method that looks to take advantage of the patterns captured by language models conducive to various tasks, and therefore it has the same shortcomings.
Revised: Prompting is a method that looks to take advantage of the patterns captured by language models conducive to various tasks, and therefore it has the same shortcomings.

Original: This being said, our approach is a more direct way to probe complex reasoning inside pre-trained LLMs, removing the confounding factor of in-context learning in prior few-shot approaches, and can lead to more unbiased study of biases in LLMs.
Revised: This being said, our approach is a more direct way to probe complex reasoning inside pre-trained LLMs, removing the confounding factor of in-context learning in prior few-shot approaches, and can lead to more unbiased study of biases in LLMs.