Original: A UTOMATIC C HAIN OF T HOUGHT P ROMPTING IN L ARGE L ANGUAGE M ODELS
Revised: A UTOMATIC C HAIN OF T HOUGHT P ROMPTING IN L ARGE L ANGUAGE M ODELS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Large language models (LLMs) can perform complex reasoning  by generating intermediate reasoning steps.
Revised: Large Language Models (LLMs) can carry out complex reasoning tasks by generating intermediate reasoning steps.

Original: Providing these steps for prompting demonstrations is called chain-of-thought (CoT) prompting. CoT prompting has two major paradigms. One leverages a simple prompt like “Let’s think step by step” to facilitate stepby-step thinking before answering a question (Zero-Shot-CoT).
Revised: These steps are triggered by what is called chain-of-thought (CoT) prompting, which comes in two flavors: one leverages a simple prompt like “Let’s think step by step” to facilitate step-by-step reasoning before answering a question (Zero-Shot-CoT).

Original: The other uses a few step-by-step demonstrations, each composed of a question and a reasoning chain that leads to an answer.
Revised: The other uses manual demonstrations, each composed of a question and a reasoning chain that leads to an answer (Manual-CoT).

Original: The superior performance of the second paradigm hinges on the hand-crafting of multiple task-specific demonstrations.
Revised: Unfortunately, the superior performance of the latter strategy crucially hinges on manually generating task-specific demonstrations.

Revised: This makes it far less scalable and more dependent on the talent of the CoT engineer.

Original: We show that this limitation may be addressed by invoking LLMs to generate  reasoning chains for demonstrations.
Revised: We show that such manual efforts may be eliminated by leveraging LLMs to generate the reasoning chains on its own.

Original: However, these generated chains often come with mistakes. To mitigate the effect of such mistakes, we find that diversity matters for automatically constructing demonstrations.
Revised: Since these generated chains often come with mistakes we propose a number of mitigation strategies.

Original: We propose an automatic CoT prompting method: Auto-CoT. It automatically samples  questionswith diversity and generates reasoning chains by Zero-Shot-CoT to construct demonstrations.
Revised: Our proposed Auto-CoT method automaticaly samples diverse questions and we perform postprocessing quality control to generate usable reasoning chains from Zero-ShotCoT.

Original: On ten public benchmark reasoning tasks, Auto-CoT performs competitively compared to Manual-CoT that requires manual designs.
Revised: On ten public benchmark reasoning tasks, Auto-CoT performs on par with Manual-CoT without the need for human intervention.

Original: Code is available at Anonymous .
Revised: Code is available at https://github.com/amazon-research/auto-cot .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Large language models (LLMs) (Brown et al., 2020; Thoppilan et al., 2022; Rae et al., 2021;
Revised: Large language models (LLMs) (Brown et al., 2020; Thoppilan et al., 2022; Rae et al., 2021;

Original: Chowdhery et al., 2022) have performed impressively on complex reasoning tasks by decomposing multi-step problems into intermediate steps before giving answers (Nye et al., 2022).
Revised: Chowdhery et al., 2022) have performed impressively on complex reasoning tasks by decomposing multi-step problems into intermediate steps before giving answers (Nye et al., 2022).

Original: This reasoning process is elicited by a recent technique: chain-of-thought (CoT) prompting (Wei et al., 2022b).[Table]
Revised: This reasoning process is elicited by a recent technique: chain-of-thought (CoT) prompting (Wei et al., 2022b).

Original: Test [Table] CoT prompting can be categorized into two major paradigms. One adds a single prompt like “Let’s think step by step” after the test question to facilitate the reasoning chains in LLMs (Kojima et al., 
Revised: CoT prompting comes in two major flavors: one is to add a single prompt such as “Let’s think step by step” after the test question to facilitate the reasoning chains in LLMs (Kojima et al., 2022).

Original: Since this prompting paradigm is task-agnostic and does not need input-output demonstrations, it is called Zero-Shot-CoT (left of Figure 1).
Revised: Since this strategy is task-agnostic and does not need input-output demonstrations, it is called Zero-ShotCoT (Figure 1 left).

Original: With Zero-Shot-CoT, LLMs have shown to be decent zero-shot reasoners.
Revised: Via Zero-Shot-CoT, LLMs have shown to be decent zero-shot reasoners.

Original: The other paradigm is  few-shot prompting with a few step-by-step reasoning demonstrations  (Wei et al., 2022b).
Revised: The other strategy is to provide few-shot prompting through manual reasoning demonstrations one by one (Wei et al., 2022b).

Original: Each demonstration has a question and a reasoning chain.
Revised: Each demonstration has a question and a reasoning chain .

Original: A reasoning chain is composed of a rationale (a series of intermediate reasoning steps) and an expected answer.
Revised: The latter is composed of a rationale (a series of intermediate reasoning steps) and an expected answer.

Original: With all the demonstrations being manually designed, thisparadigm is referred to as Manual-CoT (right of Figure 1).
Revised: With all the demonstrations being manually designed, this is referred to as Manual-CoT (Figure 1 right).

Original: In practice, Manual-CoT has obtained stronger performance than Zero-Shot-CoT (Wei et al., 2022b; Kojima et al., 2022).
Revised: In practice, Manual-CoT outperforms Zero-Shot-CoT (Wei et al., 2022b; Kojima et al., 2022).

Original: However,this superior performance hinges on the hand-drafting of effective demonstrations.
Revised: However, superior performance hinges on the hand-crafting of effective demonstrations.

Original: Specifically, the hand-drafting involves nontrivial efforts in designs of both questions and their reasoning chains for demonstrations.
Revised: This involves nontrivial efforts in designs of both questions and their reasoning chains for demonstrations.

Original: Moreover, human efforts for designing task-specific demonstrations are even more: different tasks, such as arithmetic (Roy & Roth, 2015) and commonsense reasoning (Talmor et al., 2019), require different ways of demonstrations.
Revised: Even more problematic, different tasks, such as arithmetic (Roy & Roth, 2015) and commonsense reasoning (Talmor et al., 2019), require different ways of demonstrations to be manually generated.

Original: To eliminate such manual designs, we advocate another Auto-CoT paradigm to automatically construct demonstrations with questions and reasoning chains. 
Revised: We propose Auto-CoT . It addresses the problems in Manual-CoT by automatically constructing demonstrations with questions and reasoning chains. Auto-CoT uses LLMs for this task. It generates examples using the prompt “Let’s think step by step” with Zero-Shot-CoT.

Original: However, we find that this challenge cannot be effectively addressed by simple solutions.
Revised: Unfortunately, a naive approach is insufficient.

Original: For example, given a test question of a dataset, retrieving semantically similar questions and invoking Zero-Shot-CoT to generate reasoning chains will fail.
Revised: For example, given a test question of a dataset, retrieving semantically [Figure] A Few Manual Demos

Original: Although LLMs are decent zero-shot reasoners, they are not perfect: Zero-Shot-CoT can still make mistakes in reasoning chains.
Revised: [Figure]

Original: To mitigate the effect of reasoning chain mistakes from Zero-Shot-CoT, our analysis shows that diversity ofdemonstration questions is the key. Based on this insight, we propose an Auto-CoT method to automatically construct demonstrations.
Revised: similar questions and invoking Zero-Shot-CoT to generate reasoning chains will fail. After all, ZeroShot-CoT still makes mistakes in reasoning chains. These mistakes can be mitigated by providing sufficient diversity of demonstrations.

Original: Auto-CoT consists of two main steps. First, partition questions of a given dataset into a few clusters.
Revised: Auto-CoT employs two main steps: First, partition questions of a given dataset into a few clusters.

Original: Second, select a representative question from each cluster and generate its reasoning chain using Zero-Shot-CoT with simple heuristics.
Revised: Second, select a representative question from each cluster and generate its reasoning chain using Zero-Shot-CoT with simple heuristics.

Original: We evaluate Auto-CoT on ten benchmark reasoning tasks including: (i) arithmetic reasoning (MultiArith (Roy & Roth, 2015), GSM8K (Cobbe et al., 2021), AQUA-RAT (Ling et al., 2017),
Revised: We evaluate Auto-CoT on ten benchmark reasoning tasks including: (i) arithmetic reasoning (MultiArith (Roy & Roth, 2015), GSM8K (Cobbe et al., 2021), AQUA-RAT (Ling et al., 2017),

Original: SVAMP (Patel et al., 2021)); (ii) commonsense reasoning (CSQA (Talmor et al., 2019), StrategyQA (Geva et al., 2021)); (iii) symbolic reasoning (Last Letter Concatenation, Coin Flip) (Wei et al., 2022b).
Revised: SVAMP (Patel et al., 2021)); (ii) commonsense reasoning (CSQA (Talmor et al., 2019), StrategyQA (Geva et al., 2021)); (iii) symbolic reasoning (Last Letter Concatenation, Coin Flip) (Wei et al., 2022b).

Original: Experimental results show that Auto-CoT performs competitively compared to Manual-CoT that requires  manual designs.
Revised: Experimental results show that Auto-CoT performs competitively compared to Manual-CoT that requires (less scalable) manual designs.

Original: This indicates that LLMs can perform CoT reasoning by automatically constructing demonstrations.
Revised: This indicates that LLMs can perform CoT reasoning by automatically constructing demonstrations.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: This section reviews two lines of research that form the basis of this work: chain-of-thought (CoT) prompting for multi-step reasoning and in-context learning for inducing LLMs to learn from demonstrations.
Revised: Two lines of research are key for the current work: chain-of-thought (CoT) prompting for multi-step reasoning and in-context learning for LLMs. We review both of them below.

Original: 2. C HAIN - OF - THOUGHT P ROMPTING
Revised: 2. C HAIN - OF - THOUGHT P ROMPTING

Original: CoT prompting is a gradient-free technique of inducing LLMs to produce intermediate reasoning steps that lead to the final answer.
Revised: CoT prompting is a gradient-free technique of inducing LLMs to produce intermediate reasoning steps that lead to the final answer.

Original: Wei et al.
Revised: Wei et al.

Original: (2022b)formally studiedthe topic of CoT prompting in language models.
Revised: (2022b) studied CoT prompting in language models.

Original: This technique elicits LLMs to generate a coherent series of intermediate reasoning steps that lead to the final answer to a question.
Revised: It elicits LLMs to generate a coherent sequence of intermediate reasoning steps that lead to the final answer.

Original: Studies have shown that LLMs can perform CoT reasoning with zero-shot prompting (Zero-Shot-CoT) (Kojima et al., 2022) or manually written few-shot demonstrations (Manual-CoT) (Wei et al., 2022b).
Revised: LLMs can perform CoT reasoning with zero-shot prompting (Zero-Shot-CoT) (Kojima et al., 2022) or through human generated few-shot demonstrations (Manual-CoT) (Wei et al., 2022b).

Original: Zero-Shot-CoT.Kojima et al. (2022) showed that LLMs are decent zero-shot reasoners whose generated rationales have already reflected the CoT reasoning.
Revised: Zero-Shot-CoT. LLMs are decent zero-shot reasoners whose generated rationales have already reflected the CoT reasoning.

Original: This finding inspires our work to leverage self-generated rationales for demonstrations.
Revised: This observation inspires our work to leverage self-generated rationales for demonstrations.

Original: Generating rationales by LLMs was shown to be practical in a recent work (Zelikman et al., 2022).
Revised: Generating rationales by LLMs was shown to be practical by Zelikman et al.

Original: In their work, an LLM is prompted to generate rationales and those rationales that lead to the correct answer are selected.
Revised: In their work, an LLM is prompted to generate rationales. Among them, the ones that lead to the correct answer are selected.

Original: The selection requires a training dataset of questions with annotated answers.
Revised: The selection requires a training dataset of questions with annotated answers.

Original: In contrast, our work considers a more challenging scenario where only a set of test questions are given (without a training dataset), 
Revised: In contrast, we consider a more challenging scenario where only a set of test questions are given (without a training dataset), following CoT prompting (Wei et al., 2022b; Kojima et al., 2022).

Original: Manual-CoT. Manual-CoT achieves stronger performance by elicitingthe CoT reasoning ability with effective manual demonstrations.
Revised: Manual-CoT achieves stronger performance by eliciting CoT reasoning via effective humangenerated and designed demonstrations.

Original: The demonstrations for the reasoning process are manually designed.

Original: However, the human efforts in designs of both questions and their reasoning chains are nontrivial.
Revised: However, the work required in designing both questions and their reasoning chains are nontrivial.

Original: Instead of addressing this limitation, recent studies mainly focus on hand-crafting more complex demonstrations or leveraging ensemble-like methods.
Revised: Instead of addressing this limitation, recent studies mainly focus on hand-crafting more complex demonstrations or leveraging ensemble-like methods.

Original: One trend is problem decomposition.
Revised: One trend is problem decomposition.

Original: In least-to-most prompting (Zhou et al., 2022a), complex problems are reduced to sub-problems, and then the sub-problems are solved sequentially.
Revised: In least-to-most prompting (Zhou et al., 2022a), complex problems are reduced to sub-problems, and then the sub-problems are solved sequentially.

Original: In addition, prompting with more reasoning steps shows performance boosts (Fu et al., 2022).
Revised: In addition, prompting with more reasoning steps shows performance boosts (Fu et al., 2022).

Original: The other trend is to vote over multiple reasoning paths for a test question.
Revised: A second trend is to vote over multiple reasoning paths for a test question.

Original: Wang et al.
Revised: Wang et al.

Original: (2022a) introduced a self-consistency decoding strategy to sample multiple outputs of LLMs and then took a majority over the final answers.
Revised: (2022a) introduced a self-consistency decoding strategy to sample multiple outputs of LLMs and then take a majority over the final answers.

Original: Wang et al.
Revised: Wang et al.

Original: (2022b) and Li et al.
Revised: (2022b) and Li et al.

Original: (2022) introduced randomness in the input space to produce more diverse outputs for voting.
Revised: (2022) introduced randomness in the input space to produce more diverse outputs for voting.

Original: They used manually-designed demonstrations as the seed set and generated additional rationales: leave one question from the seed set and use the remaining demonstrations to generate rationales for this question by the LLM.
Revised: Manually-designed demonstrations served the seed set to generate additional rationales: leave one question from the seed set and use the remaining demonstrations to generate rationales for this question by the LLM.

Original: More recently, multilingual (Shi et al., 2022) and multimodal CoT (Lu et al., 2022b) have also emerged.
Revised: More recently, multilingual (Shi et al., 2022) and multimodal CoT (Lu et al., 2022b) have also emerged.

Original: Unlike the aforementioned research lines that rely on manually-designed demonstrations, our work intends to eliminate manual designs with competitive performance.
Revised: Unlike the aforementioned research lines that rely on manually-designed demonstrations, our work intends to eliminate manual designs while retaining competitive performance.

Original: 2. I N -C ONTEXT L EARNING
Revised: 2. I N -C ONTEXT L EARNING

Original: CoT prompting is closely related to in-context learning (ICL)
Revised: CoT prompting is closely related to in-context learning (ICL)

Original: (Radford et al., 2019; Brown et al., 2020).
Revised: (Radford et al., 2019; Brown et al., 2020).

Original: ICL enables LLMs to perform a target task by feeding a few prompted examples as part of the input.
Revised: ICL enables LLMs to perform a target task by feeding a few prompted examples as part of the input.

Original: Without gradient update, ICL allows a single model to perform various tasks universally.
Revised: Without gradient update, ICL allows a single model to perform various tasks universally.

Original: There are various research lines to improve the performance of ICL: (i) obtaining the most effective prompts (Fu et al., 2022; Lu et al., 2022a; Zhou et al., 2022b), e.g., a popular practice is dynamically retrieving related demonstrations to a given test input (Rubin et al., 2022; Su et al., 2022); (ii) augmenting with fine-grained information, such as incorporating task instruction (Mishra et al., 2022; Wei et al., 2022a;
Revised: There are various research lines to improve the performance of ICL: (i) obtaining the most effective prompts (Fu et al., 2022; Lu et al., 2022a; Zhou et al., 2022b), e.g., a popular practice is dynamically retrieving related demonstrations to a given test input (Rubin et al., 2022; Su et al., 2022); (ii) augmenting with fine-grained information, such as incorporating task instruction (Mishra et al., 2022; Wei et al., 2022a;

Original: Sanh et al., 2022); (iii) manipulating output probabilities of LLMs instead of directly computing the likelihood of target labels (Holtzman et al., 2021;
Revised: Sanh et al., 2022); (iii) manipulating output probabilities of LLMs instead of directly computing the likelihood of target labels (Holtzman et al., 2021;

Original: Zhao et al., 2021; Min et al., 2022a).
Revised: Zhao et al., 2021; Min et al., 2022a).

Original: Despite the success of ICL, studies (Liu et al., 2022b; Lu et al., 2022d;c) have shown that the strength of ICL may vary widely depending on the choice of in-context demonstrations (Liu et al., 2022a).
Revised: Despite the success of ICL, studies (Liu et al., 2022b; Lu et al., 2022d;c) have shown that the strength of ICL may vary widely depending on the choice of in-context demonstrations (Liu et al., 2022a).

Original: In detail, the formatting of the prompt, such as wording or order of demonstrations, may lead to performance fluctuations (Webson & Pavlick, 2022; Zhao et al., 2021).
Revised: In detail, the formatting of the prompt, such as wording or order of demonstrations, may lead to performance fluctuations (Webson & Pavlick, 2022; Zhao et al., 2021).

Original: A recent work (Min et al., 2022b) even questioned the necessity of ground-truth input-output mapping: using incorrect labels in the examples only marginally lowers the performance.
Revised: Min et al. (2022b) even questioned the necessity of ground-truth input-output mapping: using incorrect labels in the examples only marginally lowers the performance.

Revised: In other words, LLMs already have an innate ability to reason.

Revised: We only need to compel it to do so rather than needing to teach it how to accomplish this goal.

Original: However, the existing analysis of ICL is mainly based on standard classification and multi-choice datasets that only have simple <input → output> mappings.
Revised: Note that existing work in ICL is mainly based on standard classification and multiple choice datasets that only have simple <input → output> mappings.

Original: We discover that those findings may not be applicable to the CoT prompting scenario with more complex <input → rationale → output> mappings.
Revised: We argue that those findings may not be applicable to the CoT prompting scenario with more complex <input → rationale → output> mappings.

Original: For example, mistakes in either the <input → rationale> mapping or the <rationale → output> mappingwill lead to a dramaticperformance drop (Appendix A.1).
Revised: As experiments confirm (Appendix A.1), mistakes in either the <input → rationale> mapping or the <rationale → output> mapping lead to a dramatic drop in accuracy.

Original:  C HALLENGE OF A UTO -C O T 
Revised:  C HALLENGES IN A UTOMATIC C O T G ENERATION

Original: As just discussed, the performance of ICL hinges on hand-crafted demonstrations.
Revised: Good performance of ICL relies on well-crafted demonstrations.

Original: As reported in Manual-CoT (Wei et al., 2022b), using demonstrations written by different annotators brings up to 28.2% accuracy disparity in a symbolic reasoning task, while changing the order of demonstrations results in less than 2% changes in most tasks.
Revised: As reported in Manual-CoT (Wei et al., 2022b), using demonstrations written by different annotators brings up to 28.2% accuracy disparity in a symbolic reasoning task, while changing the order of demonstrations results in less than 2% changes in most tasks.

Original: This suggests that the key challenge of Auto-CoT lies in automatically constructing demonstrations with good questions and their reasoning chains.
Revised: This suggests that the key challenge of Auto-CoT lies in automatically constructing demonstrations with good questions and their reasoning chains.

Original: Recall that Manual-CoT hand-crafts a few (e.g., 8) questions in demonstrations.
Revised: Recall that Manual-CoT hand-crafts a small number of questions in demonstrations, not the least due to the limited attention window size of a transformer.

Original: 1 With similaritybased retrieval methods being widely adopted for prompting LLMs (Rubin et al., 2022;
Revised: 1 With similarity-based retrieval methods being widely adopted for prompting LLMs (Rubin et al., 2022;

Original: Su et al., 2022), a promisingcandidate solution is to sample demonstration questions using similarity-based retrieval.
Revised: Su et al., 2022), a promising solution is to generate CoT demonstrations using Zero-Shot-CoT and then to pick a suitable set of them for ICL, e.g. via similarity based retrieval.

Revised: We use GPT-3 (Brown et al., 2020) with 175B parameters (text-davinci-002) for the LLM, unless stated otherwise, to generate the demonstrations.

Original: We follow the more challenging assumption in CoT studies (Wei et al., 2022b; Kojima et al., 2022) that only a set of test questions are given (without a training dataset).
Revised: We follow the more challenging assumption in CoT studies (Wei et al., 2022b; Kojima et al., 2022) that only a set of test questions are given (without a training dataset).

Original: Following Liu et al.
Revised: Following Liu et al.

Original: (2022b)
Revised: (2022b)

Original: , we use Sentence-BERT (Reimers & Gurevych, 2019) to encode questions.
Revised: , we use Sentence-BERT (Reimers & Gurevych, 2019) to encode questions.

Original: For each question q test in a test dataset, we sample demonstration questions q demo i ( i = 1 , . . .
Revised: For each question q test in a test dataset, we sample demonstration questions q demo i ( i = 1 , . . .

Original: , k ) from the rest of the questions.
Revised: , k ) from the rest of the questions.

Original: We design a Retrieval-Q-CoT (Figure 9) method to retrieve the top- k
Revised: One strategy is to use Information Retrieval. Retrieval-Q-CoT

Original: (e.g., k = 8 ) similar questions based on cosine similarity.
Revised: (Figure 9) retrieves the top- k , e.g., k = 8 , most similar questions based on cosine similarity.

Original: To compare with this similarity-based method, we also test arelatively more diversity-based method: Random-Q-CoT (Figure 10), which randomly samples k other test questions for each test question. Both Retrieval-Q-CoT and Random-Q-CoT invoke Zero-Shot-CoT (Kojima et al., 2022) to generate the reasoning chain c demo i (rationale and answer) for each sampled question q demo i , as LLMs are decent zero-shot reasoners (Kojima et al., 2022). We use GPT-3 (Brown et al., 2020) with 175B parameters (text-davinci-002) for the LLM unless otherwise stated.
Revised: Alternatively we can pick a more diverse set by drawing k of them at random. We refer to this as Random-Q-CoT (Figure 10).

Original: On a high level, both Retrieval-Q-CoT and Random-Q-CoT take the concatenation of q demo i , cdemo i pairs ( i = 1 , . . .
Revised: At a high level, both Retrieval-Q-CoT and Random-Q-CoT take the concatenation of q demo i , c

Revised: [Table]

Original:  , k ) and q test as input to predict the reasoning chain for q test , which contains the answer in the end (like right of Figure 1).
Revised: pairs ( i = 1 , . . . , k ) and q test as input to predict the reasoning chain for q test , which contains the answer in the end (like right of Figure 1).

Original: [Table]
Revised: [Table]

Original: To our surprise, Retrieval-Q-CoT underperforms
Revised: To our surprise, Random-Q-CoT outperforms

Original: Random-Q-CoT on the arithmetic datasetMultiArith (Roy & Roth, 2015)
Revised: Retrieval-Q-CoT on MultiArith, an arithmetic dataset (Roy & Roth, 2015)

Original: (Table 1).
Revised: (Table 1).

Original: Note thatthe retrieval methods were originally proposed in tasks with annotated labels (Rubin et al., 2022; Su et al., 2022), however, invoking Zero-Shot-CoT does not guarantee entirely correct reasoning chains.
Revised: Note that retrieval for CoT was originally proposed for tasks with annotated labels (Rubin et al., 2022; Su et al., 2022). Unfortunately, invoking Zero-Shot-CoT does not guarantee perfectly accurate reasoning chains (if this were the case, the point of the current paper would be moot).

Original: Thus, we hypothesize that the inferior performance of Retrieval-Q-CoT is caused by incorrect reasoning chains by Zero-Shot-CoT. To test this hypothesis, we experiment with Retrieval-Q-CoT on two other datasets GSM8K (Cobbe et al., 2021) and AQuA (Ling et al., 2017) that have training sets with annotated reasoning chains.
Revised: Thus, we hypothesize that the inferior performance of Retrieval-Q-CoT is caused by incorrect reasoning chains by Zero-Shot-CoT. To test this hypothesis, we experiment with RetrievalQ-CoT on two other datasets GSM8K (Cobbe et al., 2021) and AQuA (Ling et al., 2017) that have training sets with annotated reasoning chains.

Original: The results are shown with † in Table 1. Under the setting with annotated reasoning chains, Retrieval-Q-CoT even outperforms Manual-CoT.
Revised: Given such gold-standard data, Retrieval-Q-CoT even outperforms Manual-CoT (see the results indicated by † in Table 1).

Original: The result indicates that Retrieval-Q-CoT is effective when human annotations are available.
Revised: This indicates that Retrieval-Q-CoT is effective when human annotations are available.

Original: Although human annotations are useful, such manual efforts are nontrivial. However, automatically generating reasoning chains via Zero-Shot-CoT underperforms Manual-CoT, especially when the challenge of question sampling is not addressed.
Revised: [Figure]

Original: To design more effective Auto-CoT, we need to understand its challenge better.
Revised: In order to design a robust Auto-CoT system we need to gain a better understanding of the failures of

Original: Since Retrieval-Q-CoT uses a few prompting demonstrations like in Manual-CoT, Retrieval-Q-CoT is expected to perform competitively as well. However, reasoning chains (both rationales and answers) in Retrieval-Q-CoT are generated by Zero-Shot-CoT:  they may have mistakes that lead to wrong answers.
Revised: Retrieval-Q-CoT. Key is the fact that the reasoning chains (both rationales and answers) in RetrievalQ-CoT are generated by Zero-Shot-CoT: as such they may have mistakes that lead to wrong answers, particularly by amplification of errors through retrieval of erroneous chains.

Original: Let us simply call demonstrations with wrong answers as wrong demonstrations .
Revised: Let’s see how this works in practice: we invoke

Original: Intuitively, after similar questions to a test question are retrieved, wrong demonstrations caused by Zero-ShotCoT may mislead the same LLM to reason similarly with a wrong answer (e.g., replicating mistakes) for the test question. We refer to this phenomenon as misleading by similarity . We will investigate whether misleading by similarity contributes to the inferior performance of Retrieval-Q-CoT. [Figure]
Revised: Zero-Shot-CoT

Original: To begin with, we invoke Zero-Shot-CoT on all the 600 questions from the MultiArith dataset.
Revised: on all the 600 questions from the MultiArith dataset.

Original: Among them, we collect those 128 questions (denoted as Q ) where Zero-Shot-CoT generates wrong answers (error rate: 21 . 3% = 128 / 600 ).
Revised: Among them there arequestions (denoted as Q ) where Zero-Shot-CoT generates wrong answers (error rate: 21 . 3% = 128 / 600 ).

Original: As we mentioned, with extra demonstrations, RetrievalQ-CoT and Random-Q-CoT are expected to perform more competitively than Zero-Shot-CoT. Among
Revised: We’ll call Q the unresolving set. Let’s see how well Retrieval-Q-CoT and RandomQ-CoT perform on Q and on the remaining 472 questions.

Original: Q where Zero-Shot-CoT fails, we call those where
Revised: We find that the conditional errors for

Original: Retrieval-Q-CoT or Random-Q-CoT still fail as their unresolved questions . 
Revised: Retrieval-Q-CoT and Random-Q-CoT on Q are 46 . 9% , 25 . 8% respectively.

Original: We divide the number of unresolved questions by 128 (number of questions in Q ) to calculate the unresolving rate .
Revised: On its complement they are 9 .

Original: A higher unresolving rate means that a method more likely still makes mistakes like Zero-Shot-CoT. Figureshows that the unresolving rate of Retrieval-Q-CoT (46.9%) is much higher than Random-Q-CoT (25.8%).
Revised: 7% and 9 . 4% respectively.

Original: It indicates that with similar questions being sampled for test questions, Retrieval-Q-CoT [Table] is negatively affected by misleading by similarity.
Revised: This demonstrates that it is primarily the errors on Q that contribute to the larger error for Retrieval-Q-CoT.

Original: In other words, for hard questions like those in Figure 2, retrieval tends to retrieve similarly hard questions that the model gets wrong.
Revised: In other words, for hard questions like those in Figure 2, retrieval tends to retrieve similarly hard questions that the model gets wrong.

Original: To show that unresolved questions of Retrieval-Q-CoT tend to be similar [Table] we present one case study (e [Table]
Revised: To show that unresolved questions of Retrieval-Q-CoT tend to be similar, we present one case study (e.g., lexical overlap) in Table 2.

Original: g., lexical overlap) in Table 2.

Original: In the left part, the retrieved demonstration questions are similar to the test question and ask “ how long will it take him to cook the rest? ”
Revised: In the left part, the retrieved demonstration questions are similar to the test question and ask “ how long will it take him to cook the rest? ”

Original: The reasoning chains generated by Zero-Shot-CoT produce answers regarding “ the total of ” instead of “ the rest ”.
Revised: The reasoning chains generated by Zero-Shot-CoT produce answers regarding “ the total of ” instead of “ the rest ”.

Original: Following the demonstrations, Retrieval-Q-CoT also fails by misunderstanding the meaning of “ the rest ”.
Revised: Following the demonstrations, Retrieval-Q-CoT also fails by misunderstanding the meaning of “ the rest ”.

Original: In contrast, Random-Q-CoT correctly understands “ the rest ” better without making similar mistakes in the demonstrations, thanks to relatively more diverse (random) demonstrations
Revised: In contrast, Random-Q-CoT correctly understands “ the rest ” better without making similar mistakes in the demonstrations, thanks to relatively more diverse (random) demonstrations. [Table]

Original: . 3. E RRORS F REQUENTLY F ALL INTO THE S AME C LUSTER
Revised: 3. E RRORS F REQUENTLY F ALL INTO THE S AME C LUSTER

Original: [Figure]
Revised: [Figure]

Original: Motivated by the observations in Table 2, we use k -means to partition allthe 600 test questions into k = 8 clusters, where each cluster contains similar questions.
Revised: Motivated by the observations in Table 2, we use k -means to partition all 600 test questions into k = 8 clusters, where each cluster contains similar questions.

Original: 2 With these clusters and reasoning chains generated by Zero-Shot-CoT (in Section 3.1), now we are curious if certain clusters contain questions where Zero-Shot-CoT frequently fails.
Revised: 2 With these clusters and reasoning chains generated by Zero-Shot-CoT (see Section 3.2), we can review whether certain clusters contain questions where Zero-Shot-CoT frequently fails.

Original: Thus, we calculate the error rate (questions with wrong ZeroShot-CoT answers / total questions) for each cluster.

Original: As shown in Figure 3, there exists a cluster (Cluster 2) with frequent Zero-Shot-CoT errors (52.3%).
Revised: As can be seen in Figure 3, one of the clusters (Cluster 2) has an abnormally high Zero-Shot-CoT error rate (52.3%).

Original: The phenomenon could be generic as Zero-ShotCoT may lack some skills to solve some common problems in target tasks.
Revised: The phenomenon could be generic as Zero-Shot-CoT may lack some skills to solve some common problems in target tasks.

Original: 3 For convenience of descriptions, let us call the cluster with the highest error rate as the frequent-error cluster(e.g., Cluster 2 in Figure 3).
Revised: For convenience we will refer to cluster 2, and similar clusters on other datasets, as the frequent-error cluster

Original: Therefore, the imperfect nature of  generated reasoning chains in a zero-shot fashion poses risks of retrieving multiple similar questions inside a frequent-error cluster by using similarity-based methods. For the test question in the frequent-error cluster, Retrieval-Q-CoT more easily constructs demonstrations with multiple similar mistakes. As  a result, Retrieval-Q-CoT often makes similar mistakes like Zero-Shot-CoT, reiterated by its higher unresolving rate in Figure 2.
Revised: The imperfect nature of synthetically generated reasoning chains in a zero-shot fashion makes it more likely for incorrect demonstrations to lead to incorrect answers in Retrieval-Q-CoT. As such, a path to mitigation is to bound the influence that any single cluster might impose on the ICL model.

Original: 3.  D IVERSITYM AY M ITIGATE M ISLEADING BY S IMILARITY
Revised: 3. M ITIGATION THROUGH D IVERSITY

Original: The analysis so far compellingly shows that LLMs are still not perfect zero-shot reasoners; thus, we aim to mitigate the effect of their Zero-Shot-CoT errors, especially to mitigate misleading by similarity in the design of Auto-CoT.
Revised: The analysis so far compellingly shows that LLMs are still not perfect zero-shot reasoners; thus, we aim to mitigate the effect of their Zero-Shot-CoT errors, especially to mitigate misleading by similarity in the design of Auto-CoT.

Original: As wewill show later (Section 5.5), presenting a small portion of mistakes (e.g., 1 or 2 wrong demonstrations out of 8) would not harm the overall reasoning performance for test questions.
Revised: As we show later in Section 5.4, presenting a small portion of mistakes (e.g., 1 or 2 wrong demonstrations out of 8) does not harm the overall reasoning performance for test questions.

Original: Suppose thatquestions of all the wrong demonstrations fall into the same frequent-error cluster; then sampling one question from every different cluster will lead to a higher than 7 / 8 = 87 .
Revised: On the other hand, suppose that the wrong demonstrations fall into the same cluster (as they do in our experiments). In this case, a question similar to that cluster will be burdened by many incorrect demonstrations for ICL. On the other hand, if we enforce diversity, say, by sampling one question

Original: 5% chance to construct all the 8 correct demonstrations.
Revised: [Figure]

Original: Since different clusters reflect diverse semantics of the questions,this clustering-based sampling method can be considered as diversity-based, which is in sharp contrast to similarity-based Retrieval-Q-CoT.
Revised: Since different clusters reflect diverse semantics of the questions, clustering-based sampling enhances the diversity of demonstrations, quite the opposite of similarity-based Retrieval-Q-CoT.

Original: On one hand, sampling questions with diversity may mitigate the effect of misleading by similarity (Section 3.1).
Revised: This can help mitigate the problems encountered in Section 3.2.

Original: On the other hand, if we took each demonstration as a kind of  skill, diverse demonstrations seem to cover more alternative skills for solving target questions: even though there still exists a small portion (e.g., 1 / 8 ) of mistakes in the demonstrations, the performance will not be negatively affected (to be shown in Figure 5).
Revised: At its extreme, if we consider each demonstration to be an example of a distinct skill, diverse demonstrations seem to cover more alternative skills for solving target questions: even though there still exists a small portion (e.g., 1 / 8 ) of mistakes in the demonstrations, the performance will not be negatively affected (see Figure 5).

Original: Nevertheless, the clustering-based sampling method may still construct a small portion of wrong demonstrations, such as from questions in the frequent-error cluster. As we will show later, some ofthese wrong demonstrations may be eliminated with heuristics.
Revised: A second strategy for improving the set of demonstrations is to apply general quality heuristics.

Original: For example, wrong demonstrations often come with long questions and long rationales.
Revised: For instance, wrong demonstrations often come with long questions and long rationales.

Original: Using simple and generic heuristics, such as only considering shorter questions with shorter rationales, further helps mitigate the effect of imperfect
Revised: We thus reject long and convoluted questions and rationales.

Original: Zero-Shot-CoT capabilities (Appendix C.3).
Revised: See Appendix C.3 for details on how to further improve the set of demonstrations.

Original:  A UTO -C O T: A UTOMATIC C HAIN - OF -T HOUGHT P ROMPTING
Revised:  A UTO -C O T: A UTOMATIC C HAIN - OF -T HOUGHT P ROMPTING

Original: Based on the observations and considerations in Section 3, we propose an Auto-CoT method to construct demonstrations with questions and reasoning chains automatically.
Revised: We are now ready to introduce Auto-CoT, our algorithm for automatic Chain-of-Thought Prompting.

Original: Auto-CoT consists of two main stages: (i) question clustering: partition questions of a given dataset into a few clusters; (ii) demonstration sampling: select a representative question from each cluster and generate its reasoning chain using Zero-Shot-CoT with simple heuristics.

Original: The overall procedure is illustrated in Figure 4. [Table] [Figure] [Table]
Revised: Auto-CoT consists of two main stages (the overall procedure is illustrated in Figure 4):

Original: [Figure] [Table] [Figure] [Table]
Revised: Question clustering: given a set of questions, partition them into a small number of clusters, according to the number of demonstrations that the ICL context can support (e.g., 8 clusters).

Original: [Figure]
Revised: Demonstration sampling: select a representative question from each cluster and generate its reasoning chain using Zero-Shot-CoT with simple heuristics.

Original: 4. Q UESTION C LUSTERING
Revised: 4. Q UESTION C LUSTERING

Original: Since diversity-based clustering may mitigate misleading by similarity (Section 3.3), we perform cluster analysis for a given set of  questions Q .
Revised: Since diversity-based clustering may mitigate errors we first cluster the questions. Note that clustering entire demonstrations only leads to a minimal performance improvement, at the expense of needing to invoke the full (question, reasoning, answer) generation API for each question in the set. As such, we use questions only.

Original: We first compute a vector representation for each questionin Q by Sentence-BERT (Reimers & Gurevych, 2019).
Revised: We first compute a vector representation for each question by Sentence-BERT (Reimers & Gurevych, 2019).

Original: The contextualized vectors are averaged to form a fix-sized question representation.
Revised: The contextualized vectors are averaged to form a fix-sized question representation.

Original: Then, the question representations are processed by the k -means clustering algorithm to produce k clusters of questions.
Revised: Then, the question representations are processed by the k -means clustering.

Original: For questions in each cluster i , sort them into a list q( i) =
Revised: For questions in each cluster i , sort them into a list q i =

Original: [ q( i) 1 , q( i) 2 , . . 
Revised: [ q i 1 , q i 2 , . . .

Original:  in the ascending order of the distance to thecenter of cluster i .
Revised: ] in the ascending order of the distance to the cluster center.

Original: This question clustering stage is summarized in Algorithm 1.
Revised: We will preferentially pick from the most typical question of each cluster when it comes to generating demonstrations. See Algorithm 1 for further details.

Original: 4.  D EMONSTRATION S AMPLING
Revised: 4. G ENERATING D EMONSTRATIONS

Original: In the second stage, we need to generate reasoning chains for those sampled questions and then sample demonstrations that satisfy our selection criteria.
Revised: Next we need to generate reasoning chains for those sampled questions and then sample demonstrations that satisfy our selection criteria.

Original: More concretely, we construct a demonstration d( i) (concatenation of a question, a rationale, and an answer) for each cluster i ( i = 1 , . . . , k ).
Revised: More specifically, we construct a demonstration d i (concatenation of a question, a rationale, and an answer) for each cluster i ( i = 1 , . . . , k ).

Original: For cluster i , we iterate over questions in the sorted list q( i) =
Revised: For cluster i , we iterate over questions in the sorted list q i =

Original: [ q( i) 1 , q( i) 2 , . .
Revised: [ q i 1 , q i 2 , . .

Original: (obtained by Algorithm 1) until satisfying our selection criteria.
Revised: . ] , obtained by Algorithm 1 until satisfying our selection criteria.

Original: In other words, a question that is closer to thecenter of cluster  i is considered earlier.
Revised: A question that is closer to the cluster center i is considered earlier.

Original: Say that the j -th closest question q ( i ) j is being considered.
Revised: Say that the j -th closest question q ij is being considered.

Original: A prompted input is formulated as: [Q: q ( i ) j .
Revised: A prompted input is formulated as: [Q: q ij .

Original: A: [P] ], where [P] is a single prompt “Let’s think step to step”.
Revised: A: [P] ], where [P] is a single prompt “Let’s think step by step” .

Original: Thisformed input is fed into an LLM using Zero-Shot-CoT (Kojima et al., 2022) to output the reasoning chain consisting of the rationale r ( i ) j and the extracted answer a ( i ) j .
Revised: This is fed into an LLM using Zero-Shot-CoT (Kojima et al., 2022) to output the reasoning chain consisting of the rationale r ij and the extracted answer a ij .

Original: Then, a candidate demonstration d ( i ) j for the i -th cluster is constructed by concatenating the question, rationale, and answer: [ Q: q ( i ) j , A: r ( i ) j ◦ a ( i ) j ] .
Revised: Then, a candidate demonstration d ij for the i -th cluster is constructed by concatenating the question, rationale, and answer: [ Q: q ij , A: r ij ◦ a ij ] .

Original: Similar to the criteria of the hand-crafting demonstrations in Wei et al.
Revised: Similar to the criteria of the hand-crafting demonstrations in Wei et al.

Original: (2022b)
Revised: (2022b)

Original: , our selection criteria follow simple heuristics to encourage sampling simpler questions and rationales: set the selected demonstrationd ( i ) as d ( i ) j if it has a questionq ( i ) j with no more than 60 tokens and a rationaler ( i ) j with no more than 5 reasoning steps.
Revised: , our selection criteria follow simple heuristics to encourage sampling simpler questions and rationales: choose the selected demonstration if it has a question with no more than 60 tokens and a rationale with no more thanreasoning steps.

Revised: Since Zero-Shot-CoT often uses ‘ \ n’ for separating the reasoning steps, the rule can be easily implemented by counting the ‘ \ n’ tokens in the generated rationales.

Original: Algorithm 
Revised: Algorithm 1 Cluster

Original: Require: A set of questions Q and the number of demonstrations k
Revised: Require: Questions Q , number of chains k

Original: Ensure: Sorted questions q( i) =
Revised: Ensure: Sorted questions q i =

Original: [ q( i) 1 , q( i) 2 , . . 
Revised: [ q i 1 , q i 2 , . . .

Original:  for each cluster i ( i = 1, . . ., k ) 1: procedure C LUSTER ( Q , k ) 2: for each question q in Q do 3:
Revised: ] for each cluster i ∈ { 1 . . . k } 1: procedure C LUSTER ( Q , k ) 2: for each question q in Q do 3:

Original: Encode q by Sentence-BERT 4:
Revised: Encode q by Sentence-BERT 4:

Original: Cluster all the encoded question represen- tations into k clusters 5: for each cluster i = 1, . . .
Revised: Cluster all questions q into k clusters 5: for each cluster i ∈ { 1 . . .

Original: , k  do 6:
Revised: k } do 6:

Original: Sort questions q( i) =
Revised: Sort questions q i =

Original: [ q( i) 1 , q( i) 2 , . . 
Revised: [ q i 1 , q i 2 , . . .

Original:  in the ascending orderof the distance to the cluster center 7: return  q( i ) ( i = 1, . . .
Revised: ] in the ascending order cluster centrality 7: return all q i for i ∈ { 1 . . .

Original: , k )
Revised: k }

Original: Algorithm
Revised: Algorithm

Original: Require:
Revised: Require:

Original: Sorted questions q ( i ) = [ q ( i ) 1 , q( i) 2 , . . for each cluster i ( i = 1 , . . . , k ), empty demonstration list d
Revised: Sorted question lists q i for all k clusters

Original: Ensure: Demonstration list d = [ d (1) , . . .
Revised: Ensure: Demonstration list d = [ d 1 , . . .

Original: , d( k) ] 1: procedure C ONSTRUCT ( q( i) , . . . , q( k )) 2:  for each cluster i = 1, . . ., k do 3: for each question q ( i ) j in q ( i ) do 4: Generate rationale r
Revised: , d k ] 1: procedure C ONSTRUCT ( q i , . . . , q k ) 2: d ← ∅ 3: for each cluster i ∈ { 1 . . .

Original: ( i ) j and answer a( i ) j for q( i ) j using Zero-Shot-CoT 5: ifq ( i ) j , r( i )j satisfy selection criteria then 6:
Revised: k } do 4: for each question q ∈ q i do 5: (rationale r , answer a ) via Zero-Shot-CoT( q ) 6: if ( q, r ) satisfy selection heuristic then 7:

Original: Add d  ( i ) = [ Q: q ( i ) j , A: r ( i ) j ◦ a( i ) j ] to d 7: break 8:  return d
Revised: d ← d ∪ { ( q, r, a ) } 8: break 9: return d

Original: As summarized in Algorithm 2, after demonstration sampling for allthe k clusters, there will be k constructed demonstrations [ d (1) , . . . , d ( k ) ] .
Revised: Algorithm 2 generates demonstrations for all k clusters.

Original: The constructed demonstrations are  used to augment a test question q test for in-context learning.
Revised: These are then used to augment a test question q test for in-context learning.

Original: Specifically, the input is the concatenation of all the demonstrations [ d (1) , . . .
Revised: Specifically, the input is the concatenation of all the demonstrations [ d 1 , . . .

Original: , d( k) ] followed by [Q: q test .
Revised: , d k ] followed by [Q: q test .

Original: A: [P] ].
Revised: A: [P] ].

Original: This input is fed to LLMs to obtain the reasoning chain with the answer in the end for q test (right of Figure 4).
Revised: This input is fed to LLMs to obtain the reasoning chain with the answer in the end for q test (right of Figure 4).

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We briefly describe the experimental setup and present main experimental results.
Revised: We provide a brief description of the experimental setup and present main experimental results.

Original: More experimental details and results can be found in Appendices B-D.[Table] [Table]
Revised: More details, including ablation studies and an evaluation of alternatives can be found in Appendices B-D.

Original: E XPERIMENTAL SETUP
Revised: 5. S ETUP

Original: Tasks and Datasets.
Revised: Tasks and Datasets.

Original: Our method is evaluated on ten benchmark datasets from three categories of reasoning tasks: (i) arithmetic reasoning (MultiArith (Roy & Roth, 2015), GSM8K (Cobbe 
Revised: We evaluate Auto-CoT on ten benchmark datasets from three categories of reasoning tasks: (i) arithmetic reasoning (MultiArith (Roy & Roth, 2015), GSM8K (Cobbe et al., 2021), AddSub (Hosseini et al., 2014), AQUA-RAT (Ling et al., 2017), SingleEq (Koncel-Kedziorski et al., 2015), SVAMP (Patel et al., 2021)); (ii) commonsense reasoning (CSQA (Talmor et al., 2019), StrategyQA (Geva et al., 2021)); (iii) symbolic reasoning (Last Letter Concatenation, Coin Flip) (Wei et al., 2022b).

Original: Implementation.
Revised: Implementation.

Original: We use the public GPT-3 (Brown et al., 2020)of the text-davinci-002 version with 175B parameters for the LLM (Ouyang et al., 2022) unless otherwise stated.
Revised: We use the text-davinci-002 version of GPT-3 (Brown et al., 2020) with 175B parameters as LLM (Ouyang et al., 2022) unless stated otherwise.

Original: We select this LLM because it has the strongest CoT reasoning performance among public LLMs, as reported in Kojima et al.
Revised: We select this LLM because it has the strongest CoT reasoning performance among public LLMs, as reported in Kojima et al.

Original: (2022) and Wei et al.
Revised: (2022) and Wei et al.

Original: (2022b).
Revised: (2022b).

Original: We also evaluate the  Codex model (Chen et al., 2021) (code-davinci-002) as the LLM.
Revised: To assess the dependency on LLMs we also evaluate the codedavinci-002 of Codex as an alternative (Chen et al., 2021).

Original: Following Wei et al.
Revised: Following Wei et al.

Original: (2022b), the number  of demonstrations k isexcept for AQuA and Letter (4), CSQA (7), and StrategyQA (6).
Revised: (2022b) , the number [Table] of demonstrations k is 8 except for AQuA and Letter (4), CSQA (7), and StrategyQA (6).

Original: The constructed demonstrations of Auto-CoT are presented in Appendix E.
Revised: The constructed demonstrations of Auto-CoT are presented in Appendix E.

Original: Baselines.
Revised: Baselines.

Original: We compare our methods with four baseline methods: Zero-Shot (Kojima et al., 2022), Zero-Shot-CoT (Kojima et al., 2022), Few-Shot (Wei et al., 2022b), and Manual-CoT (Wei et al., 2022b).
Revised: We compare Auto-CoT to four baselines: Zero-Shot (Kojima et al., 2022), Zero-ShotCoT (Kojima et al., 2022), Few-Shot (Wei et al., 2022b), and Manual-CoT (Wei et al., 2022b).

Original:  Zero-Shot-CoT and Manual-CoT are illustrated in Figure 1.
Revised: For reference, Zero-Shot-CoT and Manual-CoT are summarized in Figure 1.

Original: The Zero-Shot baseline concatenates a test question with the prompt “The answer is” as the LLM input.
Revised: The Zero-Shot baseline concatenates a test question with the prompt “The answer is” as the LLM input.

Revised: The

Original: The Few-Shot baseline has the same LLM input as Manual-CoT except for removed rationales from all the demonstrations.
Revised: Few-Shot baseline has the same LLM input as Manual-CoT except for removed rationales from all the demonstrations.

Original: 5.C OMPETITIVE P ERFORMANCE OF A UTO -C O T ON T EN D ATASETS
Revised: 5. P ERFORMANCE OF A UTO -C O T ON T EN D ATASETS

Original: Table 3 compares accuracy on ten datasets from three categories of reasoning tasks.
Revised: Table 3 provides a summary of the performance of Auto-CoT on ten datasets from three categories of reasoning tasks.

Original: The Zero-Shot and Zero-Shot-CoT results are taken from Kojima et al.
Revised: The Zero-Shot and Zero-Shot-CoT results are taken from Kojima et al.

Original: (2022), the Few-Shot and Manual-CoT results are taken from Wei et al.
Revised: (2022), the Few-Shot and Manual-CoT results are taken from Wei et al.

Original: (2022b), and
Revised: (2022b), and

Original: the Auto-CoT results are averaged over three random runs.
Revised: the Auto-CoT results are averaged over three random runs.

Original: Overall, Auto-CoT performs competitively compared to Manual-CoT that requires manual designs of demonstrations.
Revised: Overall, Auto-CoT performs competitively compared to Manual-CoT that requires manual designs of demonstrations.

Original: Due to the cost of manual designs, Manual-CoT may design the same demonstrations for multiple datasets (e.g., 5 / 6 of the arithmetic datasets).
Revised: Due to the cost of manual designs, Manual-CoT may design the same demonstrations for multiple datasets (e.g., 5 / 6 of the arithmetic datasets).

Original: In contrast, Auto-CoT is more flexible and task-adaptive: every single dataset gets its own demonstrations that are automatically constructed.
Revised: In contrast, Auto-CoT is more flexible and task-adaptive: every single dataset gets its own demonstrations that are automatically constructed.

Revised: Even better, the performance is sufficiently good that no human intervention is required.

Revised: This arguably shows that LLMs are already inherently able to solve multi-step reasoning problems, they only need to be compelled to do so.

Original: 5. A BLATION STUDIES OF D ESIGN C OMPONENTS
Revised: 5. A BLATION S TUDIES

Original: To investigate the effect of clustering and simple heuristics, we conduct an ablation study by removing either of these two design components in Table 4.
Revised: To investigate the effect of clustering and simple heuristics, we conduct an ablation study by removing either of these two design components in Table 4.

Original: To ablate on clustering, we implement “w/o clustering” based on Random-Q-CoT with simple heuristics.
Revised: To test for the effect of clustering, we use RandomQ-CoT onl queries that satisfy the simple quality heuristics (simplicity of reasoning and answer) described previously.

Original: Both design components are effective and essential.
Revised: Both design components are effective and essential.

Original: More comprehensive results on all the ten datasets are in Appendix C.2.
Revised: For more comprehensive results on all the ten datasets please review Appendix C.2.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 5. G ENERAL E FFECTIVENESS U SING THE C ODEX LLM

Original: To evaluate the general effectiveness of Auto-CoT using different LLMs, here we change the LLM tothe Codexmodel (Chen et al., 2021).
Revised: To evaluate the general effectiveness of Auto-CoT using different LLMs [Table] here we change the LLM to Codex (Chen et al., 2021).

Original: As in Table 5, the Codex LLM leads to performance improvement for Manual-CoT when compared withTable 3 that uses the GPT-3 (text-davinci-002)
Revised: Note that, as illustrated in Table [Table] Codex LLM leads to performance improvement on Arithmetic reasoning tasks. This also applies to Manual-CoT when compared with

Original: LLM.
Revised: Table 3 that uses GPT-3.

Original: Nonetheless, using the Codex LLM, the overall performance of Auto-CoT isstill competitive compared to Manual-CoT, providing additional empirical evidence for the effectiveness of Auto-CoT.
Revised: Both with Codex and with GPT-3 the overall performance of Auto-CoT is competitive compared to Manual-CoT, providing additional empirical evidence for its effectiveness.

Original: 5. E FFECT OF W RONG D EMONSTRATIONS
Revised: 5. E FFECT OF W RONG D EMONSTRATIONS

Original: Recall our discussions in Section 3.3 that there can be wrong demonstrations (whose answers are wrong).
Revised: One of the key observations in our preliminary experiments in Section 3.4 was that in-context models show a modicum of robustness to incorrect (question, demonstration, answer) triples.

Original: To see if diversity mitigates this effect, we design an In-Cluster Sampling baseline that constructs demonstrations by randomly sampling questions from the same cluster that contains a test question.
Revised: To see if diversity mitigates this effect, we design an In-Cluster Sampling baseline that constructs demonstrations by randomly sampling questions from the same cluster containing the test question.

Original: Figure 5 compares accuracy with varying amounts of wrong demonstrations on MultiArith.
Revised: Figure 5 compares accuracy with varying amounts of wrong demonstrations on MultiArith.

Original: Compared with In-Cluster Sampling, Auto-CoT (using diversity-based clustering) is less affected by wrong demonstrations: its performance still does not degrade significantly even when presented with 50% wrong demonstrations.
Revised: Compared with In-Cluster Sampling, Auto-CoT (using diversity-based clustering) is less affected by wrong demonstrations: its performance still does not degrade significantly even when presented with 50% wrong demonstrations.

Original: [Figure]
Revised: [Figure]

Original: Percentage of wrong

Original: [Figure][Figure]
Revised: [Figure]

Original: 5.M ORE C HALLENGING S TREAMING S ETTING
Revised: 5. S TREAMING Q UERIES

Original:  CoT studiescommonly assume that a full dataset with test questions is given  (Wei et al., 2022b; Kojima et al., 2022).
Revised: Typically CoT studies assume that a full dataset with test questions is given at the beginning (Wei et al., 2022b; Kojima et al., 2022).

Original: Based on the given dataset, Auto-CoT samples questions to construct the demonstrations.
Revised: This is needed to generate the demonstrations used in ICL estimation. As illustrated above, Auto-Cot performs very well in this case.

Original: Nonetheless, now we consider a more challenging streaming setting wherea small batch of test questions(say m questions) arrive  at a time like in data streams.
Revised: Now let us consider a case where questions arrive in small batches of, say m questions at a time.

Original: To address this challenge, we extend Auto-CoT to a bootstrapping version Auto-CoT*: (i) Initialize an empty set M 0 ; (ii) When batch1 of questions q (1)1 , . . .
Revised: This streaming setting (Auto-CoT*) is easily addressed with the following modifications: Initialize an empty set M 0 ; 2. For the first batch of questions q 11 , . . .

Original: , q (1) m arrive, invoke Zero-Shot-CoT (no clustering due to small m ) for each q (1) i to obtain its reasoning chain c (1) i  .
Revised: , q 1 m Invoke Zero-Shot-CoT (no clustering due to small m ) for each q 1 i to obtain its reasoning chain ( q 1 i , r 1 i , a 1 i ) .

Original: Add question-chain pairs ( q (1)1 , c (1) 1) , . . 
Revised: For any subsequent batch b of questions q b 1 , . . .

Original: ,( q (1) m , c (1) m ) to M 0 and call the new set M 1 ; (iii) When batch b ( b > 1 ) of questions q ( b ) 1 , . . .
Revised: , q bm

Original: , q ( b ) m arrive, construct demonstrations with existing questions and reasoning chains in M b − 1 (like Auto-CoT) and use the demonstrations for in-context reasoning for each q ( b ) i . 
Revised: Construct demonstrations with existing questions and reasoning chains in M b − 1 (like Auto-CoT) and use the demonstrations for in-context reasoning for each q bi . 4.

Original: Add question-chain pairs ( q( b) 1 , c ( b) 1 ) , . .
Revised: Add question-chain pairs ( q b 1 , r b 1 , a b 1 ) , . .

Original: , ( q ( b ) m , c ( b ) m ) to M b − 1 and call the new set M b .
Revised: , ( q bm , r bm , a bm ) to M b − 1 to obtain M b ;

Original: Figure 6 compares the accuracy on MultiArith at each batch ( m = 30 ) in this streaming setting (extended version: Figure 15 in the Appendix).
Revised: Figure 6 compares the accuracy on MultiArith at each batch ( m = 30 ) in this streaming setting. For an extended discussion see Figure 15 in the Appendix.

Original: As expected, for batch 1 , Auto-CoT* and Zero-ShotCoT obtain equal accuracy.
Revised: As expected, for batch 1 , Auto-CoT* and Zero-Shot-CoT obtain equal accuracy.

Original: From batch 2 , Auto-CoT* performs comparably with Manual-CoT.
Revised: From batch 2 , Auto-CoT* performs comparably with Manual-CoT.

Original: Thisresult indicates that our method is still effective in themore challenging streaming setting.
Revised: This indicates that Auto-CoT* is quite effective in the streaming setting.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: LLMs have shown reasoning capabilities with CoT prompting.
Revised: LLMs have shown reasoning capabilities with CoT prompting.

Original: The superior performance of ManualCoT hinges on the hand-crafting of demonstrations.
Revised: The superior performance of ManualCoT hinges on the hand-crafting of demonstrations.

Revised: This requires subject-matter experts, it is highly dependent on the skill level of the annotator and it is costly.

Original: To eliminate such manual designs, we proposed Auto-CoT to automatically construct demonstrations favoring diversity.
Revised: Auto-CoT provides an alternative by eliminating such manual designs. Our experiments showed that Auto-CoT performs competitively.

Original: Experiments showed that 
Revised: Auto-CoT relies on the fact that ICL is robust to small amounts of error. Furthermore, we are able to limit the influence of erroneous data by drawing stratified examples from clusters of questions — in many cases errors are unevenly distributed across clusters and as such we can reduce the probability of obtaining many errors from one and the same cluster.

Original: Auto-CoT performs competitively compared to Manual-CoT that requires manual designs.
Revised: Lastly, simple heuristics (complex answers are less likely to be accurate) are suitable for assessing the quality of an answer and should be employed for an improvement in accuracy.

Original: A E XTENDED ANALYSIS FOR THE CHALLENGE OF A UTO -C O T A. I MPACT OF DEMONSTRATION ELEMENTS .
Revised: A E XTENDED ANALYSIS FOR THE CHALLENGE OF A UTO -C O T A. I MPACT OF DEMONSTRATION ELEMENTS .

Original: A demonstration is a triple composed by <question, rationale, answer> as shown in Figure 1.
Revised: A demonstration is a triple composed by <question, rationale, answer> as shown in Figure 1.

Original: We shuffle either of the demonstration components to see how the performance changes.
Revised: We shuffle either of the demonstration components to see how the performance changes.

Original: According to the results in Table 6, shuffling questions has the least performance reduction ( 91 .
Revised: According to the results in Table 6, shuffling questions has the least performance reduction ( 91 .

Original: 7% → 73 . 8% ).
Revised: 7% → 73 . 8% ).

Original: A possible reason for the decent performance is that the model may capture the rationale-answer mapping patterns.
Revised: A possible reason for the decent performance is that the model may capture the rationale-answer mapping patterns.

Original: The pattern is expected to reflect how the intermediate reasoning steps can lead to the final answer.
Revised: The pattern is expected to reflect how the intermediate reasoning steps can lead to the final answer.

Original: This finding shows that it is possible to leverage the generated rationales by
Revised: This finding shows that it is possible to leverage the generated rationales by

Original: Zero-Shot-CoT
Revised: Zero-Shot-CoT

Original: because those rationales are often logically correct even though they may lead to wrong answers (Kojima et al., 2022).
Revised: because those rationales are often logically correct even though they may lead to wrong answers (Kojima et al., 2022).

Original: [Table]
Revised: [Table]

Original: In contrast [Table] shuffling either rationales or answers reduces the accuracy significantly ( 91 .
Revised: In contrast, shuffling either rationales or answers reduces the accuracy significantly ( 91 .

Original: 7% → 43 .
Revised: 7% → 43 .

Original: The observation indicates that the rationale-answer consistency is critical.
Revised: The observation indicates that the rationale-answer consistency is critical.

Original: This kind of mismatch actually happens in Zero-Shot-CoT.
Revised: This kind of mismatch actually happens in Zero-Shot-CoT.

Original: An example is shown in Table 7.
Revised: An example is shown in Table 7.

Original: Using such demonstrations might teach the model illusion—predicting answers without basis.
Revised: Using such demonstrations might teach the model illusion—predicting answers without basis.

Original: [Table]
Revised: [Table]

Original: A. O BSERVATION OF FREQUENT - ERROR CLUSTERS
Revised: A. O BSERVATION OF FREQUENT - ERROR CLUSTERS

Original: To verify if Zero-Shot-CoT fails at similar problems, we cluster the questions into a few clusters and calculate the error rate of the answers to the questions in each cluster.
Revised: To verify if Zero-Shot-CoT fails at similar problems, we cluster the questions into a few clusters and calculate the error rate of the answers to the questions in each cluster.

Original: As shown in Figure 7, the mistakes tend to gather in one or more clusters across different datasets.
Revised: As shown in Figure 7, the mistakes tend to gather in one or more clusters across different datasets.

Original: We observe a similar phenomenon when the cluster number changes, as shown in Figure 8.
Revised: We observe a similar phenomenon when the cluster number changes, as shown in Figure 8.

Original: The phenomenon has shown to be generic that Zero-Shot-CoT may lack some skills to solve some common problems in target tasks.
Revised: The phenomenon has shown to be generic that Zero-Shot-CoT may lack some skills to solve some common problems in target tasks.

Original: We call the cluster with the highest error rate as a frequent-error cluster.
Revised: We call the cluster with the highest error rate as a frequent-error cluster.

Original: Therefore, the imperfect nature of generated reasoning chains poses risks of retrieving a set of similar questions inside the frequent-error cluster for similarity-based retrieval.
Revised: Therefore, the imperfect nature of generated reasoning chains poses risks of retrieving a set of similar questions inside the frequent-error cluster for similarity-based retrieval.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: A. C OMPARISON OF DIFFERENT C O T PROMPTING METHODS
Revised: A. C OMPARISON OF DIFFERENT C O T PROMPTING METHODS

Original: There are various CoT prompting methods discussed in this paper.
Revised: There are various CoT prompting methods discussed in this paper.

Original: To help distinguish their key characteristics, we present a comparison of those CoT prompting methods across different dimensions in Table 8.
Revised: To help distinguish their key characteristics, we present a comparison of those CoT prompting methods across different dimensions in Table 8.

Original: Note that Random-Q-CoT can be implemented with “Same Demos”, however, we find that the variance of the performance will become much larger.
Revised: Note that Random-Q-CoT can be implemented with ‘Same Demos’ , however, we find that the variance of the performance will become much larger.

Original: Therefore, we implement Random-Q-CoT with different demonstrations for each test question, in a similar manner like Retrieval-Q-CoT [Figure] [Figure]
Revised: Therefore, we implement Random-Q-CoT with different demonstrations for each test question, in a similar manner like Retrieval-Q-CoT.

Original: [Table]
Revised: [Table]

Revised: Yes

Original: For Random-Q-CoT and Retrieval-Q-CoT, we present the illustrations of their processes in Figureand 10, respectively.
Revised: For Random-Q-CoT and Retrieval-Q-CoT, we present the illustrations of their processes in Figureand 10, respectively.

Original: B E XPERIMENTAL D ETAILS B. T ASKS AND D ATASETS
Revised: B E XPERIMENTAL D ETAILS B. T ASKS AND D ATASETS

Original: Our method is evaluated on ten benchmark datasets that cover arithmetic reasoning, commonsense reasoning, and symbolic reasoning tasks.
Revised: Our method is evaluated on ten benchmark datasets that cover arithmetic reasoning, commonsense reasoning, and symbolic reasoning tasks.

Original: The statistics of the datasets are shown in Table 9.
Revised: The statistics of the datasets are shown in Table 9.

Original: Arithmetic Reasoning.
Revised: Arithmetic Reasoning.

Original: For arithmetic reasoning, we consider the following six datasets: (i)
Revised: For arithmetic reasoning, we consider the following six datasets: (i)

Original: MultiArith (Roy & Roth, 2015), (ii) GSM8K (Cobbe et al., 2021), (iii) AddSub (Hosseini et al., 
Revised: MultiArith (Roy & Roth, 2015), (ii) GSM8K (Cobbe et al., 2021), (iii) AddSub (Hosseini et al., 2014), (iv) AQUA (Ling et al., 2017), (v) SingleEq (Koncel-Kedziorski et al., 2015), and (vi) SVAMP Q: A chef needs to cook 9 potatoes. He has already cooked 7. If each potato takes 3 minutes to cook, how long will it take him to cook the rest?

Original: [Figure]
Revised: A: Let’s think step by step. The chef has already cooked 5 potatoes. That means it has taken him 30 minutes so far. There are 13 potatoes in total.have been cooked so far.

Revised: That means there are 8 left to cook.

Revised: Each potato takes 6 minutes to cook.

Revised: That means it will take the chef 48 minutes to cook the 8 remaining potatoes.

Revised: In total, it will take the chef 78 minutes to … …

Revised: Q: A chef needs to cook 9 potatoes.

Revised: He has already cooked 7.

Revised: If each potato takes 3 minutes to cook, how long will it take him to cook the rest?

Revised: A: Let’s think step by step.

Revised: The chef has already cooked 7 potatoes.

Revised: That means it has taken him 7 * 3 minutes to cook those 7 potatoes.

Revised: That means it will take him 3 more minutes to cook each of the remaining 2 potatoes …

Revised: Generating Chains

Revised: LLM

Revised: In-Context Reasoning

Revised: [Figure]

Revised: to cook 13 potatoes.

Revised: He has already…

Revised: Q

Original: [Figure]
Revised: [Figure]

Revised: to cook 9 potatoes.

Original: [Table]
Revised: He has already… [Figure] [Figure] [Figure]

Revised: Let’s

Revised: [Figure]

Revised: Demos to Each

Original: [Table]
Revised: [Figure]

Original: Generating Chains
Revised: Q: A chef needs to cook 15 potatoes. He has already cooked 8. If each potato takes 9 minutes to cook, how long will it take him to cook the rest? A

Original: [Figure]
Revised: [Figure]

Original: In-Context Reasoning
Revised: Let’s think step by step. Let’s think step by step. The chef has already cooked 8 potatoes. That means it has taken him 8 * 9 minutes to cook those 8 potatoes.

Revised: That means it will take him 9 minutes to cook each of the remaining 7 potatoes.

Revised: That means it will take him a total of 8 * 9 + 7 * 9 = 135 minutes to cook …

Revised: [Figure]

Revised: [Figure]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: If

Revised: [Table]

Revised: A:

Revised: [Table]

Revised: think

Revised: [Table]

Revised: cost $

Revised: [Table]

Revised: [Table]

Revised: Now we just need to add the two numbers together

Original: [Table]
Revised: [Table]

Revised: Generating Chains

Original: [Figure]
Revised: [Figure]

Revised: In-Context Reasoning

Revised: Q: Wendy uploaded

Original: [Table]
Revised: [Table]

Revised: She …

Revised: [Table]

Revised: : At a restaurant

Revised: [Table]

Revised: and kids…

Original: [Figure]
Revised: [Figure]

Revised: Q: Wendy uploaded

Revised: [Table]

Revised: to

Revised: [Table]

Revised: Q

Revised: [Table]

Revised: At

Revised: [Table]

Revised: [Figure]

Original:  for Each 
Revised: Demos for Each Test Q: [Table] If

Revised: [Table]

Revised: A:

Original: [Table]
Revised: [Table]

Revised: has

Original: [Figure]
Revised: [Table] [Table] Each [Table]

Revised: him

Original: 2014), (iv) AQUA (Ling et al., 2017), (v) SingleEq (Koncel-Kedziorski et al., 2015), and (vi)
Revised: [Table] minutes to [Table] answer

Original: SVAMP (Patel et al., 2021).
Revised: [Table] [Figure] (Patel et al., 2021).

Original: The first three are from the classic Math World Problem Repository (Koncel-Kedziorski et al., 2016), and the last three are from more recent benchmarks.
Revised: MultiArith, AddSub, and SingleEq are from the classic Math World Problem Repository (Koncel-Kedziorski et al., 2016), and the other three are from more recent benchmarks.

Original: Commonsense Reasoning.
Revised: Commonsense Reasoning.

Original: For commonsense reasoning, we use (i) CommonsenseQA
Revised: For commonsense reasoning, we use (i) CommonsenseQA

Original: (CSQA) (Talmor et al., 2019) and (ii) StrategyQA
Revised: (CSQA) (Talmor et al., 2019) and (ii) StrategyQA

Original: (Geva et al., 2021).
Revised: (Geva et al., 2021).

Original: CommonsenseQA asks questions with complex semantics that often require reasoning based on prior knowledge (Talmor et al., 2019).
Revised: CommonsenseQA asks questions with complex semantics that often require reasoning based on prior knowledge (Talmor et al., 2019).

Original: StrategyQA requires models to infer an implicit multi-hop reasoning to answer questions (Geva et al., 2021).
Revised: StrategyQA requires models to infer an implicit multi-hop reasoning to answer questions (Geva et al., 2021).

Original: Symbolic Reasoning.
Revised: Symbolic Reasoning.

Original: For symbolic reasoning, we use (i)
Revised: For symbolic reasoning, we use (i)

Original: Last Letter Concatenation (Wei et al., 2022b) and (ii) Coin Flip tasks (Wei et al., 2022b).
Revised: Last Letter Concatenation (Wei et al., 2022b) and (ii) Coin Flip tasks (Wei et al., 2022b).

Original: Last letter Concatenation requires the model to concatenate the last letters of each word.
Revised: Last letter Concatenation requires the model to concatenate the last letters of each word.

Original: The goal of Coin Flip is to answer whether a coin is still heads up after people either flip or do not flip the coin.
Revised: The goal of Coin Flip is to answer whether a coin is still heads up after people either flip or do not flip the coin.

Original: [Table]
Revised: [Table]

Original: B. I MPLEMENTATION D ETAILS
Revised: B. I MPLEMENTATION D ETAILS

Original: We use GPT-3 (Brown et al., 2020) of the text-davinci-002 version with 175B parameters for the
Revised: We use GPT-3 (Brown et al., 2020) of the text-davinci-002 version with 175B parameters for the

Original: LLM (Ouyang et al., 2022) unless otherwise stated.
Revised: LLM (Ouyang et al., 2022) unless otherwise stated.

Original: We select the model because it is public and is widely used to assess the ability of CoT reasoning in LLMs (Wei et al., 2022b; Kojima et al., 2022).
Revised: We select the model because it is public and is widely used to assess the ability of CoT reasoning in LLMs (Wei et al., 2022b; Kojima et al., 2022).

Original: The model is accessed via the OpenAI API.
Revised: The model is accessed via the OpenAI API.

Original: Greedy decoding is used to generate the output.
Revised: Greedy decoding is used to generate the output.

Original: We set max_tokens = 256 and temperature = 0.
Revised: We set max_tokens = 256 and temperature = 0.

Original: Following Wei et al.
Revised: Following Wei et al.

Original: (2022b), the number of demonstrations k used for in-context learning is 8 in most tasks, except for 4 in AQuA and Last Letter Concatenation, 7 in CSQA, and 6 in StrategyQA.
Revised: (2022b), the number of demonstrations k used for in-context learning is 8 in most tasks, except for 4 in AQuA and Last Letter Concatenation, 7 in CSQA, and 6 in StrategyQA.

Original: For the Codex LLM, the instructions are the same as those for GPT-3.
Revised: For the Codex LLM, the instructions are the same as those for GPT-3.

Original: We find that those instructions work well for the Codex model.
Revised: We find that those instructions work well for the Codex model.

Original: The only exception is that we set an extra argument stop=["Q:"] (as the end-of-sequence token in generation) when invoking the OpenAI API to avoid generating additional questions after answering the test question.
Revised: The only exception is that we set an extra argument stop=["Q:"] (as the end-of-sequence token in generation) when invoking the OpenAI API to avoid generating additional questions after answering the test question.

Original: We report mean and standard deviations for our main results.
Revised: We report mean and standard deviations for our main results.

Original: For Random-Q-CoT, we run the experiments with three different random seeds.
Revised: For Random-Q-CoT, we run the experiments with three different random seeds.

Original: For Retrieval-Q-CoT, we run the experiments by setting "temperature=0.7".
Revised: For Retrieval-Q-CoT, we run the experiments by setting "temperature=0.7".

Original: For Auto-CoT, there may be slight randomness in question clustering (e.g., different orders of demonstration), so we also run the experiments with three different random seeds.
Revised: For Auto-CoT, there may be slight randomness in question clustering (e.g., different orders of demonstration), so we also run the experiments with three different random seeds.

Original: C A NALYSIS C. V ISUALIZATION OF Q UESTION C LUSTERING
Revised: C A NALYSIS C. V ISUALIZATION OF Q UESTION C LUSTERING

Original: Figure 11 visualizes question clustering (with PCA projection) in ten datasets.
Revised: Figure 11 visualizes question clustering (with PCA projection) in ten datasets.

Original: The illustration indicates that there exist generic patterns, where different patterns may be characterized by questions from different clusters.
Revised: The illustration indicates that there exist generic patterns, where different patterns may be characterized by questions from different clusters.

Original: A few example questions from different clusters for MultiArith are presented in Figure 12. 
Revised: A few example questions from different clusters for MultiArith are presented in Figure 12. [Figure]

Original: [Figure]
Revised: [Figure]

Revised: C

Original: [Figure]
Revised: [Figure]

Original: C. F ULL ABLATION STUDY RESULTS ON DESIGN COMPONENTS
Revised: F ULL ABLATION STUDY RESULTS ON DESIGN COMPONENTS

Original: Following the discussion in Section 5.3 that both design components of clustering and simple heuristics are effective and essential, Table 10 shows full ablation study results [Table]
Revised: Following the discussion in Section 5.3 that both design components of clustering and simple heuristics are effective and essential, Table 10 shows full ablation study results.

Original: [Table]
Revised: [Table]

Original: As discussed in Section 3, the key challenge of Auto-CoT lies in automatically constructing demonstrations with good questions and their reasoning chains.
Revised: As discussed in Section 3, the key challenge of Auto-CoT lies in automatically constructing demonstrations with good questions and their reasoning chains.

Original: This challenge can be addressed effectively by the two design components: (i) clustering helps obtain the most representative (diverse) questions; (ii) simple heuristics reduce the risk of sampling wrong reasoning chains (more details in
Revised: This challenge can be addressed effectively by the two design components: (i) clustering helps obtain the most representative (diverse) questions; (ii) simple heuristics reduce the risk of sampling wrong reasoning chains (more details in

Original: Appendix C.3).
Revised: Appendix C.3).

Original: C. E FFECTIVENESS OF THE SIMPLE HEURISTICS
Revised: C. E FFECTIVENESS OF THE SIMPLE HEURISTICS

Original: In Section 4, we apply simple heuristics to encourage the model to use simple and accurate demonstrations.
Revised: In Section 4, we apply simple heuristics to encourage the model to use simple and accurate demonstrations.

Original: Similar to the criteria of the hand-crafting demonstrations in Wei et al.
Revised: Similar to the criteria of the hand-crafting demonstrations in Wei et al.

Original: (2022b)
Revised: (2022b)

Original: , our selection criteria follow simple heuristics to encourage sampling simpler questions and rationales: set the selected demonstrationd ( i ) as d ( i )
Revised: , our selection criteria follow simple heuristics to encourage sampling simpler questions and rationales: set the selected demonstration

Original: j if it has a question q ( i ) j with no more than 60 tokens and a rationale r ( i ) j with no more than 5 reasoning steps.
Revised: d i as d ij if it has a question q ij with no more than 60 tokens and a rationale r ij with no more than 5 reasoning steps.

Original:  For arithmetic reasoning tasks except for 
Revised: 4 For arithmetic reasoning tasks except for AQuA

Original: AQuA (because it is a multiple-choice problem), we require that a ( i ) j is not empty and appears in r ( i ) j to mitigate the risk of rationale-answer mismatches (as we find that such mistakes are harmful in Appendix A.1).
Revised: (because it is a multiple-choice problem), we require that a ij is not empty and appears in r ij to mitigate the risk of rationale-answer mismatches (as we find that such mistakes are harmful in Appendix A.1).

Original: If the question, rationale, and the answer satisfy the conditions above, then a candidate demonstration d ( i ) j for the i -th cluster is constructed by concatenating the question, rationale, and answer: [ Q: q ( i ) j , A: r ( i ) j ◦ a ( i ) j ] .
Revised: If the question, rationale, and the answer satisfy the conditions above, then a candidate demonstration d ij for the i -th cluster is constructed by concatenating the question, rationale, and answer: [ Q: q ij , A: r ij ◦ a ij ] .

Original: [Table]
Revised: [Table]

Original: We run the demonstration construction process three times before and after using simple heuristics to quantify its effect.
Revised: We run the demonstration construction process three times before and after using simple heuristics to quantify its effect.

Original: Table 11 shows the comparison.
Revised: Table 11 shows the comparison.

Original: The simple heuristics reduce the average number of wrong rationales in constructing demonstrations.
Revised: The simple heuristics reduce the average number of wrong rationales in constructing demonstrations.

Original: Figure 13 further depicts the error rate with and without the simple heuristics.
Revised: Figure 13 further depicts the error rate with and without the simple heuristics.

Original: The error rate is computed by the average number of wrong rationales divided by the number of demonstrations.
Revised: The error rate is computed by the average number of wrong rationales divided by the number of demonstrations.

Original: We see that our method can keep the error rate below 20% in most tasks ( 7 / 10 ).
Revised: We see that our method can keep the error rate below 20% in most tasks ( 7 / 10 ).

Original: [Figure]
Revised: [Figure]

Original: Simple heuristics w/o heuristics
Revised: [Figure] [Figure]

Original: C. C OMPARISONS OF CRITERIA FOR SORTING QUESTIONS
Revised: C. C OMPARISONS OF CRITERIA FOR SORTING QUESTIONS

Original: We compare different ways of sorting questions in each cluster, including: (i) minimal distance to the cluster center (In-Cluster Min Dist, as adopted in Auto-CoT), (ii) maximal distance to the cluster center (In-Cluster Max Dist), and (iii) random sampling inside the cluster (In-Cluster Random).
Revised: We compare different ways of sorting questions in each cluster, including: (i) minimal distance to the cluster center (In-Cluster Min Dist, as adopted in Auto-CoT), (ii) maximal distance to the cluster center (In-Cluster Max Dist), and (iii) random sampling inside the cluster (In-Cluster Random).

Original: To alleviate the influence of wrong demonstrations, we only sample the demonstrations with correct answers for this analysis.
Revised: To alleviate the influence of wrong demonstrations, we only sample the demonstrations with correct answers for this analysis.

Original: [Table]
Revised: [Table]

Original: Comparing the results in Table 12, we see that the demonstrations are generally better if they are closer to the cluster center.
Revised: Comparing the results in Table 12, we see that the demonstrations are generally better if they are closer to the cluster center.

Original: C. E FFECT OF THE NUMBER OF CLUSTERS
Revised: C. E FFECT OF THE NUMBER OF CLUSTERS

Original: [Figure]
Revised: [Figure]

Original: Our method is robust against k-means.
Revised: Our method is robust against k-means.

Original: The number of clusters k is equal to our desired number of demonstrations.
Revised: The number of clusters k is equal to our desired number of demonstrations.

Original: For fair comparisons with Manual-CoT, we use the same number of desired demonstrations ( k ) in Manual-CoT without tuning it.
Revised: For fair comparisons with Manual-CoT, we use the same number of desired demonstrations ( k ) in Manual-CoT without tuning it.

Original: To investigate the effect of k , we vary its value to obtain different numbers of demonstrations.
Revised: To investigate the effect of k , we vary its value to obtain different numbers of demonstrations.

Original: As shown in Figure 14, the improvement of Auto-CoT over Random-Q-CoT remains robust against the varying values of k .
Revised: As shown in Figure 14, the improvement of Auto-CoT over Random-Q-CoT remains robust against the varying values of k .

Original: Our findings in Figure 14 agree with Wei et al.
Revised: Our findings in Figure 14 agree with Wei et al.

Original: (2022b) on the number of desired demonstrations (same as the number of clusters: k ).
Revised: (2022b) on the number of desired demonstrations (same as the number of clusters: k ).

Original: Therefore, k = 8 is an effective default value, and using 8 demonstrations is far below what LLMs can handle (e.g., up to 2,048 tokens in GPTmodels).
Revised: Therefore, k = 8 is an effective default value, and using 8 demonstrations is far below what LLMs can handle (e.g., up to 2,048 tokens in GPTmodels).

Original: C. U SING OTHER LLM S
Revised: C. U SING OTHER LLM S

Original: We implement Auto-CoT using another 175B version of GPT-3 (text-davinci-001).
Revised: We implement Auto-CoT using another 175B version of GPT-3 (text-davinci-001).

Original: The text-davinci001 is a fine-tuned version of GPT-3 and it can be seen as an earlier version of text-davinci-002.
Revised: The text-davinci001 is a fine-tuned version of GPT-3 and it can be seen as an earlier version of text-davinci-002.

Original: It has a weaker Zero-Shot-CoT performance than our previously adopted text-davinci-002.
Revised: It has a weaker Zero-Shot-CoT performance than our previously adopted text-davinci-002.

Original: Tableshows that Auto-CoT still attains the most competitive performance under text-davinci-001.
Revised: Tableshows that Auto-CoT still attains the most competitive performance under text-davinci-001.

Original: [Table]
Revised: [Table]

Original: C.
Revised: E XTENDED : M ORE C HALLENGING S TREAMING S ETTING

Original: E XTENDED : M ORE C HALLENGING S TREAMING S ETTING

Original: In Section 5.6, we discussed the application of Auto-CoT in a more challenging streaming setting where a small batch of test questions (say m questions) arrive at a time like in data streams.Batch
Revised: In Section 5.5, we discussed the application of Auto-CoT in a more challenging streaming setting where a small batch of test questions (say m questions) arrive at a time like in data streams.

Original: [Figure]
Revised: [Figure]

Original: y ( % ) [Table] [Figure] Due to page space limits, we only showed the results of the first 10 batches (300 test questions in total) in Section 5.6.
Revised: Due to page space limits, we only showed the results of the first 10 batches (300 test questions in total) in Section 5.5.

Original: In Figure 15, we illustrate the accuracy of each batch on all the 600 test questions in
Revised: In Figure 15, we illustrate the accuracy of each batch on all the 600 test questions in

Original: MultiArith.
Revised: MultiArith.

Original: As expected, for batch 1 , Auto-CoT* and Zero-Shot-CoT obtain equal accuracy.
Revised: As expected, for batch 1 , Auto-CoT* and Zero-Shot-CoT obtain equal accuracy.

Original: From batch 2 , Auto-CoT* quickly performs comparably with Manual-CoT.
Revised: From batch 2 , Auto-CoT* quickly performs comparably with Manual-CoT.

Original: This result indicates that our method is still effective in the more challenging streaming setting.
Revised: This result indicates that our method is still effective in the more challenging streaming setting.

Original: D E XTENDED E XPERIMENTS D. A PPLYING A UTO -C O T TO SAMPLE DEMONSTRATIONS WITH MANUALLY DESIGNED C O T
Revised: D E XTENDED E XPERIMENTS D. A PPLYING A UTO -C O T TO SAMPLE DEMONSTRATIONS WITH MANUALLY DESIGNED C O T

Original: The AQuA and GSM datasets have training sets with manually designed reasoning chains.
Revised: The AQuA and GSM datasets have training sets with manually designed reasoning chains.

Original: We are interested in whether it is possible to achieve better results by applying Auto-CoT to sample the examples with manually designed CoT from the training set.
Revised: We are interested in whether it is possible to achieve better results by applying Auto-CoT to sample the examples with manually designed CoT from the training set.

Original: Therefore, we cluster the questions from the training set and use the gold reasoning chains as our demonstrations.
Revised: Therefore, we cluster the questions from the training set and use the gold reasoning chains as our demonstrations.

Original: Results in Tableshow that Auto-CoT can be further enhanced if manually designed CoT is available
Revised: Results in Tableshow that Auto-CoT can be further enhanced if manually designed CoT is available.

Original: [Table]

Original: [Table]
Revised: [Table]

Original: D. C OMBINING R ETRIEVAL -Q-C O T AND R ANDOM -Q-C O T
Revised: D. C OMBINING R ETRIEVAL -Q-C O T AND R ANDOM -Q-C O T

Original: Combining Retrieval-Q-CoT and Random-Q-CoT may also benefit from diversity of demonstrations.
Revised: Combining Retrieval-Q-CoT and Random-Q-CoT may also benefit from diversity of demonstrations.

Original: We investigate a new method dubbed as “Cluster-Retrieval-CoT”.
Revised: We investigate a new method dubbed as “Cluster-Retrieval-CoT”.

Original: In detail, we first cluster the questions into k clusters.
Revised: In detail, we first cluster the questions into k clusters.

Original: Then, for each cluster, we retrieve the most similar question to the test question.
Revised: Then, for each cluster, we retrieve the most similar question to the test question.

Original: [Table]
Revised: [Table]

Original: The result is shown in Table 15.
Revised: The result is shown in Table 15.

Original: With more diversity involved, Cluster-Retrieval-CoT performs better than Retrieval-Q
Revised: With more diversity involved, Cluster-Retrieval-CoT performs better than Retrieval-Q

Original: -CoT.
Revised: -CoT.

Original: However, the performance is still inferior to
Revised: However, the performance is still inferior to

Original: Auto-CoT.
Revised: Auto-CoT.

Original: This experiment provides further evidence that using the representative skills (e.g., each cluster center) is helpful (more evidence in Table 12).
Revised: This experiment provides further evidence that using the representative skills (e.g., each cluster center) is helpful (more evidence in Table 12).

Original: If we took each demonstration as a kind of skill, representative demonstrations from diverse clusters could cover more alternative skills for solving target questions
Revised: If we took each demonstration as a kind of skill, representative demonstrations from diverse clusters could cover more alternative skills for solving target questions.

Original: . D. C LUSTERING WITH C O T Q UESTION -A NSWER (QA
Revised: D. C LUSTERING WITH C O T Q UESTION -A NSWER (QA) PAIRS

Original: ) PAIRS

Original: Our clustering method is based on questions.
Revised: Our clustering method is based on questions.

Original: There are other design choices such as firstly generating the rationales for all questions and then sampling the demonstrations by clustering over rationales (i.e., using QA pairs).
Revised: There are other design choices such as firstly generating the rationales for all questions and then sampling the demonstrations by clustering over rationales (i.e., using QA pairs).

Original: We find that clustering with QA pairs slightly improves the results (Table 16).
Revised: We find that clustering with QA pairs slightly improves the results (Table 16).

Original: [Table]
Revised: [Table]

Original: However, this method needs to generate rationales for all questions.
Revised: However, this method needs to generate rationales for all questions.

Original: As there are often thousands of questions in our investigated tasks, generating the reasoning chains for those questions requires thousands of inferences.
Revised: As there are often thousands of questions in our investigated tasks, generating the reasoning chains for those questions requires thousands of inferences.

Original: Given that API access is often hidden behind the paywalls of LLM providers, there will be much more inference costs.
Revised: Given that API access is often hidden behind the paywalls of LLM providers, there will be much more inference costs.

Original: In contrast, our method only needs to generate the reasoning chain for the k (e.g., 8) sampled questions, which enjoys much more cost efficiency.
Revised: In contrast, our method only needs to generate the reasoning chain for the k (e.g., 8) sampled questions, which enjoys much more cost efficiency.

Original: Therefore, we choose the questions-only method in our work.
Revised: Therefore, we choose the questions-only method in our work.

Original: D. C OMPARISON WITH OTHER ZERO - SHOT C O T PROMPTS
Revised: D. C OMPARISON WITH OTHER ZERO - SHOT C O T PROMPTS

Original: “Let’s do this step by step” has shown to be the current best prompt for Zero-Shot-CoT yet after investigating 16 different prompts according to Kojima et al. (2022).
Revised: ‘Let’s do this step by step’ has shown to be the current best prompt for Zero-Shot-CoT yet after investigating 16 different prompts according to Kojima et al. (2022).

Original: Nonetheless, we also conduct further experiments to modify the prompts with three different ones and see if the prompt can be optimized (Table 17).
Revised: Nonetheless, we also conduct further experiments to modify the prompts with three different ones and see if the prompt can be optimized (Table 17).

Original: We find that optimizing the zero-shot prompts does not result in substantial benefits.
Revised: We find that optimizing the zero-shot prompts does not result in substantial benefits.

Original: [Table]
Revised: [Table]

Original: D. O NE - SHOT SETTING DEPENDING ON TEST QUESTIONS
Revised: D. O NE - SHOT SETTING DEPENDING ON TEST QUESTIONS

Original: It is possible to cluster samples into n -clusters ( n = 2 , . .
Revised: It is possible to cluster samples into n -clusters ( n = 2 , . .

Original: . , 10 ) and then use just one demonstration from the cluster where the test sample fits.
Revised: . , 10 ) and then use just one demonstration from the cluster where the test sample fits.

Original: As a result, we can use just one demonstration at each time.
Revised: As a result, we can use just one demonstration at each time.

Original: We find that the performance of this method (dubbed as “One-Shot”) is on par with Random-Q-CoT that leverages n demonstrations.
Revised: We find that the performance of this method (dubbed as “One-Shot”) is on par with Random-Q-CoT that leverages n demonstrations.

Original: Figure 16 shows the results on the MultiArith dataset.
Revised: Figure 16 shows the results on the MultiArith dataset.

Original: We observe that the simplified method reduces the computation cost and can be seen as a less effective yet efficient variant of Auto-CoT.
Revised: We observe that the simplified method reduces the computation cost and can be seen as a less effective yet efficient variant of Auto-CoT.

Original: [Figure]
Revised: [Figure]

Original: E A PPENDIX : A UTOMATICALLY CONSTRUCTED DEMONSTRATIONS
Revised: E A PPENDIX : A UTOMATICALLY CONSTRUCTED DEMONSTRATIONS

Original: [Table]
Revised: [Table]

Original: on

Original: [Table]

Original: many

Original: [Table]

Original: she

Original: [Table]

Original: total

Original: [Table] Let’s [Table] step by step
Revised: Let’s think step by step

Original: [Table]
Revised: [Table]

Original: There are 5 shelves of mystery books.
Revised: There are 5 shelves of mystery books.

Original: Each shelf has 8 books
Revised: Each shelf has 8 books.

Original: [Table]

Original: So that’s
Revised: So that’s

Original: [Table]
Revised: [Table]

Original: books
Revised: books.

Original: [Table]

Original: There are 4 shelves of picture books
Revised: There are 4 shelves of picture books.

Original: [Table]

Original: Each shelf has 8 books
Revised: Each shelf has 8 books.

Original: [Table] So [Table]
Revised: So that’s

Original: [Table]
Revised: [Table]

Original: books
Revised: books

Original: [Table] [Table]
Revised: [Table] [Table]

Original: uploaded 45 pictures to Facebook.
Revised: uploaded 45 pictures to Facebook.

Original: She put 27 pics into one album and put the rest into 9 different albums.
Revised: She put 27 pics into one album and put the rest into 9 different albums.

Original: How many pictures were in each album?
Revised: How many pictures were in each album?

Original: [Table]
Revised: [Table]

Original: Let’s think step by step.
Revised: Let’s think step by step.

Original: First, we know that Wendy uploaded 45 pictures in total.
Revised: First, we know that Wendy uploaded 45 pictures in total.

Original: Second, we know that Wendy put 27 pictures into one album.
Revised: Second, we know that Wendy put 27 pictures into one album.

Original: That means that Wendy put the remaining 18 pictures intodifferent [Table] [Table]
Revised: That means that Wendy put the remaining 18 pictures intodifferent albums.

Original: were born in a country [Table] and [Table]
Revised: That means that each album would have 2 pictures.

Original: immigrated to

Original: [Table]
Revised: [Table]

Original: many new

Original: [Table]
Revised: [Table]

Original: began living in the country last year

Original: [Table]
Revised: [Table]

Original: step by step

Original: [Table]

Original: First

Original: [Table]

Original: were born in the country [Table] Then [Table]
Revised: How

Original: immigrated to the country

Original: [Table]
Revised: [Table]

Original: means

Original: [Table]

Original: were
Revised: does

Original: [Table]

Original: new

Original: [Table]

Original: living in the country last year

Original: [Table]

Original: The

Original: [Table]

Original: : A construction company is repaving a damaged road .

Original: So far , they have repaved a total of 4938 inches of the road .

Original: Today , they repaved 805 inches of the road .

Original: How many inches of the road had they repaved before today ?

Original: A: Let’s think step by step.

Original: The company has repaved a total of 4938 inches of the road.

Original: Today, they repaved 805 inches of the road.

Original: So before today, they had repaved 4938 - 805 = 4133 inches of the road.

Original: The answer is 4133.

Original: [Table]

Original: : There are 5 scissors and 3 pencils in the drawer .

Original: Jason placed 4 scissors in the drawer .

Original: How many scissors are now there in total ?

Original: A: Let’s think step by step.

Original: There are 5 scissors and 3 pencils in the drawer.

Original: Jason placed 4 scissors in the drawer.

Original: Now, there are 9 scissors in the drawer.

Original: The answer is

Original: [Table]

Original: [Table]

Original: : There are 37 short bushes and 30 tall trees currently in the park .

Original: Park workers will plant 20 short bushes today .

Original: How many short bushes will the park have when the workers are finished ?

Original: [Table]
Revised: [Table]

Original: : Let’s think step by step.

Original: There are 37 short bushes and 30 tall trees currently in the park.

Original: Park workers will plant 20 short bushes today.

Original: This means that when the workers are finished, there will be a total ofshort bushes in the park.

Original: The answer is 57.

Original: [Table]

Original: Let’s think step by step.
Revised: Let’s think step by step.

Original: On the first day

Original: [Table]
Revised: [Table]

Original: Sloane counted

Original: [Table] On the second day [Table] she counted
Revised: friend gave her 24 baby outfits

Original: [Table]
Revised: [Table]

Original: more customers than she did on the first day. So, she counted 150 customers on the second day. On the third day, she counted 500 customers. Therefore, on the third day, Sloane counted 500 customers
Revised: So

Original: [Table]
Revised: [Table]

Original: friend gave her 24 baby outfits that her child no longer needed. At her baby shower, [Table] received twice theamount of new baby outfits.
Revised: received twice the

Original: Then, Laurel’s mom gifted her with another 15 baby

Original: [Table]
Revised: [Table]

Original: How many outfits does she have for her baby

Original: [Table]

Original: A: Let’s think step by step. Laurel’s friend gave her 24 baby outfits. So she has 24 baby outfits. At her baby shower, Laurel received twice the amount of new baby outfits.
Revised: baby outfits.

Original: So she has 48 baby outfits.
Revised: So she has 48 baby outfits.

Original: Then, Laurel’s mom gifted her with another 15 baby outfits.
Revised: Then, Laurel’s mom gifted her with another 15 baby outfits.

Original: So she has 63 baby outfits.
Revised: So she has 63 baby outfits.

Original: The answer is 63.
Revised: The answer is 63.

Original: Q: Pam and Fred went to a carnival.
Revised: Q: Pam and Fred went to a carnival.

Original: Pam rode the roller coaster 2 times while Fred rode it 4 times.
Revised: Pam rode the roller coaster 2 times while Fred rode it 4 times.

Original: After that, each of them decided to ride the luge 2 times.
Revised: After that, each of them decided to ride the luge 2 times.

Original: If each ride cost 6 tickets, how many tickets did they use that day?
Revised: If each ride cost 6 tickets, how many tickets did they use that day?

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: Pam rode the roller coaster 2 times.
Revised: Pam rode the roller coaster 2 times.

Original: So, that’s 2 * 6 = 12 tickets.
Revised: So, that’s 2 * 6 = 12 tickets.

Original: Fred rode the roller coaster 4 times.
Revised: Fred rode the roller coaster 4 times.

Original: So, that’s 4 * 6 = 24 tickets.
Revised: So, that’s 4 * 6 = 24 tickets.

Original: Pam rode the luge 2 times.
Revised: Pam rode the luge 2 times.

Original: So, that’s 2 * 6 = 12 tickets.
Revised: So, that’s 2 * 6 = 12 tickets.

Original: Fred rode the luge 2 times.
Revised: Fred rode the luge 2 times.

Original: So, that’s 2 * 6 = 12 tickets.
Revised: So, that’s 2 * 6 = 12 tickets.

Original: In total, they used 12 + 24 + 12 + 12 = 60 tickets.
Revised: In total, they used 12 + 24 + 12 + 12 = 60 tickets.

Original: The answer is 60.
Revised: The answer is 60.

Original: Q: The eighth-grade class held a bake-off.
Revised: Q: The eighth-grade class held a bake-off.

Original: Kelsie made two times more cookies than Josh.
Revised: Kelsie made two times more cookies than Josh.

Original: Josh made one-fourth the number of cookies that Suzanne made.
Revised: Josh made one-fourth the number of cookies that Suzanne made.

Original: If Suzanne made 36 cookies, how many did Kelsie make?
Revised: If Suzanne made 36 cookies, how many did Kelsie make?

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: Josh made one-fourth the number of cookies that Suzanne made.
Revised: Josh made one-fourth the number of cookies that Suzanne made.

Original: So if Suzanne made 36 cookies, Josh made 9 cookies.
Revised: So if Suzanne made 36 cookies, Josh made 9 cookies.

Original: Kelsie made two times more cookies than Josh.
Revised: Kelsie made two times more cookies than Josh.

Original: So if Josh madecookies, Kelsie made 18 cookies.
Revised: So if Josh madecookies, Kelsie made 18 cookies.

Original: The answer is 18.
Revised: The answer is 18.

Original: Q: Jane has saved $4 of her allowance every week for the past 8 weeks.
Revised: Q: Jane has saved $4 of her allowance every week for the past 8 weeks.

Original: If she continues to save the same amount each week, how many more weeks will it take for her to have saved a total of $60?
Revised: If she continues to save the same amount each week, how many more weeks will it take for her to have saved a total of $60?

Original: A: Let’s think step by step.
Revised: A: Let’s think step by step.

Original: Jane has saved $4 of her allowance every week for the past 8 weeks.
Revised: Jane has saved $4 of her allowance every week for the past 8 weeks.

Original: That means she has $32 in savings so far.
Revised: That means she has $32 in savings so far.

Original: $60 - $32 = $28 $28 / $4 = 7 It will take Jane 7 more weeks to have saved a total of $60.
Revised: $60 - $32 = $28 $28 / $4 = 7 It will take Jane 7 more weeks to have saved a total of $60.

Original: The answer is 7.
Revised: The answer is 7.

Original: Q: Mia and Emma are currently 16 years apart in age.
Revised: Q: Mia and Emma are currently 16 years apart in age.

Original: If Mia, who is younger than Emma, is 40 years old, what’s the average of their ages?
Revised: If Mia, who is younger than Emma, is 40 years old, what’s the average of their ages?

Original: A: Let’s think step by step
Revised: A: Let’s think step by step.

Revised: Mia is currently 16 years old.

Revised: Mia is younger than

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: had

Original: [Table]
Revised: [Table]

Revised: for

Original: [Table]
Revised: [Table]

Original: animals does each aquarium have?
Revised: animals. The aquarium had [Table] had

Revised: [Table]

Revised: same number

Original: A: We know that there are 26 aquariums and 52 animals in total.
Revised: [Table] have [Table] know that there are 26 aquariums and 52 animals in total.

Original: That means that if we divide 52 by 26, we will know how many animals are in each aquarium.
Revised: That means that if we divide 52 by 26, we will know how many animals are in each aquarium.

Original: 52 divided by 26 equals 2.
Revised: 52 divided by 26 equals 2.

Original: So, each aquarium hassaltwater animals.
Revised: So, each aquarium hassaltwater animals.

Original: The answer is 2.
Revised: The answer is 2.

Original: Q:
Revised: Q:

Original: A mailman has to give 25 pieces of junk mail to each block.
Revised: A mailman has to give 25 pieces of junk mail to each block.

Original: If he gives 5 mails to each house in a block How many houses are there in a block?
Revised: If he gives 5 mails to each house in a block How many houses are there in a block?

Original: A:
Revised: A:

Original: The mailman has to give 25 pieces of junk mail to each block.
Revised: The mailman has to give 25 pieces of junk mail to each block.

Original: That means if the mailman gives 5 pieces of junk mail to each house, he would have given away 25 pieces of junk mail after visiting 5 houses.
Revised: That means if the mailman gives 5 pieces of junk mail to each house, he would have given away 25 pieces of junk mail after visiting 5 houses.

Original: Therefore, if the mailman has to give 25 pieces of junk mail to each block, he would have to visit 5 houses in a block.
Revised: Therefore, if the mailman has to give 25 pieces of junk mail to each block, he would have to visit 5 houses in a block.

Original: The answer is 5.
Revised: The answer is 5.

Original: Q: Every day Ryan spends 3 hours on learning english and some more hours on learning chinese.
Revised: Q: Every day Ryan spends 3 hours on learning english and some more hours on learning chinese.

Original: If he spends a total of 4 hours on learning english and chinese everyday How many hours does he spend on learning chinese?
Revised: If he spends a total of 4 hours on learning english and chinese everyday How many hours does he spend on learning chinese?

Original: A:
Revised: A:

Original: First, we know that Ryan spends 3 hours on learning English every day.
Revised: First, we know that Ryan spends 3 hours on learning English every day.

Original: Second, we know that Ryan spends a total of 4 hours on learning English and Chinese every day.
Revised: Second, we know that Ryan spends a total of 4 hours on learning English and Chinese every day.

Original: From this, we can infer that Ryan spends 1 hour on learning Chinese every day.
Revised: From this, we can infer that Ryan spends 1 hour on learning Chinese every day.

Original: The answer is 1.
Revised: The answer is 1.

Original: [Table]
Revised: [Table]

Original: the answer. It also won’t fit in a juice container, so that’s not the answer.
Revised: container, so that’s not the answer.

Original: It could fit in a factory, but that’s not necessarily where you would store it.
Revised: It could fit in a factory, but that’s not necessarily where you would store it.

Original: It could also fit in a supermarket, but that’s not necessarily where you would store it.
Revised: It could also fit in a supermarket, but that’s not necessarily where you would store it.

Original: The most likely answer is a hostel.
Revised: The most likely answer is a hostel.

Original: The answer is D. Q: If there is a pond with trees around it, where it it likely located?
Revised: The answer is D. [Table] : If there is a pond with trees around it, where it it likely located [Table]

Revised: [Table]

Revised: bathroom

Original: Answer Choices: (A) ground (B) bathroom (C) forest (D) countryside(E) rural area
Revised: [Table] forest [Table] countryside

Revised: [Table]

Revised: rural

Revised: [Table]

Original: A: Let’s think step by step.
Revised: : Let’s think step by step.

Original: A pond is likely to be located near trees because trees need water to survive.
Revised: A pond is likely to be located near trees because trees need water to survive.

Original: Therefore, the most likely location for a pond with trees around it is in a forest.
Revised: Therefore, the most likely location for a pond with trees around it is in a forest.

Original: The answer is C. Q: James was meeting a friend.
Revised: The answer is C. [Table]

Original: Theyhad planed a slow day. They didn’t want to do much. They just wanted what? Answer Choices: (A) relaxation (B) panic
Revised: They

Original: [Table]
Revised: [Table]

Original: C)
Revised: They

Original: [Table]
Revised: [Table]

Original: D
Revised: They just wanted

Original: [Table]
Revised: [Table]

Original: joy
Revised: Let’s think

Original: [Table]
Revised: [Table]

Original: E) cheer A: Let’s think step by step.

Original: James was meeting a friend.

Original: They had planned a slow day.

Original: Theydidn’t want to do much. They just wanted _____. The only logical answer choice is relaxation. The answer is A. Q: what does a person do to stay healthy?
Revised: They

Original: Answer Choices: (A) fever (

Original: [Table] [Table]
Revised: [Table] [Table]

Original: every day
Revised: only

Original: [Table][Table]
Revised: [Table]

Original: headache (E

Original: [Table]
Revised: [Table]

Original: expressive A: Let’s think step by step. To stay healthy, a person should probably do things like eating healthy foods, exercising, and getting enough rest. So the answer choices that best fit are (B) eat every day, (C) excited, and (D) headache.The answer is B.
Revised: headache.

Original: [Table]
Revised: [Table]

Original: something
Revised: answer

Original: [Table]
Revised: [Table]

Original: A

Original: [Table]
Revised: [Table]

Original: Let’s think step by step.
Revised: Let’s think step by step.

Original: Chick-fil-A is a restaurant chain that specializes in chicken sandwiches.
Revised: Chick-fil-A is a restaurant chain that specializes in chicken sandwiches.

Original: So right off the bat, we can say that a vegetarian probably wouldn’t be able to find a sandwich that they could eat.
Revised: So right off the bat, we can say that a vegetarian probably wouldn’t be able to find a sandwich that they could eat.

Original: However
Revised: However,

Original: [Table]
Revised: Chick -fil-A

Original: fil-A does offer salads and sides [Table] vegetarian could eat.
Revised: does offer salads and sides that a vegetarian could eat

Original: The answer is yes.
Revised: The answer is yes

Original: [Table]
Revised: [Table]

Original: Can Justin Timberlake ride Shipwreck Falls at Six
Revised: Timberlake ride [Table] Falls [Table]

Revised: Six

Original: [Table]
Revised: [Table]

Original: A: Let’s think step by step.
Revised: Let’s think step by step

Original: First, let’s consider the height requirement for Shipwreck Falls, which is 48 inches.
Revised: First , let’s consider the height requirement for Shipwreck Falls, which is 48 inches.

Original: Next, we’ll look up Justin Timberlake’s height.
Revised: Next, we’ll look up Justin Timberlake’s height.

Original: According to various sources, Timberlake is between 5’8" and 5’11".
Revised: According to various sources, Timberlake is between [Table] and 5’ [Table]

Original: Finally, we’ll compare Timberlake’s height to the height requirement for Shipwreck Falls.
Revised: Finally, we’ll compare Timberlake’s height to the height requirement for Shipwreck Falls.

Original: Since Timberlake is taller than 48 inches, he would be able to ride Shipwreck Falls.
Revised: Since Timberlake is taller than 48 inches [Table] he would be able to ride Shipwreck Falls.

Original: The answer is yes.
Revised: The answer is yes

Revised: [Table]

Revised: have

Revised: [Table]

Original: Q: Would a snake have reasons to fear a honey badger? 
Revised: fear a [Table] badger? [Table]

Original: A: Let’s think step by step.
Revised: Let’s think step by step.

Original: A snake’s main method of attack is to coil around an animal and constrict it until it suffocates.
Revised: A snake’s main method of attack is to coil around an animal and constrict it until it suffocates.

Original: This would be difficult to do to a honey badger, which has thick skin and is very agile.
Revised: This would be difficult to do to a honey badger, which has thick skin and is very agile.

Original: In addition, honey badgers are known to be very aggressive, and they have sharp claws and teeth that could do a lot of damage to a snake.
Revised: In addition, honey badgers are known to be very aggressive, and they have sharp claws and teeth that could do a lot of damage to a snake.

Original: So, it is reasonable to say that a snake would have reasons to fear a honey badger.
Revised: So, it is reasonable to say that a snake would have reasons to fear a honey badger.

Original: The answeris yes.
Revised: The answer

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: : Will Queen Elizabeth be buried in the Pantheon?
Revised: Let’s think step

Original: [Table]
Revised: [Table]

Original: : Let’s think step by step.
Revised: step [Table] the [Table]

Original: First, the Pantheon is a church, so it’s possible that she could be buried there.
Revised: is a church [Table] so it’s possible that she could be buried [Table]

Original: Second, Queen Elizabeth II is still alive, so she hasn’t been buried yet.
Revised: is still alive [Table] so she hasn’t been buried yet [Table]

Revised: Third

Original: Third, even if she were to be buried in the Pantheon, it’s unlikely that we would know about it ahead of time, so it’s hard to say for sure.
Revised: [Table] even if she were to be buried in the [Table] it’s unlikely that we would know about it ahead of time

Original: The answer is no.
Revised: [Table] it’s hard to say for sure

Original: Q: Can you see the moon in Wembley Arena?
Revised: [Table] the moon in Wembley [Table]

Original: A: Let’s think step by step.
Revised: Let’s think step by step.

Original: Wembley Arena is an indoor venue 
Revised: Wembley Arena is an indoor venue so the moon would only be visible if there was a

Original: [Table] [Table]
Revised: [Table] [Table]

Revised: The answer

Original: [Table]
Revised: [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]