STaR: Self-Taught Reasoner Bootstrapping Reasoning With Reasoning
Anonymous Author(s)
AffiliationAddress email
Abstract
Generating step-by-step "chain-of-thought" rationales improves language modelperformance on complex reasoning tasks like mathematics or commonsensequestion-answering.
However, inducing language model rationale generation currently requires either constructing massive rationale datasets or sacrificing accuracyby using only few-shot inference.
We propose a technique to iteratively leverage asmall number of rationale examples and a large dataset without rationales, to bootstrap the ability to perform successively more complex reasoning.
This technique,the "Self-Taught Reasoner" (STaR), relies on a simple loop: generate rationales toanswer many questions, prompted with a few rationale examples; if the generatedanswers are wrong, try again to generate a rationale given the correct answer; finetune on all the rationales that ultimately yielded correct answers; repeat.
We showthat STaR significantly improves performance on multiple datasets compared to amodel fine-tuned to directly predict final answers, and performs comparably to finetuning a 30 × larger state-of-the-art language model on CommensenseQA.
Thus,
STaR lets a model improve itself by learning from its own generated reasoning.
Introduction
Human decision-making is often the result of extended chains of thought [1, 2].
Recent work hasshown that explicit intermediate reasoning (“rationales”) can improve large language model (LLM)performance as well [3–8].
For example, [5] demonstrated that LLMs explicitly trained to use“scratchpads” for intermediate steps can attain perfect in-distribution performance on arithmetic,and strong out-of-distribution generalization, while models trained to predict answers directly failto do either.
These works suggest that generating explicit rationales before giving a final answer(“rationale generation”) is valuable for LLMs across diverse tasks including mathematical reasoning,commonsense reasoning, code evaluation, social bias inference and natural language inference.
However, the two primary methods for inducing rationale generation both have serious drawbacks.
One approach to rationale generation is the construction of a fine-tuning dataset of rationales, eithermanually by human annotators or automatically with hand-crafted templates [3–5, 9].
Manualmethods are expensive, and it is infeasible to construct such a dataset for each interesting problem[3].
Meanwhile, template-based methods rely on automatically-generated rationales but only workwhen a general solution is already known [5] or reasonable hard-coded heuristics can be made [4].
An alternative is to leverage in-context learning by including only a few rationale examples in thelanguage model prompt.
This has been shown to improve accuracy on mathematical and symbolicreasoning tasks relative to prompting without rationales (“direct” prompting) [5, 6].
Yet, while fewshot techniques with rationales tend to outperform their non-reasoning counterparts, they generallysubstantially underperform models fine-tuned to directly predict answers using larger datasets [5, 6].
Question
Question, Rationale, Answer
[Figure]
A
[Figure]
The
[Figure]
something
[Figure]
can be used to carry
[Figure]
designed to hold things.
Therefore, the answer is basket (b).
In this paper, we adopt a different approach: by leveraging the LLM’s pre-existing reasoning ability,we iteratively bootstrap the ability to generate high-quality rationales.
Specifically, we few-shotprompt a large language model to self-generate rationales and refine the model’s ability further byfine-tuning on those rationales that lead to correct answers.
We repeat this procedure, using theimproved model to generate the next training set each time.
This is a synergistic process, whereimprovements in rationale generation improve the training data, and improvements in training datafurther improve rationale generation.
However, we find this loop eventually fails to solve any new problems in the training set because itreceives no direct training signal for problems it fails to solve.
To overcome this issue, we proposerationalization : for each problem that the model fails to answer correctly, we generate a new rationaleby providing the model with the correct answer.
This lets the model reason backward—given thecorrect answer, the model can more easily generate a useful rationale.
These rationales are thencollected as part of the training data, which often improves overall accuracy.
We thus develop the Self-Taught Reasoner (STaR, Fig. 1) method, a scalable bootstrapping methodallowing models to learn to generate their own rationales, while also learning to solve increasinglydifficult problems.
In our method, we repeat the following process: in each iteration, first construct afinetuning dataset by attempting to solve the dataset using the current model’s rationale generationability; then, augment this dataset using rationalization , justifying ground-truth answers to problemsthe model failed to solve; finally, finetune the large language model on the combined dataset.
Applying STaR on arithmetic, math word problems, and commonsense reasoning, we observe it isable to effectively translate a small number of few-shot prompts into a large rationale dataset, yieldingdramatic performance improvements.
On CommonsenseQA [10], we find STaR improves over botha few-shot baseline (+35.9%) and a baseline fine-tuned to directly predict answers (+12.5%) , andperforms comparably to a fine-tuned model that is 30 × larger (72.5% vs. 73.0%).
Thus, we make the following contributions:
We propose a bootstrapping mechanism to iteratively generate a rationale dataset from a few
initial examples with rationales—without needing to check new rationales’ correctness.
We complement rationale generation with rationalization , where a model is tasked with
justifying an answer and then fine-tuned as if it had come up with the rationale without anyhint.
We show rationalization accelerates and improves the bootstrapping process.
We evaluate these techniques with a variety of ablations in both mathematical and common-
sense reasoning domains.
We propose what is, to our knowledge, the first technique to allow a pre-trained large
language model to iteratively use its language modeling capacity to improve itself.
Background and Related Work
In-context Learning Recently, a collection of works has emerged exploring the capacity for largelanguage models to perform in-context learning [11, 12].
In essence, in-context learning treatsfew-shot learning as a language modeling problem, by showing a few examples in the context (i.e.prompt), and allowing the model to learn and identify the pattern to apply to new examples.
Somehave studied in-context learning based on the language modeling objective in terms of Bayesianinference [13] while others have attempted to describe the process more mechanistically in termsof “induction heads” [14].
Moreover, differences in prompt configurations have been known to havedramatic effects on few-shot performance.
Some have even found that replacing few-shot promptswith a “soft prompt” which can be optimized in embedding space results in noticeable gains [15].
Instead of emphasizing the representation of the question, we focus on the model output; in particular,we focus on the model’s ability to reason through a problem before coming to a conclusion.
Rationales One of the initial works on the impact of rationales on language model performance was[3], showing that training a language model on a dataset with explicit rationales preceding the answercould improve a model’s ability to generate the final answer.
However, this required many thousandsof training examples to be manually annotated with human reasoning.
Recently, [5] demonstratedthat step-by-step “scratchpads” can improve fine-tuned LLM performance and generalization ontasks such as arithmetic, polynomial evaluation, and program evaluation.
Similarly, [6] used a singlefew-shot “chain-of-thought” reasoning prompt in order to improve model performance on a collectionof tasks, without fine-tuning.
Finally, [16] showed that a curriculum learning approach could helpsolve formal math problems, as long as 1) they were translated into Lean (a theorem-proving language[17]), 2) one could directly evaluate the validity of the proofs, 3) one could sample numerous potentialsolutions for each problem, 4) had trained a separate value function model, and 5) started with GPT-f(a model already fine-tuned on a large math dataset [18]).
We note that there are many domains wherethese conditions do not all apply.
In addition, works have aimed to explain why rationales have thisbeneficial effect: some have analyzed their impact from the perspective of latent variable models [19]while others have provided formal proofs of the benefit of intermediate task supervision [20].
Iterated Learning A variety of iterated learning algorithms have been proposed, where solutionsor successful methods which are found are in turn used to find additional solutions [21, 22, 16]. [21]introduced Expert Iteration (ExIt), a reinforcement learning technique serving as an inspiration forour approach.
Essentially, it consists of a loop of self-play by an “apprentice,” followed by imitationlearning with feedback from a slower “expert” and then the replacement of the expert with the nowimproved apprentice.
[16] builds off of ExIt for formal reasoning, while [22] applies iterated learningto visual question answering using modular networks which can be combined compositionally.
Thereare further similarities between STaR and expert iteration methods [21].
For example, filteringgenerated examples based on whether their ultimate answer matches the target can be seen as expertfeedback.
However, we have a fixed “expert” and do not train a separate value function.
Natural Language Explanations Natural language explanations have also been discussed from theperspective of explainable machine learning, focusing on justification rather than reasoning [23, 24].
The motivation for this line of work is largely grounded in explainable decision making, and similarlyto [3], generally does not find that requiring post-hoc explanations improves model performance.
Method
3.1 Rationale Generation Bootstrapping (STaR Without Rationalization)
We are given a pretrained LLM M and an initial dataset of problems x with answers y :
D ={ ( x i , y i ) } Di =1 .
Our technique starts with a small prompt set P of examples with intermediate rationales r : P = { ( x pi , r pi , y pi ) } Pi =1 , where P ≪ D (e.g. P = 10 ).
Like standard few-shot prompting, weconcatenate this prompt set to each example in D , i.e. x i =
( x p 1 , r p 1 , y p 1 , . . .
, x pP , r pP , y pP , x i ) , whichencourages the model to produce a rationale ˆ r i for x i followed by an answer ˆ y i .
We assume thatrationales that lead to correct answers are of better quality than those that lead to incorrect answers.
Therefore, we filter the generated rationales to include only the ones which result in the correctanswer ( ˆ y i =
y i ).
We fine-tune the base model M on this filtered dataset, and then restart this processby generating the new rationales with the newly fine-tuned model.
We keep repeating this processuntil the performance plateaus.
Note that during this process, once we collect a new dataset, we trainfrom the original pre-trained model M instead of continually training one model to avoid overfitting.
We provide an outline of this algorithm in Algorithm 1.
STaR can be seen as an approximation to an RL-style policy gradient objective.
To see this, notethat M can be viewed as a discrete latent variable model p M ( y | x )
=
(cid:80) r p ( r | x ) p ( y | x, r ) ; inother words, M first samples a latent rationale r before predicting y .
Now, given the indicator rewardfunction 1 (ˆ y = y ) , the total expected reward across the dataset is
[Equation]
where the gradient is obtained via the standard log-derivative trick for policy gradients.
Note thatthe indicator function discards the gradient for all sampled rationales that do not lead to the correctanswer y
i : this is the filtering process in STaR (Line 5).
Thus, STaR approximates J by (1) greedilydecoding samples of (ˆ r i , ˆ y i ) to reduce variance of this estimate (at the cost of potentially biasedexploration of rationales), and (2) taking multiple gradient steps on the same batch of data (similarto some policy gradient algorithms [25]).
These approximations make STaR a simple and broadlyapplicable method that can be implemented with standard LLM training machinery; future workshould more closely investigate the link between STaR and the RL objective above.
3.2 Rationalization
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
(b) grocery cart (
[Equation]
) (c) super market (d) fruit basket (e) fruit market A [Equation]
answer should place where grocery items are placed before checking out.
Of the above choices, grocery cart makes the most sense for holding grocery items.
Therefore, the answer is grocery cart (b).
The rationale generation bootstrapping algorithm carries a limitation.
Since the model is only trained onthe examples which it answers correctly, improvement ends when the model fails to solve new problems in the training set.
This is fundamentally dueto the fact that the algorithm cannot obtain any training signal from failed examples.
Inspired by [3], wepropose a technique we call “rationalization”.
Specifically, we provide the answer as a hint to the modeland ask it to generate rationales in the same styleas in the previous rationale generation step.
Giventhe answer, the model is able to reason backwards,and hence more easily generate a rationale that leadsto the correct answer.
For example, in Figure 2, weprovide the hint that ”(b) grocery cart” is the correct answer in the prompt to generate the rationale.
We apply rationalization to the problems which themodel failed to solve with rationale generation.
Whenadding a rationalization-generated rationale to our dataset, we do not include the hint in its corresponding prompt, as if the model had come up with the rationale without the hint.
After filtering, wefine-tune on the previously generated dataset combined with the rationalization-generated dataset.
Algorithm generation
) ∀ i ∈ [1 , D ] # Perform rationalization 5:
D n ← { ( x i , ˆ r
i , y i ) | i ∈ [1 , D ] ∧ ˆ y i =
y i } # Filter rationales using ground truth answers 6:
D rat n ← { ( x i , ˆ r rat
i , y i ) | i ∈ [1 , D ] ∧ ˆ y i ̸
= y i
∧
ˆ y rat i
= y i } # Filter rationalized rationales 7:
M n ← train ( M, D n ∪ D rat n ) # Finetune the original model on correct solutions - inner loop 8: end for 158 Algorithm 1 describes the full algorithm, with the parts in blue corresponding to rationalization.
Without those parts, Algorithm 1 corresponds to STaR without rationalization.
Figure 1 provides anoverview diagram.
Fine-tuning on the dataset generated by rationalization has a crucial benefit ofexposing the model to difficult problems which otherwise would not have appeared in its finetuningdataset.
This can be understood as challenging the model to “think outside the box” about problemson which it was unsuccessful.
A secondary benefit of rationalization is an increase in dataset size.
Experiments
For our experiments, we focus on arithmetic, commonsense reasoning, and grade school math todemonstrate STaR’s breadth.
In particular, for arithmetic, we follow a setup inspired by [5].
Forcommonsense question-answering we follow [13, 6] and use CommonsenseQA (CQA), a widelyused multiple-choice dataset for this domain [10].
For grade school math, we use GSM8K from [9].
4.1 Experimental Protocol
We used GPT-J as our base language model, and the fine-tuning script from the GPT-J repository[26].
We chose GPT-J, a 6B-parameter model, because the checkpoint and fine-tuning code arepublicly available [26], and the model is large enough to generate rationales of non-trivial quality tobe bootstrapped from.
More hyperparameter details about GPT-J and our fine-tuning are includedin Appendix H. Following the default setting of [26], we perform a 100-step learning rate warmup,from which point we use a constant learning rate.
Unless stated otherwise, we start with 40 trainingsteps at the first outer loop, and increase the number of fine-tuning training steps by 20% with eachouter loop.
In general, we found that training more slowly at the beginning ultimately benefitsmodel performance.
We expect that further improvement is possible via a thorough hyperparametersearch—we leave this to future work due to computational constraints.
For arithmetic problems, we first generate a dataset of 50,000 randomly sampled questions (uniformlyover the digit lengths) in the format introduced by [5].
For each outer loop iteration on arithmetic,we sample 10,000 problems from the dataset.
We use 10 random few-shot rationale examples foreach digit for its corresponding few-shot prompt.
For each of the 9 , 741 questions in the training setof CommonsenseQA, we add the question to the few-shot rationale prompt, and prompt the modelto generate the rationale and answer for that question.
For few shot prompting on CQA, we startwith the same 10 questions as used in [6], with the rationales modified slightly to fix an incorrectanswer and to more explicitly reference relevant knowledge.
We include these modified prompts in
Appendix B 1 .
These prompts serve as our complete set of explanations.
We run STaR until we seeperformance saturate, and we report the best results.
When performing rationalization, we find that the choice to include or omit few-shot prompts onouter-loop iterations after the first iteration does not have a substantial impact on the method’s ultimateperformance.
However, there are some nuances which we discuss further in Section 5, leading us touse few-shot prompts unless stated otherwise.
4.2 Datasets
[Figure]
Arithmetic The arithmetic task is to calculate the sum of two ndigit integers.
We generate the dataset based on the descriptionsin [5] and visualize an example scratchpad in Figure 3.
Everythingup to and including “ Target: ” is given as part of a prompt, andthe model is asked to generate the scratchpad (start/end indicatedby “ <scratch> ”) and the final answer, as in [5].
Each line of thescratchpad corresponds to the summation of each pair of digits fromthe final digit to the first digit, the accumulating final digits of theanswer, and a carry digit corresponding to whether the previous pairsummed to at least 10.
We include few-shot prompts for 1 to 5 digits.
When performing rationalization, we include the correct answer after“ Target ” and query the model to produce the scratchpad and thenreproduce the correct answer following the scratchpad.
CommonsenseQA The multiple-choice commonsense reasoningtask, CommonsenseQA [10] (CQA), is constructed from ConceptNet, a semantic graph of conceptsand their relationships with over a million nodes [28].
[10] identified a set of “target” concepts in
ConceptNet for each question, where the target concepts share a semantic relationship to one “source”concept.
Then each question is crowdsourced to allow a reader to identify one target concept, whilementioning the source concept.
In addition, two distractor answers are added.
The dataset has 12,247questions, each with five choices, with 9,741 in the train set, 1,221 in the dev set, and 1,285 in the(withheld) test set.
[Figure]
(a
[Figure]
Without rationalization
Iterations                (
[Figure]
With rationalization
Corresponding to the broad variety of ConceptNet, CQA contains a diverse set of questions whichrequire commonsense reasoning ability building off of standard world knowledge, where humanperformance is 89% [10].
Many have pointed out that CQA contains a number of biases, alongseveral dimensions including gender [3].
We discuss how this may impact our method in Appendix G.
There are also many typos and questions which are fundamentally ambiguous 2 .
We use it despitethese issues as it is a general question-answering dataset relying on both common world knowledgeand simple reasoning, which serves as a good test-bed for our method.
Grade School Math (GSM8K)
We also evaluate on the Grade School Math (GSM8K) dataset,which contains 7,473 train and 1,319 test examples of grade-school-level word problems [9].
Thesemath problems are posed in natural language and require two to eight calculation steps to arrive at afinal answer.
This dataset combines the skills needed for arithmetic and commonsense reasoning.
4.3 Symbolic Reasoning: Results on Arithmetic
The accuracies of the model across digits 1 - 5 over each iteration of the outer loop are plotted in
Figure 4.
After running STaR for 16 iterations, the overall accuracy is 89 .
For reference, abaseline trained on 10,000 examples without rationales for 5,000 steps attains 76 .
3% accuracy.
Notably, few-shot accuracy on arithmetic problems is very low, even with rationales: accuracy on2-digit addition is less than 1% , and accuracy on more digits close to zero.
Iterations
[Figure]
[Figure]
[Figure]
With rationalization, the accuracy is able to improveespecially quickly.
After one fine-tuning iteration onthe model’s generated scratchpads, 2-digit additionimproves to 32% from less than 1%.
Without rationalization, the performance improvement is stagewise: the model generally has poor performanceon the n -digit sum until it has good performanceon the ( n − 1) -digit sum.
With rationalization, themodel can learn many lengths at once, though notwith equal accuracy.
Rationalization allows manyproblems to be solved few-shot, so we start STaRtraining with 300 steps (note, doing so without rationalization causes overfitting on 1 -digit addition),and increase training by 20 steps per iteration.
We also perform an experiment where we continue pre-training STaR with rationalization withadditional digits, starting before the 20th iteration, while keeping the total number of training examplesfixed at each iteration.
We find that not only does this appear to quickly improve performance on theinitial set of digits, but when evaluated on 9 and 10 digit examples, never seen during training, themodel successfully solves many of these out-of-distribution problems.
As visualized in Figure 5, theintroduction of these digits appears to make the training less stable, but the exact cause is unclear.
[Table]
4.4 Natural Language Reasoning: Commonsense Question Answering
The CommonsesenseQA (CQA) setting introduces several new challenges.
In the arithmetic task,an incorrect scratchpad in the reasoning step, and to a lesser degree in the rationalization step, wasextremely likely to result in an incorrect answer.
On the other hand, CQA problems are 5-waymultiple choice questions.
Thus, one will get the right answer at random approximately 20% ofthe time, regardless of the quality of reasoning.
Moreover, some simple heuristics (e
[Table]
g. semanticsimilarity) can meaningfully improve this to ≈ 30% without any reasoning, as shown by [10].
We evaluate this dataset as described in the experimental protocol and compare to several baselines.
The first baseline is to finetune GPT-J to directly output the final answer, which we call “GPT-J
Finetuned”.
We also compare to GPT-3 finetuned to directly predict the final answer from [29], and a137B parameter Lambda model few-shot prompted with chain-of-thought (CoT) rationales from [6].
We found that, as shown in Table 1, STaR without rationalization outperformed GPT-J fine-tuneddirectly on the final answer for the entire dataset, despite training on less of the data.
The inclusion ofrationalization improved this performance to 72 .
5% , far closer to the 73% of the 30 × larger GPT-3.
As expected, we also see STaR surpassed the few-shot baselines, including the much-larger 137B
LaMDA model [30, 6].
We expect accuracy would be further improved if we applied STaR to amodel with higher few-shot performance.
Case Study Note that it is harder to judge the rationale quality: for arithmetic, one can comparethem to the ground truth rationales, but for CQA the evaluation is necessarily qualitative.
For thisreason, we include a case study in Figure 7.
We observe that the rationales provided are generallycoherent and of a similar structure to the few-shot rationales.
We make the following two observations:
After training with STaR, we see the model was able to generate reasonable rationales that
solve new problems, which explains part of the observed performance gain.
We also see that there were many instances in which STaR improved the quality of rationales
over those generated in a few-shot manner.
Human Evaluation Based on the observation that STaR may improve reasoning quality forproblems even when they were initially answered correctly via few-shot prompting, we performeda preliminary qualitative analysis.
We randomly selected 50 rationales generated from few-shot CoT and STaR-generated rationales on questions which they both answered correctly, as well ashuman-generated rationales for these problems from [3].
We then presented a random subset of10 questions and rationales to each of 20 crowdworkers on Prolific [31] with the rationales in arandomized order, asking them to rank the rationales based on which they felt best justified theanswer.
The participants were 30% more likely to rank the STaR-generated rationales higher than thefew-shot rationales ( p = . 039 ).
This indicates that, as mentioned in the case study, STaR can improvethe quality of rationale generation.
We also found that the participants were 74% more likely to prefer the STaR-generated rationalesover the human-generated rationales ( p < . 001 ).
To be clear, we do not believe that this indicateshuman-level rationale-generation performance.
Instead, we feel that it speaks to the difficulty ofeliciting high-quality rationales.
We reproduce the test prompts in Appendix C and elaborate on thelimitations of the crowdsourced explanations dataset.
[Table]
Failure Cases Finally, we found a variety of interesting failure cases, many of which correspondedto standard logical fallacies.
For example, the model often made statements related to the topic of thequestion but which were not actually arguments for why the answer should be true.
Sometimes, themodel claimed the question implied the answer as an argument, without explaining why.
Other times,especially early in training, the model answered as if it has knowledge about a particular individual,instead of making a general statement - e.g. “the king’s castle is a place where he feels safe” insteadof “castles are places where kings feel safe.”
We provide examples and analyze errors in Appendix A.
Few-shot Prompt Training Including few-shot prompts during fine-tuning [12] appears to havea meaningful performance benefit (60.9% to 68.8% without rationalization, 69.9% to 72.5% withrationalization).
Thus, we generally suggest its use for at least some portion of the training, thoughwe discuss some caveats in Section 5.
4.5 Mathematical Reasoning in Language: Grade School Math
[Table] [Table]
n
[Table]
a t
[Table]
L o
[Table]
f r e q u e n c y
We again find on GSM8K that STaR substantiallyimproves performance beyond few-shot with rationales or training to directly predict the answers(without rationales), shown in Table 2 and includethe few-shot prompt in Appendix I. We observethat on this task, the use of rationalization does notsubstantially improve performance.
Note that, intraining, it was necessary to cap the of steps at the iterations
Most often, the number of calculation steps generated by the model matches the number of stepstaken by humans (generally between 53% and 57%agreement across all iterations).
We visualize thisexplicitly in Figure 6.
We see that when the groundtruth and model disagree on the number of calculation steps, the model typically uses fewer.
Sometimes this is because the model skips steps, butoccasionally it finds different solutions. We show an example in Appendix J, where the modeldisregards redundant information and solves a 7-step problem in a single step.
Discussion and Challenges
The Impact of Rationalization An essential question is exactly what role rationalization plays.
Intuitively, rationalization allows a model to reverse-engineer a solution, or provides a heuristic foridentifying whether each step makes the conclusion more likely.
This parallels real-world problemswhere the final result is known, but challenging to derive a good justification.
From a mathematicalperspective, while rationale generation samples rationales from the distribution p ( r | x ) provided byour model M , rationalization conditions on the answer, letting us access an alternative distributionp ( r | x, y ) which may be a better search space for rationales.
Then rationalization could be framed asan off-policy estimate of the objective in Equation 1, sampling from the hint-augmented model as aproposal distribution.
Future work should establish more connections between rationalization andthese RL objectives, and examine more generally when and why rationalization improves learning.
In addition, due to the low sampling temperature, the outputs without rationalization correspondto the examples where the model is most confident in its answer.
This results in these examplesproviding a weaker gradient signal than the rationalization examples, at least in the first iteration.
Since we retrain from the initial pre-trained model every time we run a fine-tuning iteration, thedegree of this effect is also difficult to measure directly.
Finally, we must point out that the methodto add the “hint” does not follow immediately from the question and answer and in some contextsproviding it may be nontrivial.
An exploration of the various impacts of different hinting techniquesand their generality is an avenue for future work.
Temperature One intuitive alternative to rationalization, if one seeks to expand the trainingdataset, is more and higher-temperature sampling.
However, in practice, we found that this iscounterproductive.
In general, it substantially increases the likelihood of a correct answer despiteincorrect reasoning, and training on bad or irrelevant reasoning prevents generalization.
This isparticularly clear in more structured tasks, like arithmetic, where the scratchpads that the model learnsto produce with a higher-temperature sampling approach diverge into meaninglessness and cause themodel to stagnate.
Overall, we found that higher temperatures as an alternative to rationalization (e.g.0 .
or 0 .
7 ) consistently led to models worse than models with reasoning alone.
In addition, as textgeneration by large language models is sequential (i.e. one cannot produce a token without producingthe preceding token), generating text is a bottleneck and this is computationally far less efficientthan rationalization.
For example, generating 10 sample outputs is approximately 10 times slowerthan generating one sample output.
However, one potentially valuable way to leverage multiplesamples would be to use the method proposed in [32], using the majority-vote result of multiple hightemperature scratchpads as a ground truth against which we compare a low-temperature scratchpad.
This may allow one to apply STaR to a dataset of only questions, without answers.
Few-shot Prompting A noteworthy phenomenon is that the inclusion of few-shot prompting duringsampling seems to dramatically reduce “drift” where later rationales become increasingly dissimilarfrom the initial few-shot set of rationales.
One benefit of this is that the model may be less constrainedby the quality and difficulty of the initial rationales, theoretically allowing it to generalize more.
Onepotentially negative consequence is that the style of the rationales may less-closely match the originalprompting style.
Another benefit is in terms of computational resources - a shorter prompt lengthallows for a shorter sequence length when sampling.
Technically, the point in training at which we“disable” few-shot prompts is another hyperparameter which we could tune, but we leave this to futurework.
In addition, by leaving prompts out after the initial outer-loop iteration, the model tends toperform gradually worse at rationalization as it trains for longer periods of time.
As a result, it maybe necessary to include some hints during training for long periods of time with this approach.
Ultimately, the choice to include few-shot prompts in later iterations of training appears to depend onthe use-case: when the goal is consistent adherence to a particular prompt style, which may benefitexplainability, include few-shot prompts in sampling; when the goal is a faster training loop, one mayremove them.
Moreover, it is possible that with other datasets or larger models there is an impact onperformance, so we encourage this to be generally treated as a hyperparameter.
Conclusion
We present the Self-Taught Reasoner (STaR), which iteratively improves a model’s ability to generaterationales to solve problems.
We few-shot prompt a model to solve many problems in a step-by-stepmanner by generating rationales, and then prompt it to rationalize the correct answer for problemsit gets wrong.
We finetune on both the initially correct solutions and rationalized correct solutions,and repeat the process.
We find that this technique significantly improves the model’s generalizationperformance on both symbolic reasoning and natural language reasoning.
There are several important limitations on STaR as presented.
In order for the first iteration of STaRto succeed, few-shot performance must be above chance, implying that the initial model must bebig enough to have some reasoning capabilities.
For instance we found that GPT-2 was not ableto bootstrap from few-shot reasoning in even the arithmetic domain.
A further limitation is thatsettings with a high level of chance performance (e.g. binary decisions) yield many poor rationales,confounding the STaR approach.
An open problem is how to filter bad reasoning in these settings.
Nonetheless, we believe using examples without reasoning to bootstrap reasoning is a very generalapproach, and that STaR can serve as the basis of more sophisticated techniques across many domains.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 5 and Appendix G(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeAppendix G. (d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] We will release
the code and all necessary details for reproduction.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 4 and Appendix H.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] Experimentation was performed
on TPUv3-8 nodes.
In total, approximately 120,000 TPU-hours of compute were used.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We primary build on
the GPT-J fine-tuning code from [26](b) Did you mention the license of the assets?
[Yes] We rely on open-source assets.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
If feasible, we will release model checkpoints.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes] It does not contain personally identifiable
information.
The CommonsenseQA dataset contains biased questions and answers, as
mentioned in Subsection 4.2 and
Appendix G.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[Yes] See Appendix C
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[Yes]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[Yes] 20 participants recruited on Prolific [31],
took a mean and median time of 8.1 & 7.2 minutes, with an average wage of $10.26/hr(after platform fees, with their compensation costing of $41.65).