When Hindsight is Not 20/20: Testing Limits on Reflective Thinking in Large Language Models (2024)

Abstract: Recent studies suggest that self-reflective prompting can significantly enhance the reasoning capabilities of Large Language Models (LLMs). However, the use of external feedback as a stop criterion raises doubts about the true extent of LLMs’ ability to emulate human-like self-reflection. In this paper, we set out to clarify these capabilities under a more stringent evaluation setting in which we disallow any kind of external feedback. Our findings under this setting show a split: while self-reflection enhances performance in TruthfulQA, it adversely affects results in HotpotQA.We conduct follow-up analyses to clarify the contributing factors in these patterns, and find that the influence of self-reflection is impacted both by reliability of accuracy in models’ initial responses, and by overall question difficulty: specifically, self-reflection shows the most benefit when models are less likely to be correct initially, and when overall question difficulty is higher. We also find that self-reflection reduces tendency toward majority voting. Based on our findings, we propose guidelines for decisions on when to implement self-reflection. We release the codebase for reproducing our experiments at https://github.com/yanhong-lbh/LLM-SelfReflection-Eval.

## introduction large language models (llms) have shown impressive performance in generating human-like text (e.g., @xcite ), and recent works demonstrate that we can further prompt llms to reflect on their own outputs to improve their capabilities on complicated reasoning, programming and planning tasks @xcite @xcite @xcite and also improve their alignment with human values (e.g., less harmful step 3: revision instructions : [task-specific instruction] question : [insert question] answer : [insert answer from exploration] reflection : [insert reflection] {concatenate all k responses and reflection} question: [insert question] answer : step 1: exploration instructions : [task-specific instruction] question: [insert question] answer : step 2: reflection instructions : [task-specific instruction] please critique your answer based on the given question. question : [insert question] answer : [insert answer from exploration] reflection : repeat k times repeat k times and more helpful) @xcite . 1 however, @xcite find that performance gains associated with self-reflection may be due to implicit usage of external feedback as a stop criterion, as well as overly-engineered prompts that bias the model outputs, casting doubt on the true effectiveness of self-reflection. to verify the extent to which llms can truly reflect on their outputs, we take a more stringent evaluation approach: in addition to excluding external feedback @xcite , we also disallow multi-round iterative prompting, which can hint to the model that its prior response is incorrect. instead, we sample multiple model responses given a prompt, and ask the model to self-reflect on these candidate outputs. with this single-round testing, we can zero in on the model's ability to use selfreflection without implicit hints about whether a given response candidate is correct or incorrect. our experiments show that, in a case study with chatgpt on different qa datasets, self-reflection in our setting yields mixed results. specifically, self-reflection improves performance on truth-fulqa @xcite , but decreases model performance in hotpotqa @xcite . through follow-up analyses, we identify that the effectiveness of self-reflection strongly depends on the confidence in accuracy of the model's initial responses, as well as overall question difficulty as judged by humans: when the model is reliably giving correct answers from the start, self-reflection is more often harmful-however, on questions of greater difficulty, self-reflection is beneficial even when a decent percent of initial model responses are correct. we also find that self-reflection reduces model tendency toward majority voting, suggesting more sophisticated decision-making (albeit sometimes resulting in lower accuracy). based on our findings, we propose a practical guideline for users to decide when to use self-reflection. ## self-reflection prompting to focus on evaluating intrinsic reflective thinking capability, we adopt the following evaluation setting: in addition to @xcite protocol of excluding external feedback and prompt optimization, we additionally disallow iterative prompting, which samples new responses based on previous responses, creating an implicit hint to bias the model behavior @xcite . 2 we call our approach single-round self-reflection verification (sr 2 v). we evaluate llms' reflective thinking capability using the following simple threestage format: 1) exploration: given an input x, we prompt llm m to generate k candidate responses r j ∼ p m (r j |x, i exploration ), 1 ≤ @xmath0. note that this generation of candidate responses differs from iterative prompting because each response is sampled without conditioning on any other candidate responses. ## preliminary study: does self-reflection prompting work under sr 2 v? we follow previous works @xcite @xcite in using two representative datasets, truthfulqa and hotpotqa, to verify the effectiveness of self-reflection under sr 2 v. experiment setup for these experiments we set @xmath1, and we prompt chatgpt-3.5 ("gpt-3.5turbo-16k-0613") with the questions from each dataset. 3 our full process for making these api calls is presented in appendix f, and all prompt templates used can be found in appendix e. we also extend our experiments to llama-2 @xcite and @xcite , finding similar results to chatgpt-3.5-we present results and discussion for llama-2 and mixtral in appendix i and appendix j. for truthfulqa we evaluate automatically (see details in appendix d). for hotpotqa, we find that traditional exact match often unfairly assigns a score of 0 for semantically correct model responses; therefore, we manually assess 1, 000 randomly chosen hotpotqa instances to check the model's answers against references. to isolate the specific effect of the generated reflections, we also include an exploration-only baseline, in which we retain the exploration stage but remove the reflection component, and only concatenate the candidate model responses in the revision prompt. 4 observations results are shown in in truthfulqa, we see that using self-reflection achieves significantly better performance than either the exploration-only baseline or standard ## why self-reflection may not work? to better understand these patterns, we conduct an error analysis drawing inspiration from the reflection conceptual model in psychology @xcite . we hypothesize that two key factors influence self-reflection's efficacy: 1) the objective question difficulty (quantifiable based on human annotations), and 2) the model's comprehension quality (quantifiable based on the proportion of correct responses). following this framework, we can predict that if a question is above average in human-annotated difficulty, self-reflection may be of greater benefit. similarly, if the model already has a strong grasp of the question, it may not benefit as much from self-reflection. to test these hypotheses, we break down model performance based on levels of question difficulty and model comprehension. we focus our analysis on hotpotqa, as this is the dataset on which we 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% response accuracy 0.2 0.4 0.6 0.8 1.0 easy medium hard accuracy decomposition with artificial responses (@xmath2: performance decomposition on question difficulty and response accuracy (artificial responses). dotted lines show "turning points" at which reflection loses effectiveness, for easy/medium/hard questions. observe significant detrimental effects of applying self-reflection prompting. additionally, this dataset contains annotated human judgments of question difficulty, and enables a clearly-defined notion of accuracy. we use these human difficulty annotations for our measure of question difficulty, and for model comprehension we use response accuracy (ra): the proportion of correct answers among the k candidate model responses sampled during exploration. the broken-down results are shown in figure 2 . the results show an interaction between our two variables. for questions judged by humans as easy, self-reflection shows a benefit only when the model's candidate responses are mostly-but not all-incorrect, with self-reflection otherwise having negligible or negative effects on performance. for questions judged as medium, there is a more even split: when most or all of the model's candidate responses are wrong, self-reflection is beneficial, but when half or more of the responses are correct, self-reflection is often harmful-with the notable exception of the 75% ra bin. a similar pattern is seen for questions judged as hard, though for this category self-reflection is more consistently beneficial through the 75% ra bin, showing harm to performance only when all candidate model responses are already correct. across a wider range of question difficulties, by sampling model responses to minimally edited versions of the prompts, and then pairing these responses with the original prompts when eliciting self-reflection. this allows us to increase the number of incorrect candidate responses, and thus to more evenly distribute ra levels across human difficulty levels. more details on this simulation process can be found in appendix b. for this experiment, we generate @xmath3, with a mix of synthetic pairings and real pairings. 5 results are shown in figure 3 . we see that the benefits of self-reflection are now limited to the lowest ra levels, and there is also now a clearer shift from beneficial to harmful effects of self-reflection as ra increases. we also see that the interaction with question difficulty remains: the turning point from beneficial to harmful falls around 50% ra for hard questions, 30% for medium questions, and 20% for easy questions. overall, this indicates that a major contributor to the effectiveness of self-reflection is the confidence of model accuracy on the question-if the model is reliably correct on initial responses, self-reflection tends to be harmful. however, this effect is further modulated by overall question difficulty: the benefits of self-reflection persist to higher levels of response accuracy if the questions are more difficult based on human judgment. though truthfulqa is not as conducive to exact quantification of our variables, based on these results we can now speculate that the effectiveness of self-reflection on that dataset may be attributable to lower rate of good initial model responses, and potentially also higher overall question difficulty. 10% 20% 30% 40% 50% 60% 70% 80% does the model have access to external feedback? what's the percentage of consistent outputs? yes no is the question too difficult for humans? use self-consistency or standard prompting use self-reflection use self-reflection use self-consistency what's the estimate of response accuracy? should you use self-reflection prompting? 🤔 <25% <50% >50% >25% yes no sample n outputs from lm ploying majority voting on the candidate responses. in figure 4 we plot the percentage of items in which the model's output is consistent with majority voting, at different ra levels (computed at @xmath4, both with and without self-reflection. the plot shows that without self-reflection, the tendency to give answers consistent with majority voting is strong and closely correlated with the strength of the accuracy trend (i.e., more majority voting when most candidate responses are either correct or incorrect, and less majority voting when candidates are more mixed). however, with self-reflection the tendency to align with majority voting is significantly reduced across ra levels, suggesting that self-reflection does encourage more sophisticated decision strategies (even if in the case of higher ra levels, this in fact has a harmful effect on accuracy). ## discussion our analyses above have found that self-reflection benefits are limited to cases in which model accuracy is unreliable on initial responses, though benefits are more persistent for harder questions. based on these findings, we propose a set of guidelines for determining when to implement self-reflection in practical applications, for a given request or prompt. the core principle involves basing decisions on estimated ra and question difficulty, and these guidelines can be applied by simply sampling responses for the target question or prompt. first, if external tools or certain access to ground truth answers are available such that ra can be reliably estimated, then self-reflection should be used when ra levels are low. next, if difficulty annotations/subjective difficulty judgements are available, self-reflection can also be promising when ra levels are interme-diate and question difficulty is high. if ra cannot be estimated, response consistency can be used as a proxy: if responses are highly consistent, selfreflection may be unlikely to provide benefit. if consistency is low, then self-reflection may be beneficial, especially for questions of higher difficulty. an illustration of these guidelines is in figure 5 . ## conclusion in this paper, we evaluate chatgpt's self-reflective capabilities under a stringent single-round multiresponse evaluation setting. we find mixed results, and further analysis shows that the effectiveness of self-reflection is impacted both by question difficulty and by model response accuracy level: benefits of self-reflection are mostly limited to cases in which the model's initial responses are unreliable in accuracy, but with more persistent benefits for harder questions. additionally, we find that self-reflection reduces the model's tendency for majority voting. we propose guidelines for when to use self-reflection, and we look forward to work further exploring impacts on self-reflection, and further refining these guidelines.