Corrective In-Context Learning: Evaluating Self-Correction in Large Language Models (2025)

Abstract: In-context learning (ICL) has transformed the use of large language models (LLMs) for NLP tasks, enabling few-shot learning by conditioning on labeled examples without finetuning. Despite its effectiveness, ICL is prone to errors, especially for challenging examples. With the goal of improving the performance of ICL, we propose *corrective in-context learning* (CICL), an approach that incorporates a model’s incorrect predictions alongside ground truth corrections into the prompt, aiming to enhance classification accuracy through self-correction. However, contrary to our hypothesis, extensive experiments on text classification tasks demonstrate that CICL consistently underperforms standard ICL, with performance degrading as the proportion of corrections in the prompt increases. Our findings indicate that CICL introduces confusion by disrupting the model’s task understanding, rather than refining its predictions. Additionally, we observe that presenting harder examples in standard ICL does not improve performance, suggesting that example difficulty alone may not be a reliable criterion for effective selection. By presenting these negative results, we provide important insights into the limitations of self-corrective mechanisms in LLMs and offer directions for future research.

## introduction in-context learning (icl; @xcite has emerged as a powerful paradigm for leveraging large language models (llms) for various nlp tasks, including text classification. unlike traditional approaches that require finetuning on taskspecific data, icl allows models to make predictions based on a small number of examples presented in the prompt, effectively transforming llms into flexible tools for few-shot learning. this paradigm has demonstrated remarkable performance in numerous scenarios, often approaching figure 1: cicl prompt example. the model is tasked with predicting the correct label based on its own prediction, using examples that include both the prediction and its correction. although effective, icl is susceptible to errors, particularly with difficult examples. we set out to further improve icl by introducing a novel extension of it, which we term corrective in-context learning (cicl). our approach is based on the idea that providing the model with its initial predictions alongside the correct ground truth labels can serve as a feedback mechanism, enabling the model to refine its understanding and improve subsequent predictions. cicl builds on the intuition that llms, when exposed to their own errors in conjunction with the correct answers, might learn from these mistakes within the confines of a single interaction. for instance, if a model predicts "positive" for a sentence that is actually "negative," presenting this prediction-error pair could prompt the model to recalibrate its internal representations and make better-informed predictions for similar inputs. however, our empirical evaluation reveals a different story. while the approach is theoretically promising, our experiments show that cicl fails to deliver the anticipated improvements. in some cases, performance even deteriorates compared to standard icl. this underscores the complexity of self-correction in llms and the importance of rigorously evaluating intuitive extensions to icl. in this paper, we present a detailed investigation of cicl for text classification tasks. we outline our methodology, describe the experimental setup, and analyze the results, focusing on understanding why this approach falls short of expectations. by sharing these negative results, we aim to contribute to the growing discourse on the limitations of llms and to inspire future research to develop more effective self-correction mechanisms. ## related work in-context learning introduced by @xcite , icl enables llms to perform tasks via few-shot prompting without parameter updates. subsequent work has explored factors influencing icl performance, including example selection @xcite and ordering @xcite , raising questions about its robustness to errors or ambiguous examples. to enhance icl, prompt tuning methods @xcite optimize soft prompts, improving adaptation while maintaining frozen model parameters, bridging the gap between icl and finetuning. ## impact of harder examples on icl although not the primary objective of this study, an intriguing finding emerges from our results. as shown in figure 3 , increasing the proportion of incorrectly classified examples (i.e., higher "corrected proportions") in the few-shot context does not improve the performance of standard icl. in fact, for some models, performance slightly declines as the corrected proportion increases. intuitively, one might expect that presenting the model with harder examples-those it initially misclassified-would enhance performance. the rationale is that such examples could help refine the model's decision boundaries and establish clearer classification thresholds, improving overall accuracy. however, our results do not support this. when examining the performance trends of standard icl (blue lines in figure 3 ), we observe that including harder examples in the few-shot context fails to yield any consistent improvement. in some cases, performance remains unchanged or even decreases slightly. furthermore, statistical analysis using the kruskal-wallis test reveals no significant variation in standard icl performance across different corrected proportions ( this observation warrants further investigation into how example difficulty interacts with icl, particularly in understanding why harder examples fail to contribute to improved model calibration or decision-making in this context. ## conclusion in this paper, we introduced cicl, a novel approach aimed at leveraging incorrect model predictions to improve performance through selfcorrection. by including misclassified examples along with their correct labels in the prompt, cicl sought to refine the model's predictions in text classification tasks. however, contrary to our initial hypothesis, cicl consistently underperformed icl across all models and datasets. our findings revealed that the corrective nature of cicl often led to confusion rather than improvement. the swapping of labels in the few-shot examples disrupted the model's understanding of the task, resulting in degraded performance as the proportion of corrected examples increased. additionally, we explored an auxiliary finding regarding the impact of harder few-shot examples on standard icl performance. despite the expectation that presenting harder, misclassified examples could enhance the model's decision-making, our results showed no significant improvement across varying proportions of examples that required correction. this challenges the assumption that harder examples inherently contribute to better generalization in few-shot learning. overall, our study highlights the challenges of incorporating self-corrective mechanisms into llms through icl and demonstrates that harder examples are not necessarily more useful than easier ones in standard icl. these findings provide valuable insights for the development of more robust and effective few-shot learning methods with llms.