Smaller Large Language Models Can Do Moral Self-Correction (2025)

Abstract: Self-correction is one of the most amazing emerging capabilities of Large Language Models (LLMs), enabling LLMs to self-modify an inappropriate output given a natural language feedback which describes the problems of that output. Moral self-correction is a post-hoc approach correcting unethical generations without requiring a gradient update, making it both computationally lightweight and capable of preserving the language modeling ability. Previous works have shown that LLMs can self-debias, and it has been reported that small models, i.e., those with less than 22B parameters, are not capable of moral self-correction.However, there is no direct proof as to why such smaller models fall short of moral self-correction, though previous research hypothesizes that larger models are skilled in following instructions and understanding abstract social norms.In this paper, we empirically validate this hypothesis in the context of social stereotyping, through meticulous prompting.Our experimental results indicate that (i) surprisingly, 3.8B LLMs with proper safety alignment fine-tuning can achieve very good moral self-correction performance, highlighting the significant effects of safety alignment; and (ii) small LLMs are indeed weaker than larger-scale models in terms of comprehending social norms and self-explanation through CoT, but all scales of LLMs show bad self-correction performance given unethical instructions.

## introduction socially safe technology has attracted attention from both research and industry communities due to the increasingly wide application of llm-based systems. unethical outputs, e.g., we cannot accept ladies' opinions, from those systems can cause serious social issues @xcite . in the context of social stereo-typing, a conventional method for mitigating social stereotypes is to fine-tune llms with an antistereotype corpus @xcite . however, computational resource availability is a significant limitation for fine-tuning models as the size of llms increases. on the other hand, safety alignment, e.g., reinforcement learning from human feedback, has been the default method used in the pretraining stage to avoid generating toxic or unethical outputs during downstream applications @xcite . recently, the superficial alignment hypothesis revealed the ineffectiveness of alignment @xcite . @xcite further proves that alignment helps llms avoid generating undesired content by bypassing the typical toxicityrelevant region of the parametric space. however, the toxicity learned during pretraining is not removed from parameters. due to the aforementioned issues of alignment, moral self-correction @xcite @xcite has the potential to be a promising solution for ethical purpose, leveraging the inner capability of llms to prevent unethical outputs given a natural language feedback. moral self-correction is a post-hoc method and enjoys several advantages over conventional finetuning-based methods, specifically, computational efficiency and protection of the language modeling ability @xcite . technically, the feedback in the self-correction instructions should be actionable and specific @xcite . unlike self-correction in other tasks such as code synthesis @xcite , dialogue @xcite , question answering @xcite , and reasoning @xcite , natural language feedback with ethical judgement is hard to acquire without human annotations due to the high level of abstraction and implication present in language @xcite @xcite . therefore, for moral self-correction, previous works mainly focus on mitigating toxicity @xcite , which can be more easily extracted from text. however, social biases and stereotypes are often implied by language. additionally, @xcite challenges that the given natural language instruction directly tells llms the answer to a given reasoning question, thus explaining why self-correction with external feedback can work so well. the authors also empirically validate the intrinsic selfcorrection of llms for reasoning tasks, showing llms cannot effectively self-correct reasoning errors without external feedback of ground-truth answers. in this paper, we also focus on the intrinsic selfcorrection capability for morality. in specific, we explore to what extent small llms, i.e., those with less than 22b parameters, can, if at all: (1) understand abstract social norms; (2) follow instructions; the capacity for instruction-following emerges in large language models (llms) through instruction-tuning @xcite . while there is no conclusive evidence explaining the acquisition of instruction-following capacity in llms, @xcite suggests that instruction-tuning enhances llms' ability to recognize instruction tokens, facilitating the retrieval of relevant latent knowledge for a given task. additionally, @xcite advocates for the metaevaluation of llms-based evaluators via evaluating the instruction-following capacity, emphasizing the importance of assessing instruction-following capacity in current llms-based research and applications. other studies focus on evaluating the intrinsic instruction-following capacity of llms @xcite @xcite by instructing llms to perform tasks such as label flipping for classification or assessing their understanding of negated prompts. more details about related works are discussed in appendix a.1. ## experimental setting in this study, we use various scales of llms foot_0 : gpt2 (355m and 774m) @xcite , olmo foot_1 (1b) @xcite , phi-3 foot_2 (3.8b) @xcite , and llama-2 (7b, 13b and 70b) (tou- ## analysis figure 1 shows the fairness performance of all considered llms over the winogender benchmark and the physical and religion bias dimensions of bbq (additional results are available in appendix 4.). it is obvious that all llms with over 3.8b parameters can achieve positive gains from self-correction and outperform the baseline performance. for llms with smaller scales, self-correction does not contribute to improvement and even leads to worse performance, e.g., 1b model. for those two llms of 335m and 775m, they can not even follow instructions to give correct answer format and their baseline fairness score is around 0. interestingly, the 3.8b model of phi-3 outperforms all llama-2 models, in both baseline performance and selfcorrection performance for bbq. notably, phi-3 is fine-tuned with safety alignment, indicating the significant help from safety alignment when it comes to have better self-correction performance. this is aligned with the conclusion of @xcite . in summary, the empirical observations shows that the model scale threshold for the emergence of moral self-correction capability is 3.8b. for the cot setting, the 70b model demonstrates a positive gain with the cot approach across all evaluated tasks, with cot performance notably surpassing self-correction. nonetheless, other scales of llms have varying performances given cot explanations. for the 13b model, cot causes a performance decrease compared to selfcorrection, but cot helps 7b model acquire better performance among religion and physical bias dimensions, the similar phenomenon is observed for the 1b model as well. the 3.8b model only has better performance with cot on the physical bias but the cot performance is marginally better than that of self-correction. therefore, we can conclude that llms, with less than 70b parameters, can not give informative explanations based on their cot capability w.r.t. morality-relevant questions. in the appendix a.4, we show an example about the cot explanation from llama2-7b. per the dimension of specificity shown in figure 2 , the least specific instruction does help all model scales improve significantly, and the improvement is more apparent for the 3.8b and 7b models. this indicates that smaller models, with no less than 3.8b parameters, can understand abstract social norms of stereotyping. by increasing the specificity level from 1 to 2, the fairness performance of smaller models is further improved, while the change of the 70b version is slight since it is al-ready very unbiased. this demonstrates that more specific social norms in instructions can indeed help both small and large llms perform better self-correction. given the instruction (specificity-3) clearly containing a correct answer, all scales, except those less than 3.8b, can achieve a perfect fairness performance. this aligns with the conclusion from @xcite about the significant effect of ground-truth answers in instructions. remarkably, the 70b model demonstrates a propensity to approach optimal fairness with regard to instruction of specificity-2 (in the absence of access to the correct answer), thereby underscoring its proficiency in instruction following and understanding of social norms. overall, llms with scales no less than 3.8b can understand abstract social norms in the instruction and instructions with higher specificity levels indeed benefit intrinsic self-correction. the experimental results w.r.t. negation are shown in figure 3 , the considered llms with various scales perform rather differently across tasks, except for the 70b and 7b llama2 which show worse performance than that of the baseline setting among all tasks. this suggests that the 70b/7b models have a strong capability to follow instructions, but also indicates that safety alignment does not ensure llms can detect unethical instructions and refuse to follow them. interestingly, the performances of 13b and 3.8b models are not consistent with the given negation instruction, across tasks. the 3.8b model shows declined performance for religion and physical biases, yet its performance improves in the winogender benchmark. we believe this is because the excellent safety alignment performance of 3.8b model phi-3. the 13b llama2 follows the negation instruction and has a significant performance drop w.r.t. winogender, but its performance is better than that of the baseline setting within the religion and physical bias dimensions. we guess this is because, given the religion and physical bias of the bbq benchmark, the safety alignment process can motivate the 13b model to recognize the unethical purpose in the negation instructions can refute to follow that. we propose to uncover how llms react differently to the identical negation instruction among different tasks in future research. considering the superior performance of the 3.8b model phi-3, and the varying behaviors of llms given the negation instruction, it is reasonable to believe the significant role of safety alignment in determining the post-hoc selfcorrection performance. in essence, all considered scales of llms can not have a completely appropriate performance given an unethical instruction, the capability to recognize and refute unethical instructions should be enhanced through better safety alignment. ## discussions previous studies on the mechanism of selfcorrection @xcite reveal that intrinsic self-correction is superficial and is not an innate capability in llms, therefore there are various issues brought by intrinsic selfcorrection @xcite this work serves as complementary evidence supporting previous studies, demonstrating that even very small llms, when carefully fine-tuned, can perform well in intrinsic self-correction. on the other hand, several studies have shown that llms struggle with tasks requiring social and moral intelligence. in particular, @xcite argues that llms fail to develop true moral reasoning capabilities due to the gap between their distributional semantic learning and the inherently pragmatic nature of morality. given the aforementioned findings from previous studies and the historical evaluation showed in this paper, it is rational to argue that intrinsic moral self-correction is not an instance of moral reasoning in llms. instead, it can only be enhanced through additional fine-tuning @xcite or figuring out optimal self-correction instructions. ## conclusion in this paper, we demonstrate that smaller llms with no less than 3.8b parameters do possess the capability for moral self-correction and are able to follow instructions with social norms, and that enhancing the specificity level of instructions positively impacts self-correction performance. our experimental evidence supports the significant role of safety alignment in the success of moral selfcorrection, besides the impact of model scales.