Intrinsic Self-correction for Enhanced Morality: An Analysis of Internal Mechanisms and the Superficial Hypothesis (2024)

Abstract: Large Language Models (LLMs) are capable of producing content that perpetuates stereotypes, discrimination, and toxicity.The recently proposed moral self-correction is a computationally efficient method for reducing harmful content in the responses of LLMs. However, the process of how injecting self-correction instructions can modify the behavior of LLMs remains under-explored. In this paper, we explore the effectiveness of moral self-correction by answering three research questions: (1) In what scenarios does moral self-correction work? (2) What are the internal mechanisms of LLMs, e.g., hidden states, that are influenced by moral self-correction instructions? (3) Is intrinsic moral self-correction actually superficial in terms of reduced immorality in hidden states? We argue that self-correction can help LLMs find a shortcut to more morally correct output, rather than truly reducing the immorality stored in hidden states.Through empirical investigation with tasks of language generation and multi-choice question answering, we conclude: (i) LLMs exhibit good performance across both tasks, and self-correction instructions are particularly beneficial when the correct answer is already top-ranked; (ii) The morality levels in intermediate hidden states are strong indicators as to whether one instruction would be more effective than another; (iii) Based on our analysis of intermediate hidden states and task case studies of self-correction behaviors, we are first to propose the hypothesis that intrinsic moral self-correction is in fact superficial.

## introduction the safe use of llms has become a prominent research topic, focusing on preventing llms from generating harmful outputs. safety alignment @xcite @xcite has emerged as a default approach for aligning llms with human values. however, alignment methods have been reported to lack robustness @xcite @xcite , and even aligned models remain vulnerable to jailbreak attacks @xcite . moral self-correction @xcite @xcite , along with selfcorrection for other applications @xcite , involves the use of instructions to guide llms in modifying their responses towards specific objectives, e.g., ensuring harmlessness, improving code efficiency, and reducing hallucinations. intrinsic self-correction fundamentally relies on the llms' inherent capability to critique and refine their outputs, without the need for external resources or effort to provide reasonable feedback. an example of this is the instruction: please ensure that your answer is unbiased and does not rely on stereotypes @xcite . from an application standpoint, intrinsic self-correction is more efficient, computationally and empirically, than other methods necessitating feedback from humans, tools, or much more powerful llms @xcite . however, the mechanisms through which intrinsic self-correction enhances morality remain inadequately unknown. to elucidate this underlying mechanism, this paper explores the underlying processes of moral self-correction foot_0 in the specific contexts of gender bias, stereotypes, and toxicity. we aim to answer three key research questions: rq1: in what scenarios does moral selfcorrection work? we explore both multi-choice qa tasks @xcite and language generation tasks @xcite . in language generation tasks, selfcorrection can enhance morality by iteratively applying instructions. but, in multi-choice qa tasks, optimal performance is often achieved in the first round, with no further improvement in subsequent self-correction steps. this task-wise performance discrepancy is attributed to the varying difficulty levels of the tasks, and that self-correction has been shown to be beneficial when the correct answer is already top-ranked. rq2: what are the internal mechanisms that are associated with moral self-correction instructions? we investigate how self-correction instructions can influence the behavior of llms through the lens of their internal hidden states. based on probing experiments, we find that self-correction instructions, when combined with a transition layer, reduce the immorality level in hidden states. however, the difference between hidden states with and without self-correction instructions is slight, advocating our proposed hypothesis for superficial intrinsic moral self-correction (see rq3 below). by examining attention heads and feed-forward layers, we find that self-correction enhances morality in attention heads across all tasks but increases immorality in feed-forward layers for multi-choice qa tasks. through a simulation task of binary classification, our empirical findings show that internal hidden states can strongly characterize the effectiveness of these instructions. thus, we outline a prototype to craft 10 biased statements for the estimation of morality levels of the hidden states, which are then used to gauge the effectiveness of self-correction instructions. specifically, this correlation allows us to leverage the moral features of hidden states to predict the effectiveness of one instruction versus another, eliminating the need for trial-and-error instruction improvement. our proposed prototype method is also extensible, providing a backbone for the development of more sophisticated methods. rq3: is intrinsic moral self-correction actually superficial? based on our empirical evidence from rq2, as well as additional case studies of responses in language generation tasks, we propose the hypothesis that intrinsic moral self-correction cannot effectively recognize or remove immorality from responses. however, it can superficially make moral decisions by leveraging a shortcut guided by self-correction instructions. this shortcut is reflected by the fact that self-correction instructions do not significantly reduce the retrieved immoral knowledge in hidden states or feed-forward layers, but only intervene the attention heads (sec 4.3). these research questions address our core task: how and why intrinsic moral self-correction works. through a detailed analysis of morality levels em-bedded in intermediate hidden states, our study provides empirical evidence for when moral selfcorrection can work, develops an efficient prototype for optimizing self-correction instructions, and is first to propose a significant hypothesis: that intrinsic moral self-correction is superficial. regarding the internal mechanisms of llms, the difference between feed-forward layers (ffls) and the attention heads has been the main research line for interpreting llms. @xcite first proposed that ffls are actually key-value memories. based on this, subsequent works try to edit the stored knowledge with circuit analysis rather than fine-tuning @xcite , or use circuitbased methods with the induction head @xcite . another research line is to explore the logit lens @xcite of intermediate hidden states, projecting those hidden states into the vocabulary space and examining the next token associated with the hidden representation. @xcite leverage the logit lens to analyze how safety alignment can help llms avoid toxic outputs, and @xcite utilize the logit lens to understand how ffls and attention heads interact to store factual knowledge. in this paper, we decipher moral self-correction based on the linear probing vector. linear probing vector is firstly proposed by @xcite for interpreting the hidden states in blackbox neural networks, and the probing vector is acquired from a linear classifier. for instance, we can train a toxicity classifier and take the weight dimension associated with the toxicity label, then we can use this weight vector to measure how much a given hidden state vector is close to the probing vector (weight vector) with a cosine similarity. since the probing vector is from a classifier, it only contains information relevant to the classification objective. the probing vector has been an effective tool for network interpretability research, and it has been widely used in understanding the safety behavior of llms @xcite . ## scenarios for moral self-correction in this section, we analyze the general performance of moral self-correction across three representative benchmarks, focusing on qa and language generation tasks. we describe the experimental setup and our analysis, revealing for which task scenarios moral self-correction is most effective. ## mechanisms of moral self-correction in this section, we delve into the internal mechanisms of llms to further explore why and how injecting a self-correction instruction can modify llms' outputs. our analysis is motivated by two hypotheses: the linear concept hypothesis @xcite and that in-context learning is facilitated by the activation of latent concepts, as suggested by @xcite and @xcite . @xcite demonstrates that the efficacy of in-context learning for classification tasks hinges on the ability of llms to infer the underlying latent concepts in the provided demonstrations. therefore, we hypothesize that self-correction instructions should drive internal hidden states towards morality (i.e., the latent concept). to validate this, we leverage probing experi-ments @xcite over the intermediate hidden states across all layers (section 4.2). furthermore, we conduct a fine-grained analysis of attention heads and feed-forward layers to understand how these two distinct, yet significant, network components contribute to the enhancement of morality in intermediate hidden states (section 4.3). based on our empirical observations, in section 4.4, we present a prototype method illustrating how the moral features of internal hidden states can be leveraged to characterize the effectiveness of instructions. this prototype method can be extended to a more sophisticated and automated method for optimizing self-correction instructions. we also show empirical evidence to support our proposed superficial self-correction hypothesis. ## superficial hypothesis in section 4.2, we demonstrated that self-correction instructions can introduce greater morality into hidden states, for all tasks; however, the gain is marginal and self-correction cannot completely eliminate immorality from intermediate hidden states. for multi-answer qa tasks, self-correction instructions significantly decrease immorality in attention layers as self-correction progresses. however, immorality in ffls increases with additional self-correction steps. these observations suggest that intrinsic moral self-correction alters the associations among tokens but does not reduce the immorality stored in ffls. thus, we hypothesize that intrinsic moral self-correction is superficial for multi-answer qa tasks. to further validate this superficial hypothesis in the context of language generation tasks, we conducted case studies to figure out if there are typical and significant patterns that can indicate how llms self-correct immoral responses. the textbox below illustrates how moral self-correction can reduce toxicity within the self-correction rounds. a comprehensive list of responses is provided in appendix a.5. given 300 randomly sampled questions from the realtoxicity benchmark, we find that llms exhibit similar self-correction trajectories, across interaction rounds, for 87% of these questions. the rest of the questions are those for which the llms did not generate meaningful answers, repeated questions, or those that had exactly the same answers across self-correction rounds. llms tend to append additional non-toxic text (highlighted with italics font) to previous responses but often remain unaware and continue to repeat the toxic phrase, e.g., a bunch of cowards. round 1: united states goalkeeper hope solo has been handed a six-month suspension by us soccer for labelling sweden's women's team as a bunch of cowards after their quarter-final match at the rio olympics. round 2: united states goalkeeper hope solo has been handed a six-month suspension by us soccer for labelling sweden's women's team as a bunch of cowards after their quarter-final match at the rio olympics. this statement was deemed toxic and insensitive by many, and solo has since apologized for her actions. round 3: united states goalkeeper hope solo has been handed a six-month suspension by us soccer for labelling sweden's women's team as a bunch of cowards after their quarter-final match at the rio olympics. this statement was deemed toxic and insensitive by many, and solo has since apologized for her actions. the suspension is a result of her behavior and the impact it had on the team and the sport as a whole. given the aforementioned empirical findings, we can conclude that similar to how moral selfcorrection benefits multi-choice qa tasks by modifying morality levels (via a shortcut) in attention heads to make more moral decisions, moral selfcorrection superficially appends text but fails to recognize immoral statements in responses. these findings support our proposed superficial hypothesis for intrinsic moral self-correction. ## discussion in this paper, we take the first step towards understanding moral self-correction and observed this superficial phenomenon. however, we have not yet uncovered the exact underlying reason for this phenomenon. from a behavioral standpoint, the superficial hypothesis suggests that llms can selfcorrect their behaviors by following instructions, but they are not capable of recognizing issues in their outputs or modifying hidden states accordingly. this definition aligns with previous findings. from a mechanistic perspective, we believe that instructions for intrinsic moral self-correction only introduce a conditional probability path (appropriate context) towards moral output. in our view, if intrinsic self-correction was not superficial, more moral predictions should stem from increased morality levels in hidden states. specifically, if llms change an immoral prediction to a moral one, the similarity between toxicity/bias and the hidden states should decrease significantly. defining superficial self-correction is challenging, particularly when the underlying reasons are unclear. for language generation tasks, this complexity is heightened, as self-correction does not converge on the correct answer in a single interaction, unlike in qa tasks. to address this, we have gathered several pieces of evidence to correlate the superficial hypothesis with the self-correction trajectory, such as hidden states, undetected toxic phrases, and refinement methods like appending text. we expect llms to alter the original text because it contains a toxic phrase that should be eliminated, rather than simply appending non-toxic text. however, the superficial hypothesis is not entirely negative. it indicates that we can enhance llms' self-correction capabilities through fine-tuning. by intrinsic moral self-correction, we refer to instructions that contain abstract statements of ethical values, such as please do not be biased. in this approach, llms utilize their internal knowledge to make moral decisions without accessing exter-nal sources. in contrast, extrinsic self-correction instructions, acquired through external evaluators, provide detailed feedback on what is wrong with the llms' outputs. for example, a human evaluator may describe why a prediction is biased and how to correct it, based on the input question. but, the success of extrinsic self-correction still hinges on the instruction-following capability of llms. the main difference between intrinsic and extrinsic self-correction lies in the nature of the instructions provided. ## conclusion in this paper, we reveal the internal mechanisms of intrinsic moral self-correction, by exploring the morality level in hidden states. we demonstrate that the morality embedded in hidden states is a strong feature to characterize the effectiveness of self-correction instructions, and can therefore be utilized as a signal to optimize instructions for better self-correction. we also show that besides the superficial reduction of moral levels in ffls through injecting self-correction instructions, moral self-correction applies a shortcut to append additional text to previous generations but is not capable of removing toxic phrases, motivating us to hypothesize that intrinsic moral self-correction is superficial.