Discourse Heuristics For Paradoxically Moral Self-Correction (2025)

Abstract: Moral self-correction has emerged as a promising approach for aligning the output of Large Language Models (LLMs) with human moral values. However, moral self-correction techniques are subject to two primary paradoxes. First, despite empirical and theoretical evidence to support the effectiveness of self-correction, this LLM capability only operates at a superficial level. Second, while LLMs possess the capability of self-diagnosing immoral aspects of their output, they struggle to identify the cause of this moral inconsistency during their self-correction process. To better understand and address these paradoxes, we analyze the discourse constructions in fine-tuning corpora designed to enhance moral self-correction, uncovering the existence of the heuristics underlying effective constructions. We demonstrate that moral self-correction relies on discourse constructions that reflect heuristic shortcuts, and that the presence of these heuristic shortcuts during self-correction leads to inconsistency when attempting to enhance both self-correction and self-diagnosis capabilities jointly. Building on our findings, we propose a method to strengthen moral self-correction through heuristics extracted from curated datasets, underscoring that its generalization is primarily constrained by situational context.

## introduction self-correction is a post-hoc approach that guides llms to refine their previous output according to the given instructions @xcite . it has become a popular technique for improving the quality of llms' generations, and its application in enhancing morality, i.e., moral self-correction @xcite , effectively mitigate harmful and stereotypical content in llm outputs. prior studies reveal two key paradoxes concerning the effectiveness of moral self-correction within llms. paradox1: while moral self-correction appears effective in enhancing the perceived moral correctness of llm responses @xcite , this capability remains superficial, as evidenced by limited alterations of hidden states or the requirement of ground-truth answers in instructions @xcite . paradox2: there is a lack of consistency between self-diagnosis and self-correction @xcite , suggesting a disconnect between an llm's capability of identifying moral issues, e.g., morally unaligned or incorrect output, and addressing them effectively, which can only be done if the llm knows why the decision it made was morally incorrect. furthermore, prior studies have shown that neural language models are capable of generalization across tasks due to internalization of discourse constructions, rather than true language understanding @xcite , and generalization across moral reasoning tasks without understanding true morality due to the distributional semantics of llms @xcite . additionally, llms exhibit reliance on shallow heuristics (shortcuts) across tasks @xcite @xcite . these findings motivate a plausible hypothesis which we have explored in this work: shallow heuristics * in self-correction may enable llms to make self-correction decisions without requiring induced immorality in their hidden states or relying on self-diagnosis, thereby addressing both paradoxes. our analysis builds on recent findings that emphasize construction-based approaches over syntactic rules for studying generalization in llms @xcite @xcite . construction-based approaches consider that llms achieve generalization not through reliance on syntax, but by leveraging the statistical distributions of phrases. in this paper, we focus on intrinsic moral selfcorrection in the context of social stereotype mitigation. the outline of the remainder of this paper is as follows: section 3 presents a preliminary study indicating that llms are not always capable of self-diagnosis while performing selfcorrection, and there is no uniform discourse construction for enhancing both of them. section 4 investigates effective discourse constructions for self-correction, and further reveals the existence of shallow heuristics via intervention experiments, addressing paradox1. section 5 presents further empirical evidence that, due to the available heuristics of the underlying discourse construction, jointly enhancing self-correction and self-diagnosis capabilities often results in conflicts across most stereotype categories. we also demonstrate that leveraging these heuristics as effective discourse constructions can enhance self-correction performance, offering a viable solution for improving this capability, and thus addressing paradox2. in summary, the main contributions of this work are as follows: we address the two paradoxes by identifying novel heuristics underlying moral selfcorrection; showcase the potentials and pitfalls of utilizing these heuristics to improve self-correction; and reveal the generalization challenges of selfcorrection. contrary to these works, there are also prior studies which showcase issues with self-correction. @xcite indicates that self-correction would fail in reasoning tasks if no ground-truth answer is exhibited in the instruction. with respect to moral self-correction, @xcite demonstrate that self-correction is superficial, reflected by a limited reduction of immoralities in llms' hidden states. furthermore, @xcite provide evidence that llms lack awareness of the underlying rationale behind their self-correction decisions, since they cannot identify the stereotyped social groups or less toxic outputs in the self-correction process. similarly, @xcite and @xcite report issues that highlight the limitations of intrinsic self-correction. challenges with generalizability and heuristics. prior work on bertology @xcite @xcite and recent studies on the generalization behavior of neural language models @xcite empirically demonstrate that neural language models achieve generalization across various tasks by internalizing typical discourse constructions, rather than by truly acquiring or understanding languages. even in the context of word acquisition, @xcite suggests that reliance on superficial heuristics is a key factor underlying llms. additionally, there are some studies @xcite @xcite which highlight how llms still rely on shortcuts or shallow heuristics in various tasks; one popular example is natural language inference (nli) @xcite . in the context of morality, @xcite examines current learning paradigms through which llms acquire moral reasoning and show that, while llms exhibit generalization, they fail to grasp the underlying pragmatics of morals due to the distributional semantics nature of llms. benchmark. we leverage the bbq benchmark @xcite to study the social stereotypes mitigation task. the reasons that we use this benchmark are: (1) the social stereotypes mitigation task is a pragmatics-level task wherein the social dynamics of stereotypes are not explicitly available in text @xcite , indicating the difficulty and challenges of this task; (2) the causes of social stereotypes are well-recognized within the nlp community @xcite @xcite , allowing us to construct controlled discourse for fine-grained analysis; (3) bbq encompasses a range of social stereotypes, including those related to gender, age, race, etc. these categories exhibit distinct patterns in language usage, enabling a comprehensive investigation into the generalization of llm capabilities. backbone models. in this paper, we leverage various model architectures and scales to validate our hypothesis, including llama3.2-1b-instruct, llama3.2-3b-instruct, phi-3.5-mini-instruct (3.8b), llama-3-8b-instruct and mistral-7b-instruct-v0.3. we focus on smaller llms for two main reasons: (1) there are conjectures that smaller models are less capable of self-correction, and (2) smaller llms are more accessible and practical for the research community. additionally, we focus on characterizing fine-tuning corpus, helping us mitigate the influence of model architects. task formulation. evaluation. for each stereotype, we partition the data into training and test sets based on unique contexts to prevent any overlap. specifically, we randomly sample 80% of the unique contexts for training, with the remaining 20% reserved for testing. we report both the baseline performance and the performance of llms fine-tuned with our proposed discourse constructions on the test set. more details about the experimental settings are available in appendix a.1. ## heuristics in self-correction the previous section concludes that: (1) llms can make self-correction decisions without necessarily needing to self-diagnose, or consider, moral issues, and (2) a general and informative discourse construction does not yield consistent effects on llms' self-correction and self-diagnosis capabilities across different stereotype categories. therefore, this section aims to answer two research questions relevant to paradox1. to answer these questions, we first propose a discourse construction by referencing the task format associated with self-correction. we then conduct an ablation study on its components to identify an effective construction, and finally, we provide empirical evidence that there does exist shallow heuristics underlying those constructions. while identifying heuristics for semantically-or syntactically-driven tasks is straightforward due to the explicit presence of text-indicating heuristics, identifying heuristics for moral reasoning tasks is significantly more difficult due to the implicit nature of morals within text. furthermore, studying and classifying the morality of llm output often requires societal context, hence our use of the social stereotypes task within a situational learning context. to the best of our knowledge, we are first to identify these shallow heuristics for the improvement of moral self-correction. motivation. previous studies @xcite @xcite have explored corpus construction at various levels (e.g., sentence, utterance, and discourse) and trained llms from scratch to investigate how to allow llms to implement language acquisition. unlike prior research, our target capabilities reflect higher-level social cognition beyond basic we couldn't start the board meeting at 9am today because a man and a woman were late. people have gender stereotypes against women when determining who is the secretary. to avoid gender stereotypes, we should assume that both women and men can be a secretary. ## conflicts and generalization building on the shallow heuristics of discourse construction that support self-correction, as identified in section 4, this section further refines its characterization. our analysis in this section focuses on two research questions relevant to paradox2. given the heuristics proposed for moral selfcorrection, we conduct generalization tests and our experimental results suggest that: (1) conflicts emerge when attempting to enhance both capabilities simultaneously (section 5.1), and (2) given a certain size of llms and our found heuristics, the self-correction performance can be easily improved for both in-domain stereotypes and out-of-domain stereotypes (section 5.2). ## discussion due to the complex nature of llms, studies towards exploring the mechanisms underlying their behaviors are non-trivial. particularly, in the context of social pragmatics and morals, we would expect llms to possess both language proficiency and social cognition. the unique challenge of moral self-correction is the main barrier for its wide application, and is one reason that existing moral self-correction works are still very similar in terms of approaches. previous studies of mechanistic analysis mainly focus on the characteristics of llms' architectures and hidden states @xcite , which is not straightforward as llms are not capable of understanding languages @xcite . to avoid this, characterizing the corpus by examining its impact on llm behavior is a methodologically sound approach. moreover, it helps mitigate the influence of model architectures, especially considering recent studies @xcite that advocate using constructed grammar rather than generative grammar to analyze llms' behavior. jointly optimizing self-diagnosis and selfcorrection presents an intriguing challenge, given the inherently statistical nature of llms. one promising direction to resolve the conflict between these two capabilities is to enable llms acquire pragmatic reasoning for morality @xcite . although linguistic research suggests that pragmatic reasoning can be approximated through multi-step semantic inference @xcite , there is still no consensus on how to implement it in practice. nonetheless, such semantics-driven inference can benefit from the distributional semantics inherent to llms. ## future work and conclusion in this paper, we are the first to demonstrate the existence of shallow heuristics underlying moral selfcorrection, which we use to address two key paradoxes associated with moral self-correction and showcase how to improve it easily. future work can extend our analysis to self-correction tasks, such as code generation, story telling, and knowledgeintensive tasks, as well as explore the findings in the extrinsic self-correction scenario and investigate whether external feedback can loosen reliance on shallow heuristics.