Diagnosing Moral Reasoning Acquisition in Language Models: Pragmatics and Generalization (2025)

Abstract: Ensuring that Large Language Models (LLMs) return just responses which adhere to societal values is crucial for their broader application. Prior research has shown that LLMs often fail to perform satisfactorily on tasks requiring moral cognizance, such as ethics-based judgments. While current approaches have focused on fine-tuning LLMs with curated datasets to improve their capabilities on such tasks, choosing the optimal learning paradigm to enhance the ethical responses of LLMs remains an open research debate. In this work, we aim to address this fundamental question: can current learning paradigms enable LLMs to acquire sufficient moral reasoning capabilities? Drawing from distributional semantics theory and the pragmatic nature of moral discourse, our analysis indicates that performance improvements follow a mechanism similar to that of semantic-level tasks, and therefore remain affected by the pragmatic nature of morals latent in discourse, a phenomenon we name the pragmatic dilemma. We conclude that this pragmatic dilemma imposes significant limitations on the generalization ability of current learning paradigms, making it the primary bottleneck for moral reasoning acquisition in LLMs.

## introduction given the widespread usage of llms across all facets of society, enabling such models with moral reasoning capabilities has become a significant research goal. though ai alignment @xcite has become the de-facto method to align llms with human values, its effectiveness has been @xcite . one significant complaint is that alignment with human preference does not allow llms to achieve intrinsic alignment, resulting in various safety issues, e.g., jailbreak attacks @xcite and propagation of social biases to downstream tasks @xcite . however, enabling llms to develop moral reasoning capabilities is a non-trivial task; it is both a pragmatics-level task @xcite , as well as philosophically challenging, due to debate over the correct representation of human morals and ethics @xcite . @xcite and @xcite represent pioneering efforts to enable llms to acquire ethical judgment capabilities by fine-tuning them on curated textual data that jointly depicts various moral situations alongside corresponding judgments. @xcite introduces an in-context learning method to help llms perform moral reasoning, based on a top-down framework driven by the moral foundation theory @xcite . @xcite introduce a social sandbox wherein llms can learn how to be moral through interactions. @xcite propose a moral alignment framework to make llm agents behave morally through a newly designed intrinsic moral reward function based on the iterated prisoner's dilemma foot_0 . in addition to those efforts proposing solutions, new benchmarks have also been proposed @xcite @xcite . there are also several studies which highlight the inefficiency of llms on tasks requiring moral reasoning. @xcite has criticized the @xcite work described above, because while their intended goal was normative ethics, they instead leveraged a bottom-up approach for learning descriptive ethics @xcite . @xcite empirically demonstrate that the current learning paradigm for moral reasoning tasks relies on a large training dataset. @xcite also show the failure of llms on social intelligence tasks such as theory-of-mind. in cognitive science, @xcite sug-gest that while llms excel in formal language competence, they struggle with functional language competence which is an essential requirement for acquiring moral reasoning capabilities. more fundamentally, @xcite and other studies in bertology @xcite , argue that transformers cannot achieve true language acquisition, as it necessitates physical grounding and situated communicative intent @xcite , which extends beyond the distributional semantics captured by transformers @xcite @xcite . previous studies @xcite demonstrate that llms do not have consistent moral or ethical orientations across various instances, which is contrary to the moral consistency principle @xcite . appendix a.1 contains additional related works and motivation pertaining to machine ethics. to address this debate, in this paper we pursue a deeper understanding of the mechanisms underlying current learning paradigms for moral reasoning acquisition. we argue that while existing paradigms can improve llms' performance on morality-related tasks, this enhancement: (1) primarily arises from distributional similarities between seen and unseen ethical situations, and (2) faces challenges in generalization due to the inherently pragmatic nature of morality. we name this phenomenon as the pragmatic dilemma @xcite of moral reasoning acquisition, which arises from the inherent gap between the nature of distributional semantics in llms and the pragmatic nature of morality. significant consequences of this pragmatic dilemma include poor generalization and a lack of intrinsic alignment. specifically, we employ three fundamental tasks, moral foundations classification, rule of thumb generation, and ethical judgment prediction, as downstream evaluations of moral reasoning acquisition. we then compare their generalization characteristics with a representative semantics-driven task, sentiment analysis. motivated by the distributional semantics theory, we: (1) empirically show the generalization and convergence pitfalls of moral foundations classification; (2) given the characteristic of autoregressive language models, propose a representational likelihood algorithm (rla) to statistically correlate representational similarity between seen and unseen situations with the prediction likelihood of unseen situations; and (3) using rla, perform mechanistic analysis of llm performance gains for unseen situations. section 2 introduces the prevalent learning paradigm for moral reasoning acquisition and highlights the generalization challenges in fine-tuning masked language models for moral foundation prediction. section 3 presents experimental results across different learning paradigms, and section 4 provides a detailed mechanistic analysis. based on our experimental results, we conclude that the pragmatic dilemma blocks the effectiveness of current learning paradigms. ## preliminary background in this section, we begin by introducing the benchmarks and dataset annotation used in our study. we then present the prevailing learning paradigm for moral reasoning acquisition. finally, we use the moral foundations prediction task with a masked language model, specifically bert @xcite , as a case study, to illustrate the generalization challenges of this task by drawing comparisons to the semantics-level task of sentiment analysis. ## fine-tuning for moral reasoning acquisition in this section, we introduce fine-tuning strategies and experimental results of existing learning paradigms for moral reasoning acquisition, focusing on the tasks of rot generation and ethical judgment prediction. experimental settings. we take mistral-7b foot_3 and llama3-8b foot_4 as the backbone models and leverage the lora method to fine-tune them through a supervised fine-tuning loss. for each benchmark, we employ two fine-tuning strategies for rot generation and four fine-tuning strategies for ethical judgment prediction. for rot generation, we fine-tune llms: (1) to directly socialchem bertscore rouge1 rouge2 rougel mic bertscore rouge1 rouge2 rougel rot .777 .229 .096 .213 rot .768 .175 .077 .168 moral-rot .836 .416 .205 .401 moral-rot .826 .393 .192 .379 judg .7240 .230 .137 .230 judg .671 .071 .000 .071 moral-judg .7632 .464 .346 .464 moral-judg .762 .314 .000 .314 rot-judg .7626 .464 .346 .464 rot-judg .660 .061 .000 .061 moral-rot -judg .7628 .463 .345 .463 moral-rot -judg .761 .306 .000 .306 the moral-judg is the best strategy for both socialchem and mic, in terms of the judgment prediction task. additional results for llama3 are availabe in the prompting format and lora fine-tuning settings are available in appendix a.4. we consider 10000 samples with only one underlying moral foundation for analytical convenience. in the process of fine-tuning, we take the check point with the least loss on the development set, and report its performance on the test set. during inference, we prompt fine-tuned llms to first generate intermediate predictions before producing the final rot or ethical judgment, following the same prompting strategy used during fine-tuning. for example, in the moral-rot strategy, llms are instructed to first predict the moral foundation based on the given situation and subsequently generate the rot using both the situation and the predicted moral foundation. following @xcite , we report the performance of the bertscore @xcite , rouge-1, rouge-2, and rouge-l metrics. rot generation and ethical judgment prediction align with the core capabilities essential for morality-related scenarios and serve as prototypical formats for moral reasoning. by incorporating moral foundations into rot generation, we aim to guide llms to first identify the moral foundation associated with a given situation, thereby improving the quality of the generated rot. rots serve as instances of evidence and explanation for ethical judgments, aligning with previous studies that seek to enhance llms' social intelligence through social interaction environments @xcite . ## mechanistic analysis in the previous sections, we introduced preliminary studies regarding the generalization pitfalls of the moral foundations classification task (section 2), and the performance of fine-tuning llms for two moral reasoning tasks (section 3). in this section, we: (1) propose the representational likelihood algorithm (rla) which can uncover supportive training samples for a given test sample; (2) explore the characteristics of supportive training samples, demonstrating that the introduction of additional information to enhance generalization aligns with the generalization mechanism of the semantics-level task; (3) showcase that the pragmatic dilemma still holds even though fine-tuned llms perform better in rot generation and ethical judgment prediction. motivation. our study builds on the representational learning nature of llms and the widely accepted principle in generalization theory that a welltrained machine learning model can generalize effectively when the training and test set distributions are closely aligned in the feature space @xcite . since neural language models capture distributional semantics, representational similarity can be interpreted as equivalent to distributional similarity. recall from section 2 that we highlighted the generalization pitfalls of the moral foundation classification task. we argue that similar pitfalls should also exist for rot generation and ethical judgment prediction. our hypothesis is that for a given test sample, the llm can generalize effectively only if highly similar training samples have been adequately learned during finetuning. to test this hypothesis, we propose a novel algorithm to identify the training samples most conducive to the generalization of a given test sample within the representation space. ## discussion generalization remains a significant challenge in the acquisition of moral reasoning, and no optimal solution has yet been identified. recently, @xcite proposed a hybrid approach that combines bottom-up and top-down methods. however, their method still relies on a substantial number of training samples. @xcite demonstrated that pragmatic reasoning can be approximated through semantic inferences, highlighting a linguistic foundation for this connection. nevertheless, how to formally structure a semantic inference framework for moral reasoning remains an open question. one promising direction is to ground such a framework in the human moral decisionmaking process. @xcite introduced the first benchmark in the nlp community focused on how humans make moral decisions. their benchmark is based on an intuitionist model: participants are first asked to make a moral judgment and then provide an explanation for their decision. this type of annotation presents challenges for llms, as human explanations are expressed in free-text form and often lack enough situated semantic information @xcite . despite these difficulties, the benchmark offers a valuable opportunity for exploring methods that aim to derive semantic inferences from human rationales, potentially bridging the gap in pragmatic reasoning for morality. @xcite propose a social bias inference framework to identify social implications, but their approach still relies heavily on semanticslevel signals, as the social implications in their work are primarily associated with explicit toxicity. @xcite introduce a pragmatic inference framework targeting implicit toxicity, where metaphorical expressions of toxicity pose particular challenges for llms. they provide empirical evidence that their framework enhances the detection of implicit toxicity in off-the-shelf llms, though it still depends on the powerful distributional semantics captured by these models. we argue that the study of moral reasoning acquisition should focus on smaller llms with 3b-8b parameters, as these models are less powerful than very large ones yet still exhibit satisfactory instruction-following capabilities. the powerful distributional semantics of large models make it difficult to disentangle whether performance gains stem from the proposed pragmatic inference framework or from learned statistical correlations between situations and moral reasoning objectives. ## conclusion in this paper, we answered the question can current learning paradigms enable llms to acquire moral reasoning? based on distributional semantics and the pragmatic nature of morality, we demonstrate that (1) the pragmatic dilemma of llms make them inefficient in moral reasoning acquisition tasks; (2) the improved performance still stems from the realm of distributional semantics; (3) the current learning paradigm for moral reasoning acquisition impairs llms' language modeling capability more than semantics-level tasks. we conclude that the pragmatic dilemma is the primary bottleneck for moral reasoning acquisition.