Probabilistic Aggregation and Targeted Embedding Optimization for Collective Moral Reasoning in Large Language Models (2025)

Abstract: Large Language Models (LLMs) have shown impressive moral reasoning abilities. Yet they often diverge when confronted with complex, multi-factor moral dilemmas. To address these discrepancies, we propose a framework that synthesizes multiple LLMs’ moral judgments into a collectively formulated moral judgment, realigning models that deviate significantly from this consensus. Our aggregation mechanism fuses continuous moral acceptability scores (beyond binary labels) into a collective probability, weighting contributions by model reliability. For misaligned models, a targeted embedding-optimization procedure fine-tunes token embeddings for moral philosophical theories, minimizing JS divergence to the consensus while preserving semantic integrity. Experiments on a large-scale social moral dilemma dataset show our approach builds robust consensus and improves individual model fidelity. These findings highlight the value of data-driven moral alignment across multiple models and its potential for safer, more consistent AI systems.

## introduction large language models (llms) have demonstrated a growing ability to analyze intricate social contexts and provide novel insights into human behavior and moral decision-making @xcite @xcite . recent work shows that, when given carefully designed prompts, llms can handle a range of moral judgments in straightforward scenarios @xcite . yet moral reasoning is profoundly contextual: competing ethical principles, convoluted personal narratives, and diverse social norms can all reshape how a dilemma should be interpreted @xcite . as a result, even strong llms frequently differ in 1 our code and data are available at: https://github. com/yuanchencn/collective-moral-reasoning . ## problem setup let @xmath0, 2, . . . , n } index a collection of moral scenarios, and let @xmath1, 2, . . . , m } index a set of moral philosophical theories (i.e., virtue, justice, deontology, utilitarianism, and commonsense morality). the goal is to obtain moral judgments from l large language models for each scenariotheory pair (i, j). specifically, each model m provides a continuous annotation a m,j,@xmath2, 1], indicating the degree to which it deems scenario i morally acceptable under theory j. this continuous formulation allows for more nuanced interpretations than binary annotations: values near 0.5 reflect ambiguity or moral tension, while values closer to 0 or 1 reflect clearer moral signals. although these continuous annotations yield rich information about each model's stance, they can vary significantly across models. we therefore introduce a collective opinion γ j,@xmath3, 1], which integrates the annotations {a m,j,i } l @xmath4: ## methodology our approach comprises two major components (figure 2 ). first, we propose a probabilistic aggregator based on a truncated-normal formulation. this aggregator derives a consensus probability γ j,i for each scenario-theory pair by modeling both the reliability and variance of each llm's annotations. second, for models exhibiting significant misalignment, we apply a targeted embedding optimization on theory-related tokens. this twofold strategy allows us to both establish a meaningful moral consensus and refine individual models' embeddings when they diverge from that consensus. ## experimental evaluation two key questions are examined: (1) does the truncated-normal em approach produce a coherent collective opinion across llms? (2) can targeted embedding optimization effectively reduce misalignment for specific theories and models? ## conclusion we propose a truncated-normal em framework to synthesize moral judgments from multiple llms and introduce targeted embedding optimization to reduce misalignment in theory representations. experiments on a large-scale moral dilemma dataset highlight that our method effectively aggregates continuous judgments and models like llama benefit substantially from local embedding refinements, achieving closer alignment to the consensus. overall, our work demonstrates a systematic path toward reconciling divergent model opinions and improving moral reliability in multi-llm settings.