HISTOIRESMORALES : A F rench Dataset for Assessing Moral Alignment (2025)

Abstract: Aligning language models with human values is crucial, especially as they become more integrated into everyday life. While models are often adapted to user preferences, it is equally important to ensure they align with moral norms and behaviours in real-world social situations. Despite significant progress in languages like English and Chinese, French has seen little attention in this area, leaving a gap in understanding how LLMs handle moral reasoning in this language. To address this gap, we introduce HistoiresMorales, a French dataset derived from MoralStories, created through translation and subsequently refined with the assistance of native speakers to guarantee grammatical accuracy and adaptation to the French cultural context. We also rely on annotations of the moral values within the dataset to ensure their alignment with French norms. HistoiresMorales covers a wide range of social situations, including differences in tipping practices, expressions of honesty in relationships, and responsibilities toward animals. To foster future research, we also conduct preliminary experiments on the alignment of multilingual models on French and English data and the robustness of the alignment. We find that while LLMs are generally aligned with human moral norms by default, they can be easily influenced with user-preference optimization for both moral and immoral data.

## introduction recently, there has been a growing interest in assessing and identifying the emergent properties of large language models (llms) @xcite ). with their extensive pre-trained knowledge, llms such as mistral @xcite auto-regress by predicting subsequent tokens based on provided conditions or instructions. however, llms still struggle with multilingual complex instructions, often requiring additional customization or alignment steps to better meet user expectations for input requests. a significant aspect of alignment is ensuring that llms adhere to human moral values and principles, such as humility, honesty, helpfulness, and competitiveness, to make their interactions safer and more reliable @xcite @xcite . learning from user preferences in multilingual settings is a complex task, further challenged by the varying performance across different target languages @xcite . while a few papers explored this alignment in languages other than english, the study of such case is still limited to few languages due to a lack of data @xcite , and, to the best of our knowledge, no such work has been conducted for french. in the line of works such as the french crows-pairs dataset @xcite for stereotypes, we contribute to resources for evaluating llms' capabilities in social reasoning tasks in french. this paper introduces histoiresmorales, the first corpus for situated social reasoning in french, consisting of 12,000 stories that encompass moral norms, intentions, situations, actions (both deviating from norms and not), and the consequences of these actions. histoiresmorales is adapted to french from the widely used moral-stories dataset @xcite . we first translate the moralstories dataset and then refine the translations through multi-step manual annotations. motivated by recent advances in cultural awareness in nlp @xcite , we develop a translation approach that ensures grammatical fluency and culture-specific translation of named entities and activities, to build a semantic space consistent with the french cultural context. validation by native speakers suggests that his-toiresmorales is generally aligned with the moral values commonly shared in france. our main contributions are the following. (i) in §3, we introduce histoiresmorales, a first dataset of narratives describing moral behaviour in french, which can be used alongside parallel english data for comparative analysis. then, we explain the translation pipeline we build using errorexplanation prompts supplied with manual annotations and human feedback to achieve high-quality translations. (ii) we ensure the quality of texts in histoiresmorales dataset, and assess the alignment of the values contains in it with the ones of french human annotators ( §4). (iii) we compare llms' moral alignment with human norms using sentence likelihood and classification of moral actions with declarative prompts ( §5). finally, (iv) we investigate the robustness of llms' multilingual moral alignment by making it shift to favour either moral or immoral actions using direct preference optimisation (dpo, §6). the first results show that llms align better with moral norms in english (en) than in french (fr), with low robustness of this alignment, paving the way for further research. ## related work human values alignment of lms the emerging abilities of llms in language understanding have raised questions about their moral biases @xcite or whether they may perform well on moral reasoning tasks. @xcite and @xcite evaluate the moral biases llms encode and their aptitudes to apply moral values. likewise, @xcite investigate the generative capacities of an llm to produce descriptions of actions and consequences aligned with human shared values. other research explores applications of llms trained on tasks involving morality challenges @xcite . the problem of moral alignment of llms with human values is also investigated under the perspective of various moral schools-of-thought @xcite . although most research on alignment focuses on us-centred moral values, @xcite show that llms encode different moral biases depending on the target language in german, czech, arabic, chinese, and english. sim-ilarly, @xcite explore the alignment of llms with different branches of normative ethics in english, spanish, russian, chinese, hindi, and swahili. @xcite investigate whether english-based llms accurately infer moral norms across cultures, finding better performance for western cultures over non-western ones. finally, at the intersection of these ideas, @xcite study multilingual models in a multicultural setting, concluding that reliance on a few dominant languages often leads to conceptual inconsistencies on the encoding of culture and moral values. this concern highlights the need for diversity of languages and moral norms resources when studying the moral understanding of llm. while some works aim to emphasize pluralistic values @xcite , they restrict their objective to english data. to the best of our knowledge, our work is the first attempt to create a dataset to assess llm's morality in french, the 5th spoken language in the world with 321 millions of speakers foot_1 . ## the histoiresmorales dataset we introduce histoiresmorales, a french dataset built upon a corpus of human-written moral stories in english called moralstories. 3 this dataset was introduced by @xcite . his-toiresmorales and moralstories consist of short narratives that describe moral and deviant behaviour in social situations centred around personal relationships, education, commerce, domestic affairs, and meals. we provide details about corpus statistics for both datasets in appendix c. each story begins with a context: a moral norm, a description of the social situation and its participants, and the actor's intention. subsequently, each story is followed by two continuations: a moral action and its consequence and an action that deviates from the norm. we provide an example from the dataset in translation setup we use gpt-3.5-turbo-16k model for translations, accessed via the api in november 2023. we initiate the data translation process with a simple prompt and refine it through human feedback. below, we describe the construction of the prompt body and the corresponding data annotation procedures. ## model moral alignment in this section, we show that the dataset can be used to investigate the alignment of llms with human values across languages. we demonstrate how our dataset, combined with the one from @xcite , can serve to investigate 1) the alignment of llms with human moral norms and 2) the impact of language (english and french) on it. ## influencing llm with direct preference optimization in this section, we probe whether the models' alignment is robust to external influence, an important task to ensure that decision-support models do not produce immoral content. methodology using direct preference optimization (dpo) @xcite , we aim to influence the model to prefer either moral (dpo m ) or immoral (dpo i ) actions. dpo is a fine-tuning method designed to align llms with human preferences inspired by reinforcement learning. it is based on two models, a reference model and the main model, that is fine-tuned with an objective to increase the likelihood of preferred responses while decreasing that of dispreferred responses. thus, dpo also relies on pairs of entries, the preference data, where one entry is considered preferable to the other. we replace those pairs with moral and immoral actions to evaluate whether the model can be influenced to prefer ones over the others. furthermore, we investigate the number of examples required to shift the model toward a specific leaning, which serves as a measure of the model's robustness to moral influence. ## conclusion this work introduces histoiresmorales, the first dataset for social reasoning informed by behavioural guidelines in the french language. the introduced dataset is an augmentation of the moralstories dataset with a bilingual addition of french. the dataset is created through prompting with human-crafted demonstrations, complemented by detailed error explanations to guarantee high-quality translations. we also conduct an analysis of dataset quality, including the cultural value alignment of social norms and actions with the moral principles shared in france. our dataset encourages practitioners to explore potential applications of bilingual data for grounded social reasoning. we perform initial investigations into potential applications and demonstrate how datasets can be used to compare the alignment of moral values in llms across two languages. our experiment results indicate a substantial difference in action choices among existing llms between english and french. we demonstrate how our dataset can be leveraged to adapt to user preferences using dpo, requiring less than 100 examples. future work may explore the models' capacity for generating action consequences based on input actions. another potential research direction is studying multilingual alignment with dpo using the bilingual dataset we introduced.