CM oral E val: A Moral Evaluation Benchmark for C hinese Large Language Models (2024)

Abstract: What a large language model (LLM) would respond in ethically relevant context? In this paper, we curate a large benchmark CMoralEval for morality evaluation of Chinese LLMs. The data sources of CMoralEval are two-fold: 1) a Chinese TV program discussing Chinese moral norms with stories from the society and 2) a collection of Chinese moral anomies from various newspapers and academic papers on morality. With these sources, we aim to create a moral evaluation dataset characterized by diversity and authenticity. We develop a morality taxonomy and a set of fundamental moral principles that are not only rooted in traditional Chinese culture but also consistent with contemporary societal norms. To facilitate efficient construction and annotation of instances in CMoralEval, we establish a platform with AI-assisted instance generation to streamline the annotation process. These help us curate CMoralEval that encompasses both explicit moral scenarios (14,964 instances) and moral dilemma scenarios (15,424 instances), each with instances from different data sources. We conduct extensive experiments with CMoralEval to examine a variety of Chinese LLMs. Experiment results demonstrate that CMoralEval is a challenging benchmark for Chinese LLMs.

## introduction recent years have witnessed remarkable progress achieved by large language models in both natural language understanding and generation @xcite . despite such progress, a variety of risks have been found in the content yielded by llms, e.g., toxicity, unfaithfulness with hallucination @xcite . as llms become increasingly applicable to and integrated into real-world scenarios, the moral and ethical implications of their outputs should be regulated to ensure alignment with societal values and norms @xcite . to evaluate such alignment capabilities of llms, a wide variety of datasets have been proposed to examine dimensions like toxicity @xcite , bias @xcite and fairness @xcite . among them, the assessment of morality can be traced back to the moral foundation theory (mft) @xcite . mft categorizes moral precepts into five distinct domains, each comprising both positive and negative manifestations, e.g., care/harm or fairness/cheating. over time, mft has evolved into the foundational framework for subsequent specifications of datasets aimed at moral and ethical evaluation of llms @xcite . however, in striking contrast to the remarkable development of chinese llms, moral benchmarks tailored to chinese culture for evaluating the moral alignment capacity of chinese llms remains underexplored. to bridge this gap, we propose cmoraleval, a multiple-choice qa dataset grounded in the moral norms of chinese society. cmoraleval is meticulously curated through manual annotation on raw data collected from a chinese legal and ethical tv program "observations on morality" foot_0 and a set of "chinese moral anomies" foot_1 , followed by a rigorous quality review process. specifically, from 833 episodes of the tv program over the past three years, we generadditionally, from a set of 229 collected instances of chinese moral anomies, 615 templates are produced. with these templates, we create 30,388 data instances. all these instances are categorized into five groups of morality according to a pre-developed taxonomy: familial morality, social morality, professional ethics, internet ethics and personal morality. cmoraleval covers both explicit moral scenarios and moral dilemma scenarios. data statistics of cmoraleval is displayed in our main contributions are summarized as follows: ## related work the proposition of mft @xcite has laid the foundation for numerous subsequent datasets related to morality and ethics. some of these datasets are tailored to specific domains such as politics @xcite , social media @xcite and social sciences ( e.g., social chemistry 101 @xcite ). social chemistry 101 @xcite , based on mft, annotates experiential norms from 12 dimensions, ultimately resulting in a dataset comprising 292k roles of thumbs (rot). construction of the fine-grained lexical resource mfd @xcite involves meticulous refinement and expansion based on the foundation of mft, carried out by thoughtful deliberation by a specialized team of experts. mfd is later extended to emfd @xcite due to its limitations: mfd is formulated by a small group of experts, thus lacking the coverage of moral principles prevalent in the general population. furthermore, it does not account for the variability of a single word that belongs to different categories defined by mft in various contexts. ethics @xcite , similar to social chemistry 101 @xcite , establishes ethical benchmarks for specific scenarios based on several dimensions, including justice, deontology, virtue ethics, utilitarianism, and commonsense moral judgements. additionally, there are derivative moral benchmarks from previous datasets, such as moral stories @xcite , which builds on rots from social chemistry 101 and serves as a crowdsourced collection of structured, branching narratives for the study of grounded, goal-oriented social reasoning. prosocialdialog @xcite , derived from ethics @xcite and social chemistry 101 @xcite , captures scenarios and employs artificial intelligence to generate responses encouraging prosocial behavior based on common-sense social rules (i.e., experiential rules, rots). similarly, mic @xcite , another dialogue dataset, draws inspiration from rots but sources you're a local news reporter who gets a tip that a father in a spider-man costume is looking for his missing son, and after learning the basics about his son, you realize that you may have heard about his son. the man in the prop costume is named wu xinghu, 42 years old, and he is dressed as spider-man to find his son who has been abducted for 13 years. … perspective rot tv program introductions or moral anomies generating options explicit moral scenarios templates moral dilemma scenarios templates basic scenes contravening reasons perspective rot narrator rot experts checking template gallery final dataset high-speed rail hegemony … new scenes you're a local news reporter who gets a tip that a father in a spider-man costume is looking for his missing son, and after learning the basics about his son, you realize that you may have heard about his son. but not saying it keeps the news hot. figure 1 : the overall pipeline for collecting questions in cmoraleval. scene denotes an objective description of an event; narrator encompasses various characters involved in the event; rot refers to a descriptive cultural norm structured as the judgment of an action @xcite . each narrator corresponds to a specific rot, and this pairing is referred to as a narrator-rot pair. contravening reasons are legitimate justifications that may be perceived as contradicting the "rots". a narrator-rot pair is used for generating options, which uses chatgpt for assistance in the generating process. the highlighted text with yellow background represents different narrators in the basic scene. the highlighted text with grey background denotes a contravening reason in the new scene. the detailed generating process is described in appendix a.2. in addition to these datasets, benchmark evaluations have been also conducted on llms. trust-gpt @xcite has evaluated the ethical performance of certain llms, although it does not account for some exceptional ethical cases. in contrast, moralexceptqa @xcite encompasses moral exceptions, illustrating the complexities and uncertainties of ethical choices, albeit limited to three representative cases of moral exceptions. scruples @xcite , a dataset related to moral dilemmas, annotates one of two behaviors as less ethical, yet the lack of clear association between these two behaviors underutilizes the model's reasoning and contextual comprehension abilities. unlike most moral benchmarks, cmoraleval encompasses two distinct moral scenarios, offering diverse perspectives for assessing the morality of llms. furthermore, cmoraleval comprises five moral categories pervasive in chinese society. @xcite propose a datast similar to cmoraleval, which includes only two options per data instance. in some highly ambiguous situations, both options violate moral norms, potentially contaminating the dataset. in cmoraleval, we ensure that there is only one correct option in different scenarios. additionally, unlike existing moral benchmarks, each data instance in cmoraleval includes three options, with one option unrelated to morality, thereby adding complexity. cmoraleval also incorporates varying perspectives from different narrators (multiple parties and bystanders) on the same scene. ## dataset curation cmoraleval encompasses two distinct moral scenarios, each accompanied by questions derived from two data sources, and features a unique annotation process for each scenario. we have established a comprehensive annotation platform, enabling annotators to label various moral situations effectively. simultaneously, stringent quality control measures are implemented to ensure the integrity and reliability of cmoraleval. figure 1 illustrates the overall pipeline for creating the dataset. ## experiments we evaluated 26 open-source chinese llms with the created benchmark cmoraleval, ranging in size from chatyuan-large-v2 (0.7b) (xuanwei zhang and zhao, 2022) to yi-34b-chat (34b). all selected models have been fine-tuned, with some possibly undergoing value-aligned fine-tuning, such as chinese-alpaca-2-7b-rlhf @xcite and chinese-alpaca-2-1.3b-rlhf @xcite , while others may have been fine-tuned for chat or other purposes. we provide extra information for the chosen llms in appendix a.5.1. ## conclusion in this paper, we have presented cmoraleval, a dataset comprising over 30,000 entries that span five moral categories, two types of scenarios and two data sources. the range of options for evaluating chinese llms has been significantly expanded. this high-quality dataset, produced under stringent annotation standards, reveals that current chinese llms exhibit considerable disparities and underperformance in moral reasoning, indicating substantial room for improvement.