Does Cross-Cultural Alignment Change the Commonsense Morality of Language Models? (2024)

Abstract: Alignment of the language model with human preferences is a common approach to making a language model useful to end users.However, most alignment work is done in English, and human preference datasets are dominated by English, reflecting only the preferences of English-speaking annotators.Nevertheless, it is common practice to use the English preference data, either directly or by translating it into the target language, when aligning a multilingual language model.The question is whether such an alignment strategy marginalizes the preference of non-English speaking users.To this end, we investigate the effect of aligning Japanese language models with (mostly) English resources.In particular, we focus on evaluating whether the commonsense morality of the resulting fine-tuned models is aligned with Japanese culture using the JCommonsenseMorality (JCM) and ETHICS datasets.The experimental results show that the fine-tuned model outperforms the SFT model. However, it does not demonstrate the same level of improvement as a model fine-tuned using the JCM, suggesting that while some aspects of commonsense morality are transferable, others may not be.

## introduction while large language models (llms) trained on massive datasets have been demonstrated to possess remarkable capabilities in natural language understanding and generation, these models have also been shown to generate responses containing toxic, untruthful, biased, and harmful outputs @xcite @xcite @xcite . the challenge for the field is thus to align the behavior of the llms with human values, steering the models to generate responses that are informative, harmless, and helpful @xcite @xcite . however, existing studies in this field have primarily focused on english. the common approach to align multilingual llms is to translate an english preference dataset to the target language or to synthesize a dataset using highly capable llms (e.g., gpt-4) @xcite @xcite @xcite . indeed, previous research has demonstrated that it is possible to align a multilingual chat llm in languages with limited resources if the preference data in english is sufficiently large @xcite @xcite . the question is whether such alignment strategies result in language models marginalizing the culture and values of non-english-speaking communities @xcite . in this paper, we focus on studying the effect of the alignment of language models on their sense of commonsense morality as a case study. in particular, we investigate japanese llms fine-tuned with multilingual datasets on the understanding of commonsense morality in japan. the term commonsense morality refers to the body of moral standards and principles that most people in a given community intuitively accept @xcite . it is important to note that commonsense morality is known to be culturally dependent @xcite . for instance, @xcite points out that the delphi classification model for judging the commonsense morality @xcite ) outputs it's normal when prompted with the question greeting by kissing on the cheek in japan, yet it is typically considered impolite in japan. the objective of this paper is to evaluate the effect of aligning japanese llms with english resources. in particular, we investigate how the alignment process affects the commonsense morality of the models. the initial step is to assess the impact of aligning llms with the jcommon-sensemorality (jcm) dataset and the commonsense morality subset of the ethics dataset. we then evaluate three japanese llms aligned with the development set of the jcm and show that they achieve higher accuracy on the jcm than the models aligned with the ethics dataset. interestingly, we observe that the llms aligned with the englishtranslated jcm dataset achieve higher accuracy than the llms aligned with the japanese-translated ethics dataset. this result suggests that cultural differences may be more challenging to learn and generalize than language differences for the llms. then, the impact of aligning llm models with primarily english resources, which is currently the most prevalent approach for training multilingual models, is evaluated. the experimental results demonstrate that incorporating an english dataset and a multilingual reward model significantly enhances the instruction-following capability of the japanese llm, including the jcm dataset. nevertheless, the model trained on the development set of the jcm dataset outperforms the model trained on english resources in the test set of the jcm dataset. this suggests that aligning with non-japanese resources can facilitate the improvement of shared commonsense morality. however, it is possible that this may not generalize to the specific commonsense morality observed in japanese culture. ## related work while cross-lingual transfer has been successful in various nlp tasks @xcite @xcite @xcite , cross-cultural transfer presents a significant challenge @xcite @xcite @xcite @xcite . previous studies have demonstrated that the alignment can influence the language model to prioritize specific values or groups of people @xcite . a number of studies have been conducted with the objective of investigating the diversity of human preference @xcite @xcite @xcite @xcite . the prism alignment project is designed to collect preference data from annotators with a variety of backgrounds @xcite . @xcite posits that pluralistic alignment is of significant importance in serving people with diverse language annotator japanese english japan jcm jcm-en us, canada, gb ethics-ja ethics several studies have examined the moral beliefs and commonsense morality of nlp systems @xcite @xcite @xcite . @xcite introduces the ethics dataset, which is used to evaluate the moral judgments of language models, including commonsense morality. the data is collected from english speakers in the united states, canada, and great britain. @xcite examined the capabilities of llms in the context of cultural commonsense tasks. while their experiments focus on evaluating the performance of the instruction-tuned llms, our study focuses on the effect of alignment process on the cultural commonsense understanding of the llms. ## evaluation of alignment with japanese commonsense morality dataset we first assess the impact of aligning llm models with english and japanese commonsense morality datasets. datasets. the effect of alignment with cultural commonsense morality is evaluated using the jcm and a subset of the ethics dataset @xcite . the jcm dataset follows the protocol of collecting short sentences from the commonsense morality subset of the ethics dataset, with the exception that the crowd workers are required to speak japanese and are from japan. the jcm dataset comprises only short sentences, therefore, for evaluation purposes, we utilise the first 2000 short sentences of the commonsense morality subset of the ethics dataset. in order to isolate the cross-cultural and cross-lingual differences, we translate the jcm into english (jcm-en) and the ethics into japanese (ethics-ja) using wmt 21 x-en and en-x models @xcite ( the development sets are employed for model #params #tokens instruction tuning calm2 7b 1.3t of japanese and english (not disclosed) llm-jp 13b 138b of japanese and 140b of english japanese and english swallow 7b 100b of japanese + llama 2 (2.4t, primarily english) english setup. we use three japanese sft models, calm2, foot_0 llm-jp, foot_1 and swallow-7b foot_2 for evaluation. while calm2 and llm-jp are pretrained from scratch to construct a japanese llm, swallow is a japanese continual pre-training model of llama 2 ( as a reference, we evaluate the accuracy of gpt-3.5 turbo on the jcm dataset using the same prompt. 4 the accuracy of gpt-3.5 turbo is 0.757. @xcite reports that the accuracy of gpt-4 on the short sentences of the commonsense morality subset of the ethics dataset is 0.95. ## evaluation of alignment using real-world user's prompts in section 3, we observe that commonsense morality may be culturally dependent, and the alignment with a certain dataset may bias the llm. the question is whether the same bias occurs when aligning with a more generic preference dataset rather than a dataset explicitly tuned to train commonsense morality. in this section, we align a japanese llm with english resources translated into japanese and evaluate its effect on its commonsense morality. dataset. the chatbot arena conversations dataset is selected for use in this study because it contains real-world user prompts @xcite . the instructions written in english are translated into japanese using the wmt21 en-x nmt model @xcite . the translated instructions are then input into calm2, resulting in two responses per input. we use the oasst reward model to label the preference over the two responses @xcite . the oasst reward model is employed to label the preference between the two responses. the model is trained on approximately 40% english and 40% spanish messages, with japanese messages comprising approximately 0.4%. consequently, while the model is capable of understanding japanese sentences, its primary training is on english-or spanish-speaking annotators. this approach yields a japanese preference dataset (chatbotarena-ja) derived from an en- setup. we use calm2 for this experiment. we fine-tune the model on chatbotarena-ja using dpo @xcite used, where one of the entries is positive (incorrect) and the other is negative (not incorrect). the sentences are input into the reward model, and the accuracy is computed by the ratio of paired entries with the negative entry having a higher reward score than the positive entry. paired entries with the same labels are discarded for this experiment. the oasst reward model achieves an accuracy of 0.692 on the jcm dataset. result. the fine-tuning process has resulted in a significant enhancement in japanese mt-bench performance ( ## conclusions the objective of this study is to evaluate the effect of aligning an llm with english annotations to the commonsense morality of the japanese llm. three japanese llms are trained using the training set of the jcm and the ethics. interestingly, the models trained on the english-translated jcm dataset achieve higher accuracy than the models trained on the japanese-translated ethics dataset, indicating that cross-cultural transfer may be more challenging than cross-lingual transfer. we then evaluate a model trained using the chatbot arena conversations dataset translated to japanese with preferences annotated by a multilingual reward model (oasst) @xcite . the accuracy improved on both ethics and jcm, but was lower than that aligned with the datasets directly. the result shows that translating rich english resources into japanese can be beneficial in aligning japanese llms, even improving the accuracy of japanese commonsense morality. nevertheless, the results indicate the potential for further enhancement of the model's comprehension of cultural commonsense morality by using the annotations provided by members of the communities.