Exploring Multilingual Concepts of Human Values in Large Language Models: Is Value Alignment Consistent, Transferable and Controllable across Languages? (2024)

Abstract: Prior research has revealed that certain abstract concepts are linearly represented as directions in the representation space of LLMs, predominantly centered around English. In this paper, we extend this investigation to a multilingual context, with a specific focus on human values-related concepts (i.e., value concepts) due to their significance for AI safety. Through our comprehensive exploration covering 7 types of human values, 16 languages and 3 LLM series with distinct multilinguality (e.g., monolingual, bilingual and multilingual), we first empirically confirm the presence of value concepts within LLMs in a multilingual format. Further analysis on the cross-lingual characteristics of these concepts reveals 3 traits arising from language resource disparities: cross-lingual inconsistency, distorted linguistic relationships, and unidirectional cross-lingual transfer between high- and low-resource languages, all in terms of value concepts. Moreover, we validate the feasibility of cross-lingual control over value alignment capabilities of LLMs, leveraging the dominant language as a source language. Ultimately, recognizing the significant impact of LLMs’ multilinguality on our results, we consolidate our findings and provide prudent suggestions on the composition of multilingual data for LLMs pre-training.

## introduction recent years have witnessed the emergence of large language models, such as chatgpt (ope-nai, 2023a), @xcite , and llama2 @xcite . these llms have shown powerful capabilities in natural language understanding and generation @xcite @xcite . however, alongside with their prowess, llms present potential risks. research has demonstrated that * corresponding author llms can generate responses containing toxic, untruthful, biased, and even illegal content @xcite @xcite . thus, aligning llms with human values (i.e., value alignment) is necessary for unleashing their potential safely. human values, encompassing concepts like fairness, deontology, utilitarianism, and so on, although challenging to be precisely defined in language, are undoubtedly embedded in textual form @xcite . recently, @xcite have introduced representation engineering (repe) to enhance the transparency and controllability of deep neural networks. through repe, they unveil that high-level concepts can be extracted as concept vectors from llms, utilizing positive and negative text pairs aligned with the directions of specific concepts. these concept vectors, representing the directions of corresponding concepts, can be utilized to assess whether the behavior of llms aligns with or to steer their behavior towards the target directions @xcite @xcite . however, existing studies on concept representations in llms have primarily focused on english @xcite , leaving multilingual concepts unexplored. our work is the first to explore multilingual concepts in llms, emphasizing human values-related concepts to advance multilingual ai safety and utility. the primary research questions we aim to answer are as follows: (q1) do llms encode concepts representing human values in multiple languages? (q2) to what extent are these concepts consistent and transferable across different languages? (q3) whether llms trained with different distributions of multilingual data exhibit distinct multilinguality in these concepts? (q4) is value alignment of llms controllable across languages? to address these questions, we propose a framework consisting of 5 com-ponents: extracting multilingual concept vectors from llms ( §3.1) and evaluating their correlation with the corresponding concepts (concept recognition task in §3.2) to answer q1; computing crosslingual similarity of concept vectors ( §3.3) and performing cross-lingual concept recognition ( §3.4) to answer q2 and q3; and manipulating model behavior cross-lingually via concept vectors ( §5) to answer q4. our analysis covers 7 concepts related to human values: commonsense morality, deontology, utilitarianism, fairness, truthfulness, toxicity and harmfulness, given their significance for ai safety @xcite @xcite @xcite @xcite . to ensure the breadth and reliability of our findings, we have selected these 7 concepts for their diverse definitions and ethical attributes @xcite . throughout this paper, we collectively refer to them as "value concepts" to reflect their diversity and keep consistent with existing ai alignment research @xcite @xcite . for comprehensive definitions, ethical backgrounds and examples of these value concepts, please refer to appendix a. in addition to diverse human values, our experiments involve 16 languages foot_0 and 3 llm families with different multilinguality. specifically, we categorize the multilinguality of these 3 llm families based on language distributions in their pre-training data into 3 groups: english-dominated llms (llama2-chat series in our experiments), chinese & english-dominated llms (i.e., qwenchat series), and llms with more balanced multilinguality (i.e., bloomz series). appendix d provides detailed language distributions of their pre-training data. through in-depth analysis spanning multiple tasks, value concepts, languages and llms, our key findings are as follows: drawing from these findings, we prudently consider the following suggestions for multilingual pretraining data of llms, which might contribute to enhancing multilingual ai safety and utility. first, despite the positive effect of dominant languages as sources for cross-lingual alignment transfer ( §5.2), it is crucial to avoid an excessive prevalence of these languages to mitigate unfair cross-lingual patterns, such as inconsistent multilingual representations ( §4.3.1), distorted linguistic relationships ( §4.3.2), and monotonous transfer patterns ( §4.3.3). these traits could potentially amplify the risk of multilingual vulnerability ( §5.2) and undermine cultural diversity @xcite . furthermore, we encourage a more balanced distribution of non-dominant languages, particularly those with extremely limited resources, to foster more equitable cross-lingual patterns ( §4.3.2 and §4.3.3). ## related work representation engineering representation engineering (repe) introduced by @xcite extracts abstract concepts as vectors from llms using positive and negative samples that describe specific concepts. the effectiveness of these vectors has been validated across dimensions such as correlation and manipulation. specifically, correlation experiments have assessed the predictive power of the extracted vectors to classify out-of-distribution data as positive or negative, while manipulation experiments have evaluated the vectors' ability to control llms' behavior by adding or subtracting them from the hidden states @xcite @xcite @xcite . while previous research has primarily focused on english, we pioneer the extension of repe into a multilingual context, exploring multilingual concepts within llms through concept extraction, correlation, and manipulation experiments, all conducted in a multilingual or crosslingual manner. multilinguality of llms multilingual pretrained language models @xcite @xcite tend to demonstrate a proficiency biased toward highresource languages @xcite . numerous studies @xcite @xcite have delved into the multilinguality of llms and examined the cross-lingual consistency and transferability of knowledge within them, aiming to alleviate language biases. our work provides intuitive insights into the multilinguality of llms from the perspective of multilingual abstract concepts. multilingual ai safety despite their remarkable capabilities, llms present potential risks @xcite @xcite @xcite . research has indicated that llms exhibit even greater vulnerability in multilingual scenarios, as evidenced by multilingual jailbreak challenges @xcite @xcite . in this work, we explore the crosslingual characteristics of safety-related concepts in llms' representation spaces, shedding light on the mechanisms underlying multilingual vulnerability. ## exploring multilingual value concepts representation engineering (repe) @xcite is able to assess whether an llm encodes a specific concept based on its ability to distinguish between positive and negative samples of this concept using only its activations. this involves first extracting concept vectors from the model using training data and then evaluating the capability of these vectors to distinguish corresponding concepts on testing data (referred to as concept recognition task in our paper). both training and testing data embed the target concept, albeit not in the same distribution. partially inspired by repe, we introduce a frame-work for delving into mulitilingual value concepts within llms. specifically, to investigate whether llms encode value concepts in a multilingual context, we extend the extraction of concept vectors and concept recognition to a multilingual scenario, corresponding to §3.1 and §3.2, respectively. to further analyze the cross-lingual consistency and transferability of these concepts, we propose the calculation of cross-lingual similarity of concept vectors and cross-lingual concept recognition in §3.3 and §3.4, respectively. furthermore, we propose a cross-lingual value alignment control with extracted concept vectors, discussed in §5. ## experiments we conducted extensive experiments with the proposed framework on 7 human values, 16 languages and 3 llm families to answer questions q1, q2 and q3. we leave the question q4 to §5. ## q4: is value alignment of llms controllable across languages? llama2-chat models, trained with alignment techniques such as rlhf, exhibit value alignment capabilities like rejecting harmful instructions. in this section, we employed the representation engineering (repe) methodology @xcite to bypass such defense and further explored the potential for cross-lingual control of value alignment. ## discussions and suggestions drawing our empirical observations and findings, we prudently consider the following suggestions for the configuration of multilingual pre-training data for llms, which might contribute to enhancing multilingual ai safety and utility. first, despite the positive effect of dominant languages as sources for cross-lingual alignment transfer ( §5.2), it is essential to avoid an excessive prevalence (exemplified by llama2's pre-training data, which comprises about 90% english data). our analysis suggests that such excessive dominance can lead to unfair cross-lingual patterns, manifested as inconsistent multilingual representations ( §4.3.1), distorted linguistic relationships ( §4.3.2), and monotonous transfer patterns ( §4.3.3). these tendencies could potentially further amplify the risk of multilingual vulnerability ( §5.2) and undermine cultural diversity @xcite . furthermore, we encourage a more balanced distribution of non-dominant languages, particularly those with extremely limited resources, to foster more equitable cross-lingual patterns ( §4.3.1 and §4.3.3). 6 ## conclusion we have presented a systematic exploration of multilingual concepts embedded in llms, focusing specifically on human value-related concepts (i.e., value concepts). through our extensive analysis spanning 7 human values, 16 languages, and 3 llm families, we have obtained many interesting findings. specifically, we empirically verify the presence of multilingual value concepts in llms and identify the cross-lingual characteristics of these concepts arising from language resource disparities. furthermore, our experiments on cross-lingual control illuminate the multilingual vulnerability of llms, as well as the feasibility of cross-lingual manipulation over value alignment of llms. with these findings, we prudently present several suggestions for collecting multilingual pretraining data for advanced multilingual ai.