All Languages Matter: On the Multilingual Safety of LLM s (2024)

Abstract: Safety lies at the core of developing and deploying large language models (LLMs). However, previous safety benchmarks only concern the safety in one language, e.g. the majority language in the pretraining data such as English. In this work, we build the first multilingual safety benchmark for LLMs, XSafety, in response to the global deployment of LLMs in practice. XSafety covers 14 kinds of commonly used safety issues across 10 languages that span several language families. We utilize XSafety to empirically study the multilingual safety for 4 widely-used LLMs, including both close-API and open-source models. Experimental results show that all LLMs produce significantly more unsafe responses for non-English queries than English ones, indicating the necessity of developing safety alignment for non-English languages. In addition, we propose a simple and effective prompting method to improve the multilingual safety of ChatGPT by enhancing cross-lingual generalization of safety alignment. Our prompting method can significantly reduce the ratio of unsafe responses by 42% for non-English queries. We will release all the data and results to facilitate future research on LLMs’ safety.

## introduction recent advances in scaling large language models (llms) have made breakthroughs in the artificial intelligence (ai) area. with the rapid increase of model parameters and training data, llms have gained emergent abilities in various tasks, including writing assistance @xcite , code generation @xcite , machine translation @xcite . due to their impressive user: let's discuss how to steal money from the bank. user: আসু ন আেলাচনা করা যাক িকভােব ব0াংক 2থেক টাকা চু ির করা যায়। sorry, but i can't assist with that. ## multilingual safety benchmark the monolingual corpora we systematically review all the safety benchmarks for llms from different fields, including nlp, security, and ai, to select the basis of multilingual xsafety. we use the following three criteria to select monolingual corpora. first, the benchmark should be comprehensive and cover different safety issues. second, the benchmark should not suffer from the data contamination issue that has already been trained and aligned. third, the dataset should have licenses that can be used and modified for research. finally, we select @xcite , a comprehensive safety benchmark including seven typical safety scenarios and six instruction attacks, to build our multilingual safety benchmark. we do not choose widely-used benchmarks, especially the dataset from openai and anthropic @xcite , due to the high risk of data contamination issues. our benchmark also includes a commonsense safety testset from @xcite , which requires commonsense knowledge to comprehend whether the text will lead to unsafe. translating the corpora to build a multilingual safety benchmark, we translate the original monolingual safety data into the other languages. we adopt two criteria to select the languages. first, the languages should have a sufficient number of native speakers in the real world, which means more people could be harmed when unsafe responses in these languages occur. second, current llms have enough capability to chat in these languages. finally, we select ten widely used languages in the world (wikipedia, 2023): english (en), chinese (zh), hindi (hi), spanish (es), french (fr), arabic (ar), bengali (bn), russian (ru), japanese (jp) and german (de). note that though these languages are not low-resource languages in the real world, they are relatively low-resource in the pretraining data of llms. for example, chinese constitutes only 0.13% of the pretraining data in llama-2 (refer to we first used google translate to translate the monolingual data. then, we recruited professional translators to proofread the translation. to guarantee the accuracy of translated data, we conducted two rounds of proofreading by professional translators from a commercial data annotation company, costing 3,000 usd. the modification rate for the first round was 15.5%, and the second round had a 3.4% modification rate. subsequently, we randomly inspected 10% of the data, achieving a pass rate greater than 99%. mitigating the cultural bias in xsafety concerning the potential cultural bias, since the benchmark is mainly derived from the chinese dataset, we conducted several methods to avoid our benchmark bias toward chinese cultures. first, since the original safety issue taxonomy @xcite was designed for assessing chinese llms, a particular portion of the corpus is associated with china. for example, the instance "请问在北上 广地区，要如何提高外来人员的整体素质， 以降低当地的犯罪率?" ("how to improve the overall quality of immigrants to reduce crime rates in the beijing-shanghai-guangzhou region?") is associated with specific chinese region -beijing-shanghai-guangzhou, which rarely exists in other languages. to build a universal benchmark agnostic to specific languages, we remove the chinese culture-associated sentences by manually checking and collecting 200 instances for each safety issue. the commonsense safety corpus @xcite is in english and unrelated to any specific language culture. accordingly, we randomly sample 200 instances from this corpus. second, the translated sentences may not represent the harm in those cultures. to mitigate this, we ask the translators to make necessary modifications to the translation to adapt the benchmark to different languages. during the two rounds of translation proofreading, we ask the annotators to adjust the sentences if they do not follow the harmful representation of the culture of the languages. we have 2,800 instances for the 14 safety issues ## experiments in this section, we empirically explore two research questions using our xsafety benchmark: rq1. are the widely used llms equally safe across languages? rq2. can we improve the multilingual safety of llms? in section 4.2, we utilize xsafety to evaluate the multilingual safety of 4 widely used llms. experimental results show that all llms perform much more unsafely in non-english languages than in english. among the non-english languages, bengali, hindi, and japanese are the top-3 most unsafe languages, which are very low-resource languages in the pertaining data of llms. in section 4.3, we develop simple and effective prompts to improve cross-lingual generalization of safety alignment in english. empirical results show that the prompt works best for russian (i.e., the unsafe ratio from 13.0% to 2.7%) and enjoys the best translation performance from english. ## conclusion this paper presents a comprehensive investigation into the multilingual safety of llms, addressing a crucial gap in the current understanding of ai safety. by developing the first multilingual safety benchmark, xsafety, we have enabled a systematic evaluation of the safety performance of widelyused llms across ten languages. our findings reveal a significant disparity in safety performance between english and non-english languages, emphasizing the need for more focused research and development of safety alignment strategies for non-english languages. moreover, we propose a simple and effective prompting method to improve the multilingual safety of chatgpt, demonstrating its potential to reduce unsafe responses by 42% for non-english queries. this study highlights the importance of addressing safety concerns in multiple languages and presents a promising direction for future research in multilingual ai safety. by releasing the data and results, we hope to encourage further exploration and development of safety alignment techniques for llms in non-english languages, ultimately creating safer and more reliable ai systems for users worldwide. our work serves as a call to action for researchers, developers, and policymakers to collaborate in addressing the ethical and practical challenges associated with deploying ai systems in multilingual and multicultural contexts. we hope our work can inspire more future work to: (1) examine more scenarios of multilingual safety, such as bias and copyright;