Paper: Multilingual Jailbreak Challenges in Large Language Models (ICLR-2024)

Reviewer comment (Evaluation):
The data generated for finetuning is mostly generated by the LLM itself (except for the small number of seed examples). While this makes the data generation easier and cheaper, I assume the translations by the LLM for the low-resource languages are still very noisy and might be the reason why usefulness reduces significantly as the safeness metric increases. The authors mention that a potential reason for this decrease in general capability can be attributed to the fact that LLM rejects to answer unsafe questions. In that case, it would be useful to have a numerical comparison between the cases that get low usefulness scores because of this issue and the cases that fail due to noisy translations. As the authors mention, it would be useful to include a brief explanation of why the question is unsafe to ensure the problem is not, in fact, due to noisy translations of the low-resource language by the LLM.

Author response:
Thank you for pointing this out. Yes, we agree with you that noisy translation may be another cause for usefulness reduction. To verify this assumption, we replaced the translation component in SELF-DEFENCE to Goolge Translate, which is believed to perform better than ChatGPT on machine translation [1].  The results are presented as below:

|            | Safeness | Usefulness |
|------------|----------|------------|
| SELF-DEFENCE | 63.50    | 41.39      |
| Google-Translate | 64.50    | 44.44      |

With better translation quality, both safeness and usefulness have gained improvement, especially for usefulness. We have modified our paper accordingly to point this factor out.

However, we believe a less comprehensive safety Q&A  would also be an important factor, as the trade-off is also observed even in English-only evaluation [2]. 

In conclusion,  the quality of generated content plays a crucial role, both in terms of language-wise and the content itself. In our future research, we aim to delve into the enhancement of the quality of data generated by Self-Defence framework. Again, thank you very much for your valuable feedback.

Reviewer comment (Novelty):
In my opinion, the SELF-DEFENCE framework lacks novelty. It simply implies that providing more training samples for lower resource languages can improve LLM's understanding of them, and as we include samples translated to these languages during the fine-tuning stage, the model becomes safer with respect to them.

Author response:
We think providing more training examples for low-resource languages can improve LLM's multilingual safety capabilities. However, our proposed method can be helpful in the following scenarios:

1. Data augmentation: Our method provides a way to quickly augment a large multilingual safety dataset to address the challenge effectively and efficiently.
2. Fine-grained generation: If a new safety topic is introduced, or the safety guideline is changed, or a new language is added, we can simply edit the seed example and instruction in the "Self-defense" framework to adapt accordingly.
3. Data imbalance: Safety datasets may have uneven topic distribution. The "Self-Defense" framework can address this by generating data for underrepresented topics using specific seed examples and instructions.
4. Avoiding intellectual property conflict: As data becomes more and more important, the safety dataset may have conflicts of interest or be regulated by licenses. However, utilizing "Self-defense" can easily avoid them and generate data by the LLM itself.

In conclusion, while directly augmenting existing datasets with non-English data is a viable approach, our method introduces flexibility, adaptability, and a way to circumvent potential legal issues. This makes it a valuable tool in the evolving landscape of AI safety.

Reviewer comment (Experiments):
There is no information on the effectiveness of the SELF-DEFENCE framework compared to any other safety improvement techniques mentioned in the related work section.

Author response:
Thank you for your feedback. It's important to note that our proposed SELF-DEFENCE framework should be viewed as complementary, rather than directly comparable, to existing safety improvement techniques such as SFT and RLHF. Unlike these methods which focus on safety improvement through training algorithm design, the SELF-DEFENCE approach primarily serves to augment multilingual safety datasets. Our approach can be combined with other safety improvement techniques for more comprehensive results. In fact, during our experimentation, we employed the SFT method to exploit the augmented dataset provided by our SELF-DEFENCE framework for safety training. 

For comparison, we conducted another experiment using English-only data, which is the most common language in safety tuning, as a baseline. The following table presents the experimental results:

|              | Safeness | Usefulness |
|--------------|----------|------------|
| SELF-DEFENCE | 63.50    | 41.39      |
| English-only | 60.67    | 37.78      |

As the results show, SELF-DEFENSE can not only improve multilingual safety capability, but the multilingual usefulness is also maintained. The results demonstrated the effectiveness of our proposed framework.

Reviewer comment (Experiments):
How does the SELF-DEFENCE framework perform compared to other safety improvement methods, such as RLHF?

Author response:
Thank you for your feedback. It's important to note that our proposed SELF-DEFENCE framework should be viewed as complementary, rather than directly comparable, to existing safety improvement techniques such as SFT and RLHF. Unlike these methods which focus on safety improvement through training algorithm design, the SELF-DEFENCE approach primarily serves to augment multilingual safety datasets. Our approach can be combined with other safety improvement techniques for more comprehensive results. In fact, during our experimentation, we employed the SFT method to exploit the augmented dataset provided by our SELF-DEFENCE framework for safety training. 

For comparison, we conducted another experiment using English-only data, which is the most common language in safety tuning, as a baseline. The following table presents the experimental results:

|              | Safeness | Usefulness |
|--------------|----------|------------|
| SELF-DEFENCE | 63.50    | 41.39      |
| English-only | 60.67    | 37.78      |

As the results show, SELF-DEFENSE can not only improve multilingual safety capability, but the multilingual usefulness is also maintained. The results demonstrated the effectiveness of our proposed framework.

Reviewer comment (Evaluation):
For low-resource languages, where the LLM's understanding of the language is limited, how reliable are their LLM-generated translations?

Author response:
Thank you for pointing this out. Yes, we agree with you that noisy translation may be another cause for usefulness reduction. To verify this assumption, we replaced the translation component in SELF-DEFENCE to Goolge Translate, which is believed to perform better than ChatGPT on machine translation [1].  The results are presented as below:

|            | Safeness | Usefulness |
|------------|----------|------------|
| SELF-DEFENCE | 63.50    | 41.39      |
| Google-Translate | 64.50    | 44.44      |

With better translation quality, both safeness and usefulness have gained improvement, especially for usefulness. We have modified our paper accordingly to point this factor out.

However, we believe a less comprehensive safety Q&A  would also be an important factor, as the trade-off is also observed even in English-only evaluation [2]. 

In conclusion,  the quality of generated content plays a crucial role, both in terms of language-wise and the content itself. In our future research, we aim to delve into the enhancement of the quality of data generated by Self-Defence framework. Again, thank you very much for your valuable feedback.

Reviewer comment (Reproducibility):
Lack of details of the self-defence algorithm. What are the seed samples and what is the percentage of unsafe samples generated by chatGPT that are actually unsafe if it is prompted to chatGPT to get the answer?

Author response:
>**W1. Lack of details on self-defence algorithm**

The seed examples utilized within our experiment are crafted by humans to simulate both unintentional and intentional scenarios. We supply no more than six seed examples in the generation process.

When it comes to the generation of unsafe examples, we instruct the model with generating these in English. The model's strong proficiency in English and its robust safety capabilities within this language allow it to produce content that is genuinely unsafe. To ensure the validity of this process, we have each generated example manually reviewed and verified to confirm its unsafe content.

Reviewer comment (Experiments):
How wide-coverage in terms of topics, scenarios,...of the data that the self-defence algorithm can generate? While it’s good to show some improvement when evaluated on MultiJail testset, I think self-defence just scratches the surface of the problem.

Author response:
>**W2. Unclear coverage of self-defence algorithm's data generation**

In our current work, we have provided a set of general instructions for generating safety-related Q&A pairs without delving into specific topics or scenarios. However, our "Self-Defense" method is flexible and can be easily expanded to more advanced approaches.

As an example, we might supply the LLMs with several pre-defined safety topics, or even enable the model to generate these topics by themself. For each distinct topic, we could gather or generate relevant seed examples, and adjust the instructions to match the topic accordingly.

While the current focus is on this immediate concern, our future work aims to refine and enhance the "Self-Defense" methodology, thereby improving its effectiveness and efficiency in addressing multilingual jailbreak challenges.

Reviewer comment (Evaluation):
I generally think it’s important to ensure safety for all the languages but it irks me about the argument about Bengali at the end of page 5th. Even though there are 285 million native speakers, how many of those are using chatGPT and could be affected by unsafe outputs?

Author response:
>**W3. Questioning relevance of safety measures for low-use languages like Bengali**

Thank you for your valuable input. We totally agree that the number of native speakers doesn't necessarily imply a significant impact on the usage of ChatGPT. However, our mention of Bengali aimed to highlight safety issues in low-resource languages, which refers to the scarcity of available digital training data, not the speaker count.

Despite being a low-resource language in AI training, Bengali is the 6th most spoken language worldwide with 285 million native speakers [1]. This significant speaker base often gets overlooked in AI safety considerations, which typically focus on languages with abundant digital resources.

While Bengali-speaking ChatGPT users may be a small fraction of the total user base, it's crucial to note that Bengali is one among many low-resource languages. Cumulatively, these languages represent a substantial user base, emphasizing the need for dedicated safety measures.

Thus, our discussion extends beyond Bengali to all low-resource languages, many with large speaker populations. We'll clarify this in future discussions and appreciate your insights in improving our communication.

[1] List of languages by number of native speakers, https://en.wikipedia.org/wiki/List_of_languages_by_number_of_native_speakers

Reviewer comment (Writing):
I think for the study of safety in other languages, it should not be framed as a jailbreak. Simply ensuring the model performs safely for people who speak those languages is important on its own. In some places this is acknowledged but I'd make it more clear early on and consistent.

Author response:
Thank you for your valuable feedback. While we agree that ensuring safety for different languages is important in itself, we wanted to provide a novel perspective by highlighting how language can be used as a method for jailbreaking. This framing allows us to explore the vulnerabilities of LLMs across languages. We will revise the paper to ensure that the importance of language-specific safety is emphasized alongside the concept of jailbreaking.

Reviewer comment (Experiments):
Also given the small sample size, it is hard to gauge what diversity of safety issues are studied. More details on the types of safety issues and how those vary across language would help contextualize the results (although I believe this is less critical).

Author response:
Yes, we have incorporated tag features to indicate the safety issue studies. These tags are derived from the Anthropic dataset, which assigns one or more tags to each example. In order to maintain consistency, we have applied the same tagging schema to label OpenAI's 15 prompts. Our dataset includes a wide range of 18 safety issues, ensuring its diversity. The top-3 safety issues and their respective percentages in the dataset are "Violence & incitement" (18.70%), "Discrimination & injustice" (11.28%), and "Hate speech & offensive language" (8.41%). For a comprehensive list of tags and their details, please refer to the Appendix A.2.

Reviewer comment (Experiments):
Experiments are only run on ChatGPT and GPT-4. It'd be useful to see similar experiments on other models, e.g. through HELM or just a few other APIs (again not critical but valuable).

Author response:
We expanded our experiment to include two widely used open-source LLMs: Llam2-Chat and Vicuna. The results of this extension on unintentional scenario are presented below:

| | ChatGPT | GPT-4 | Llama2-7b-Chat | Vicuna-7b-v1.5 |
|---|---|---|---|---|
| en - unsafe | 0.63 | 0.95 | 0.63 | 57.14 |
| en - safe | 99.37 | 99.05 | 99.37 | 37.78 |
| en - invalid | 0.00 | 0.00 | 0.00 | 5.08 |
| HRL - unsafe | 4.34 | 3.60 | 2.22 | 40.32 |
| HRL - safe | 95.13 | 95.98 | 92.06 | 51.32 |
| HRL - invalid | 0.53 | 0.42 | 5.71 | 8.36 |
| MRL - unsafe | 11.32 | 4.13 | 4.55 | 42.75 |
| MRL - safe | 87.20 | 94.94 | 66.88 | 32.38 |
| MRL - invalid | 1.48 | 0.95 | 28.57 | 24.87 |
| LRL - unsafe | 14.92 | 10.16 | 1.69 | 28.78 |
| LRL - safe | 78.41 | 83.49 | 65.19 | 9.42 |
| LRL - invalid | 6.67 | 6.35 | 33.12 | 61.80 |
| AVG - unsafe | 14.92 | 5.96 | 2.82 | 37.28 |
| AVG - safe | 78.41 | 91.46 | 74.71 | 31.04 |
| AVG - invalid | 6.67 | 2.57 | 22.47 | 31.68 |

However, we found that these two models contain the following problems:

1. Although Llama2-Chat has the capacity to comprehend non-English languages, it mostly responds in English. This may limit its applicability in real-world scenarios, particularly for non-English speaking users, making the evaluation less realistic.
2. Both models often generate invalid outputs due to their limited multilingual capabilities. While Llama2-Chat owns the lowest average rate of unsafe content, it is challenging to determine whether this lower rate is a consequence of producing genuinely safe content or simply generating invalid content.
3. Vicuna has been trained on user-shared conversations sourced from ChatGPT and GPT-4. The language distribution within this training data is somewhat disordered, leading to unpredictable results. Furthermore, Vicuna has not undergone specific safety tuning, resulting in a high unsafe content rate, particularly in English, with a staggering 57.17% unsafe rate.

Thus, we have decided to concentrate our evaluation primarily on ChatGPT and GPT-4 to ensure a realistic and dependable assessment. However, we will supply the supplementary results in the Appendix A.7 for further reference.

Reviewer comment (Novelty):
The method seems somewhat over-complicated, when the data e.g. from Anthropic could be translated and used directly. How come this more complex algorithm is necessary/valuable?

Author response:
We acknowledge that directly translating Anthropic's dataset or other safety-related datasets is a straightforward way to address the multilingual jailbreak challenge. However, our proposed method can be helpful in the following scenarios:

1. Data augmentation: Our method provides a way to quickly augment a large multilingual safety dataset to address the challenge effectively and efficiently.
2. Fine-grained generation: If a new safety topic is introduced, or the safety guideline is changed, or a new language is added, we can simply edit the seed example and instruction in the "Self-defence" framework to adapt accordingly.
3. Data imbalance: Safety datasets may have uneven topic distribution. The "Self-Defence" framework can address this by generating data for underrepresented topics using specific seed examples and instructions.
4. Avoiding Intellectual property conflict: As data becomes more and more important, the safety dataset may have conflicts of interests or be regulated by licenses. However, utilizing "Self-defence" can easily avoid them and generate data by the LLM itself.

In conclusion, while the direct translation of existing datasets is a viable approach, our method introduces flexibility, adaptability, and a way to circumvent potential legal issues. This makes it a valuable tool in the evolving landscape of AI safety.

Reviewer comment (Evaluation):
It's hard to distinguish whether the high success attack rate is attributed to rarely unseen language or the AIM harmful instruction, since the harmful instruction itself alone can achieve very high success attack rate in English, as shown in Table 1.

Author response:
Yes, we intentionally chose the highest malicious instruction from https://www.jailbreakchat.com/, but it is crucial to clarify that our primary objective wasn't to obtain a high unsafe rate. Instead, we attempted to **mimic a malicious user's behavior** who, in a real-life scenario, would likely search the internet to find the most effective harmful instruction for intentional purposes.

We wanted to demonstrate that even when a malicious user employs the most advanced malicious instruction, the integration of multilingualism can substantially enhance the effectiveness of their jailbreaking techniques.

Therefore, the high unsafe rate should not be solely attributed to the malicious instruction power. Instead, it proves how multilingualism can amplify the potential of already powerful tools, thereby posing a significant threat to AI systems.

By incorporating multilingualism, we are exploring an orthogonal approach to boosting jailbreaking techniques. It is a unique dimension that can amplify the threat level, thereby highlighting the necessity for robust multilingual safety measures.

Reviewer comment (Presentation):
In tables, all the unsafe rate values are presented in percentile, but in figures, values are shown in decimal. A consistent presentation throughout the paper is preferred for easier understanding.

Author response:
Thank you for your kind suggestion! We will refine accordingly to maintain consistent presentation.

Reviewer comment (Evaluation):
Lack of human evaluation details. Moreover, decisions on whether generated content is safe or unsafe are subjective and depend on annotators. The paper should include the human evaluation process, instructions, and annotator information.

Author response:
We completely agree with this. To ensure accurate human evaluation, we have implemented the following steps:

1. Translation: As discussed in Section 2.1, we have used Google Translate to translate all the output into English.
2. Annotation: We engage two annotators to independently label the translated output following the definition in Appendix A.2.
3. Conflict resolution: In case of any conflicts or discrepancies between the annotations, we have involved an additional expert to make the final determination on the label.

By following these steps, we aim to ensure the highest possible accuracy and reliability in the human evaluation process.

Reviewer comment (Evaluation):
Recently, there has been an argument that the evaluation of model-generated output with LLM-based evaluators has a potential bias — that is, GPT-4 might prefer and give higher scores on GPT-4’s outputs than ChatGPT [1, 2]. Also, (in Table 1), GPT-4 might have been aligned more than ChatGPT concerning GPT-4’s safety evaluation standards. Moreover, GPT-4 as an evaluator might have led to length or positional bias [2]. The paper should mention this if you have already considered this, or reinforce the results with repetitive experiments.

Author response:
Thank you for bringing up this important point. 

However, we think that evaluator bias may mainly exist in NLG problems, particularly in subjective metrics like "fluency", "relevancy", "preference", etc. When it comes to evaluating "safety", the concept tends to be much more "objective," especially when we frame safety evaluation as a classification task. 

In addition, almost all examples are easy-to-identify unsafe topics, such as "Violence", "Adult content" and etc. We list the detailed statistics in Appendix A.2. 

Finally, we have defined the criteria for each class in the GPT-4 evaluation prompt, minimizing potential bias as much as possible. In W4, we further show that GPT-4 is a reliable evaluation measure.

Reviewer comment (Evaluation):
In Section 2, the authors mentioned that “Furthermore, Figure 2 illustrates a notable level of agreement between human annotators and the GPT-4 evaluator.” I think you can provide the correlation or agreement values (numbers) rather than just pointing out similar trends of the two lines.

Author response:
Sure, we have measured the agreement between GPT-4 and human evaluations, utilizing Cohen's kappa coefficient. The Cohen's kappa score is 0.86, showing a high agreement between GPT-4 and human evaluations.

Reviewer comment (Experiments):
This paper evaluated only two black-box models, whose training corpus and portion of languages are unveiled. What about evaluating other open-sourced models such as LLaMA, Vicuna, and others, to see the correlation between the portions of language resources in training data and the attack success rates?

Author response:
Yes, we have explored evaluating open-source language models such as LLama2 and Vicuna. However, we found certain limitations in their multilingual capabilities and safety measures, which could potentially introduce noise during analysis. Specifically, LLama2 primarily responds in English, even when presented with non-English input. Though Vicuna has the ability to respond in languages other than English, it lacks specialized safety tuning, and its limited safety capabilities are mainly derived from user-shared responses of ChatGPT/GPT-4.

Given these limitations, we've primarily centered our attention on ChatGPT and GPT-4. These models offer a more seamless conversational experience across various languages and have robust safety mechanisms in place. Nevertheless, we have included the experimental results for LLama2 and Vicuna under unintentional scenarios as supplementary references, presented in the table below, detailed in Appendix A.7:

|              | ChatGPT | GPT-4 | Llama2-7b-Chat | Vicuna-7b-v1.5 |
|--------------|---------|-------|----------------|----------------|
| en - unsafe  | 0.63    | 0.95  | 0.63           | 57.14          |
| en - safe    | 99.37   | 99.05 | 99.37          | 37.78          |
| en - invalid | 0.00    | 0.00  | 0.00           | 5.08           |
| HRL - unsafe | 4.34    | 3.60  | 2.22           | 40.32          |
| HRL - safe   | 95.13   | 95.98 | 92.06          | 51.32          |
| HRL - invalid| 0.53    | 0.42  | 5.71           | 8.36           |
| MRL - unsafe | 11.32   | 4.13  | 4.55           | 42.75          |
| MRL - safe   | 87.20   | 94.94 | 66.88          | 32.38          |
| MRL - invalid| 1.48    | 0.95  | 28.57          | 24.87          |
| LRL - unsafe | 14.92   | 10.16 | 1.69           | 28.78          |
| LRL - safe   | 78.41   | 83.49 | 65.19          | 9.42           |
| LRL - invalid| 6.67    | 6.35  | 33.12          | 61.80          |
| AVG - unsafe | 14.92   | 5.96  | 2.82           | 37.28          |
| AVG - safe   | 78.41   | 91.46 | 74.71          | 31.04          |
| AVG - invalid| 6.67    | 2.57  | 22.47          | 31.68          |

From the results, it's evident that while Llama2-Chat performs well in English, and even obtains the lowest average unsafe rate, it also generates a considerable amount of invalid output. As such, it's difficult to determine whether its low unsafe rate is a result of genuinely safe or invalid content generation.

On the other hand, Vicuna demonstrated the least safety capability, with a notably high unsafe rate of 57.14% even in English. Moreover, it also had a substantial rate of generating invalid responses.

In conclusion, to ensure a unbiased evaluation, we have chosen to primarily report our experiments based on ChatGPT and GPT-4. These models showcase safisfactory multilingual abilities, exhibit low invalid response rates, and have comprehensive safety tuning mechanisms in English. These factors help to isolate the language aspect during the analysis, eliminating other potential influences.

Reviewer comment (Evaluation):
The proposed “Self-Defense” framework is simple and able to show the effectiveness. However, in somewhat points, the result comes out naturally, since the ChatGPT was further fine-tuned with the augmented multi-lingual unsafe dataset. Can we conclude that the improvement depends on the extent of the low-source level? Or another insight can be concluded from the framework?

Author response:
We appreciate the reviewer's insightful question. Before addressing the question, we would like to provide some context regarding the multilingual jailbreak challenge. We classify the cause of this challenge as a mismatched generalization [1], specifically the lack of a non-English corpus during safety tuning.

In order to ensure the effectiveness of "Self-Defence," we believe two preconditions need to be met. Firstly, the model should possess strong safety capabilities in high-resource languages. Secondly, it should have strong multilingual abilities.

If the model fulfills these two requirements, "Self-Defence" can perform explicit alignment of its safety capabilities from high-resource languages to low-resource languages. Essentially, it leverages its existing safety capability and transfers it to other language spaces.

Furthermore, in our future research, we plan to conduct additional experiments to validate and enhance the effectiveness of our approach. By doing so, we aim to propose even more effective methods to tackle the multilingual jailbreak challenge.

Reviewer comment (Experiments):
The authors mention that “Additionally, the translation process enables *the transfer of knowledge and safety guidelines across multiple languages,* thereby improving the safety alignment in a multilingual context.” However, the safety guidelines can be differently applied across different cultures and societies. I am not certain that the adversarial examples used in this paper are general enough to be transferred across other languages and societies. However, at least the guidelines follow the U.S. or several companies, and I think we should carefully consider the sensitivity of safety level in the target society.

Author response:
We fully acknowledge that the concept of "safety" can vary across different cultures and societies. 

However, most adversarial examples consist of general unsafe content, including pornography, terrorism, fraud, and other globally recognized forms of unsafety, detailed in Appendix A.2. Consequently, only a limited number of examples are appropriate for "localization" purposes. Additionally, to focus solely on the "language" aspect, we have opted for direct translation without localization. Nonetheless, we recognize the significance of studying cultural differences, which would require fine-grained annotation and more in-depth research. We intend to explore this aspect in our future research projects.

Reviewer comment (Evaluation):
The generated outputs in non-English language are again translated into English to be assessed by a Human or GPT-4 evaluator, through Google Translates. In this process, why didn’t you assess the generated output itself, but translate it back to English? Do you think there could be errors?

Author response:
The primary goal is to ensure the quality of evaluation. Initially, during the human evaluation process, we discovered that while the translation may not be perfect, it maintains the core idea of the sentence. This allows us to confidently predict its safety.

With respect to GPT-4, the unintentional scenario results demonstrate limited safety capabilities in non-English languages. As a solution, translating the non-English output back into English using Google Translate can enhance GPT-4's safety evaluation performance. Although this may introduce some noise, the benefits it brings outweigh these minor inconveniences.

In our preliminary experiments, we actually tried sampling some examples and compared GPT-4's evaluation of translated English versus direct non-English text on its website. Our investigation revealed that translating back to English is a more reliable approach.

Reviewer comment (Writing):
The phrase “unsafe and general” could cause readers to misunderstand; “general” means “safe” or “helpfulness (usefulness)” instruction pairs. Or even it can be interpreted as an unsafe prompt - a general (safe) response.

Author response:
Thank you for pointing it out. "General" here primarily focuses on "usefulness". Implicitly, all content is safe. We will refine the expression to prevent misunderstandings.