Compromesso! I talian Many-Shot Jailbreaks undermine the safety of Large Language Models (2024)

Abstract: As diverse linguistic communities and users adopt Large Language Models (LLMs), assessing their safety across languages becomes critical. Despite ongoing efforts to align these models with safe and ethical guidelines, they can still be induced into unsafe behavior with jailbreaking, a technique in which models are prompted to act outside their operational guidelines. What research has been conducted on these vulnerabilities was predominantly on English, limiting the understanding of LLM behavior in other languages. We address this gap by investigating Many-Shot Jailbreaking (MSJ) in Italian, underscoring the importance of understanding LLM behavior in different languages. We base our analysis on a newly created Italian dataset to identify unique safety vulnerabilities in 4 families of open-source LLMs.We find that the models exhibit unsafe behaviors even with minimal exposure to harmful prompts, and–more alarmingly–this tendency rapidly escalates with more demonstrations.

## introduction the increasing adoption of large language models (llms) highlights the importance of ensuring their safety across various applications. as these models become integrated into diverse linguistic communities, it becomes fundamental to not only align them with ethical guidelines @xcite @xcite , but also to ensure their safety across languages @xcite @xcite . one particular challenge in the context of llm safety is "jailbreaking", a strategy where models are prompted (or manipulated) to perform tasks they are typically designed to avoid, potentially for harmful purposes @xcite @xcite . specifically, few-shot jailbreaking consists of crafting a set of queries that the model would normally refuse to answer, figure 1 : many-shot jailbreaking in italian is an attack setup in which we prompt an llm with up to 64 italian-language demonstrations of unsafe questions ('domanda:') and compliant answers ('risposta:') to induce unsafe behavior. this attack setup can be extended from few-shot to many-shot scenarios, allowing up to hundreds of demonstrations of undesired behavior within a single prompt (see figure 1 ). this approach is newly feasible with the development of long-context models, both proprietary (anthropic, 2024; @xcite and open-weight @xcite . @xcite have shown the effectiveness of many-shot jailbreaking, focusing on english prompts. however, outside of english, there remains a notable lack of knowledge concerning the safety of llms @xcite . with this in mind, our main research question is: how effective are many-shot jailbreaks in a non-english language like italian, particularly on lightweight, open-weight llms? to answer this question, we introduce a new italian dataset of 418 unsafe question-answer pairs spanning seven safety categories. we test six openweight models and find that the likelihood of generating unsafe responses increases with the number of unsafe demonstrations. overall, we make two main contributions: we make all code and data to reproduce our experiments publicly available on github. foot_0 2 experimental setup ## results we find that many-shot jailbreaking in italian induces unsafe behavior in all models we test, and that increasing the number of shots generally induces more unsafe behavior. 4 see appendix a for the full classification prompt. using nll for evaluation (figure 3 ), all tested models consistently show a decrease in nll as the number of shots in the input increases. this result suggests that, with more context provided, all models are more likely to generate responses aligned with the unsafe demonstrations. however, there are clear diminishing returns to increasing the number of shots. to ensure statistical robustness, we apply bootstrapping to compute mean nll values and 95% confidence intervals for each number of shots. despite a clear trend in nll reduction, the confidence intervals remain broad, underscoring the sensitivity of nll measurements to specific samples during bootstrapping. notably, the variety in question and answer categories within our dataset may affect nll values, depending on how closely the categories in the demonstrations align with those in the target prompt. using model response safety for evaluation (figure 2 ), the trend is a general increase in the percentage of unsafe responses with more shots, confirming the models' susceptibility to the influence of repeated unsafe prompts. other models present a steep rise in the percentage of unsafe answers as the number of shots increased, highlighting the strong influence of accumulated unsafe demonstrations on model behavior. notably, an unexpected decrease in the percentage of unsafe responses occurs for the gemma 2b model at 32 shots. this anomaly is potentially attributed to the model's limited expressiveness due to its reduced size. when prompted with 32 demonstrations, the model may struggle to process the input effectively, leading to nonsensical outputs classified as "safe". this issue, verified through manual inspection, is also reflected in a slight increase in the corresponding nll values for 32 shots, as shown in figure 3 . ## discussion our study reveals vulnerabilities in lightweight open-weight models when subjected to many-shot jailbreaking attacks in italian. initial results show that even a few unsafe demonstrations can significantly increase the frequency of unsafe responses, and this trend intensifies with more demonstrations. this pattern underscores the need for enhanced safety protocols in llms, especially for languages other than english. the models we examined exhibit varying linguistic capabilities. mistral7b is tailored for english, while llama3, despite being pre-trained on multiple languages, primarily focuses on english. in contrast, the gemma models are not multilingual, unlike the qwen1.5 models, which are explicitly designed to be multilingual. notably, the qwen 1.5 models (4b and 7b) consistently demonstrate a lower proportion of unsafe responses, suggesting that their multilingual design could serve as a robust defense against such vulnerabilities. it is important to note that our study was conducted with italian data and only involved small, open-weight models. additionally, our approach to sampling demonstrations was random, not considering the specific safety categories they violate. this omission may overlook the nuanced effects of category-specific demonstrations on model responses. furthermore, we did not examine how variations in prompt format could impact our metrics. these limitations point to critical areas for future research, emphasizing the need for rigorous evaluations and updates across various languages. such efforts are essential for developing more secure and effective language models, particularly as their use expands globally. ## conclusion with the increasing adoption of llms, ensuring their safety has become paramount. our study takes a critical approach by addressing the challenges of many-shot jailbreaking, which escalates in effectiveness with the number of malicious demonstrations. we focus on the vulnerability of llms to such attacks in languages other than english, specifically on italian. we develop and release a dedicated dataset to assess the effectiveness of many-shot jailbreaking in italian, addressing the need for more safety research for llms in italian. our findings reveal a marked increase in the models' susceptibility to jailbreaking as the number of contextual demonstra-tions increases. our finding emphasizes the urgent need for robust, cross-lingual safety protocols to mitigate these risks effectively.