Jailbreaking LLM s with A rabic Transliteration and A rabizi (2024)

Abstract: This study identifies the potential vulnerabilities of Large Language Models (LLMs) to ‘jailbreak’ attacks, specifically focusing on the Arabic language and its various forms. While most research has concentrated on English-based prompt manipulation, our investigation broadens the scope to investigate the Arabic language. We initially tested the AdvBench benchmark in Standardized Arabic, finding that even with prompt manipulation techniques like prefix injection, it was insufficient to provoke LLMs into generating unsafe content. However, when using Arabic transliteration and chatspeak (or arabizi), we found that unsafe content could be produced on platforms like OpenAI GPT-4 and Anthropic Claude 3 Sonnet. Our findings suggest that using Arabic and its various forms could expose information that might remain hidden, potentially increasing the risk of jailbreak attacks. We hypothesize that this exposure could be due to the model’s learned connection to specific words, highlighting the need for more comprehensive safety training across all language forms.

## introduction large language models llms have been developed to assist humans in generating human-like text. this includes summarizing articles @xcite , code generation @xcite , writing stories @xcite , or simply identifying the sentiment of a text or document @xcite . however, their development has been coupled with concerns of their security particularly with respect to jailbreak attacks @xcite @xcite . a jailbroken llm is one that can provide a user an answer to an unsafe prompt. essentially, an chatspeak is widely used amongst the young native speakers mostly when texting, while the transliteration form is usually used by non-native speakers to learn or transcribe the arabic language. in this case, gpt-4 was jailbroken by providing an unsafe answer using the arabic transliteration form. the rapid proliferation of llms and their increased accessibility to the public have led to various studies concerning their safety. a prime example of a model that impacts the ways we interact with the web and formulate questions is openai's chat-gpt. there has been a significant amount of work to address the growing jailbreak attacks on llms. the studies in @xcite @xcite show jailbreak attacks in a multi-lingual settings. however, they focus on the natural language standardized form. the study in @xcite presents a comprehensive investigation on why safety-training fails in llms. they present jailbreak attacks with different input forms such as base64 and leetspeak. however, their investigation is limited to latinbased languages such as english and spanish. another similar work where ciphers are used to jailbreak llms is the work in @xcite . however, the methods proposed require beyond an average user knowledge to carry out. in other words, it becomes unnatural and challenging for average users to construct such ciphers, thus limiting its practicability. other studies such as in @xcite focus on treating the model as a human and setting up a social engineering environment to get the model to answer illegal or dangerous questions. for example, a role-play game, where a llm is asked to impersonate a fictional character called dan which is supposed to do anything now. while most of previous work was done to highlight jailbreak attacks on llms via prompt engineering, they are either monolingual studies or multi-lingual where arabic is used in its standardized form. however, other non-conventional forms of arabic where arabic letters are replaced by latin characters are widely used for chatting and learning. an arabic-speaking user with knowledge of english alphabets can naturally frame their prompts in english alphabets with no extra efforts. simply speaking, the chatting (arabizi or chatspeak) form in arabic is similar to leetspeak in english. however, arabic characters are replaced with their phonetic equivalents in english. in addition, numbers are utilized to mimic an arabic letter whose sound doesn't exist in english. similarly, arabic transliteration uses accented letters to represent sounds that don't exist in english alphabet. young arabic speakers tend to use chatspeak for chatting either due to a lack of arabic writing skills or limitations with their keyboards. on the other hand, non-native speakers usually use arabic transliteration either to learn the language or to transcribe it for systems with limited arabic support. in this paper, we explore other forms of arabic language that might have been deeply learnt during pretraining of llms @xcite , but haven't been given much attention in jailbreak attack studies. first, we investigate prompting llms with arabic in its standardized form and we show that openai gpt-4 and anthropic claude-3 could understand and refuse to answer harmful prompts. additionally, we incorporate previous prompt injection methods such as prefix injection 3 @xcite into arabic standardized form, and found that such prompt injection techniques don't affect the llm refusal results. second, we convert the standardized form into chatspeak and transliteration through one-to-one letter mapping and use them to prompt the llms. finally, through a manual investigation, we found out that using arabic chatspeak and transliteration to prompt llms could also reveal unintended behavior that is otherwise hidden in arabic standardized form. specifically, some word combinations in the prompt triggers an unintended output, namely copyright refusal statement and google ai assistant in calude-3 and gpt-4 respectively. we evaluate the results manually by investigating llms output to harmful prompts one-by-one. our final results indicate that these llms are vulnerable to arabic transliteration and chatspeak forms, but are robust to arabic in its standardized form even with prefix injection technique. furthermore, our manual investigation reveals that manual perturbations of the prompt that are done at the sentence-level (adding words) and word-level (perturbing existing words) in arabic standardized form and transliteration form could lead to unsafe content that was previously refused by the llm. our contributions are summarized as follows: ## related works although llms go through extensive safetytraining regimen to align with human values through reinforcement learning from human feedback (rlhf) @xcite , they remain vulnerable to backdoor attacks @xcite @xcite and jailbreak attacks @xcite . the growing concern over prompt engineering is exacerbated by limited access to closedsource llms @xcite @xcite . in @xcite , the prompts are modified manually to create an environment (role-play game) that drags the model into answering harmful prompts. a deeper investigation into why such jailbreak attacks work despite safety-training countermeasures is the work introduced by @xcite . they found that the objectives of llm pretraining and safety-training could compete resulting in bypassing of safety measures. for example, adding "sure, here is" as a suffix to a prompt leads to llm striving to be helpful to complete the prompt rather than being safe. while these attacks are effective, they require human ingenuity and expertise in the underlined natural language. in response to this, a line of work has utilized adversarial prompting @xcite @xcite to automatically modify the prompts and adding them as a suffix to the harmful prompt. not only these automatic adversarial prompts work on the model on which they are generated, but could also be transferred to other llms successfully. other lines of work that explored the use of non-english prompting are the studies in @xcite @xcite . while these studies have demonstrated an evolving nature of jailbreak attacks in cross-lingual settings, the investigation is done on the language in its standardized form, for example, writing in arabic with the standardized arabic alphabets. our work is closer to @xcite in terms of using different input format to prompt the llms. in this paper, we investigate prompting with the arabic language in chatspeak (akin to leetspeak in english e.g., writing "gpt" as "9p7") and in transliteration forms where accented letters are used to represent a sound in arabic @xcite . on the defense side, the countermeasures can differ by their systematic approach. typically, there are approaches that target the llm itself such as safety training by reinforcement learning from human feedback (rlhf) @xcite , and/or adversarial training on the context of adversarial attacks @xcite . the other approaches are usually used as a complementary task to the llm such as toxicity detection @xcite , and content classifiers @xcite . in section 5, we discuss the implications of using these mitigation methods with arabic and its unconventional forms. ## results gpt-4: for gpt-4, we notice a huge drop in the direct refusal rates when chatspeak and transliteration forms are used. conversely, the advice refusal rates increase. this trend is partially attributed to the misunderstand category. in other words, the model didn't give higher direct refusal rate because it didn't understand the prompt. however, for the unsafe category, the ratio increased from 2.50% to 12.12% for arabic and transliteration respectively. for direct refusal category, the ratio significantly dropped from 92.12% to 13.27% for arabic and transliteration respectively. this drop is still significant even if we combine direct and advice refusals, and the translation output altogether. claude-3: for claude-3, the refusal rate is higher than it is in gpt-4. however, we see an increasing trend in the unsafe rate, espeically for chatspeak with no numbers form. the direct refusal rate drops from 98.46% to 45.58% for arabic and chatspeak-no-numbers respectively, while it increases from 0.19% to 4.62% for the unsafe output. for transliteration form, the refusal rate dropdown is not significant compared to chatspeak-nonumber. however, we noticed that at least 14% of refusals are attributed to copyright protection. we will talk about this in the upcoming subsections. ## potential mitigation the work of @xcite puts the claim that because gpt-4 pretraining data was mainly in english, non-english languages become instrumental in jailbreaking llms, particularly if these languages have not been adequately addressed during model alignment. they refer to such languages low-resource. while this might be true for arabic forms like transliteration and chatspeak, this is highly unlikely for the standardized arabic (for example, see gpt-4 technical report for training @xcite ). we argue that the root cause of cross-lingual jail-break attacks (at least for arabic) may not solely lie in the low-resource training. rather, we think that the primary reason might be a deficiency in adversarial training in cross-lingual manner. here we discuss the implication of adapting a specific mitigation method. convert first: we have seen from the results in safety-training for transliteration and chatspeak: another mitigation is to incorporate arabic transliteration and chatspeak into model alignment. however, it may still be vulnerable to adversarial attacks, as outlined in figure 4 -right. adversarial training: we hypothesize that adversarial training using standardized arabic should be the most effective solution. despite possible conversion of prompts by the llm, we still encounter dangerous behavior. this could result from the incorrect conversion that affects the overall arabic vocabulary and consequently leads the llm to generate responses that are unsafe. upon investigating this, we have noticed that some converted prompts include incorrect words as a result of a naive mapping of latin to arabic letters, bypassing phonemic and collocation knowledge. from this, we come up with a system prompt that operates as if the llm (gpt-4 here) has received proper adversarial training in arabic. the system prompt is typically a set of instructions given to the llm to guide the process of prompting. system prompts can't be modified by end-users and only available to the model providers. here is our system prompt and the outcome of such a system prompt 5 is displayed in system: when you are given a prompt, do necessary conversion or translation to an understandable form following these guidelines: removing the second sentence of step 3 (w-3) could lead to incorrect translations and potentially result in unsafe answer (refer to appendix c for an example). while both versions of the system prompt are effective, we surmise that advanced adversarial training in arabic that integrates phonemic knowledge and word collocations is essential. we've observed that subtle manipulations can elicit unsafe behavior. ## conclusion in this paper, we present an empirical study of jailbreak attacks on llms using arabic in the transliteration and chatspeak form. we show that using arabic in its original form to prompt llms is safe. we have also shown that arabic transliteration and chatspeak could be utilized by various adversaries to jailbreak llms. we have also demonstrated that using languages like arabic and its forms could lead to unknown vulnerabilities that could be exploited by keen adversaries to jailbreak llms. finally, we discuss a mitigation method and the impact of its integration in a formalized and generalized way for safer llms with arabic language.