A rabizi vs LLM s: Can the Genie Understand the Language of Aladdin? (2025)

Abstract: In an era of rapid technological advancements, communication continues to evolve as new linguistic phenomena emerge. Among these is Arabizi, a hybrid form of Arabic that incorporates Latin characters and numbers to represent the spoken dialects of Arab communities. Arabizi is Widely used on social media and allows people to communicate in an informal and dynamic way, but it poses significant challenges for machine translation due to its lack of formal structure and deeply embedded cultural nuances. This case study is motivated by a growing need to translate Arabizi for gisting purpose. It evaluates the capacity of different LLMs’ to decode and translate Arabizi, focusing on multiple Arabic dialects that have rarely been studied up until now. Using a combination of human evaluators and automatic metrics, this research project investigates the model’s performance in translating Arabizi into both Modern Standard Arabic and English. Key questions explored include which dialects are translated most effectively and whether translations into English surpass those into Arabic.

## introduction although there are approximately 420 million arabic speakers worldwide, an intriguing linguistic paradox emerges: modern standard arabic (msa), the standardized form of the language, is the mother tongue of none. instead, arabs communicate through their regional dialects, which are vibrant linguistic hybrids influenced by arabic and the historical languages of each region. these dialects have been honed by geographic, cultural, and historical factors, and can vary significantly even within a single country, resulting in a mosaic of over 60 distinct varieties. arabizi (a fusion of "arabic" and englizi, the arabic word for english) is an informal, non-standard writing system that emerged in the 1990s when arabic keyboards were not widely available. it uses latin characters and numbers, combining both transliteration and transcription mappings. primarily used in online communication-such as short messages and comments on social media-arabizi varies significantly across dialects and even within the same dialect @xcite . for instance, the transcription of the following sentence in msa (i want to talk to you about something) in arabizi can be "badde e7kik bi mawdu3" in levantine arabic, or "rani hab nahdar m3ak f wahd sujet" in algerian arabic. the idea of romanizing the arabic language is not a new concept, as there have already been several attempts to do so over the last century. however, these efforts largely failed, as they were perceived as colonialist initiatives aimed at suppressing cultural and religious identity. more recently, the international organization for standardization (iso) introduced two norms, @xcite , to standardize the romanization of arabic. these standards aimed to facilitate the international exchange of information. nevertheless, their adoption remained limited due to their impracticality, with usage restricted primarily to official contexts @xcite . conversely, arabizi has become the dominant written form of communication among arabic speakers in informal settings. its rise reflects a crucial sociolinguistic reality: while msa remains the language reserved for academic, religious, and formal settings, it is often perceived as inaccessible or overly formal for daily use. arabizi, by contrast, offers a dynamic and flexible medium for self-expression that aligns with the fluidity of arabic dialects @xcite . despite its widespread use across digital platforms, and the recent focus on informal language and low-resource languages, arabizi remains an unexplored area in natural language processing (nlp). it poses particular challenges due to its colloquial nature, variation across dialects and lack of standardization, as well as the scarcity of digital resources. in nlp, research on arabizi has mainly focused on transliteration into arabic (deromanization) at the character or word level, using different approaches @xcite , and on the creation of a parallel annotated corpus of sms and chat messages written in arabizi and their corresponding arabic script transliterations by @xcite . some studies explore interlinguistic machine translation (mt) techniques to and from arabizi, employing various architectures and pipelines, mainly between english and egyptian dialects (see @xcite for a summary up until 2017). while some recent datasets for low resource language translation include romanization, they are not specifically focused on arabizi. flores benchmark @xcite , for example, is limited to the romanized transcription of msa or arabic dialects in arabic scripts. the terjamabench dataset @xcite is an exception and includes entries in darija, the moroccan arabic dialect, written in both latin alphabet (arabizi) and arabic script, and their corresponding english translations. since there is a growing need to translate arabizi into resource-rich languages on social media and other digital platforms, we conducted a case study to evaluate the feasibility of using large language models (llms) for out-of-the-box machine translation. the project began when the language technology company iguanodon.ai received a request from a client who wanted to know if short arabizi texts could be translated for gisting purposes. the study involves a collaboration between the start-up and a professional translator with previous experience in arabizi. our contribution includes aladdinbench , an authentic dataset in arabizi for three dialects publicly available on huggingface 1 and a comparative evaluation of five llms using different prompting strategies. to our knowledge, this is the first study that has explored the direct translation from different dialects in arabizi to msa or english without prior deromanization. the rest of this paper is organized as follows. we describe the data production methodology and resulting dataset in section 2, followed by the experimental setup in section 3 and the results in section 4. finally, discussion of results and the limitations of this study are presented in section 5 and section 6. ## experimental setup we carried out a systematic evaluation of translation quality using an automated protocol. for each dialect, we created a combination of parameters defined as follows: all models were prompted with a temperature of 0.5. a discussion of the chosen models and prompts is available in 3.1 and in 3.2. evaluation metrics are presented in 3.3. our code is available. 3 ## conclusion and discussion models struggle significantly with arabizi. gpt-4o is the best-performing translation model, followed by gemini. mistral large and gemma perform moderately well, while llama 3 and jais are the weakest models (see appendix h). interestingly, gemma performed surprisingly well in translation tasks despite being a 27b parameter model. its results, particularly in english, were competitive with larger models, suggesting that model size is not the only determinant of translation quality-architectural optimizations and training data also play a crucial role. for model prompting, few-shot approaches improved performance but was more effective for english than for arabic. english prompts worked better overall and in all prompting scenarios, though the difference was much less stark for gpt-4o and gemini and, to a lesser extent, for gemma. despite a large variation in average segment length between different dialects, no clear pattern emerged in terms of automatic scores. this hints that translation quality does not directly depend on segment length. the llm-as-a-judge scenario aligned with expert human raters, making it a relevant tool in this setting. this study further shows that while far from perfect, using "out-of-the-box" llms to translate arabizi is a viable solution for gisting, especially when combined with an llm-as-a-judge.