Benchmarking LL a MA -3 on A rabic Language Generation Tasks (2024)

Abstract: Open-sourced large language models (LLMs) have exhibited remarkable performance in a variety of NLP tasks, often catching up with the closed-sourced LLMs like ChatGPT. Among these open LLMs, LLaMA-3-70B has emerged as the most recent and the most prominent one. However, how LLaMA-3-70B would situate itself in multilingual settings, especially in a rich morphological language like Arabic, has yet to be explored. In this work, we focus to bridge this gap by evaluating LLaMA-3-70B on a diverse set of Arabic natural language generation (NLG) benchmarks. To the best of our knowledge, this is the first study that comprehensively evaluates LLaMA-3-70B on tasks related to Arabic natural language generation. Our study reveals that LLaMA-3-70B lags behind the closed LLMs like ChatGPT, both in modern standard Arabic (MSA) and dialectal Arabic (DA). We further compare the performance of LLaMA-3-70B with our smaller and dedicated finetuned Arabic models. We find that both LLaMA-3-70B and ChatGPT are outperformed by comparatively smaller dedicated Arabic models, indicating the scope for potential improvement with Arabic-focused LLMs.

## introduction in the last few years, the emergence of large language models (llms) @xcite @xcite @xcite has emerged as powerful systems to accomplish diverse set of natural language processing (nlp) tasks. such success of these models primarily stems from pretraining them on large datasets with the next token prediction objective. the current state-of-the-art performance is however, dominated mostly by 'closed' models @xcite @xcite . that is, little to no information about them is known. this includes details about model architectures, pretraining data, languages involved, and training configurations. such llms are also expensive both to pretrain and deploy. to alleviate these concerns, 'open' llms such as llama-3 @xcite , mistral @xcite , and olmo @xcite were introduced to facilitate research and (non-) commercial deployment. rigorous evaluations @xcite @xcite have already been conducted to analyze llms performance on a wide range of english benchmarks. however, a limited number of studies can be found when it comes to evaluating llms on non-english languages @xcite @xcite . this limitation becomes more acute, in the case of arabic; a rich morphological language comprised of a wide range of dialects and varieties. @xcite evaluate closed models like chatgpt and gpt-4 on a wide range of arabic nlp tasks, while @xcite experiment with different commercially available closed models (e.g., bard) on various arabic machine translation tasks. however, the performance of the open-sourced models on arabic natural language generation (nlg) benchmarks is still an unexplored territory to this date. to address this gap, in this study, we benchmark llama-3-70b (ai@meta, 2024), the most recent and the best open llm, on a diverse set of 20 arabic nlg tasks. notably, our study reveals that llama-3-70b lags behind closed-sourced llms like chatgpt. we further show that even smaller, dedicated, finetuned arabic models can outperform llama-3-70b. our contributions can be summarized as follows: ## related work this section reviews recent developments in large language models (llms), focusing on their applications in machine translation, question answering, and multilinguality. it particularly emphasizes tasks involving non-english languages such as arabic. please see appendix a for a comprehensive guide. ## dataset we conduct a comprehensive evaluation of llama-3-70b and chatgpt on arabic natural language generation (nlg) tasks. this evaluation encompasses 20 distinct tasks, organized into 10 clusters, compiled from dolphin benchmark @xcite . machine translation. this cluster evaluates translation performance across various languages, focus-ing on capturing nuanced meanings and contexts. key tasks include translating from the six official un languages to modern standard arabic (msa), using the un parallel corpus @xcite and the multiun corpus for training @xcite . another task involves translating arabizi, a romanized version of arabic dialects, into english and french using the darija (outchakoucht and es-samaali, 2021) and narabizi datasets @xcite . additionally, the cluster includes translating texts from six arabic dialects into english utilizing the mdp corpus @xcite , with zero-shot training supported by additional msa-english sentences. code-switching. this task cluster addresses translating arabic dialect texts, including codeswitching, into the corresponding foreign languages. specifically, it includes translating 300 code-switched arabic-french tweets from twitter users in algeria, morocco, and tunisia, and 300 code-switched arabic-english posts from users in egypt, jordan, and palestine. these translations are conducted in a zero-shot setting, utilizing 50,000 msa-english and msa-french sentences from the araopus-20 dataset for monolingual training @xcite ). ## prompt design in this study, we define a prompt as a customized set of instructions that shapes the functionality and enhances the output of a large language model, aligning with the framework proposed by @xcite . the design of a prompt is critical as it influences how the model interacts with users and the quality of its responses by establishing specific operational rules. therefore, to ensure the model's responses meet desired outcomes, it's essential to articulate these rules and objectives. in our initial experiments, we employed a variety of prompts in both english and arabic to evaluate their efficacy across several tasks. these included dialect identification, detection of machinegenerated texts, and toxicity detection. our observations indicated that prompts in english consistently outperformed their arabic counterparts. we developed a standardized english prompt template to optimize performance. this template systematically guides the model by (1) assigning a specific role to the model, such as acting as a translator for natural language generation (nlg) tasks; (2) naming the task to be undertaken, like machine translation; (3) describing the expected outcome, such as accurately translating text from one language to another; (4) incorporating k-shot learning examples if applicable, and; (5) providing a test input to generate an output. we showcase the structure of our prompts in figure 1 . this structured approach allows us to fully utilize the capabilities of llms by ensuring they operate within well-defined parameters that meet the complexities of the tasks at hand. together with simple posthoc-processing, e.g., removing whitespace. models comparisons. for benchmarking, we use llama-3-70b foot_0 from huggingface (wolf et al., 2020 ## evaluation result we present the results of our evaluation on natural language generation tasks in ## conclusion and future work in this work, we conduct a systematic evaluation of llama-3-70b on a wide range of arabic nlg tasks. we benchmark llama-3-70b and comparing against closed llm like chatgpt, we show that open llms like llama-3-70b noticeably lags behind its counterpart. additionally, we compare both closed and open-sourced llms with smaller arabic-focused finetuned model to demonstrate the superiority of such smaller but finetuned models. task cst (jo-@xmath0, i told the exam, it will be 'very nice', my feeling turns to be true llama3-70b i still haven't taken the exam, it will be'very nice', my feeling is that it will come out in its place chatgpt just yesterday, i told the exam, it will be 'very nice', my feeling turns to be true task dialect translation (@xmath1, it says in the end this is not true at all chatgpt i came to say where is he, but i found the message saying in the end it was not true. task diacritization prompt gold prediction llama3-70b chatgpt task transliteration prompt gold prediction topmen llama3-70b tubmin chatgpt tubmin