LA ra B ench: Benchmarking A rabic AI with Large Language Models (2024)

Abstract: Recent advancements in Large Language Models (LLMs) have significantly influenced the landscape of language and speech research. Despite this progress, these models lack specific benchmarking against state-of-the-art (SOTA) models tailored to particular languages and tasks. LAraBench addresses this gap for Arabic Natural Language Processing (NLP) and Speech Processing tasks, including sequence tagging and content classification across different domains. We utilized models such as GPT-3.5-turbo, GPT-4, BLOOMZ, Jais-13b-chat, Whisper, and USM, employing zero and few-shot learning techniques to tackle 33 distinct tasks across 61 publicly available datasets. This involved 98 experimental setups, encompassing ~296K data points, ~46 hours of speech, and 30 sentences for Text-to-Speech (TTS). This effort resulted in 330+ sets of experiments. Our analysis focused on measuring the performance gap between SOTA models and LLMs. The overarching trend observed was that SOTA models generally outperformed LLMs in zero-shot learning, with a few exceptions. Notably, larger computational models with few-shot learning techniques managed to reduce these performance gaps. Our findings provide valuable insights into the applicability of LLMs for Arabic NLP and speech processing tasks.

## introduction generative pre-trained transformer (gpt) models are examples of large language models (llms) 1 trained on massive datasets and using hundreds of millions of parameters. several llms have been recently released for use through apis or pre-trained zero-shot bloomz zero-shot gpt-3.5 zero-shot gpt-4 few-shot gpt-4 (3-shot) sota 0.0 0.2 0.4 0.6 0.8 avg score 0.429 0.535 0.641 0.675 0.700 figure 1: average performance of the models as compared to sota across 21 unique nlp tasks and 31 testing setups. several research initiatives have evaluated these large models' performance on standard nlp and speech processing tasks. the helm project @xcite assessed english llms across various metrics and scenarios. big-bench @xcite introduced a large-scale evaluation with 214 tasks, including low-resource languages. gpt2.5 @xcite , chatgpt (ope-nai, 2023), and bloom @xcite , were recently evaluated by @xcite ; @xcite ; @xcite ; @xcite . large speech models such as whisper @xcite and universal speech model (usm) @xcite were also explored for speech recognition and translation tasks. initiatives such as superb @xcite were introduced to support benchmarking tools and leaderboards for several speech-related tasks. @xcite explored gpt-4's capabilities to determine if it surpasses mere memorization, possessing a profound and adaptable comprehension of concepts, skills, and domains. their results indicate that gpt-4 demonstrates a higher level of general intelligence compared to its predecessors. larabench study fulfills an important objective of assessing the llms capabilities for supporting arabic language processing tasks, for modern standard arabic (msa) and dialectal arabic (da), at the same level of depth and breadth as for english tasks. our evaluation involves 61 publicly available datasets and 98 test setups used to perform and evaluate language processing tasks in both msa and dialectal arabic across various genres (e.g., news articles, tweets, meetings, telephony, and broadcast content). our evaluation focuses on assessing the capabilities of gpt-3.5-turbo, gpt-4, jais-13bchat @xcite 2 and bloomz (176b) for nlp tasks, and of whisper (large, 1.55b) and usm (2b) for speech processing, in both zero and few-shot settings. we investigate: (i) can llms effectively perform arabic nlp and speech processing tasks without prior task-specific knowledge (zero-shot)? (ii) how does performance vary across tasks with different complexities in zero-and fewshot settings? (iii) how do llms compare to current sota models, and are open llms as effective as the commercially available (closed) models? our investigation reveals unique insights about llms' performance on arabic nlp and speech tasks: a. zero-shot multi-task performer. gpt-4 outperforms other models in majority of the nlp tasks (see figure 1 ). however, a large performance gap between gpt-4 and sota models remains due to the higher quality sota models. for speech tasks, usm outperforms all the whisper variants and performs comparably with sota. b. few-shot and sota. gpt-4 reduces the performance gap with sota in the few-shot (only 3-shots) setting (see figure 1 ). this significant performance gain is noticed for almost all tasks, particularly for more complex semantic and questionanswering tasks compared to syntactic and segmentation tasks. similarly, whisper models exhibit promising results in speech recognition with just 2 hours of speech examples in few-shot finetuning. open models (bloomz and whisper) performed poorly w.r.t. to their commercially available counterparts. however, fine-tuning with more instruc-tions may help these open models to achieve closer performance to sota and other closed llms. c. msa vs dialect. the gaps in llms' performance between msa and dialectal datasets (e.g., for machine translation (mt) and speech recognition task) are more pronounced, indicating ineffectiveness of llms for under-represented dialects. ## methodology for benchmarking of arabic nlp and speech processing tasks, we use zero-and few-shot learning involving gpt-3.5-turbo, gpt-4, bloomz and jais-13b-chat for nlp, and whisper (small, medium, and large), usm and amazon polly for speech. we also compared llm's performance with the respective sota models. the use and evaluation of llms involve prompting and post-processing of output to extract the expected content. therefore, for each task, we explored a number of prompts, guided by the same instruction and format as recommended in the azure openai studio chat playground, and promptsource @xcite . after obtaining a reasonable prompt, foot_3 we used it to complete the evaluation of the task using modality-specific api services, e.g., openai api from azure for nlp tasks and google's usm api for speech tasks. as for bloomz and jais-13b-chat, we use an onpremises hosted version. we based our model selection on factors like performance, language support, and accessibility. for nlp tasks, we chose openai models because they consistently outperformed others for english tasks. initially, we used gpt-3.5 and later transitioned to gpt-4 when it became available. limited budget and lack of arabic language support led us to avoid other closed models. among open models, we selected bloomz because it's a large multilingual model, including 4% arabic content. recently released arabic-focused models, such as jais @xcite and acegpt @xcite , have become available as we carry out this study. in our experiments, we benchmarked the jais-13b-chat model across datasets related to seven tasks. a comprehensive exploration of these newly released arabic models will be incorporated into our future studies. for asr, we chose whisper and usm due to their excellent performance in recent studies. ## results and discussion in tables 1, 2, 3 and 4, we report the results of different nlp and speech related tasks. in the below sections, we summarize the results and challenges specific to the task groups. ## conclusion and future studies this study is the first large-scale benchmark that brings together both speech and nlp tasks under the same study. we report the performance of llms covering different domains and dialects. our study also considers tasks with a wide range of complexity ranging from token to text classification, different application settings, ner to sentiment, factuality and disinformation, asr, and tts among others. we evaluate 33 tasks and 61 datasets with 98 test setups, which are very prominent for arabic ai. we compare and report the performance of each task and dataset with sota, which will enable the community and practitioners of large language models to decide on their uses of these models. future work aims to investigate open models and explore ways to reduce the performance gap with sota; enhance prompts for better performance; and expand datasets and tasks.