A ra R easoner: Evaluating Reasoning-Based LLM s for A rabic NLP (2025)

Abstract: Large language models (LLMs) have shown remarkable progress in reasoning abilities and general natural language processing (NLP) tasks, yet their performance on Arabic data, characterized by rich morphology, diverse dialects, and complex script, remains underexplored. This paper presents a comprehensive benchmarking study of multiple reasoning-focused LLMs, with a special emphasis on the newly introduced DeepSeek models, across a suite of fifteen Arabic NLP tasks. We experiment with various strategies, including zero-shot, few-shot, and fine-tuning. This allows us to systematically evaluate performance on datasets covering a range of applications to examine their capacity for linguistic reasoning under different levels of complexity. Our experiments reveal several key findings. First, carefully selecting just three in-context examples delivers an average uplift of over 13 F1 points on classification tasks—boosting sentiment analysis from 35.3% to 87.5% and paraphrase detection from 56.1% to 87.0%. Second, reasoning-focused DeepSeek architectures outperform a strong GPT o4-mini baseline by an average of 12 F1 points on complex inference tasks in the zero-shot setting. Third, LoRA-based fine-tuning yields up to an additional 8 points in F1 and BLEU compared to equivalent increases in model scale. The code is available at https://anonymous.4open.science/r/AraReasoner41299

## introduction arabic is spoken by more than 400 million people across 22 countries, making it one of the most widely used languages in the world. despite its global significance, arabic has traditionally been underrepresented in natural language processing (nlp) research @xcite . this gap is partially due to the inherent complexity of the language, which features a rich morphological system, multiple dialects, and a non-latin script. for instance, modern standard arabic (msa) differs considerably from various colloquial dialects in syntax and vocabulary, increasing the difficulty for models to generalize across tasks such as sentiment analysis, summarization, and translation. moreover, diacritization and transliteration tasks pose unique challenges, as they require a nuanced understanding of phonological rules and script variations. these factors underscore the importance of dedicated research efforts to improve arabic nlp capabilities. large-scale language models (llms) have demonstrated outstanding performance across a wide range of nlp tasks @xcite . models from the gpt @xcite , gemini @xcite , deepseek @xcite , and llama @xcite families have shown exceptional capabilities in different nlp tasks. these models exhibit a strong understanding of natural language and express fluently in hu-man language @xcite . this performance can be attributed mainly to their model architecture and the pre-training on large-scale corpora. recently, the newly introduced reasoning-based models, such as deepseek r1 and gpt-4o, have demonstrated impressive performance in several nlp tasks @xcite . although these models showed a good performance in english and other high-resource languages, their adaptability to arabic remains an open question as arabic remains a morphologically rich yet low-resource language @xcite . this study aims to systematically evaluate reasoning-based llms on fifteen critical arabic nlp tasks. these tasks have been categorized into sentence classification (sc), natural language generation (nlg), and linguistic analysis (la) tasks. the sc cluster involves sentiment analysis, dialect, hate speech, offensive speech, sarcasm, claim, stance, and paraphrase detection. the nlg cluster involves generation-based tasks, such as summarization, paraphrasing, transliteration, machine translation, and question answering. the la tasks are word sense disambiguation (wsd) and part-of-speech (pos) tagging. these tasks have been evaluated under various learning paradigms, such as zero-shot, in-context learning, and finetuning. moreover, we compare the performance of the reasoning-based models with other recent llms. through this comprehensive evaluation, we seek to identify both the potential and the shortcomings of these advanced reasoning-based models in handling arabic's linguistic complexities, ultimately guiding further research and development in this area. ## related work evaluating the performance of these llms requires robust, realistic, and task-specific benchmarks to ensure fair and meaningful comparisons. due to the unique characteristics of the arabic language, its complex morphology, rich diacritic system, and numerous dialects, arabic-specific benchmarks have been developed to better capture these challenges. for instance, diacritization @xcite , the task of restoring diacritical marks to arabic text, is a crucial task specific to arabic. transliteration @xcite ) is another arabic-specific task, often involving converting arabizi (romanized arabic script) into standard arabic script. additional tasks and benchmarks tailored to arabic are discussed in more detail in section 3.1. prior studies have examined the performance of llms across multiple arabic nlp tasks. for instance, @xcite introduced recently orca, a challenging benchmark for arabic language understanding, consisting of 60 different datasets across seven natural language understanding tasks. they used orca to offer a comprehensive comparison between 16 multilingual and arabic pretrained language models. however, their evaluation is restricted to bert-based models, focusing solely on understanding tasks and excluding generative capabilities and newer architectures such as decoder-only llms. taqyimitw @xcite assessed the performance of gpt-3.5 and gpt-4 across seven arabic nlp tasks: sentiment analysis, translation, transliteration, paraphrasing, pos tagging, summarization, and diacritization, proving the outperformance of gpt-4 on six tasks. they also provide an evaluation pipeline that facilitates the evaluation of such models. however, their study is limited to just two closed-source models from the same family, leaving out a broader range of open-source and multilingual models. @xcite for evaluating arabic nlp and speech processing, comparing multiple models, including gpt-3.5, gpt-4, bloomz, jais-13b-chat, whisper, and usm, using zero-shot and few-shot learning techniques to address 33 distinct tasks across 61 publicly available datasets. they found that state-of-the-art (sota) models generally outperformed the evaluated models in both zero-shot and few-shot settings, with a few exceptions in few-shot prompting. although prior studies such as orca, taqyim, and larabench have significantly advanced the evaluation of arabic nlp, there remains a notable gap in assessing the reasoning capabilities of more recent llms. prior efforts have either focused on nlu using bert-based models or a limited set of closed-source generative models. in this paper, we address this gap by evaluating a new set of sota reasoning-oriented models including deepseek-v3 @xcite , deepseek-r1 @xcite , gpt-4o, qwen @xcite , and qwq (qwen-team, 2025) on a diverse suite of arabic nlp tasks. ## benchmark figure 2 shows the pipeline of arareasoner. we selected fifteen arabic nlp tasks to evaluate how well reasoning-based llms perform across a diverse range of language understanding challenges. a prompt instruction selection process was followed for each task to identify the most effective prompt (see appendix c). subsequently, extensive evaluation experiments were conducted under various prompting strategies, including zero-shot and in-context learning. in addition, we compared the reasoning-based models not only against each other but also against non-reasoning llms to highlight the impact of reasoning abilities on task performance. ## conclusion in this work, we have delivered the first in-depth benchmark of reasoning-oriented llms on 15 arabic nlp datasets, demonstrating how prompt engineering, model architecture, and scale interact to shape performance across classification, generation, and la tasks. by showing that just three well-chosen examples can yield double-digit f1 improvements, that deepseek's reasoning-focused design outstrips a top gpt o4-mini baseline by 12 points in zero-shot inference, and that lora fine-tuning can add up to 8 additional points over sheer scaling, we provide a clear roadmap for maximizing results on morphologically rich, dialectally varied arabic data. our comparative analysis also quantifies the remaining 10-15 point gap between reasoning and non-reasoning models, underscoring where future architectures must evolve. future work will focus on evaluating additional open-source models, exploring methods to enhance performance on specific tasks, and expanding both the range of datasets and the diversity of tasks considered.