A rabic S ense: A Benchmark for Evaluating Commonsense Reasoning in A rabic with Large Language Models (2025)

Abstract: Recent efforts in natural language processing (NLP) commonsense reasoning research have led to the development of numerous new datasets and benchmarks. However, these resources have predominantly been limited to English, leaving a gap in evaluating commonsense reasoning in other languages. In this paper, we introduce the ArabicSense Benchmark, which is designed to thoroughly evaluate the world-knowledge commonsense reasoning abilities of large language models (LLMs) in Arabic. This benchmark includes three main tasks: first, it tests whether a system can distinguish between natural language statements that make sense and those that do not; second, it requires a system to identify the most crucial reason why a nonsensical statement fails to make sense; and third, it involves generating explanations for why statements do not make sense. We evaluate several Arabic BERT-based models and causal LLMs on these tasks. Experimental results demonstrate improvements after fine-tuning on our dataset. For instance, AraBERT v2 achieved an 87% F1 score on the second task, while Gemma and Mistral-7b achieved F1 scores of 95.5% and 94.8%, respectively. For the generation task, LLaMA-3 achieved the best performance with a BERTScore F1 of 77.3%, closely followed by Mistral-7b at 77.1%. All codes and the benchmark will be made publicly available at https://github.com/.

## arabicsense: a new benchmark dataset the aim of this work is twofold: to create a dataset for evaluating arabic commonsense reasoning in llms and to improve their performance in this area. to achieve this, we generate diverse, high-quality data specifically designed for training llms in arabic commonsense reasoning. this section outlines the methodology used to create the arabicsense dataset, followed by the human validation process and an analysis of the dataset. ## conclusion in this paper, we introduced arabicsense, the first comprehensive benchmark designed to evaluate the commonsense reasoning abilities of large language models (llms) in arabic. through the creation of three distinct tasks: commonsense validation (task a), commonsense explanation (task b), and commonsense explanation generation (task c), we