ORCA : A Challenging Benchmark for A rabic Language Understanding (2023)

Abstract: Due to the crucial role pretrained language models play in modern NLP, several benchmarks have been proposed to evaluate their performance. In spite of these efforts, no public benchmark of diverse nature currently exists for evaluating Arabic NLU. This makes it challenging to measure progress for both Arabic and multilingual language models. This challenge is compounded by the fact that any benchmark targeting Arabic needs to take into account the fact that Arabic is not a single language but rather a collection of languages and language varieties. In this work, we introduce a publicly available benchmark for Arabic language understanding evaluation dubbed ORCA. It is carefully constructed to cover diverse Arabic varieties and a wide range of challenging Arabic understanding tasks exploiting 60 different datasets (across seven NLU task clusters). To measure current progress in Arabic NLU, we use ORCA to offer a comprehensive comparison between 18 multilingual and Arabic language models. We also provide a public leaderboard with a unified single-number evaluation metric (ORCA score) to facilitate future research.

## introduction pretrained language models (plms) @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite have become a core component of the natural language understanding (nlu) pipeline, making it all the more important to evaluate their performance under standardized conditions. for this reason, several english-based benchmarks such as glue @xcite , superglue @xcite , syntaxgym @xcite , evaluation harness @xcite , @xcite , nl-augmenter @xcite , dynabench @xcite , @xcite , natural instructions @xcite , big-bench @xcite , and multilingual benchmarks such as xtreme @xcite , xglue @xcite , and masakhane @xcite have been introduced. benchmarks for a few other languages also followed, including flue @xcite for french, clue @xcite for chinese, in-donlu @xcite for indonesian, @xcite for korean, and jglue @xcite . this leaves behind the majority of the world's languages, and relies on multilingual benchmarks which often have limited coverage of dialects and naturally-occurring (rather than machine translated) text. this motivates us to introduce a benchmark for arabic. one other reason that lends importance to our work is that arabic is a rich collection of languages with both standard and dialectal varieties and more than 400m native speaker population. to the best of our knowledge, there have only been two attempts to provide arabic nlu evaluation benchmarks. these are arlue @xcite and alue @xcite . although useful, both of these have major limitations: alue has modest coverage (only eight datasets covering only three task clusters) and arlue involves datasets that are not publicly available. our goal is to rectify these limitations by introducing orca, which expands task coverage using fully public datasets, while also offering an accessible benchmark with a public leaderboard and processing tools as well as wide geographical and linguistic coverage. orca exploits 60 different datasets, making it by far the most extensive benchmark for arabic nlu and among the most extensive for any language. we present detailed analyses of the data comprising orca and evaluate a wide range of available pretrained language models (plms) on it, thus offering strong baselines for future comparisons. in summary, we offer the following contributions: (1) we introduce orca, an extensive and diverse benchmark for arabic nlu. orca is a collection of 60 datasets arranged into seven task clusters, namely: sentence classification, text classification, structured prediction, semantic similarity, natural language inference (nli), question-answering (qa), and word sense disambiguation (wsd). (2) we provide a comprehensive comparison of the performance of publicly available arabic plms on orca using a unified orca score. (3) to facilitate future work, we design a public leaderboard for scoring plms on orca. our leaderboard is interactive and offers rich meta-data about the various datasets involved as well as the language models we evaluate. (4) we distribute orca with a new modular toolkit for pretraining and transfer learning for nlu. the toolkit is built around standard tools including pytorch @xcite and huggingface datasets hub @xcite . the rest of the paper is organized as follows: in section 2, we provide an overview of related work. section 3 introduces orca, our arabic nlu benchmark. in section 4, we describe multilingual and arabic pretrained language models we evaluate on orca, providing results of our evaluation in section 5. section 6 is an analysis of model computational cost as measured on orca. we conclude in section 7. ## related work most recent benchmarks propose a representative set of standard nlu tasks for evaluation. these can be categorized into english-centric, multilingual, arabic-specific, and x-specific (x being a language other than english or arabic such as chinese, korean, or french). we briefly describe each of these categories next. we also provide a comparison of benchmarks in the literature in terms of task clusters covered and number of datasets in ## orca benchmark we present orca, a benchmark for arabic nlu that is challenging and diverse. orca involves 60 datasets arranged into 29 tasks and seven task clusters. in the following, we will first introduce our design principles for developing orca then introduce the different task clusters covered. ## language models in this section, we list multilingual plms that include arabic in its coverage only by name, for space but provide a description of each of them in appendix a. multilingual lms. these are mbert @xcite , xlm-r @xcite , gigabert @xcite , and mt5 @xcite . arabic lms. these are arabert (antoun et al., 2020), arabicbert (safaya et al., 2020), arabic-albert (safaya, 2020), qarib chowdhury et al. (2020), arbert & marbert (abdul-mageed et al., 2021), camelbert (inoue et al., 2021), jaber and saber (ghaddar et al., 2021), and arat5 (nagoudi et al., 2022). table a.1 (appendix a) shows a comparison between the multilingual as well as the arabic plms in terms of (1) training data size, (2) vocabulary size, (3) language varieties, and (4) model configuration and architecture. ## model evaluation on orca this section shows experimental settings and performance of 18 multilingual and arabic language models on orca downstream tasks. 6 baselines. for comparison, we finetune the multilingual language models mbert and xlm-r base on all training data of orca benchmark. evaluation. for all models and baselines, across all tasks, we identify the best model on the respective development data split (dev) and blind-test on the testing split (test). we methodically evaluate each task cluster, ultimately reporting a single orca score following @xcite ; @xcite . orca score is simply the macro-average of the different scores across all tasks and task clusters, where each task is weighted equally. we compute the orca score for all 18 language models. results. we present results of all language models and the baselines on each task cluster of orca independently using the relevant metric, for both dev (see ## analysis of model computational cost we also compare the arabic language models in terms of computational cost using the average time needed for convergence (in minutes) and average number of epochs to convergence as identified on dev sets. for this, we finetune all models for a maximum of 25 epochs on all orca tasks. we report results in terms of average of three runs. ## conclusion we presented orca, a large and diverse benchmark for arabic natural language understating tasks composed of 60 datasets that are arranged in seven task clusters. to facilitate future research and adoption of our benchmark, we offer a publicly-available interactive leaderboard with a useful suite of tools and extensive meta-data. in addition, we provide a comprehensive and methodical evaluation as well as meaningful comparisons between 18 multilingual and arabic language models on orca. we also compare the models in terms of computing needs. as our results show, orca is challenging and we hope it will help standardize comparisons and accelerate progress both for arabic and multilingual nlp.