A ra MUS : Pushing the Limits of Data and Model Scale for A rabic Natural Language Processing (2023)

Abstract: Developing monolingual large Pre-trained Language Models (PLMs) is shown to be very successful in handling different tasks in Natural Language Processing (NLP). In this work, we present AraMUS, the largest Arabic PLM with 11B parameters trained on 529GB of high-quality Arabic textual data. AraMUS achieves state-of-the-art performances on a diverse set of Arabic classification and generative tasks. Moreover, AraMUS shows impressive few-shot learning abilities compared with the best existing Arabic PLMs.

## introduction scaling-up pre-trained language models (plms) has led to astonishing performance gains on a vast variety of natural language processing (nlp) tasks @xcite @xcite . it has also opened new perspectives for studying the opportunities and limitations of large plms @xcite @xcite , as well as their social and ethical impacts @xcite @xcite @xcite . although for some languages such as english and chinese, several plms with even more than hundred billions of parameters have been developed @xcite @xcite , little or no progress has been made on this direction for many other languages including arabic. 1 while there have recently been few attempts to develop multibillion parameters arabic plms @xcite @xcite , still, their performances and abilities have not been well investigated. the largest well-studied arabic plm has no more than 370m parameters @xcite . in this work, we introduce aramus, an 11b parameter encoder-decoder t5 @xcite model, which is pre-trained on 529gb of highquality arabic text (filtered out of 8.8tb). to the best of our knowledge, aramus is the largest arabic plm in terms of pre-training data and model size. furthermore, it is the first time a multi-billion parameter arabic plm is systematically evaluated, against the existing state-of-the-art models, on a diversified set of discriminative and generative task models. more precisely, aramus achieves new state-of-the-art performances of 79.8% on the alue @xcite benchmark, which is a collection of 8 discriminative tasks. in addition, it significantly outperforms the best available encoder-decoder models on multiple generative tasks. finally, aramus shows remarkable abilities to maintain its performance under few-shot settings. ## related work recently, there has been a growing body of the literature on very large-scale english plms by thoroughly studying different aspects of their scaling. these efforts can be summarized into scaling their pre-training data @xcite and model size @xcite @xcite , designing efficient architectures @xcite and pre-training objectives @xcite , democratizing their access @xcite , and making them useful in real-world applications @xcite . besides english, there have been multiple attempts to develop multilingual @xcite , as well as non-anglocentric @xcite @xcite multi-billion plms. unfortunately, the development of arabic plms does not follow the same pace as that of english. the earliest released arabic plms @xcite were based on the bertbase (as well as -large) architecture @xcite and pre-trained on less than 100gb of unfiltered data. successive works tried to improve arabic bert-base models performance by scaling up the pre-training data up to 197gb and 167gb of unfiltered arabic text for marbert @xcite and camelbert @xcite respectively. in addition, other works focused on developing arabic plms to support other architectures like araelectra @xcite , @xcite , arat5 @xcite , and @xcite which are equivalent to english electra @xcite , gpt @xcite , t5 @xcite , and bart @xcite respectively. recently, @xcite developed stateof-the-art arabic bert (jaber and saber) and t5 models (at5s and at5b) by improving the pre-training data quantitatively and qualitatively. more precisely, they pre-trained arabic bertbase/large and t5-small/base models on 115gb of high-quality arabic text data (filtered out of 514gb). @xcite , jasmine (nagoudi et al., 2022a), noor @xcite are the only existing multi-billion arabic plms. these are decoder-only gpt models with 1.5b, 6.7b, and 10b parameters respectively. however, these aforementioned works suffer from the absent (e.g. in aragpt, noor) or limited (e.g. jasmine) comprehensive evaluation on nlp endtasks. moreover, some of these models (such as noor and jasmine) are not publicly available for custom evaluations. foot_2 evaluation is a key factor for understanding the strengths and limitations of these models, without which the progress of the arabic nlp field is hindered. ## conclusion in this paper, we introduced aramus which is not only the largest arabic plm in terms of pretraining data and model size, but also the first multibillion arabic plm to be extensively evaluated on a wide range of nlp tasks. since our work gives clues on the benefits and limitations of scaling up data and model sizes, we hope that it will pave the way for the arabic nlp community to focus on problems that are beyond the reach of plm scaling.