Revisiting Pre-trained Language Models and their Evaluation for A rabic Natural Language Processing (2022)

Abstract: There is a growing body of work in recent years to develop pre-trained language models (PLMs) for the Arabic language. This work addresses two major problems in existing Arabic PLMs that limit the progress of the Arabic NLU and NLG fields. First, existing Arabic PLMs are not well-explored and their pre-training can be improved significantly using a more methodical approach. Second, there is a lack of systematic and reproducible evaluation of these models in the literature. We revisit both the pre-training and evaluation of Arabic PLMs. In terms of pre-training, we explore the impact of the quality of the pretraining data, the size of the model, and the incorporation of character-level information on Arabic PLM. As a result, we release three new Arabic BERT-style models ( JABER, Char-JABER, and SABER), and two T5-style models (AT5S and AT5B). In terms of evaluation, we conduct a comprehensive empirical study to systematically evaluate the performance of existing state-of-the-art models on ALUE, a leaderboard-powered benchmark for Arabic NLU tasks, and on a subset of the Arabic generative tasks. We show that our models significantly outperform existing Arabic PLMs and achieve a new state-of-the-art performance on discriminative and generative Arabic NLU and NLG tasks. Our models and source code to reproduce results will be made available upon acceptance.

## introduction pre-trained language models (plms) such as bert @xcite , gpt @xcite , and t5 @xcite have become the default standard architectures for modern natural language understanding (nlu) systems in both academic @xcite and industrial settings (chakravarti et al., 2020; * equal contribution @xcite . on the evaluation side, the community has widely adopted the leaderboard paradigm foot_0 as a reliable and fair tool to track the progress on various nlp tasks @xcite @xcite . recent years have seen tremendous efforts to develop language-specific plms @xcite @xcite and leaderboards @xcite @xcite for languages other than english. these language-specific plms have proven to be more accurate than multilingual ones in monolingual evaluation settings @xcite @xcite . moreover, creating high-quality human-curated benchmarks is considered to be of utmost importance for reliable evaluation @xcite . for some high-resource languages like chinese, the community has been able to be on par with english nlu in terms of developing plms @xcite @xcite and evaluating them on publicly available leaderboards @xcite . however, we find that the nlp community is unfortunately lagging behind for other languages like arabic. despite the wide availability of arabic plms @xcite @xcite and datasets @xcite @xcite , there are two major issues that constrain the progress of arabic nlu field. first, we observe that the latest techniques for improving pre-training @xcite @xcite are under-explored in the context of arabic plms. in this work, we investigate three ways to improve on the existing arabic plms: quality of the pre-training data, size of the model, and morphology. we propose jaber, a bert-base model pre-trained on high-quality filtered data, that significantly outperforms the best arabic plm baseline by 1.5% on alue @xcite , a newly proposed benchmark with a leaderboard for sequence classification arabic tasks. 2 we also explore two other variants of jaber and report further gains in performance: (i) char-jaber which exploits character-level information and (ii) saber which involves a bert-large model. second, there is a lack of systematic and reproducible evaluation. as a matter of fact, most of the existing work on arabic plms does not follow the recommended evaluation protocols @xcite which include extensive hyperparameter tuning, performing multiple runs on the development set, and reporting performance on hidden test sets. to address this issue, we systematically compare five popular bert-based arabic plms by carefully assessing their performance on the alue leaderboard. we find that the performance ranking of models drastically changes when measured on dev sets as compared to the leaderboard test sets, thereby calling for caution when comparing models without a leaderboard setting. furthermore, we extend our work to t5 encoderdecoder models and arabic generative tasks. we pre-train two t5 small and base models for arabic: at5s and at5b. at5b achieves state-of-the-art results on several generative tasks @xcite by outperforming the recently proposed arat5-base model @xcite both on automatic and human evaluations. we further observe that t5-based arabic plms perform worse than the bert-based models on the alue benchmark which is in contrast to the powerful performance of t5-models on english language tasks @xcite . we conclude with a set of suggestions and directions to explore for pushing progress forward in the arabic nlu community. ## related work there have been several efforts to improve on the pre-training paradigm by scaling up the model size @xcite and data size @xcite , exploring new a wide range of methods have been proposed lately to enrich plms with character-level information, as it has been shown to be beneficial for morphologically rich languages like arabic @xcite @xcite . @xcite proposed noisy language modeling, a new unsupervised pre-training objective for learning character representations. pinter et al. (2021) proposed their xrayemb method that involves adding character-level information to existing plms without the need for pretraining them from scratch. @xcite uses a character-cnn module to learn representations for entire words by consulting the characters of each token, thus avoiding to recourse to word-pieces @xcite . our characterenhanced bert-base model, char-jaber, uses a simple and efficient method to inject character-level representations alongside the sub-tokens representations only at the input layer of bert, with minimal additional parameters and no computational overhead. recent efforts have also been made to develop benchmarks for arabic nlu tasks. @xcite proposed the arlue benchmark which is a collection of 42 discriminative classification tasks. nagoudi et al. (2022) proposed the argen benchmark which consists of 19 datasets for generative tasks. however, both benchmarks have certain limitations which make it challenging to meaningfully evaluate arabic plms. for many tasks, the authors use their own train-devtest splits which are not made publicly available, as of may 10, 2022. in addition, the access to some datasets is not available free of cost. furthermore, none of the tasks include privately-held test data which is important to ensure that a benchmark is used fairly @xcite . therefore, we adopt the alue benchmark @xcite for evaluating our models on classification tasks because this benchmark has a public leaderboard and includes privately-held test sets for many tasks. for evaluating our arabic t5 models, we select a subset of generative tasks from the argen benchmark whose results are freely reproducible (see section 4.1). ## conclusion in this work, we revisit the pre-training and evaluation of arabic plms. we introduced five new arabic language models using both bert-and t5style pre-training schemes. our models outperform all existing arabic models on the generative tasks as well as on the alue benchmark, with saber setting a new state-of-the-art on alue. in order to accelerate the progress in arabic nlu, we advocate for the creation of more leaderboard-based benchmarks with privately-held evaluation sets that cover a wide array of tasks. moreover, we strongly recommend researchers follow a systematic approach similar to the one we propose when evaluating arabic models, with extensive hyperparameter tuning and multiple runs with different random seeds. in the future, our research will mainly focus on scaling up arabic plms to tens (and hundreds) of billions of parameters in an energyefficient manner @xcite as well as scaling up with high-quality pre-training data @xcite . having met all the other conditions in the reproducibility checklist @xcite , we make our source code and models freely available at https://github.com/huawei-noah/pretrainedlanguage-model/tree/master/jaber-pytorch .