ARBERT & MARBERT : Deep Bidirectional Transformers for A rabic (2021)

Abstract: Pre-trained language models (LMs) are currently integral to many natural language processing systems. Although multilingual LMs were also introduced to serve many languages, these have limitations such as being costly at inference time and the size and diversity of non-English data involved in their pre-training. We remedy these issues for a collection of diverse Arabic varieties by introducing two powerful deep bidirectional transformer-based models, ARBERT and MARBERT. To evaluate our models, we also introduce ARLUE, a new benchmark for multi-dialectal Arabic language understanding evaluation. ARLUE is built using 42 datasets targeting six different task clusters, allowing us to offer a series of standardized experiments under rich conditions. When fine-tuned on ARLUE, our models collectively achieve new state-of-the-art results across the majority of tasks (37 out of 48 classification tasks, on the 42 datasets). Our best model acquires the highest ARLUE score (77.40) across all six task clusters, outperforming all other models including XLM-R Large ( 3.4x larger size). Our models are publicly available at https://github.com/UBC-NLP/marbert and ARLUE will be released through the same repository.

## introduction language models (lms) exploiting self-supervised learning such as bert @xcite and roberta @xcite have recently emerged as powerful transfer learning tools that help improve a very wide range of natural language processing (nlp) tasks. multilingual lms such as mbert @xcite and xlm-roberta (xlm-r) @xcite have also been introduced, but are usually outperformed by monolingual models pre-trained with larger vocabulary and bigger language-specific datasets (virtanen et al., 2019; antoun et al., 2020; dadas et al., 2020; † all authors contributed equally. since lms are costly to pre-train, it is important to keep in mind the end goals they will serve once developed. for example, (i) in addition to their utility on 'standard' data, it is useful to endow them with ability to excel on wider real world settings such as in social media. some existing lms do not meet this need since they were trained on datasets that do not sufficiently capture the nuances of social media language (e.g., frequent use of abbreviations, emoticons, and hashtags; playful character repetitions; neologisms and informal language). it is also desirable to build models able to (ii) serve diverse communities (e.g., speakers of dialects of a given language), rather than focusing only on mainstream varieties. in addition, once created, models should be (iii) usable in energy efficient scenarios. this means that, for example, medium-to-large models with competitive performance should be preferred to large-to-mega models. a related issue is (iv) how lms are evaluated. progress in nlp hinges on our ability to carry out meaningful comparisons across tasks, on carefully designed benchmarks. although several benchmarks have been introduced to evaluate lms, the majority of these are either exclusively in english (e.g., @xcite , glue @xcite , superglue @xcite ) or use machine translation in their training splits (e.g., xtreme @xcite ). again, useful as these benchmarks are, this circumvents our ability to measure progress in real-world settings (e.g., training and evaluation on native vs. translated data) for both cross-lingual nlp and in monolingual, non-english environments. context. our objective is to showcase a scenario where we build lms that meet all four needs listed above. that is, we describe novel lms that (i) excel across domains, including social media, (ii) can serve diverse communities, and (iii) perform well compared to larger (more energy hungry) mod-els (iv) on a novel, standardized benchmark. we choose arabic as the context for our work since it is a widely spoken language (∼ 400m native speakers), with a large number of diverse dialects differing among themselves and from the standard variety, modern standard arabic (msa). arabic is also covered by the popular mbert @xcite and xlm-r @xcite , which provides us a setup for meaningful comparisons. that is, not only are we able to empirically measure monolingual vs. multilingual performance under robust conditions using our new benchmark, arlue, but we can also demonstrate how our base-sized models outperform (or at least are on par with) larger models (i.e., xlm-r large , which is ∼ 3.4× larger than our models). in the context of our work, we also show how the currently best-performing model dedicated to arabic, @xcite , suffers from a number of issues. these include (a) not making use of easily accessible data across domains and, more seriously, (b) limited ability to handle arabic dialects and (c) narrow evaluation. we rectify all these limitations. our contributions. with our stated goals in mind, we introduce arbert and mar-bert, two arabic-focused lms exploiting largeto-massive diverse datasets. for evaluation, we also introduce a novel arabic natural language understanding evaluation benchmark (arlue). arlue is composed of 42 different datasets, making it by far the largest and most diverse arabic nlp benchmark we know of. we arrange ar-lue into six coherent cluster tasks and methodically evaluate on each independent dataset as well as each cluster task, ultimately reporting a single arlue score. our models establish new stateof-the-art (sota) on the majority of tasks, across all cluster tasks. our goal is for arlue to serve the critical need for measuring progress on arabic, and facilitate evaluation of multilingual and arabic lms. to summarize, we offer the following contributions: the rest of the paper is organized as follows: in section 2, we provide an overview of arabic lms. section 3 describes our arabic pre-tained models. we evaluate our models on downstream tasks in section 4, and present our benchmark ar-lue and evaluation on it in section 5. section 6 is an overview of related work. we conclude in section 7. we now introduce existing arabic lms. ## arabic lms the term arabic refers to a collection of languages, language varieties, and dialects. the standard variety of arabic is msa, and there exists a large number of dialects that are usually defined at the level of the region or country @xcite . a number of arabic lms has been developed. the most notable among these is @xcite , which is trained with the same architecture as bert @xcite and uses the bert base configuration. arabert is trained on 23gb of arabic text, making ∼ 70m sentences and 3b words, from arabic wikipedia, the open source international dataset (osian) @xcite (3.5m news articles from 24 arab countries), and 1.5b words corpus from el-khair (2016) (5m articles extracted from 10 news sources). antoun et al. (2020) evaluate arabert on three arabic downstream tasks. these are (1) sentiment analysis from six different datasets: hard @xcite , astd @xcite , arsentd-lev @xcite , labr @xcite , and arsas @xcite . (2) ner, with the anercorp @xcite , and (3) arabic qa, on arabic-squad and arcd @xcite datasets. another arabic lm that was also introduced is ara-bicbert @xcite , which is similarly based on bert architecture. arabicbert was pretrained on two datasets only, arabic wikipedia and arabic osacar @xcite . since both of these datasets are already included in arabert, and arabic osacar foot_0 has significant duplicates, we compare to arabert only. gigabert @xcite , an arabic and english lm designed with code-switching data in mind, was also introduced. foot_1 3 our models ## conclusion we presented our efforts to develop two powerful transformer-based language models for arabic. our models are trained on large-to-massive datasets that cover different domains and text genres, including social media. by pre-training marbert and marbert-v2 on dialectal arabic, we aim at enabling downstream nlp technologies that serve wider and more diverse communities. our best models perform better than (or on par with) xlm-r large (∼ 3.4× larger than our models), and hence are more energy efficient at inference time. our models are also significantly better than arabert, the currently best-performing arabic pre-trained lm. we also introduced aralu, a large and diverse benchmark for arabic nlu composed of 42 datasets thematically organized into six main task clusters. arlue fills a critical gap in arabic and multilingual nlp, and promises to help propel innovation and facilitate meaningful comparisons in the field. our models are publicly available. we also plan to publicly release our arlue benchmark. in the future, we plan to explore self-training our language models as a way to improve performance following @xcite . we also plan to investigate developing more energy efficient models.