The Interplay of Variant, Size, and Task Type in A rabic Pre-trained Language Models (2021)

Abstract: In this paper, we explore the effects of language variants, data sizes, and fine-tuning task types in Arabic pre-trained language models. To do so, we build three pre-trained language models across three variants of Arabic: Modern Standard Arabic (MSA), dialectal Arabic, and classical Arabic, in addition to a fourth language model which is pre-trained on a mix of the three. We also examine the importance of pre-training data size by building additional models that are pre-trained on a scaled-down set of the MSA variant. We compare our different models to each other, as well as to eight publicly available models by fine-tuning them on five NLP tasks spanning 12 datasets. Our results suggest that the variant proximity of pre-training data to fine-tuning data is more important than the pre-training data size. We exploit this insight in defining an optimized system selection model for the studied tasks.

## introduction pre-trained language models such as bert @xcite and roberta @xcite have shown significant success in a wide range of natural language processing (nlp) tasks in various languages. arabic has benefited from extensive efforts in building dedicated pre-trained language models, achieving state-of-the-art results in a number of nlp tasks, across both modern standard arabic (msa) and dialectal arabic (da) @xcite . however, it is hard to compare these models to understand what contributes to their performances because of their different design decisions and hyperparameters, such as data size, language variant, tokenization, vocabulary size, number of training steps, and so forth. practically, one may empirically choose the best performing pre-trained model by fine-tuning it on a particular task; however, it is still unclear why a particular model is performing better than another and what design choices are contributing to its performance. to answer this question, we pre-trained various language models as part of a controlled experiment where we vary pre-training data sizes and language variants while keeping other hyperparameters constant throughout pre-training. we started by scaling down msa pre-training data size to measure its impact on performance in fine-tuning tasks. we then pre-trained three different variants of arabic: msa, da, and classical arabic (ca), as well as a mix of these three variants. we evaluate our models along with eight other recent arabic pre-trained models across five different tasks covering all the language variants we study, namely, named entity recognition (ner), part-ofspeech (pos) tagging, sentiment analysis, dialect identification, and poetry classification, spanning 12 datasets. our contributions can be summarized as follows: ## pre-training camelbert we describe the datasets and the procedure we use to pre-train our models. we use the original implementation released by google for pre-training. 2 3.1 data msa training data for msa, we use the arabic gigaword fifth edition @xcite , osian corpus @xcite , arabic wikipedia, foot_2 and the unshuffled version of the arabic oscar corpus @xcite . da training data for da, we collect a range of dialectal corpora: ldc97t19-callhome transcripts @xcite ; ldc2002t38-callhome supplement transcripts @xcite ; ldc2005s08-babylon levantine arabic transcripts @xcite ; ldc2005s14-cts levantine arabic transcripts @xcite ; ldc2006t07-levantine arabic transcripts @xcite ; ldc2006t15-gulf arabic transcripts (appen pty ltd, 2006a); ldc2006t16-iraqi arabic transcripts (appen pty ltd, 2006b); ldc2007t01-levantine arabic transcripts (appen pty ltd, 2007); ldc2007t04-fisher levantine arabic transcripts @xcite ; arabic online commentary dataset (aoc) (zaidan and callison-burch, 2011); ldc2012t09-english/arabic parallel text @xcite ; arabic multi dialect text corpora @xcite ; a multidialectal parallel corpus of arabic @xcite ; multi-dialect, multi-genre corpus of informal written arabic @xcite ; youdacc @xcite ; padic @xcite ; curras @xcite ; werd @xcite ; ldc2017t07-bolt egyptian sms @xcite ; shami @xcite ; suar @xcite ; arap-tweet (zaghouani and charfi, 2018); gumar @xcite ; madar @xcite ; habibi (el-haj, 2020); nadi @xcite ; and qadi @xcite . ca training data for ca, we use the openiti corpus (v1.2) @xcite . ## fine-tuning tasks we evaluate our pre-trained language models on five nlp tasks: ner, pos tagging, sentiment analysis, dialect identification, and poetry classification. specifically, we fine-tune and evaluate the models using 12 datasets (corresponding to 12 subtasks). we used hugging face's transformers @xcite to fine-tune our camelbert models. 5 the fine-tuning was done by adding a fully connected linear layer to the last hidden state. ## evaluation results and discussion we first present an experiment where we investigate the effect of pre-training data size. we then report on camelbert models pre-trained on msa, da, and ca data, in addition to a model that is pretrained on a mixture of these variants. we then provide a comparison against publicly available models. ## conclusion and future work in this paper, we investigated the interplay of size, language variant, and fine-tuning task type in arabic pre-trained language models using carefully controlled experiments on a number of arabic nlp tasks. our results show that pre-training data and subtask data variant proximity is more important than pre-training data size. we confirm these results on existing models. we exploit this insight in defining an optimized system selection model for the studied tasks. we make all of our created models and fine-tuning code publicly available. in future work, we plan to explore other design decisions that may contribute to the fine-tuning performance, including vocabulary size, tokenization techniques, and additional data mixtures. we also plan to utilize camelbert models in a number of other arabic nlp tasks, and integrate them in the open-source toolkit, camel tools @xcite .