Adapting MARBERT for Improved A rabic Dialect Identification: Submission to the NADI 2021 Shared Task (2021)

Abstract: In this paper, we tackle the Nuanced Arabic Dialect Identification (NADI) shared task (Abdul-Mageed et al., 2021) and demonstrate state-of-the-art results on all of its four subtasks. Tasks are to identify the geographic origin of short Dialectal (DA) and Modern Standard Arabic (MSA) utterances at the levels of both country and province. Our final model is an ensemble of variants built on top of MARBERT that achieves an F1-score of 34.03% for DA at the country-level development set—an improvement of 7.63% from previous work.

## introduction the arab world is a vast geographical region that covers north africa and southwest asia, boasting a population of around 400m that speak different derivatives of a common language. however, by virtue of its size and cultural variety, there exists a dialect continuum across the region wherein the language varieties of neighboring peoples may differ slightly, but distant regions can become mutually unintelligible. this continuum is referred to as dialectal arabic (da) and is the "low" variety of modern arabic diglossia. on the other hand, the "high" variety is referred to as modern standard arabic (msa) and is used in formal settings such as academia, mass media, and legislationand is taught through the formal education system in most arab countries. this standard variant emerged gradually, but most notably with the advent of the printing press in the 19th century. it diverged from classical arabic (ca) into a more simple version that is now used across the arab world. the modern vernacular dialects (da) differ along several dimensions, including pronunciation, syntax, morphology, vocabulary, and even orthography. dialects may be heavily influenced by previously dominant local languages. for example, egyptian variants are influenced by the coptic language, while sudanese variants are influenced by the nubian language. in this paper, we study the classification of such variants and describe our model that achieves stateof-the-art results on all of the four nuanced arabic dialect identification (nadi) subtasks @xcite . the task focuses on distinguishing both msa and da by their geographical origin at both the country and province levels. the data is a collection of tweets covering 100 provinces from 21 arab countries. the code has been made open-source and available on github foot_0 . ## related work the first efforts to collect and label dialectal arabic data goes back to 1997 @xcite . however, studying da in nlp started to gain traction in recent years as more digital arabic data became available, especially with the rise of online chatting platforms that place no restrictions on the syntax, style, or formality of the writing. @xcite labelled the arabic online commentary dataset (aoc) through crowd-sourcing, then built a model to classify even more crawled data. @xcite provided the qadi dataset by automatically collecting dialectal arabic tweets and labelling them based on descriptions of the author accounts, while trying to reduce the recall of written msa and inappropriate tweets. abu farha and magdy (2020) provided arsarcasm: a dataset of labelled tweets. originally for sarcasm detection, it also contains labels for sentiment and dialect detection. labelled dialectal arabic challenges such as madar @xcite and nadi @xcite (which started in 2020) shed light on the underlying challenges of the task. both comprise labelled arabic tweets but with class sets of different granularities. @xcite presents a solution that won the 2020 nadi shared task (subtask 1.2) by continuing training @xcite using masked language modelling (mlm) on 10m unlabelled tweets, then fine-tuning on the dialect identification task. on the other hand, the solution that won subtask 2.2 uses a hierarchical classifier that takes as input a weighted combination of tf-idf and arabert features to first classify the country, then invokes an arabbert-based, country-specific, province-level classifier to detect the province @xcite . the large size of pretrained transformer models hinders their applicability in many use cases. the usual number of parameters in such a model lies between 150m and 300m and needs between 500mb and 1gb of space to be stored. a novel technique to address these issues is proposed by @xcite -bottleneck layers ("adapters") are added to each transformer layer as an alternative to fine-tuning the entire pre-trained model when optimizing for downstream tasks @xcite @xcite . only these additional parameters (which can be 1% or less of the size of the main model) need to be stored per downstream task, given that they are the only layers changing. besides being light-weight and scalable, adapters offer several advantages over traditional approaches of transfer-learning: (1) they learn modular representations that are compatible with other layers of the transformer, (2) they avoid interfering with pre-trained knowledge, mitigating catastrophic forgetting and catastrophic inference-two common downsides of multi-task learning @xcite . the second nadi shared task consists of four subtasks on two datasets (see ## system description our model builds on marbert-a publicly available transformer model trained on 1b multidialectal arabic tweets @xcite . it follows the bert base architecture @xcite with 163m parameters and similarly uses wordpiece tokenization @xcite . to optimize it for the task at hand, we introduce changes to the architecture and training regimen as described below. note that, due to time and compute constraints, all hyperparameters were optimized on the development set of subtask 1.2 then applied as-is to the other three subtasks. we report the result of the best ensemble for each subtask. ## discussion the confusion matrix of the ensemble in subtask 1.2 ( considering the sequence length in words for correct and wrong predictions, we find that correctly predicted sentences tend to be longer than the means of all and of wrongly predicted sentences (fit as an erlang distribution). the numbers were computed after removing the special user and url tokens. ## conclusion in this paper, we present a solution that provides a new state-of-the-art on all of the nadi subtasks. inserting adapters at each of the marbert transformer layers preserved the original pre-trained knowledge, stemming from a rich corpus of tweets, while still embedding task knowledge. to further improve the model's performance, we vertically attend on all of the cls token hidden-states coming out of each of the transformer layers instead of just using the top layer's output. we ensemble models on two design variables: whether fine-tuning is full or through adapters, and whether vertical attention is used. the bias towards dominating classes in the task dataset remains a significant issue that is still not mitigated. as a future work, we would like to employ adapter fusion @xcite to attenuate this bias.