BERT -based Multi-Task Model for Country and Province Level MSA and Dialectal A rabic Identification (2021)

Abstract: Dialect and standard language identification are crucial tasks for many Arabic natural language processing applications. In this paper, we present our deep learning-based system, submitted to the second NADI shared task for country-level and province-level identification of Modern Standard Arabic (MSA) and Dialectal Arabic (DA). The system is based on an end-to-end deep Multi-Task Learning (MTL) model to tackle both country-level and province-level MSA/DA identification. The latter MTL model consists of a shared Bidirectional Encoder Representation Transformers (BERT) encoder, two task-specific attention layers, and two classifiers. Our key idea is to leverage both the task-discriminative and the inter-task shared features for country and province MSA/DA identification. The obtained results show that our MTL model outperforms single-task models on most subtasks.

## introduction the arabic language is spoken by approximately 400 million people and characterized by different varieties. on the one hand, people of the arab world tend to use modern standard arabic (msa) as a communication channel in formal situations (e.g. media, religion, education). on the other hand, arabic dialects are usually used for informal communication in daily life. these dialects differ, to varying degrees, from one region to another. generally, existing research works categorized da into four regions (maghreb, egypt, gulf, and levant) based on the coarse-grained taxonomy @xcite . recently, several research studies classified da into more finegrained varieties such as country-level dialects @xcite @xcite . in the last few years, arabic dialect identification has gained much attention @xcite @xcite . identifying the dialect of an end-user is a very important task in many applications such as user profiling, personalized customer support, etc. nevertheless, due to the nature and the structure of da as well as msa @xcite @xcite , this task faces several challenges. previous works on da identification mainly focused on the use of traditional machine learning models @xcite and single-task deep learning models @xcite . @xcite introduced hierarchical models that perform the training and prediction of the country-level and province-level classification based on a sequential process. recently, @xcite shown the effectiveness of mtl using marbert, a transformer-based language model pre-trained on a massive dataset of 128gb arabic tweets, for both country-level and province-level da identification. in this paper, we tackle both the da and msa identification of the second nadi (nuanced arabic dialect identification) shared tasks: countrylevel msa identification, country-level da identification, province-level msa identification, and province-level da identification (abdul-mageed et al., 2021). our submitted system consists of an end-to-end deep mlt model that predicts both the province and the country of a given arabic tweet. our model leverages marbert's contextualized word embedding @xcite with two task-specific attention layers that extract the task-discriminative features. the latter features are then concatenated with the encoder's pooled embedding ([cls] embedding) and are fed to the task classifiers. thus, the core idea of our approach is to combine both task-discriminative and inter-task shared features for country and province msa/da identification. the obtained results show that our mtl model outperforms its single-task counterpart on all evaluated subtasks. these results prove the effectiveness of combining task-specific features and inter-task shared features for countrylevel and province-level msa and da identification. the rest of this paper is organized as follows. section 2 describes the nadi shared task's datasets. section 3 presents our proposed method. section 4 summarizes the obtained results for both subtask 1 and subtask 2. in section 5, we discuss these results. finally, the conclusion is given in section 6. ## data the second nadi shared task consists of four subtasks on msa and da country-level as well as province-level identification. ## method our multi-task model, for both tasks, consists of three main components: bert encoder (mar-bert pre-trained language model), two taskspecific attention layers, and two task classifiers. ## results in our experiments, we have investigated multiple models, starting from traditional machine learning techniques to transformer-based approaches. the obtained results show that marbert significantly outperforms the other approaches. for a fair comparison, our single-task model employs an attention layer over the contextualized word embedding of marbert and concatenates its outputs with the [cls] token embedding for msa and da identification subtasks. it is worth mentioning that incorporating an attention layer into single-task and mtl models improves the results compared to performing the classification using only the [cls] token embedding representation. ## discussion in order to analyze the results and explain the lower performance of our mtl model on some subtasks, figure 2 draws the confusion matrices for da and msa identification at the country-level on the dev set. the matrices highlight a number of strengths and weaknesses of our final model. moreover, the msa country-level identification is a very challenging subtask since it is hard to find patterns to discriminate between countries and provinces based on standard language. also, the model tends to make incorrect predictions for countries that are geographically close since their dialects have some minor differences (e.g. the gulf countries) compared to other arab countries. ## conclusion in this paper, we introduced our submitted system to the second nadi shared task. we proposed an mtl model for joint country-level and province-level identification of msa and da tweets. the model is based on the state-of-the-art marbert encoder and uses two task-specific attention layers to extract the task-discriminative features. the obtained results have shown that our mtl model outperforms the single-task model on all subtasks for both evaluation measures (macro-f1 and accuracy). future research work will focus on developing task-interaction and class-interaction modules and mechanisms for coarse-grained and fine-grained da and msa identification.