A rabic dialect identification: An A rabic- BERT model with data augmentation and ensembling strategy (2020)

Abstract: This paper presents the ArabicProcessors team’s deep learning system designed for the NADI 2020 Subtask 1 (country-level dialect identification) and Subtask 2 (province-level dialect identification). We used Arabic-Bert in combination with data augmentation and ensembling methods. Unlabeled data provided by task organizers (10 Million tweets) was split into multiple subparts, to which we applied semi-supervised learning method, and finally ran a specific ensembling process on the resulting models. This system ranked 3rd in Subtask 1 with 23.26% F1-score and 2nd in Subtask 2 with 5.75% F1-score.

## introduction with the increasing internet access to arab populations, their contributions to internet content are growing in a remarkable way. indeed, the internet penetration rate increased from 30.3% to 51.6% between 2012 and 2019 in arab countries @xcite . additionally, there are multiple social networking platforms facilitating the sharing of content for all users. this has led to the appearance of arabic dialect on internet platforms, which for a long time remained limited to oral conversations of everyday life, unlike modern standard arabic (msa), which is the only structured arabic language that serves as the official language of writing and communication in all arab countries. as a result, there is a growing interest in the treatment and exploitation of these dialects, which differ substantially from msa and also differ between different countries @xcite . we can differentiate between two types of works related to arabic dialects identification (did): coarse-grained and fine-grained, where the former focuses on binary classifications @xcite , or large groups of dialects such as egyptian, gulf, iraqi, maghrebi and levantine @xcite @xcite . works for fine-grained identification has widened the field by incorporating a multitude of dialects reaching, among others, 17 arab countries @xcite ) and 25 different cities @xcite . the nuanced arabic dialect identification (nadi) shared task (abdul-mageed et al. 2020) reinforces this type of did by offering two subtasks, namely subtask 1 (country-level dialect identification) with 21 different countries dialects, and subtask 2 (province-level dialect identification) with 100 different province dialects. in this paper, we present our contribution for both subtasks. indeed, we confirm in our work the performance of the bert models, which, contrary to traditional statistical methods or classical machine learning techniques, have not been often used for did problems. in this paper, we expand upon the work of @xcite with an arabic specific bert model and a new data augmentation approach concluded with an ensemble model. this approach allowed us to go from an f1-@xmath0.41% for the baseline model to an f1-@xmath1.01%; an improvement of 9.1 percentage points. the semi-supervised stage contributed with an improvement of 1.56 percentage points (from a score of 23.45%) scores obtained on the dev dataset, see results section. in the next sections, we describe used data in section 2, describe our system in section 3, present our results in section 4, discuss the data in sections 5, and finally summarize our work in section 6. as for the content of the train and dev datasets, they contained the tweet ids, their texts, the two variables to be predicted, namely country labels (subtask 1) and province labels (subtask 2), for respectively 21 countries and 100 provinces. finally, the organizers also provided us with the unlabeled test dataset, intended for the final evaluation of the system. the score obtained on this dataset was used for system ranking. dataset number of tweets country labels provinces labels train 21,000 21 100 dev 4,957 21 100 unlabeled-10m 9,999,978 1 --test 5,000 -- ## system the adopted system consists of three main steps, namely, (i) bert-type model training, (ii) data augmentation with semi-supervised learning, and (iii) ensembling methods. these three steps will be described in this section. ## conclusion in this paper, we have described our contribution to nadi shared task 2020 subtasks. subtask 1 dealt with arabic dialects identification (did) at the country level (21 arab countries) and subtask 2 focused on dialects by provinces (100 provinces). our system is based on an arabic-specific bert model, a semi-supervised method for data augmentation and an ensembling of different models following the data augmentation. by recording an f1-@xmath2.26% at subtask 1 (ranked third) and 5.75% at subtask 2 (ranked second), this approach demonstrates the efficiency of our bert models in the field of did.