On The A rabic Dialects’ Identification: Overcoming Challenges of Geographical Similarities Between A rabic dialects and Imbalanced Datasets (2022)

Abstract: Arabic is one of the world’s richest languages, with a diverse range of dialects based on geographical origin. In this paper, we present a solution to tackle subtask 1 (Country-level dialect identification) of the Nuanced Arabic Dialect Identification (NADI) shared task 2022 achieving third place with an average macro F1 score between the two test sets of 26.44%. In the preprocessing stage, we removed the most common frequent terms from all sentences across all dialects, and in the modeling step, we employed a hybrid loss function approach that includes Weighted cross entropy loss and Vector Scaling(VS) Loss. On test sets A and B, our model achieved 35.68% and 17.192% Macro F1 scores, respectively.

## introduction the arabic language is spoken in many regions of the world, including north africa, asia, and the middle east. it is the official language of over 25 nations and one of the most widely used languages on the internet, with 164 million and 121 million internet users from the middle east and north africa, respectively. the expansion of the arabic language over the centuries formed widely dispersed groups, which in turn transformed the language through time and separated it into different dialects, which are a specialized form of the arabic language that is specific to a given region or social group, such as egyptian, jordanian, lebanese, and palestinian, etc. the closer the countries are geographically, the less variance between their dialects. furthermore, in formal situations such as the media and education, all arab nations use modern standard arabic (msa), but arabic dialects are used in informal everyday life communication. due to the intricacy of the language morphology and the scarcity of relevant datasets as the majority of the available datasets are data imbalanced, arabic research received little attention in its early phases, particularly in the arabic dialect identification task, because of the many challenges posed by the high similarity of dialects, especially in short phrases, as the same words are all commonly used in all dialects, in fact, the same word can have different meanings in the same dialect. however, it is a significant problem in many applications since being able to recognize the dialect effectively helps enhance specific applications and services, such as automatic speech recognition, remote access, ehealth, and e-learning. the majority of the research is focused on classifying the language into four regions: gulf, egyptian, maghrebi, and levantine, because it's less challenging than country-level dialect identification. recent studies, however, have concentrated on classifying the language into finergrained variants such as country-level dialects. in this paper, we present our approach to solving nuanced arabic dialect identification (nadi) shared task 2022 subtask-1 (country-level dialect identification) @xcite . our approach is divided into two main phases. the first step is in the prepossessing phase, where we removed the most frequent terms from all sentences across dialects to decrease the model confusion as the same word can appear in different dialects. the second step is in the modeling phase, where we employed a hybrid loss function technique combining weighted cross entropy loss and vs loss @xcite to overcome the imbalanced data problem. the rest of the paper is organized as follows: section 2 provides a review of previous arabic dialect identification literature, section 3 describes the proposed dataset, section 4 proposes the model of arabic dialect identification, in section 5 and section 6 we show the results of the proposed model and discuss the experiments of the different parameter settings and various loss functions. finally, we conclude in section 7. ## related work this section discusses previous research addressing arabic dialect identification challenges in the arabic language, the methodologies, strengths, and drawbacks. @xcite labeled the arabic online commentary dataset (aoc) through crowd-sourcing by collecting reader's comments from three arabic newspapers: al-ghad, al-youm al-sabe, and al-riyadh each of which represents one of the three dialects egyptian, gulf, and levantine. the final dataset is composed of 108,173k comments. they built a model to classify even more crawled data and achieved an accuracy of 82.45%. @xcite gathered dialectal arabic tweets and labeled them based on account descriptions. the resulting dataset comprises 540k tweets from 2,525 users spread over 18 arab nations. @xcite present a solution that won the 2020 nadi shared task (subtask 1.2) (abdul-mageed et al., 2020) by adapting to the task's unlabeled data (task-adaptive pretraining), then fine-tuning the dialect identification task using arabert on 10m unlabeled tweets. @xcite the solution that won subtask 2.2 employed a hierarchical classifier with a combination of tf-idf and arabert features to classify the country at the first level then at the second level to classify the province. @xcite the solution that won the 2021 nadi shared @xcite employed an ensemble learning model by fine-tuning the mar-bert model with adapters and vertical attention (vatt). embedding two additional layers at each transformer block at the marbert model allows for preserving the pre-trained embedded knowledge in the marbert layers. at nadi-2021, an end-to-end deep multi-task learning (mtl) approach (el mekki et al., 2021) was used to address both country-level and province-level identification. the mtl model combines the contextualized word embedding of marbert with two task-specific attention layers that extract task-discriminative characteristics. the results of this study show that most studies relied on the robustness of the model to solve the issue of high word similarity in different dialects rather than conducting additional research to address the issue. in order to reduce the likelihood of model confusion between the different dialects, this study aims to overcome past limitations by adopting a strategy to exclude the most common phrases from the tweets in the dataset. in the modeling phase, we also employed a hybrid loss function combining vs loss and weighted cross entropy loss to solve the issue of data imbalance, which is a common issue in most arabic datasets. ## dataset the proposed nadi-2022 dataset -country-level dialect identification (sub-task 1) (abdul-mageed et al., 2022) has 18 distinct dialects with a total of 20k tweets for training, 4871 tweets for development, and two test sets for testing, test-a with 18 dialects and 4758 tweets, and test-b with an unknown number of dialects and 1474 tweets. however, the dataset's distribution is significantly uneven and skewed (see figure 1 ), with egypt being the most common dialect with a total of 4283 tweets and sudan, qatar, and bahrain being the least common classes with a total of 215 tweets for each dialect. arabic dialect identification has two major challenges. first, there is a significant degree of similarity between dialects in short words; numerous short phrases are utilized in all dialects. second, there is an imbalance in data distribution. to overcome these issues, we eliminated the most frequently occurring phrases from the corpus and used a hybrid loss function composed of weighted cross-entropy and vs loss. ## system description this section will outline the methods we followed in developing our approach to solving subtask 1, starting with data pre-processing to the model's experiments with different loss functions. ## results in the two main steps of the suggested technique, we tested with various settings. the best macro f1-score is obtained by eliminating the most common terms during the pre-processing step and then feeding the pre-processed data into marbert-v2 using the suggested hybrid loss function. the proposed methodology achieved an f1 score of 39% on the development set, 35.68% on test set a, and 17.1924% on test set b, and the average macro f1 score between the two test sets is 26.44%. ## discussion as indicated in the ## conclusion this paper outlines our method for solving nuanced arabic dialect identification (nadi) shared task 2022 subtask-1 (country-level dialect identification). we eliminated the most frequent words from all tweets to reduce the likelihood that the model would become confused between the different dialects. additionally, we used marbertv2 with a hybrid loss function approach during the modeling phase to effectively address the class imbalance issue in our dataset. the findings demonstrated that our method outperforms a standalone loss function and tweets without removing the most common terms, with an f1 score of 35.68% on test set a and 17.1924% on test set b, and the average macro f1 score between the two test sets is 26.44%. to further improve the model performance, we aim to develop better methods to handle the removal process of the standard terms. also, collecting more data for the least common dialects may be significant in the performance.