TF - IDF or Transformers for A rabic Dialect Identification? ITFLOWS participation in the NADI 2022 Shared Task (2022)

Abstract: This study targets the shared task of Nuanced Arabic Dialect Identification (NADI) organized with the Workshop on Arabic Natural Language Processing (WANLP). It further focuses on Subtask 1 on the identification of the Arabic dialects at the country level. More specifically, it studies the impact of a traditional approach such as TF-IDF and then moves on to study the impact of advanced deep learning based methods. These methods include fully fine-tuning MARBERT as well as adapter based fine-tuning of MARBERT with and without performing data augmentation. The evaluation shows that the traditional approach based on TF-IDF scores the best in terms of accuracy on TEST-A dataset, while, the fine-tuned MARBERT with adapter on augmented data scores the second on Macro F1-score on the TEST-B dataset. This led to the proposed system being ranked second on the shared task on average.

## introduction arabic is a semitic language spoken in more than 26 countries by more than 350 million people with at least 30 dialects foot_0 . some previous studies attempted to use hierarchical deep learning for a finegrained dialect classification @xcite . arabic has its own, letter based writing system which is used mostly for only those consonants which could denote a wide range of pronunciation alternatives. a single letter in this alphabet can have various forms depending on the context, and its position within the word which are encoded by different characters. there are also single character ligatures which are formed by two or more characters (e.g., from this corpus: u+fefb ("al") denotes u+0627 (a) and (u+0644 (l), or words like u+fd71 ("aspired") or u+fdf2 ("allah") are also represented by a single character). simi-larly to the latin alphabet, arabic letters can denote, e.g., urdu, ottoman turkish, sindhi, malay, uyghur, or even english and french words which are not uncommon. this article targets the nuanced arabic dialect identification (nadi) 2022 shared task @xcite . it more specifically focuses on subtask 1 aimed at identifying country-level dialects by providing ∼20k twitter data which are labeled by geo-location (i.e. country) from where the tweets were posted. in this shared task no external labeled data sources were allowed to be used, however, a large unlabeled dataset was also provided. the training set remains relatively small to encourage competitors to use few or zero-shot learning models. solutions were tested on two datasets using macro-averaged f1-score: according to the systems developed and presented in this work, dialect identification can be modeled at the character, word, expression, or phrase level. each of these levels was modeled by the traditional tf-idf method, a pre-trained transformer called marbert @xcite , and by using marbert with word level augmentation. these methods were analyzed individually as well as by using their combination in order to find the most informative parameter regarding the dialects. on test-a dataset the traditional approach produced the best accuracy, while on test-b dataset our approach won the runner-up award on macro-averaged f1-score. ## data the nadi 2022 shared task subtask 1 dataset contained a total of 20,398 tweets in the training set, 4,872 validation samples from 18 dialects, while unlabeled test sets, test-a and test-b contained 4,758 and 1,473 tweets respectively. dialects are identified based on geolocation data where the tweets were originated instead of the linguistic analysis. this in itself leads to the contamination of the data since people might reside in a country other than their country of origin. moreover, sometimes the words are used in or borrowed from other languages, e.g., english or urdu. additionally, there is an imbalance in the class distribution in the training dataset (see ## system three models were proposed for this subtask out of which the first model was a traditional approach without using any language models or deep neural network architecture, i.e., tf-idf based. in the second approach, the data augmentation was performed with fine-tuned mar-bert @xcite with and without adapters @xcite . ## results as shown in on the test-b dataset, where the number of country-level dialects is unknown, fine-tuned-adapter-marbert (aug) performs the best in every metric. however, the performance difference among transformer-based approaches with or without augmented data tested on either test-a or test-b dataset is not as noticeable as the difference between the traditional approach and the transformer-based approaches tested on test-b. this indicates the superiority of zero-shot transfer of the pre-trained transformer. ## discussion results show noticeably high variance in precision and overall accuracy between development and test data sets, regardless of which submitted model one cross-references. under-sampling could explain that because in small samples words can either be interpreted as dialectal use of another, more common concept or simply another topic, stance, or key communication element which focuses the attention. in both cases, the word embedding, and tf-idf could see clear alternatives for the same concept which is the basis of the classification. moreover, the traditional approach suffers significantly less in accuracy in comparison with transformers-based models, and its performance is relatively consistent between the two differently sampled test sets. while tf-idf models have no background knowledge of the language, there is no pretraining available, it still can outperform transformers in terms of accuracy, especially for dialects with large samples. in that sense, the tf-idf approach is more stable, and therefore its power for generalization is stronger which means it can grab some important features of dialectal arabic. there is a strong indication to improve or to create a subword based, or a transliteration and sub-word based transformer for arabic. further analysis of predictions made be the best performing model show an expected over prediction of dialects with higher presence within the training data figure 1 . the over prediction showed a tendency towards dialects that are more similar. for instance, uae was predicted more as oman or ksa rather than egypt. on the other hand, coun-tries with small presence such as qatar and bahrain had no correct predictions on the dev set. ## conclusion this paper targets the problem of arabic dialect detection based on a traditional approach as well as the pre-trained transformers in a dataset where few-shot learning was encouraged, and no large training set was provided. while the tf-idf based approach performs less than the pre-trained transformer based approach on the nadi 2022 corpus which was expected, the accuracy of the tf-idf approach surprisingly remained competitive on the whole (test-a) test set. tf-idf obviously underperforms as compared to the marbert-based approach for low sampled dialects due to a lack of enough data for stable fingerprinting which explains test-b results. since tf-idf and mar-bert target different levels of the written language, so the most reasonable explanation is that dialect is more likely determined at the sub-word level. this hypothesis, however, needs further investigation.