ELYADATA at NADI 2024 shared task: A rabic Dialect Identification with Similarity-Induced Mono-to-Multi Label Transformation. (2024)

Abstract: This paper describes our submissions to the Multi-label Country-level Dialect Identification subtask of the NADI2024 shared task, organized during the second edition of the ArabicNLP conference. Our submission is based on the ensemble of fine-tuned BERT-based models, after implementing the Similarity-Induced Mono-to-Multi Label Transformation (SIMMT) on the input data. Our submission ranked first with a Macro-Average (MA) F1 score of 50.57%.

## introduction dialect identification (di) is the task of automatically determining the specific dialect (country) or regional variation that a text belongs to. in recent years, nadi 1 has been dedicated to provide diverse datasets and modeling opportunities to advance arabic natural language processing (nlp), including arabic dialects. nadi subtasks are not limited to di, it includes other nlp tasks such as sentiment analysis @xcite , etc... there are 3 subtasks for nadi's competition this year, we chose to participate in multi-label country-level dialect identification (mldid). this shared task started from 2020, and the di task has been consistent in all previous editions of nadi. it has shown a significant improvement over the past years. as an instance, in 2020, the best system for the di task, achieved a macro-average f1 score of 26.78% @xcite , while in 2023, the macro-average f1 score increased to 87.27% @xcite . however, this year represents an exception for the shared subtask, nadi extended the challenge to cover the task of mldid which is identifying and categorizing regional or social variations in a 1 https://nadi.dlnlp.ai/ language. when looking at today's research literature, as far as we know, there are no published systems on mldid for arabic dialects. when trying to solve such a task, multiple challenges rise: the main contributions of this work can be summarized as follows: ## data the provided nadi dataset is balanced and mono-labled accross 18 arabic dialects. each class consists of 1000 tweets where the average tweet length comprises 16 words. it is important to note that it is the same training set as the nadi-2023 dataset. nadi-2020 and nadi-2021 datasets were also provided by the shared task organizers. these datasets exhibit distinctions from the nadi-2023 dataset, characterized by uneven label distribution. notably, bahraini and qatari dialects were underrepresented compared to the predominantly represented saudi arabian and egyptian dialects. however, it should be noted that the tweets were classified based solely on their posting locations, which may not always be precise. the new addition in this year's competition is a dev set introduced for the multi-label classification @xcite task. this dev set comprises of 120 entries spread over 8 classes: algeria, egypt, jordan, palestine, sudan, syria, tunisia, and yemen. ## systems in this work, several approaches were experimented. some of them are data-centric, such as data preprocessing and simmt, while others are model-centric, including ensemble methods, binary relevance, and staged fine-tuning. but before delving into these approaches, the first step was to choose the model architecture to work with. bert-based models @xcite , which are masked language models (mlm) @xcite , seemed the most adequate. since the dataset is in da, language models that have already been trained on arabic were explored. ## experiments and results as already mentioned in section 3, bert-based models trained on arabic language were chosen as base models for our experiments. several pretrained models were tested, mainly: camelbert @xcite , marbert, arbert, mar-bertv2 (abdul-mageed et al., 2020), and their variations. to ensure equality amongst experimented systems, the same hyper-parameters in ## discussion after analyzing the results and exploring the dev set further, problems with wrongly labeled phrases were found. some labels were mistakenly assigned to many instances, creating inconsistencies in the dataset. for example, one sentence labeled as palestine " " (please i want one cup of tea), reflects a linguistic nuance, using " " (i want), which is more a characteristic of gulf arabic rather than palestinian arabic. another challenge faced was determining labels for transliterated sentences, like " " (i say let's face to face after this), which can be confusing even for native speakers to classify. mistakes in labeling do not just add noise but also make it hard to judge how well a model performs accurately. it is crucial to refine data anno-tation procedures and implement rigorous quality checks to mitigate the impact of mislabeled data on model training and evaluation. ## conclusion this paper presents our team's submission to the multi-label country-level dialect identification of the 2024 nadi shared task. our submission relies on the usage of similarity-induced mono-to-multi label transformations a data-centric approach and max probabilities ensemble on fine-tuned models which is a model-centric approach. this allowed us to be ranked first in the official evaluation with a macro-average f1 score of 50.57%.