Mavericks at NADI 2023 Shared Task: Unravelling Regional Nuances through Dialect Identification using Transformer-based Approach (2023)

Abstract: In this paper, we present our approach for the “Nuanced Arabic Dialect Identification (NADI) Shared Task 2023”. We highlight our methodology for subtask 1 which deals with country-level dialect identification. Recognizing dialects plays an instrumental role in enhancing the performance of various downstream NLP tasks such as speech recognition and translation. The task uses the Twitter dataset (TWT-2023) that encompasses 18 dialects for the multi-class classification problem. Numerous transformer-based models, pre-trained on Arabic language, are employed for identifying country-level dialects. We fine-tune these state-of-the-art models on the provided dataset. Ensembling method is leveraged to yield improved performance of the system. We achieved an F1-score of 76.65 (11th rank on leaderboard) on the test dataset.

## introduction dialects, which are variations of a language, often differ in their vocabulary, grammar, pronunciation, and occasionally even cultural quirks. the practice of identifying the particular dialect or regional variety of a language that is used in a text or speech sample is known as dialect identification. the goal of dialect identification is to categorize a text or speech into one of the many dialects or regional adaptations that may exist. for many nlp applications, including language modeling, speech recognition, and data retrieval, this task may be vital. arabic, with its plethora of dialects, is a rich language. however, many of these dialects are not studied in depth because of a dearth of monetary backing and available datasets. arabic dialect identification can assist in perpetuating linguistic diversity by acknowledging and valuing various dialects. it contributes to addressing the gap between * equal contribution existing nlp techniques and the rich fabric of regional dialectal differences in a globalized setting. rule-based strategies for arabic dialect identification have given way to data-driven techniques, with a focus on machine learning, deep learning, and the creation of corpora of languages and datasets. the accuracy of dialect detection has risen significantly with the use of multilingual pre-trained models such as bert and its derivatives. this paper presents our approach for subtask 1: country-level dialect identification, which poses a multiclass classification problem @xcite . multiclass classification is a form of statistical modeling or machine learning problem where the objective is to classify data into more than two unique classes or labels. we aim to classify the tweets and map them into their respective dialect labels. we have demonstrated the use of various transformer-based models on the given arabic data. the ensembling method has been leveraged to enhance the performance of the proposed system. ## related work dialect detection in arabic is an arduous task due to several factors, including the lack of a consistent spelling system, the medium's characteristics, and the scarcity of data. surveys on deep learning and natural language processing methods for processing arabic data were presented in 2015 @xcite and 2017 @xcite , focusing on the identification of arabic dialects. however, only 6 arabic dialect classes had been examined until that time. the madar project was launched in 2018 to provide a large corpus of 25 arabic city dialects @xcite . a study on the classification of dialects in 25 arab cities used multi-label classification methods and examined a wide range of features, yielding promising results @xcite . employing supervised machine learning methods on arabic nlp tasks was found to be a difficult feat because of the lack of resources in the arabic language ( @xcite . as a result, scholars and researchers have introduced plenty of initiatives to make new datasets available and encourage more people to work in the field of arabic nlp. one of the initiatives, nuanced arabic dialect identification (nadi) shared tasks, was started in 2020 which comprised country-level and province-level dialect detection @xcite . bert (bidirectional encoder representation from transformers) @xcite models have been commonly used for these dialect detection tasks. a multilingual bert model was pretrained on unlabeled tweets and fine-tuned for the classification task by @xcite . arabert was finetuned on an additional dataset produced by reverse translating the nadi dataset and employed for the dialect detection task by @xcite . furthermore, @xcite utilized ensembling methods and semi-supervised methods along with arabic-bert. a system comprising an ensembling of multiple models was created using marbert as the base model, which yielded promising results @xcite . in the nadi 2022 shared task, alshenaifi and azmi (2022) pretrained arabert model and bilstm model for dialect detection. various models were combined and performance was enhanced using a combination of tf-idf and n-grams. an ensembling of transformerbased models, predominantly using variations of marbert was employed for dialect detection as well as sentiment analysis, in the nadi 2022 shared tasks @xcite , @xcite . @xcite addressed the issue regarding an imbalance in the classes of the nadi dataset by using focal loss and employed various arabic bert-based models. this paper proposes a system that employs an ensemble of transformer-based models, specifically variations of bert for the classification task. ## system the given subtask tackles the problem of countrylevel dialect identification. this comes under the umbrella of multi-class classification problems for which language models have been extensively used and have achieved impressive results. the models are trained for 10 epochs with a learning rate of 1e-5, a batch size of 32, and the adamw optimizer. we experiment and use several language models and ensembling methods in our research, as shown in figure 1 . ## ensembling ensembling is a technique that integrates the output of multiple models to get the system's eventual outcome. for this, both statistical and non-statistical methods are employed. ensembling is beneficial since it contributes to the production of results that are superior to those provided by the individual models. we note that the "hard voting" ensemble strategy emerges as the most effective and precise among the many strategies used for ensembling. in hard voting, the final prediction is chosen based on the majority vote or the "mode" of all the predictions. it reduces the volatility in the outcomes and aids in strengthening the system's robustness. ## results this section discusses the results obtained by our system and analyses its performance. ## conclusion this paper compares several transformer-based models on the task of nuanced arabic dialect identification (nadi). arabertv02-twitter-base is found to outperform other models for this task. it achieves an f 1 score of 76.65. we use hard voting-based ensembling as the final approach for our system as it generates predictions that are stable while also improving the overall performance. with higher computational resources at hand, the performance of the system can be improved by training it for longer and by using bigger models for the system. models that are specifically pretrained on data that is similar to the data used in the task at hand can help enhance understanding and in turn, give better performance. we can also experiment with other suitable ensembling methods and gauge their efficiency for our task.