dz NLP at NADI 2024 Shared Task: Multi-Classifier Ensemble with Weighted Voting and TF - IDF Features (2024)

Abstract: This paper presents the contribution of our dzNLP team to the NADI 2024 shared task, specifically in Subtask 1 - Multi-label Country-level Dialect Identification (MLDID) (Closed Track). We explored various configurations to address the challenge: in Experiment 1, we utilized a union of n-gram analyzers (word, character, character with word boundaries) with different n-gram values; in Experiment 2, we combined a weighted union of Term Frequency-Inverse Document Frequency (TF-IDF) features with various weights; and in Experiment 3, we implemented a weighted major voting scheme using three classifiers: Linear Support Vector Classifier (LSVC), Random Forest (RF), and K-Nearest Neighbors (KNN).Our approach, despite its simplicity and reliance on traditional machine learning techniques, demonstrated competitive performance in terms of accuracy and precision. Notably, we achieved the highest precision score of 63.22% among the participating teams. However, our overall F1 score was approximately 21%, significantly impacted by a low recall rate of 12.87%. This indicates that while our models were highly precise, they struggled to recall a broad range of dialect labels, highlighting a critical area for improvement in handling diverse dialectal variations.

## introduction arabic, with its rich linguistic diversity encompassing numerous dialects alongside modern standard arabic (msa), presents a significant challenge for natural language processing (nlp) tasks. despite its importance, many arabic dialects remain understudied due to resource constraints such as limited research funding and datasets. addressing this gap, the nuanced arabic dialect identification (nadi) shared task series @xcite @xcite serves as a pivotal initiative to overcome these hurdles. by providing extensive datasets and modeling opportunities, nadi aims to facilitate dialect identification and processing tasks, thereby advancing the understanding and utilization of arabic dialects. in the context of feature extraction for arabic text analysis, standard approaches frequently employ term frequency-inverse document frequency (tf-idf). our previous work @xcite in the madar'2019 shared task @xcite demonstrated the effectiveness of using tf-idf for efficient feature extraction in arabic text classification. drawing inspiration from this prior research, we adopted a similar strategy in our current study. specifically, in our first experiment, we employed a union of tf-idf features with various n-gram analyzers (word, character, and character with word boundaries) to enhance the representation of arabic text. moreover, the utilization of weighted feature fusion has emerged as a promising technique for enhancing classification accuracy. @xcite demonstrated the efficacy of weighted feature fusion in multilingual text classification, as showcased in our earlier study in nadi'2023 for arabic dialect identification @xcite , emphasizing its adaptability across various linguistic datasets. this methodology closely corresponds to our experiment 2, which focuses on the weighted concatenation of tf-idf features. these earlier studies, along with recent advancements in tf-idf-based feature extraction and weighted feature fusion, have collectively inspired innovative ideas. in our third experiment, we implemented a weighted hard major voting approach. this paper, instead of introducing innovative solutions or groundbreaking insights for nadi 2024 @xcite , serves as a concise consolidation of existing knowledge. the rest of the paper is structured as follows: section 2 provides an overview of the dataset used in our study. in section 3, we present our proposed system, followed by discussing the findings and their significance in section 4. finally, our paper concludes in section 5, summarizing the key takeaways and contributions. ## description of the dataset the nuanced arabic dialect identification (nadi) 2024 shared task provided a valuable dataset for subtask 1: multi-label country-level dialect identification (mldid). the dataset encompassed dialects from various arabic-speaking countries, including egyptian, saudi arabian, algerian, syrian, palestinian, and lebanese. the dataset provided for mldid consisted of two parts: a development set, comprising 100 samples, served as a training ground for system development and evaluation, and a test set, containing 1,000 samples, was used for the final evaluation of participant models. additionally, participants were granted access to nadi datasets from 2020, 2021, and 2023 for training purposes. these datasets provided a larger pool of arabic text for system development and refinement. ## proposed system our proposed system is based on two concatenation or union process of features vs classifiers where we opted for three experiments: in this instance, we combine the three tf-idf features using a weight vector comprising three distinct values (w1, w2, w3) corresponding to the word, char, and char_wb analyzer, respectively. the lsvc classifier was then trained. ## obtained results during our participation in the mldid (closed track) project, we iteratively explored various approaches to enhance our multi-label dialect identification models (see as a baseline experiment, we utilized a basic tf-idf representation with 1-gram features and a linear kernel svc classifier. despite its simplicity, it achieved a low f1-score of 19.43%. in the first experiment, we expanded the feature space by incorporating a combination of word and character-level n-grams (word, char and char_wb) in tf-idf representation. we employed the linear support vector classifier (lsvc) classifier with balanced class weights, resulting in notable improvements in f1-score of 20.64%. in the second experiment, we experimented with a different combination of n-grams while maintaining the balanced class weights. we also explored variations in the weights assigned to different feature types within the tf-idf representation. the obtained f1-score ranged from 20.51% to 22.51% (see in the third experiment, we introduced ensemble methods, namely hard voting and weighted hard voting, combining svc, rf, and knn classifiers. interestingly, despite the complexity of these ensemble approaches, their f1-score were lower compared to well-tuned single lsvc configurations (16.33% and 21.44%, respectively). this suggests that the ensemble methods may not have effectively leveraged the complementary strengths of the individual classifiers or their voting strategies might not have been optimal for this task. ## conclusion our analysis of the nadi 2024 shared task highlights the critical role of feature engineering and model optimization in arabic dialect identification. key findings include integrating characterlevel features alongside word-level features consistently improved performance, balanced class weights within the lsvc classifier significantly enhanced f1-score, surpassing 20.64%, strategic assignment of transformer weights yielded the highest f1-score of 22.51%, and ensemble methods like hard voting achieved moderate f1-score but were surpassed by finely tuned single svc configurations. in conclusion, our study emphasizes the importance of incorporating character-level information, utilizing balanced class weights, and exploring advanced feature weighting techniques to advance arabic dialect identification systems. these insights provide valuable guidance for future research in this domain.