NLP _ DI at NADI 2024 shared task: Multi-label A rabic Dialect Classifications with an Unsupervised Cross-Encoder (2024)

Abstract: We report the approaches submitted to the NADI 2024 Subtask 1: Multi-label country-level Dialect Identification (MLDID). The core part was to adapt the information from multi-class data for a multi-label dialect classification task. We experimented with supervised and unsupervised strategies to tackle the task in this challenging setting. Under the supervised setup, we used the model trained using NADI 2023 data and devised approaches to convert the multi-class predictions to multi-label by using information from the confusion matrix or using calibrated probabilities. Under unsupervised settings, we used the Arabic-based sentence encoders and multilingual cross-encoders to retrieve similar samples from the training set, considering each test input as a query. The associated labels are then assigned to the input query. We also tried different variations, such as co-occurring dialects derived from the provided development set. We obtained the best validation performance of 48.5% F-score using one of the variations with an unsupervised approach and the same approach yielded the best test result of 43.27% (Ranked 2).

## introduction arabic is a language spoken by a large community of about 400 million people, which is widely distributed around different countries and regions. modern standard arabic (msa), the official language in many arabic-speaking countries, differs from the regional varieties lexically, syntactically, and phonetically @xcite . arabic dialect identification (adi) deals with identifying the dialect of a given arabic input utterance. some of the most popular datasets in adi include: the adi vardial dataset @xcite , which includes arabic text that is both speech transcribed and transliterated @xcite , arabic online commentary (aoc), which includes a large-scale repository of arabic dialects obtained from reader commentary of online arabic newspapers @xcite , multi arabic dialect applications and resources (madar) corpus constitutes parallel sentences written in different arabic city dialects from travel domain @xcite etc. the nadi shared task started in 2020 and presented continued efforts in the adi, including country-wise, province-wise, and region-wise dialect identification tasks @xcite @xcite . the main limitation of these aforementioned datasets is that they are monolabeled, which means they are multi-class, where each input sample belongs exactly to one of the classes/dialects. the 2024 nadi adi task focuses on multi-label dialect identifications, meaning an input sample can belong to more than one dialect class. in general, the approaches for dialect classifications ranged from n-gram and machine learning approaches @xcite @xcite @xcite to ensembles el @xcite and pre-trained neural models @xcite @xcite . in this paper, we describe our solutions submitted to the nadi shared task 2024 (abdul-mageed et al., 2024), subtask-1, which targets the challenging task of multi-label country-level dialect identification (mldid). the unavailability of multi-label training data poses a major roadblock in this task. hence, the main challenge is to adapt the available multi-class training data for multi-label predictions. this paper is organized as follows: the data statistics are described in section 2, methods used are discussed in section 3, experimental results are reported in section 4, followed by conclusions in section 5. 2 data subtask 1 of nadi 2024 provided a development set with 8 country dialects constituting 100 samples. the samples were multi-labeled, i.e., each sample can belong to more than one country dialect. the label distribution is shown in figure 1 , which depicts that we have 13 samples that did not belong to any of the 8 dialects and about 12 samples belonging to all of the 8 dialects. most of the samples were belonging to at least two classes. the test set included 1000 samples, which are supposed to be multi-dialectal, while the number of dialects was kept unknown, with a maximum number of dialects specified as 18. hence, the number of multi-label classes,l, for a single instance can be @xmath0. ## models and methods this section describes the techniques used for nadi-2024 mldid subtask 1. the task at hand considers an input utterance that can belong to multiple dialect classes, which is inherently challenging. the challenge becomes multi-faceted by factors such as the unavailability of multi-label training data and the unknown number of classes in test data. the training data from previous nadi dialect identification tasks is multi-class. hence, we must devise approaches to utilize the patterns or label information in these data to provide a multi-label classification for the test data. further, the development set provided had only 8 dialects (multilabeled), while in the test set, the number of dialects is unknown, with a maximum number of dialects specified as 18. this restricts us from using any label-specific statistical observational patterns derived from the development set to be adapted directly to the test set. under these challenging settings, we experimented with supervised and unsupervised approaches to derive multi-label predictions on the test set. for the final submission, we used the approaches that provided the best scores on the dev set. ## experimental settings and results for implementations, we used the models and libraries from huggingface and sentence trans-to find the overlaps. we also tried the simple ir algorithm, bm25 lexical retrieval model, m25 ir model, a ranking algorithm that determines document relevance to a query and ranks them based on a relevance score @xcite , for comparisons. it can be observed that the best performing approach is with the unsupervised set-up with post-filtering, giving an f-score of 48.45%. similar performance was obtained with a combination of co-occurrence based label enhancements. it can also be noted that the post-processing approaches facilitated performance improvement. we observed that our approaches provided better precision while recall was less. the best-performing approach presented a recall of 41.39%, with a precision of 63.72%. based on these results, we used the best models to evaluate the test data. the results are shown in a better recall could be obtained by adjusting the post-processing parameters such as filtering n values. however, this can decrease precision, and hence, we need to explore strategies to achieve a better recall-precision trade-off. as expected, the unsupervised approaches outperformed the supervised settings since the supervision was based on a dataset under multi-class settings. considering the challenging setting, the proposed approach performed reasonably well. ## conclusion in this paper, we described our experimental approaches for the nadi 2024 subtask 1, which deals with multi-label dialect identifications. the absence of multi-label datasets poses a major challenge in this setup. the idea is to use multi-class datasets and adapt them for multi-label predictions. the complexity becomes multi-folded since the number of dialects in the test set was kept unknown. we use a cross-encoder-based approach in the unsupervised set-up with heuristics based on label count thresholds. further, filtering and label enhancements were applied in the post-processing step. in the supervised setup, we used the trained classifiers for label predictions and then applied label enhancements or used binary classifiers for predictions. we observed that the cross-encoderbased approach with post-processing performed the best. a generalized conclusion of the best approach to mldid cannot be made without sufficient properly annotated data. hence, continued efforts in this direction must be made while experimenting with suitable multi-class adaptation strategies.