NLP DI at NADI Shared Task Subtask-1: Sub-word Level Convolutional Neural Models and Pre-trained Binary Classifiers for Dialect Identification (2022)

Abstract: In this paper, we describe our systems submitted to the NADI Subtask 1: country-wise dialect classifications. We designed two types of solutions. The first type is convolutional neural network CNN) classifiers trained on subword segments of optimized lengths. The second type is fine-tuned classifiers with BERT-based language specific pre-trained models. To deal with the missing dialects in one of the test sets, we experimented with binary classifiers, analyzing the predicted probability distribution patterns and comparing them with the development set patterns. The better performing approach on the development set was fine-tuning language specific pre-trained model (best F-score 26.59%). On the test set, on the other hand, we obtained the best performance with the CNN model trained on subword tokens obtained with a Unigram model (the best F-score 26.12%). Re-training models on samples of training data simulating missing dialects gave the maximum performance on the test set version with a number of dialects lesser than the training set (F-score 16.44%)

## introduction arabic natural language processing (nlp) is traditionally faced with the problem of dialect identification. although arabic is spoken by a large community of about 400 million people, this community is distributed around different countries and extremely diverse in term of regional linguistic varieties, often called dialects. modern standard arabic (msa), which is the official language in many arabic speaking countries is highly formal language used in books and official communication, but newspapers and online writing already show considerable diversification, which is greatly increased in the spoken language of everyday communication. msa differs from regional varieties lexically, syntactically and phonetically @xcite . in the long history of arabic dialect identification (adi), multiple datasets have been developed. some of the most popular datasets include: the adi vardial dataset @xcite , which includes arabic text that is both speech transcribed and transliterated @xcite . arabic online commentary (aoc) is another dataset, which includes a large-scale repository of arabic dialects obtained from reader commentary of online arabic newspapers @xcite . multi arabic dialect applications and resources (madar) corpus constitutes parallel sentences written in different arabic city dialects from travel domain @xcite . classification methods tried out on these datasets range from feature-based machine learning approaches @xcite @xcite , n-gram based language models @xcite and ensemble models el @xcite to neural and pretrained models @xcite @xcite . in this paper, we describe the solutions submitted by our team to the nuanced arabic dialect identification (nadi) shared task 2022 @xcite ## data the subtask 1 of nadi 2022 provides training and development sets with 18 country dialects. the training set constitutes 20,398 instances and development set 4871 instances. in the evaluation phase, two test sets were provided, test-a with 4871 instances and test-b with 1474 instances. test-a had all the 18 dialects as in the training set, while test-b had k missing dialects, where @xmath0. ## models and methods we tried two kinds of solutions described in the following subsections. ## adaptation to unknown set of dialects (test-b) to deal with the missing dialects in test-b, we apply two additional techniques to the multi-dialect-arabic-bert model as the baseline. these techniques are described in the remainder of this subsection. ## experimental settings and results the results obtained on the development set are reported in in test-a (with all the 18 dialects), we used the four models: unigram_cnn, bpe_cnn, arabert and multi-dialect-arabic-bert. in test-b (with missing dialects), we submitted five models: unigram_cnn, bpe_cnn, multi-dialect-arabic-bert, multi-dialect-arabic-bert_ls (multi-dialect-arabic-bert with label smoothing with α = 0.1) and binary classifiers + multi-dialect-arabic-bert (binary classifiers with multi-dialect-arabic-bert). in binary classifiers + multi-dialect-arabic-bert, we use the binary classifier approach as discussed in section 4.2 for identifying the possible missing dialects and further retraining the model. it can be observed that the best performance on test-a was achieved with multi-dialect-arabic-bert model. on test-b, pretrained models work better with the best result achieved in the last setting (binary classifiers + multi-dialect-arabic-bert model). now, we discuss briefly the different outcomes on the two test sets. in both test sets, the best results are obtained by language specific pre-trained models. in test-b, all the scores are higher and the results with pretrained models are much better. we believe that this difference can be attributed to two factors. first, the smaller number of classes seems to make the task easier for all the models. second, our adaptation techniques are better suited to the setting with pretrained models. label smoothing helped in improving the performance slightly (multi-dialect-arabic-bert_ls) and binary classifiers with model retraining brings additional improvement. overall, based on the official results, we achieved a f-score of 21.28%. ## conclusion in this paper, we described and discussed two kinds of solutions for the nadi shared task, subtask 2: automatic country-wise identification of arabic dialects. among the solutions that we submitted, the language specific pre-traiend models gave the best performance in both test-a and test-b. label smoothing and simulating the missing dialect scenario with binary classifiers were our techniques for test-b with unknown set of labels. these techniques improve the performance compared to the baseline setting. in test-b, adaptation techniques enable better performance on this set, but there is still a lot of room for improving the perfor-mance. in future work, we aim to pursue the development of cnn architectures for fine-grained discrimination. we plan to investigate self-attention mechanisms with cnn and unsupervised deep embedding clustering.