Optimizing Naive B ayes for A rabic Dialect Identification (2022)

Abstract: This article describes the language identification system used by the SUKI team in the 2022 Nuanced Arabic Dialect Identification (NADI) shared task. In addition to the system description, we give some details of the dialect identification experiments we conducted while preparing our submissions. In the end, we submitted only one official run. We used a Naive Bayes-based language identifier with character n-grams from one to four, of which we implemented a new version, which automatically optimizes its parameters. We also experimented with clustering the training data according to different topics. With the macro F1 score of 0.1963 on test set A and 0.1058 on test set B, we achieved the 18th position out of the 19 competing teams.

## introduction this paper describes the system used by the suki team at the nuanced arabic dialect identification (nadi) shared task 2022 @xcite . the task was the third in a series of language identification shared tasks focusing on arabic languages @xcite . in 2020, the first subtask of country-level classification was won by @xcite using multi-dialect arabic bert model @xcite and the second subtask of province-level classification by el @xcite using an ensemble of a bert-based and a stochastic gradient descent (sgd) based @xcite identifiers. the various subtasks of the 2021 edition were won by @xcite using marbertbased systems @xcite . a recent literature review of language identification for dialectal arabic was conducted by @xcite and a more general survey of language identification techniques by @xcite . deep learning, specifically bert-based, systems dominated the two previous nadi shared tasks. as the suki team, we have participated in various language identification (li) related shared tasks throughout the years with our shallow heli or naive bayes-based systems. in 2016, we participated in the arabic dialect sub-task of the 3rd edition of the discriminating between similar languages (dsl) shared task, which featured four arabic dialects in addition to modern standard arabic (msa) @xcite . using the heli li method, we arrived at the seventh position, which was poor in contrast to the shared first place we reached in the first sub-task of dsl that year. the experiments described in this paper are the first time we have returned to the identification of various arabic languages after that. in these experiments, we use a naive bayes (nb) based identifier instead of one based on the heli method. we implemented it and used it as a baseline in the 2019 cuneiform language identification (cli) shared task @xcite . during the same year, we adapted our language model adaptation scheme @xcite to work with the nb implementation and won one of the two tracks in the discriminating between the mainland and taiwan variation of mandarin chinese (dmt, @xcite ) shared task @xcite . more recently, we also won the romanian dialect identification (rdi, chakravarthi et al. ( 2021 )) 2021 @xcite and the identification of languages and dialects of italy (itdi, @xcite ) 2022 @xcite shared tasks using the adaptive version of the nb identifier. for the nadi shared task, we set out to find out whether our current nb implementation would be more competitive when distinguishing between close arabic languages than our heli-based identifier in 2016. additionally, we were trying to develop a way to use unlabeled data to improve the identifier results. the experiments to utilize unlabeled data were inconclusive and did not improve the identification results on the development set, so we did not end up using them in the one run we submitted. also, as the language identification accuracy was already relatively low, using language model adaptation did not prove advantageous with the development data. thus we submitted our only run using the non-adaptive nb identifier. ## shared task evaluation setting the third nadi shared task foot_0 featured 18 countrylevel dialects of arabic. the official ranking metric was the macro-averaged f1 score. the shared task participants were given separate training and development sets consisting of tweets labeled with their respective country-level dialects. the training set was the same as in the nadi 2021 shared task @xcite . according to the shared task instructions, the provided development set was not to be used as training data for the identifier used for the test data. the set sizes are seen in the participants were also given the tweet ids of 10 million additional unlabeled arabic tweets that could be used in training and developing the language identification system. the organizers provided a python script that could be used to download the corresponding tweets using a twitter api and their credentials. currently, twitter allows academic users to download 10 million monthly tweets for research purposes. due to the twitter service being repeatedly over capacity and terminating the connection, the download had to be made in 16 parts, which took almost a week. of the 9,999,998 downloaded tweets, 2,005,682 were tagged @xmath0. the participants were expected to provide results on two test sets; test set a featuring new unseen tweets for each of the 18 dialects and test set b featuring tweets from a subset of unknown size from the 18 languages. we only used the nadi-labeled training and development sets for the submitted run. we did not use the development set for training the final identifier; we used it only to determine the method's optimal parameters. ## system the system uses a naive bayes-based method using the observed relative frequencies of multiple-size character n-grams as probabilities. as described by @xcite , the naive bayes type method adds together logarithms of the relative frequencies of character n-gram combinations f i in the training data c g as defined in equation 1 the exact range of the used character n-grams is optimized using the development data. in previous versions of the identifier, we have semi-manually identified the optimal character n-gram ranges and the penalty modifier. however, on this occasion, we decided to implement an automatic optimizer to streamline experimentation. the automatic optimizer is first given initial character n-gram and penalty modifier ranges which it then uses to populate a todo-table. the parameters in the todo-table are evaluated, and the results are stored in a master results list. an additional top ten list of macro f1 scores is created with the parameters used to obtain them. the parameter instances used in the top ten list are checked, and nearby parameter combinations are added to a new todo-table if they are not found in the master results list. in the case of ngram ranges, the optimizer tries one higher and one lower for both the minimum and maximum n-gram sizes. for the penalty modifier, it adds and subtracts 0.5 from the current one if there are no other penalty modifiers for the respective n-gram range in the master results list. if a "neighboring" penalty modifier exists in the results list, the halfway between the penalty modifiers is tried if the distance between modifiers is larger than 0.1. the cycle of evaluating the todo-table, creating a top ten list, and creating a new todo-table is continued as long as the top ten list changes between cycles. an ex ## experiments manually optimizing the parameters for the nb system, we arrived at the macro f1 score of 0.2046 with n-grams from two to four and the penalty modifier of 1.40. after this, we did some experiments with language model adaptation using the same parameters, but adding more splits to adaptation worsened the results, as seen in clustering experiments dividing languages into topic-or dialect-based clusters has proven fruitful in our earlier experiments @xcite . we expected the training data to contain tweets on many different topics and hypothesized that dividing the training data into several clusters might be advantageous. each dialect would then be divided into several language models based on these clusters. we created a custom clustering software based on the naive bayes identifier. it chose a random tweet among all the tweets and created language models from it. then every other tweet was scored using those language models, and the one furthest from the original tweet was selected. additional language models were also created from the second tweet, and then again, all the tweets were identified using both models. if the model claimed only one tweet, e.g., itself, the model was dropped out of the repertoire as an outlier. then the tweet being as far as possible from both models was selected as the material for the third model. and again, all the tweets were re-scored, one chosen for new models, and so on until none of the models claimed more than half of all the tweets (max 10k tweets). this resulted in 214 clusters for all the dialects, as seen in ## results we ended up submitting only one run on each of the test sets using the non-adaptive version of the language identifier. first, we treated both the training and the test data with the farasa segmenter and then ran them through the naive bayes language identifier using character n-grams from one to four with a penalty modifier of 1.375. with the macro f1 score of 0.1963 on test set a and 0.1058 on test set b, our submissions reached the 19/19 and 15/19 positions for the respective test sets. the final ranking for the whole shared task combined the results of the two test sets. we were ranked 18th out of the 19 participating teams, which shows that our results were not competitive against most other submitted results. as of this writing, we have not received the gold-standard labels for the test set. ## discussion there are still several avenues worth exploring when using the nb-based identifier in classifying arabic tweets. we intend to continue exploring different kinds of topic clustering methods to divide the training data into different models. currently, we have no efficient means to utilize additional unannotated data, and developing such means remains a high priority. ## conclusion we have presented the experiments we conducted when participating in the nadi 2022 shared task. many of the experiments provided interesting results for further research. we were successful in implementing a new version of the nb identifier, which automatically optimizes its parameters, thus leaving more time to explore ideas to improve the identification accuracy. we reached the 19th and 15th places in the shared task.