A rabic Dialect Identification Using BERT Fine-Tuning (2020)

Abstract: In the last few years, deep learning has proved to be a very effective paradigm to discover patterns in large data sets. Unfortunately, deep learning training on small data sets is not the best option because most of the time traditional machine learning algorithms could get better scores. Now, we can train the neural network on a large data set then fine-tune on a smaller data set using the transfer learning technique. In this paper, we present our system for NADI shared Task: Country-level Dialect Identification, Our system is based on fine-tuning of BERT and it achieves 22.85 F1-score on Test Set and our rank is 5th out of 18 teams.

## introduction arabic is a very complex language, a fast-growing one with more than 400 million speakers around the world. online social media platforms (twitter in our case) are a very good place where arabic speakers can communicate, express, and share their thoughts. it will be very useful if we can study people's behavior so we can help them in various ways, and dialect identification is the first step to identify a given country to study the behavior of its people. text classification is considered an easy task, not because it is simple in general, but it is considered one of the oldest research areas in natural language processing (nlp). if we have two classes separated with obvious features, it is a fairly simple process, but in our task, we have 21 classes with a lot of common features so it gets a lot more challenging. we will discuss that in the data section. we have found that the madar arabic dialect corpus and lexicon @xcite is a data set quite similar to the one provided in the task but instead of 21 labels for countries, there are 25 city labels. top-ranked systems proposed by @xcite on the madar data set were built on traditional machine learning algorithms. in our first trial we have used n-gram tf-idf characters and words features for train linear and nonlinear support vector machine, bernoulli naive bayes followed by ensembling all of the three aforementioned classifiers, we reached a score of 18.1 on the development set of nadi shared task @xcite . our final model was built by fine-tuning a pretrained bidirectional encoder representations from transformers (bert) @xcite . the original trained model provided from @xcite and trained on around 95 gigabytes of arabic text from different resources, all the parameters of training will be described in the system section. ## data the data set was created for this shared task from organizers. the provided training data set consists of 21,000 tweets with 21 different labels representing 21 (egypt, bahrain, iraq, saudi arabia, algeria, oman, syria, united arab emirates, libya, morocco, yemen, tunisia, lebanon, jordan, kuwait, palestine, qatar, mauritania, djibouti, somalia, sudan). for reporting results, there are 4,957 labeled tweets for development, 5,000 for testing, and an extra 10 million tweets for other purposes. we split training data into 19,950 tweets for training and 1,050 for validation so we can make use of most of the given data, for testing we have used development test. this work is licensed under a creative commons attribution 4.0 international license. license details: http: //creativecommons.org/licenses/by/4.0/ . a simple pre-processing function was used to remove links, punctuation, and repeated characters. we also tried removing directs but the performance was slightly better without removing them. looking into the data set distribution in figure 1 , we can notice that the data is totally imbalanced. for example, egypt's tweets represent 21.3% of the data but bahrain represents 0.01% of the data set. what makes the task even more challenging is the similarity between classes as the gulf countries (bahrain, kuwait, iraq, oman, qatar, saudi arabia, and the united arab emirates) have a very similar dialect. it is already hard for humans to differentiate between them and this data represents 37% of the data set. 3 systems in this section, we are going to introduce two systems with totally different approaches. the bert system, which ranked as 5th in the leader board, and another system using feature extraction and ensemble of different algorithms. ## results and discussion for the task, we have built several systems to achieve the models described above. a comparison between the results described in ## conclusion this paper described two methods applied to nadi shared sub-task 1 to predict an arabic dialect from tweets. our experiments show to show that the bert model achieved the best f1 score betting other machine learning models. using the unlabeled tweets to fine-tuning the language model is a great way to increase the performance but needs a lot of time and computation power. we had achieved a relatively high f1 score of 24.05 on the development set and 22.85 on the test set.