QADI : A rabic Dialect Identification in the Wild (2021)

Abstract: Proper dialect identification is important for a variety of Arabic NLP applications. In this paper, we present a method for rapidly constructing a tweet dataset containing a wide range of country-level Arabic dialects —covering 18 different countries in the Middle East and North Africa region. Our method relies on applying multiple filters to identify users who belong to different countries based on their account descriptions and to eliminate tweets that either write mainly in Modern Standard Arabic or mostly use vulgar language. The resultant dataset contains 540k tweets from 2,525 users who are evenly distributed across 18 Arab countries. Using intrinsic evaluation, we show that the labels of a set of randomly selected tweets are 91.5% accurate. For extrinsic evaluation, we are able to build effective country level dialect identification on tweets with a macro-averaged F1-score of 60.6% across 18 classes.

## introduction twitter is one of the most popular social media platforms in the middle east and north africa (mena) region with almost two thirds (63%) of arab youth indicating that they look first to facebook and twitter for news @xcite . the popularity of twitter in mena is reflected by approximately 164 million active monthly users, who produce a massive volume of arabic tweets, many of which are in dialectal arabic (da). hence, many researchers have been using twitter as a major data source that is representative of current language usage and linguistic phenomena @xcite @xcite . though arabic is the lingua franca of most of the mena region, different dialects of arabic are used in different countries. while some dialects may differ significantly from each other (e.g. egyptian dialect (eg) and moroccan maghrebi dialect (ma) foot_0 ), others, particularly those in close geographic proximity, may be more difficult to tweak apart (e.g. variants of the levantine dialect such as syrian (sy) and lebanese (lb)). figure 1 highlights the dialectal variations across the arab world. the figure shows that dialects are a continuum that often transcends geographical regions and borders. automatically distinguishing between the different dialectal variations is valuable for many downstream applications such as machine translations @xcite , pos tagging @xcite , geo-locating users, and author profiling @xcite . though there has been prior work on performing arabic dialect identification (adi), much of the work was conducted on datasets with significant limitations in terms of genre @xcite , number of dialects @xcite , or focus @xcite , where often the focus was on geo-locating and profiling users as opposed to dialect identification. in this work, we expand beyond these efforts by utilizing tweets from across the mena region to build a large, non-genre specific, fine-grained, and balanced country-level dialectal arabic dataset that we use to build effective arabic dialect identification. we rely on two main features to build the dataset. the first feature is the twitter user profile description, where we identify users who self-declare themselves as belonging to a specific country in different forms such as showing signs of loyalty and pride (e.g. "proud egyptian"). in the second, we use a classifier that utilizes distant supervision to accurately discriminates between msa and di-alects. in doing so, we can identify users who self-declare their identity, mostly tweet in dialectal arabic, and only retain dialectal user tweets. further, we use our newly constructed dataset to build models that can effectively distinguish between 18 country-level arabic dialects. we didn't consider four arab countries, namely mauritania, somalia, djibouti, and comoros, because we were not able to find a sufficient number of twitter users tweeting in arabic. this could be due to the limited use of twitter in these countries, or that users may tweet primarily in other languages. for automated dialect identification, our models use a variety of features, such as character-level and word-level n-gram, static word embeddings, and contextual embeddings (e.g. multilingual bert (mbert) and arabert), and two classification techniques, namely support vector machines (svm) classification and fine-tuned transformer models. the contributions of this work are: this method can be completely automated such that it can be used in the future to collect fresh dialectal tweets. the dataset contains more than 540k tweets covering 18 country-level dialects with an associated test set containing 182 tweets per country on average that was manually labeled by native speakers from 18 arab countries. most efforts in building resources for arabic dialect identification are limited either in terms of genre, granularity, or the size of the data. zaidan 2 qadi dataset, is freely available for research purposes from: http://alt.qcri.org/resources/qadi/ 3 source: https://en.wikipedia.org/wiki/ varieties_of_arabic , country codes and regions are added and callison-burch (2011) curated the arabic online commentary dataset, a resource of more than 52m-words. they annotated over 108k sentences (41%) of the dataset with one of 5 possible dialects, namely: maghrebi, egyptian, levantine, gulf, and iraqi. similarly, alshutayri and atwell (2017), el-haj et al. (2018), and alsarsour et al. (2018) annotated collections of texts using the five regions/dialects. elfardy and diab (2013) and darwish et al. (2014) identified whether a sentence is modern standard arabic (msa) or egyptian. in recent years, there have been more efforts to cover more countries with finer granularity. @xcite constructed a dataset that covers 10 countries. @xcite built the "arap-tweet" dataset, which includes tweets from 15 countries. they retrieved tweets containing distinct dialectal words and expressions, and then given the users who authored these tweets, they crawled their timelines @xcite . we approach the problem from a different angle, where we start with self-declared users rather than a pre-defined keywords, which may have limited coverage of dialectal lexical variations. the madar (multi-arabic dialect applications and resources) project @xcite produced several resources among which two corpora were used for the shared task on fine-grained dialect identification @xcite . the resource includes a lexicon and a 1,000 parallel sentences from the travel domain that were translated into local dialects of 26 arab cities. additionally, the project released another set of tweets by searching twitter using a set of 25 seed hashtags corresponding to the 22 states of the arab league (e.g., #algeria, #egypt, #kuwait, etc.) and relevant hashtags such as: "#arabworld", "#arableague", and "#arab". the approach resulted in a collection of 2,980 profiles. when inspecting the profiles, the majority of the obtained users were from saudi arabia, representing 36% of the total. this was another motivation to curate a more balanced and representative dataset to use for dialect identification. further, as we show later, this dataset is sub-optimal for tweet-level dialect identification. @xcite built a large tweet collection containing more than 200 million geo-tagged tweets that were collected over 5 years @xcite @xcite @xcite . the resulting collection included tweets from 29 cities from 10 arab countries, of which 2,500 were manually annotated. the average inter-annotator agreement, cohen's kappa (k), was 67%, where the annotators reported not being able to distinguish between dialects from neighboring cities or countries @xcite . multiple approaches have been used for dialect id that exploit a variety of features, such as character or word n-grams @xcite @xcite , and techniques such as multiple kernel learning (ionescu and popescu, 2016) and distributed representation of dialects @xcite to name a few. @xcite used semi-supervised learning using multilingual bert for user-level dialect identification on the madar shared task. arabic tranformers-based approaches @xcite showed competitive results in nadi (abdul-mageed et al., 2020) shared task. ## data collection it is common for users on social networks to disclose social and linguistic information about themselves in their profiles. in twitter, the user profile provides a header and a short biography. both fields allow users to freely describe themselves. surveying arabic speaking profiles, it is customary to see users declaring their patriotism and national belonging by using their county's flag or explicitly naming the city or country that they are from (e.g. "kuwait is my home country", "i am a libyan citizen"). to build our dataset, we obtained a collection of arabic tweets that was crawled using the twitter streaming api, where we set the language filter to arabic ("lang:ar"), during the entirety of march and april, 2018. in all, the collection contains 25m tweets from which we extracted the profile information of all the users who authored these tweets. we applied three filters on user profiles and tweets as we describe in the next subsections. country identification for the first stage, to identify a user's country, we filtered user profiles using a gazetteer that includes: ## corpus statistics and analysis upon constructing the dataset, we attempted to explore its characteristics. first, we extracted features that are distinctive for each dialect. to do so, we computed the so-called valence score for each word in each dialect @xcite . the score helps determine the distinctiveness of a given word in a specific dialect in reference to other dialects. given n (t, d i ), which is the frequency of the term t in dialect d i , valence is computed as follows: where n (d i ) is the total number of occurrences of all words in dialect d i . figure 3 lists the words with highest valence scores per country. though the majority of the top words were in fact distinctive dialectal words (typically function words), there were three other prominent categories of words that were not. the first was names of locations inside these countries, which implies that geographic locations in a user's twitter timeline can be a strong features in identifying the country of the user. the second had words that appear in multiple dialects, which is expected given the overlap between dialects from different countries. the third category included msa words. though we intentionally excluded all tweets that were identified as msa, the appearance of such words was expected given the large overlap between msa and dialects and the frequent context switching between msa and dialects in user tweets. next, we computed the similarity between dialects to ascertain if similarities are consistent with reports in prior literature by visualizing the similarity between different country-level dialects. for such, we constructed a list of the top 10k words with the highest valence scores across all dialects. the resulting list can be viewed as a vector of 19 valence values for each word corresponding to the valence of 18 different country-level dialects in addition to msa. for msa data, we used the 3.09m msa tweets that we used earlier to train the msa/dialect classifier. then given the word vectors, we applied shc bottom-up hierarchical agglomerative clustering @xcite . the algorithm treats each dialect as a singleton cluster at the outset and then successively merges (or agglomerates) clusters until all clusters have been merged into a single cluster that contains all dialects. figure 4 shows the results of hierarchical clustering. the figure reflects the similarity and the geographical proximity of various dialects. at higher levels, dialects are grouped per region, where we can identify the major dialectal groups, namely gulf, maghrebi, egyptian, and levantine. this is aligned with geographical distribution of the dialects as well as the findings of prior work @xcite . ## experimental setup given our new dataset, we conducted a battery of experiments on the dataset to build effective country-level arabic dialect identification. we experimented with several tweet representation and classification models. for tweet representations, we used: surface features, namely words and character n-grams, static embeddings, and deep contextual embeddings, namely arabert and mbert. for classification, we used two different classifiers, namely an svm classifier and a fine-tuned transformer model. for comparison, we conducted the same experiments on madar dataset. in the following subsections, we present tweet representations and classification models. ## conclusion in this paper, we presented a method for building a country-level dialectal tweet corpus. the construction of the corpus relied on a cascade of filters, where user accounts were filtered on keywords indicating country, and tweets were filtered to remove users who predominantly tweet in msa or vulgar language. we built a large corpus containing 540k tweets from 2,525 twitter accounts that cover 18 arab countries. based on a manual inspection of a random sample of tweets from the corpus, the estimated accuracy of country-level dialectal tags was 91.5%. we also showed that the resultant corpus can be effective in training a country-level dialect classifier for tweets that achieves a macroaveraged f1-score of 60.6% across 18 different classes. we compared to training on a publicly available dataset, namely madar dataset, and madar results were significantly lower. based on our error analysis, we discovered that a large source of errors was due to the naturally occurring overlap between dialects from neighboring countries and to code switching between different dialects. while overlap between dialects is potentially an intractable problem, detecting code switching between dialects is a future direction that can further help filter training data and identify tweets that may include multiple dialects simultaneously. for future work, we plan to investigate code switching and examine the efficacy of extending our dataset to perform user-level geotagging. though identifying a user's country may depend on multiple signals, accurate dialect identification is likely a strong signal that can aid classification. 9 https://en.wikipedia.org/wiki/ egyptian_arabic