Toward Micro-Dialect Identification in Diaglossic and Code-Switched Environments (2020)

Abstract: Although prediction of dialects is an important language processing task, with a wide range of applications, existing work is largely limited to coarse-grained varieties. Inspired by geolocation research, we propose the novel task of Micro-Dialect Identification (MDI) and introduce MARBERT, a new language model with striking abilities to predict a fine-grained variety (as small as that of a city) given a single, short message. For modeling, we offer a range of novel spatially and linguistically-motivated multi-task learning models. To showcase the utility of our models, we introduce a new, large-scale dataset of Arabic micro-varieties (low-resource) suited to our tasks. MARBERT predicts micro-dialects with 9.9% F1, 76 better than a majority class baseline. Our new language model also establishes new state-of-the-art on several external tasks.

## introduction sociolinguistic research has shown how language varies across geographical regions, even for areas as small as different parts of the same city @xcite . these pioneering studies often used field work data from a handful of individuals and focused on small sets of carefully chosen features, often phonological. inspired by this early work, researchers have used geographically tagged social media data from hundreds of thousands of users to predict user location @xcite @xcite @xcite @xcite or to develop language identification tools @xcite @xcite . whether it is possible at all to predict the micro-varieties foot_1 of the same general language is a question that remains, to the best of our knowledge, unanswered. in this work, our goal is to investigate this specific question by introducing the novel task of micro-dialect identification (mdi). given a single sequence of characters (e.g., a single tweet), the goal of mdi is to predict the particular micro-variety (defined at the level of a city) of the community of users to which the posting user belongs. this makes mdi different from geolocation in at least two ways: in geolocation, (1) a model consumes a collection of messages (e.g., 8-85 messages in popular datasets @xcite and (2) predicts the location of the posting user (i.e., user-level). in mdi, a model takes as input a single message, and predicts the micro-variety of that message (i.e., message-level). while user location and micro-dialect (md) are conceptually related (e.g., with a tag such as seattle for the first, and seattle english, for the second), they arguably constitute two different tasks. this is because location is an attribute of a person who authored a wikipedia page @xcite or posted on facebook @xcite or twitter @xcite , whereas md is a characteristic of language within a community of speakers who, e.g., use similar words to refer to the same concepts in real world or pronounce certain sounds in the same way. to illustrate, consider a scenario where the same user is at different locations during different times. while a geolocation model is required to predict these different locations (for that same person), an mdi model takes as its target predicting the same micro-variety for texts authored by the person (regardless of the user location). after all, while the language of a person can, and does, change when they move from one region to another, such a change takes time. concretely, although to collect our data we use location as an initial proxy for user md, we do not just exploit data where n number of posts (usually @xmath0. rather, to the extent it is possible, we take the additional necessary step of manually verifying that a user does live in a given region, and has not moved from a different city or country (at least recently, see section 2). we hypothesize that, if we are able to predict user md based on such data, we will have an empirical evidence suggesting md does exist and can be detected. as it turns out, while it is almost impossible for humans to detect md (see section 2.3 for a human annotation study), our models predict varieties as small as those of cities surprisingly well (9.9% f 1 , ∼ 76× higher than a majority class baseline based on a single, short message) (sections 6 and 7). context. mdi can be critical for multilingual nlp, especially for social media in global settings. in addition to potential uses to improve machine translation, web data collection and search, and pedagogical applications @xcite , mdi can be core for essential @xcite , recommendation systems @xcite , event detection @xcite , and disaster response @xcite . further, as technology continues to play an impactful role in our lives, access to nuanced nlp tools such as mdi becomes an issue of equity @xcite . the great majority of the world's currently known 7,111 living languages, 3 however, are not nlpsupported. this limitation also applies to closely related languages and varieties, even those that are widely spoken. we focus on one such situation for the arabic language, a large collection of similar varieties with ∼400 million native speakers. for arabic, currently available nlp tools are limited to the standard variety of the language, modern standard arabic (msa), and a small set of dialects such as egyptian, levantine, and iraqi. varieties comprising dialectal arabic (da) differ amongst themselves and from msa at various levels, including phonological and morphological @xcite , lexical @xcite @xcite , syntactic @xcite , and sociological @xcite . most main arabic dialects, however, remain understudied. the situation is even more acute for mds, where very limited knowledge (if 3 source: https://www.ethnologue.com . ## data acquisition and labeling we first acquire a large user-level dataset covering the whole arab world. we then use information in user profiles (available only for a subset of users) to automatically assign city, state, and country labels to each user. since automatic labels can be noisy (e.g., due to typos in city names, use of different languages in user profiles), we manually fix resulting errors. to further account for issues with human mobility (e.g., a user from one country moving to another), we manually inspect user profiles, tweets, and network behavior and verify assigned locations. finally, we propagate city, state, and country labels from the user to the tweet level (each tweet gets the label assigned to its user). we now describe our data methods in detail. to develop a large scale dataset of arabic varieties, we use the twitter api to crawl up to 3,200 tweets from ∼2.7 million users collected from twitter with bounding boxes around the arab world. overall, we acquire ∼6 billion tweets. we then use the python geocoding library geopy to identify user location in terms of countries (e.g., morocco) and cities (e.g., beirut). 4 out of the 2.7 million users, we acquired both 'city' and 'country' label for ∼233k users who contribute ∼ 507m tweets. the total number of cities initially tagged was 705, but we manually map them to only 646 after correcting several mistakes in results returned by geopy. geopy also returned a total of 235 states/provinces that correspond to the 646 cities, which we also manually verified. we found all state names to be correct and to correspond to their respective cities and countries. foot_3 ## marbert: a new language model we introduce marbert, a new language model trained with self-supervision on 1b tweets from from our unlabaled twitter collection (described in section 2). we train marbert on 100k word-piece vocabulary, for 14 epochs with a batch size of 256 and a maximum sequence length of 128. training took 14 days on 8 google cloud tpus. we use the same network architecture as mbert, but without the next sentence prediction objective since tweets are short. marbert has much larger token count than bert @xcite (15.6b vs. 3.3b), and is trained on 5× bigger data than arabert @xcite ) (126gb vs. 24gb of text). unlike arabert, which is focused on msa, marbert has diverse coverage of dialects in addition to msa. as ## model generalization for the experiments reported thus far, we have split our micro-ara (monolingual) dataset randomly at the tweet level. while this helped us cover the full list of our 319 cities, including cities from which we have as few as a single user, this split does not prevent tweets from the same user to be divided across train, dev, and test. in other words, while the tweets across the splits are unique (not shared), users who posted them are not unique. we hypothesize this may have the consequence of allowing our models to acquire knowledge about user identity (idiolect) that interact with our classification tasks. to test this hypothesis, we run a set of experiments with different data splits where users in test are never seen in train. to allow the model to see enough users during training, we split the data only into train and test and use no dev set. we use the same hyper-parameters identified on previous experiments. an exception is the number of epochs, where we report the best epoch identified on test. to alleviate the concern about absence of a dev set, we run each experiment 3 times. each time we choose a different test set, and we average out performance on the 3 test sets. this is generally similar to cross-validation. for this set of experiments, we first remove all cities from which we have only one user (79 cities) and run experiments across 3 different settings (narrow, medium, and wide). 13 we provide a description of these 3 settings in section d.1 in the appendix. for the narrow setting only, we also run with the same code-switching and diaglossic auxiliary tasks (individually and combined) as before. we use mbert fine-tuned on each respective train as our baseline for the current experiments. as importantly, although results in ## discussion as we showed in sections 6 and 7, our models are able to predict variation at the city-level significantly better than all competitive baselines. this is the case even when we do not remove msa content, but better results are acquired after removing it. a question arises as to whether the models are indeed capturing micro-linguistic variation between the different cities, or simply depending on different topical and named entity distributions in the data. to answer this question, we visualize attention in ∼ 250 examples from our test set using one of our marbert-narrow models reported 16 to filter out msa, we apply an in-house msa vs. dialect classifier (@xmath2.1%, @xmath3.6%) on the data, and remove tweets predicted as msa. more information about the msa vs. dialect model is in section d.3 of the appendix. we cast more extensive investigation of the interaction between dialects and msa vis-a-vis our classification tasks, including based on manually-filtered msa, as future research. in ## comparisons and impact comparisons to other dialect models. in absence of similar-spirited nuanced language models, we compare our work to existing models trained at the country level. these include the tweet-level 4-country shami @xcite which we split into train (80%), dev (10%), and test (10%) for our experiments, thus using less training data than @xcite (who use crossvalidation). we also compare to @xcite , the winning team in the the 21country madar shared task-2 @xcite . note that the shared task targets user-level dialect based on a collection of tweets, which our models are not designed to directly predict (since we rather take a single tweet input, making our task harder). 19 for the purpose, we train two models, one on madar data (shared tasks 1 and 2) and another on our micro-ara+madar data. we also develop models using the 17-country arap-tweet @xcite , noting that authors did not preform classification on their data and so we include a unidirectional 1-layered gru, with 500 units as a baseline for arap-tweet. note that we do not report on the dataset described in @xcite since it is automatically labeled, and so is noisy. we also do not compare to the dataset in @xcite since it is small, not naturally occurring (only 2,000 translated sentences per class), and the authors have already reported linear classifiers outperforming a deep learning model due to small data size. as geolocation, variation, and mtl. research on geolocation is also relevant, whether based on text @xcite @xcite , user profile @xcite , or network-based methods @xcite . models exploiting network information, however, do not scale well to larger datasets @xcite . @xcite exploit geotagged twitter data to study how words spread geographically. @xcite uses representations based on geolocation to improve semantic similarity. @xcite studies syntactic variations in 7 languages based on geolocated data. @xcite visualizes regional variation across europe using twitter. @xcite find that twitter data are representative of actual population. mtl has been successfully applied to many nlp problems, including mt and syntactic parsing @xcite , sequence labeling @xcite , and text classification @xcite . ## conclusion we introduced the novel task of mdi and offered a large-scale, manually-labeled dataset covering 319 city-based arabic micro-varieties. we also introduced several novel mtl scenarios for modeling mds including at hierarchical levels, and with linguistically-motivated auxiliary tasks inspired by diaglossic and code-switching environments. we have also exploited our own data to train mar-bert, a very large and powerful masked language model covering all arabic varieties. our models establish new sota on a wide range of tasks, thereby demonstrating their value. ultimately, we hope our work can open up new horizons for studying mds in various languages. investigating examples of the geolocated data, we observed geopy made some mistakes. to solve the issue, we decided to manually verify the information returned from geopy on all the 705 assumed 'cities'. for this purpose of manual verification, we use wikipedia, google maps, and web search as sources of information while checking city names. we found that geopy made mistakes in 7 cases as a result of misspelled city names in the queries we sent (as coming from user profiles). we also found that 44 cases were not assigned the correct city name as the first 'solution'. geopy provided us with a maximum of 7 solutions for a query, with best solutions sometimes being names of hamlets, villages, etc., rather than cities.