Identifying Token-Level Dialectal Features in Social Media (2023)

Abstract: Dialectal variation is present in many human languages and is attracting a growing interest in NLP. Most previous work concentrated on either (1) classifying dialectal varieties at the document or sentence level or (2) performing standard NLP tasks on dialectal data. In this paper, we propose the novel task of token-level dialectal feature prediction. We present a set of fine-grained annotation guidelines for Norwegian dialects, expand a corpus of dialectal tweets, and manually annotate them using the introduced guidelines. Furthermore, to evaluate the learnability of our task, we conduct labeling experiments using a collection of baselines, weakly supervised and supervised sequence labeling models. The obtained results show that, despite the difficulty of the task and the scarcity of training data, many dialectal features can be predicted with reasonably high accuracy.

## introduction language variation is a pervasive phenomenon in human language. these varieties can differ on phonemic, lexical, or syntactic levels, among others, and often vary on several levels at a time @xcite . one common type of language variation stems from geographical location, as people actively use regional variations to mark their identity. when a language variety indicates where a speaker is from, we call this variety a dialect, or more precisely a geolect or topolect, as the word 'dialect' can also refer to social background or occupation. in this work, we use 'dialect' to denote geographical variation. dialectal variation in norwegian is widespread and, in contrast to many languages, the use of spoken and written dialects in the public sphere is generally viewed positively @xcite . although norwegian can be broadly divided into four dialectal regions, many dialectal features are shared across these regions (see figure 1 ). therefore, rather than seeing dialects as discrete categories, we should view them as a combination of correlated dialectal features @xcite . the under-resourced status of dialects, however, makes it difficult to build nlp tools from scratch. this is exacerbated by the growing reliance on pretrained language models, which often encounter few examples of dialectal data during training. if nlp models fail to process dialectal inputs, their deployment may reinforce existing inequalities, as those who use a non-standard variety will either receive worse service or be forced to adopt a standard variety to interact. those who advocate for maintaining dialectal variation also depend on tools to help them monitor the use of dialects on social media. this motivates the development of fine-grained models of dialectal features. previous work on dialectal nlp has classified dialects, geographical location, or provided training and testing resources for various dialects. in this paper, we take a different viewpoint on identifying dialects, opting to label the token-level dialectal features of a text rather than classifying or predicting the geolocation of the entire text. we first propose a fine-grained annotation scheme for token-level dialectal features in norwegian. we then annotate a corpus of norwegian dialectal tweets using this scheme, and finally validate its use for fine-tuning neural sequence labeling models in norwegian. our contributions are 1) we introduce the novel task of token-level dialect feature identification, 2) provide a novel corpus of norwegian dialectal tweets annotated for 21 token-level features, foot_0 and 3) describe extensive experiments demonstrating the learnability and difficulty of the task. 2 related work in contrast to more formal writing, social media abounds with dialectal variation, ranging from variation between racial groups @xcite , to variation within online communities @xcite . while not all levels of variation are equally present, often due to a speaker's lack of awareness of sociolinguistic indicators @xcite , a substantial share of dialectal variation is reliably transcribed in social media posts @xcite . for nlp, dialectal data presents both a challenge and an important area to improve upon. previous work in nlp has included descriptive corpus studies @xcite , dialect classification @xcite , geolocation of tweets based on their dialectal features @xcite or quantifying the spatial dependence of linguistic variables @xcite . there have also been a series of workshops (var-dial) (e.g. @xcite @xcite that include work on discriminating similar languages @xcite , identifying dialects @xcite , and geolocation of tweets @xcite . the workshops have also held several shared tasks with the aim to identify languages and dialects @xcite , as well as morpho-syntactic tagging @xcite . another series of shared tasks have focused on the identification of arabic dialects @xcite @xcite . while each of these shared tasks proposed dialect identifications on different level of granularity (region, country, and city-levels), they all approached dialect identification as a sentence classification task. work on code-switching (e.g. @xcite @xcite , on the other hand, has focused on word-level classification, but usually casts this a binary decision, rather than identifying fine-grained labels. regarding norwegian dialects specifically, linguistic work is long and varied. @xcite described the main dialect regions, while @xcite describes several factors that drive language change in modern norwegian dialects, e.g. urban jumping @xcite , the prestige of certain dialects, or the general tendency towards simplification. within nlp, @xcite present the nordial corpus, a curated collection of 1,073 tweets classified as either bokmål, nynorsk (which are the two standardized written forms for norwegian), dialectal, or mixed. the authors experimented with classifying these tweets with norwegian bert models @xcite and found that the resulting models achieved reasonably good performance at identifying tweets written in dialectal norwegian. @xcite introduce the task of dialect feature detection at the phrase/sentence level. they use available annotations on the ice-india english data @xcite and annotate a small amount of this data with separate set of 18 dialectal features. as they have no training data for their annotated features, they propose to use a minimal pairs framework as a kind of weak supervision. they find that even with minimal supervision, their models are able to reliably predict many of the features. however, they do not predict which tokens carry the features, choosing to label the entire phrase instead. to address these limitations, we propose a new approach where we annotate dialectal features at the token level. we contend that this annotation strategy provides a more fine-grained view of the actual use of dialectal features in social media. ## dialectal tweet collection in order to increase the number of dialectal tweets, we expand upon the nordial corpus @xcite and collect a further 3,000 tweets to annotate. during the initial collection, we used the twitter api without a search query and confined the search to tweets from the geographical area of norway. this first collection, however, yielded relatively ## annotation of fine-grained dialectal features figure 2 shows an example from texan english with three main dialectal features: y'all, which is the non-standard second person plural pronoun and fixin' to, which contains the lexical feature 'fixing to' which means 'about to', and the morphological feature of 'g-dropping'. in the rest of this section, we detail the inventory of dialectal features used in our annotation. as each example highlights a minimal pair example of a single dialectal feature, we do not include the labels below the relevant tokens. ## experiments we now describe the experimental setup employed to validate our annotations. as early results indicated that standard models had difficulty learning multi-label sequence labelling tasks, we merge occurrences of multiple labels, yielding a total of 159 combinations (including 'ø', the null label). for each possible combination of labels in our dataset, we create a new merged label that represents them. this increases the number of total classes to be predicted, but reduces the task to a much simplified multi-label sequence labelling problem. formally, the task is then given a sequence of n tokens @xmath0, t 2 , . . . , t n } to predict the sequence of token-wise labels @xmath1, l 2 , . . . , l n }, where these labels can be either single labels, e.g., 'vowel_shift' or a merged label, e.g., 'lexical-vowel_shift'. for all experiments with neural models, we train an set of five models with different random seeds and report both micro-averaged f 1 and standard deviation. ## results the majority label baseline ('vowel shift') achieves a low f 1 score of 4.4. while the handcrafted functions obtain slightly higher f 1 scores than these baselines, the scores demonstrate that the proposed task is challenging and that simple rule-based approaches are insufficient. all supervised models perform better than the weak supervision models, with the bilstm achieving 45.5 f 1 , the svm 47.7, norbert 52.9, and nb-bert 58.9. in general, the results of the svm follow a high-precision low-recall pattern (e.g., hv: precision 90/recall 42, interjection: 100/8.3, palatalization: 100/13.3) displaying this model's inability to generalize to new examples, while the neural models tend to generalize better. the good performance of nb-bert follows previous trends for classification of tweets @xcite . those results differs from @xcite , who found that the weak supervision provided by several hundred minimal pairs was often enough to outperform supervised approaches. this discrepancy may be due to differences in the training set size or the increased difficulty of labeling the tokens rather than the full utterance. ## error analysis we provide here an error analysis of the results from the best performing model, namely nb-bert. it is clear from the confusion matrix in figure 4 that the model confuses most labels with the label 'ø'. the other label that is regularly overpredicted is 'vowel shift', which suggests that frequency plays a strong role in prediction. when it comes to multiple labels, the performance of nb-bert can be characterized as high-precision and low-recall, with only 30% of the test tokens with multiple labels being predicted as such by the model, with a micro f 1 of 88.1. to establish the importance of context, we compare the performance of the svm and nb-bert models on context-free labels (h-v, functional, vowel shift, voicing, palatalization, shortening, interjection, nominal declension, conjugation, marked, lexical) and context-sensitive labels (phonemic spelling, contraction, pronoun subject, pronoun object, present marker deletion, apocope, adjectival declension, demonstrative pronoun, copula, gender). we compare these two groups by taking the average difference between the f1 scores for each label. for the context-free labels, there is an average 14.0 percentage points difference between the two models, while for the context-sensitive labels, this difference is 24.9. this implies that including context via contextual embeddings is especially important for the context-sensitive labels. ## conclusion and future work in this paper, we have presented a new dataset for token-level dialect feature prediction, composed of norwegian tweets classified as dialectal, which we annotated for 21 dialectal features achieving good another promising direction is to predict regional dialects based on the token-level features. as dialectal traits are correlated with certain regions, it may be possible to create hierarchical representations of dialects on different levels of granularity. the guidelines, models, and annotations will be made publicly available. 7 as for potential risks, the dataset was compiled from social media posts. therefore, complying with the gdpr regulations, authors of these posts 7 https://github.com/jerbarnes/nordial must have the right to be forgotten if they wish to remove previous posts. we will therefore only release the annotations with the original tweets upon request. in this way, if they have been deleted, they will also not be recuperated for our dataset.