Variation and Instability in Dialect-Based Embedding Spaces (2023)

Abstract: This paper measures variation in embedding spaces which have been trained on different regional varieties of English while controlling for instability in the embeddings. While previous work has shown that it is possible to distinguish between similar varieties of a language, this paper experiments with two follow-up questions: First, does the variety represented in the training data systematically influence the resulting embedding space after training? This paper shows that differences in embeddings across varieties are significantly higher than baseline instability. Second, is such dialect-based variation spread equally throughout the lexicon? This paper shows that specific parts of the lexicon are particularly subject to variation. Taken together, these experiments confirm that embedding spaces are significantly influenced by the dialect represented in the training data. This finding implies that there is semantic variation across dialects, in addition to previously-studied lexical and syntactic variation.

## dialects and embedding spaces this paper investigates the degree to which embedding spaces are subject to variation according to the regional dialect or variety that is represented by the training data. the experiments train character-based skip-gram embeddings on gigaword corpora representing four regional dialects of english (north america, europe, africa, and south asia). while there is a robust tradition of discriminative modelling of dialects and varieties within nlp @xcite @xcite @xcite , there has been much less work on the influence which the dialectal composition of the training data (upstream) has on embedding spaces after training (downstream). the basic idea in this paper is to train five iterations of character-based skip-gram embeddings on dialect-specific corpora in order to measure both variation (across dialects) and instability (within dialects); this is visualized in figure 1 . in order to find out whether specific parts of the lexicon are especially influenced by the dialect represented in the training data, the lexicon used for comparing embedding spaces is annotated for frequency, concreteness, part-of-speech, semantic domain, and age-of-acquisition. if the specific dialect represented in the training corpus has no influence on embedding spaces, then variation across regions will be the same as variation within regions. in other words, we must control for instability (operationalized as variation across embeddings from the same dialect) to avoid false positives. however, if the dialect represented in the training data does have an influence on embedding spaces after training, then there will be a clear distinction between variation across dialects and instability within dialects. the contribution of this paper is to show (i) that dialectal variation in character-based embedding spaces is significantly stronger than the noise caused by background instability and (ii) that this variation remains concentrated in certain parts of the lexicon. to accomplish this, we model the impact of dialect-specific training corpora on embeddings by controlling for background instability and organizing the experiments around the lexical attributes of frequency, concreteness, part-of-speech, semantic domain, and age-of-acquisition. we begin by reviewing related work on dialectal variation and embedding stability (section 2), before describing the main experimental questions (section 3), the data (section 4), and the methods (section 5). we then compare variation within and between dialect-specific embeddings (section 6) before modelling the influence of lexical factors on such dialectal variation (section 7). taken together, these experiments confirm that regional dialect or variety has a significant influence on embedding spaces that far exceeds baseline instability. ## related work this section discusses previous work in which models trained on data from different varieties (upstream) become significantly different after training (downstream). it also presents previous work on instability in embedding spaces. ## experimental questions this paper focuses on two questions: first, are there significant differences in embeddings trained from corpora representing different dialects when accounting for baseline instability in the embeddings? second, if so, are these dialectal differences specific to a certain part of the vocabulary, such as words belonging to a specific semantic domain? the basic idea here is to compile four gigaword corpora representing english as used in north america, europe, africa, and south asia. these areas represent different dialect regions. for example, while there are smaller differences between american english and canadian english, these two dialects are more similar to one another than to other national dialects like irish english. for ex-ample, work on syntactic variation has shown that american and canadian english, at least in digital contexts, are closely related while uk and irish english form a separate closely related pair @xcite . based on the distribution of errors within a confusion matrix, other work has shown that indian and pakistani english are likewise more similar to one another than to other dialects @xcite . the dialects included represent both innercircle and outer-circle varieties. the concept of inner-circle vs outer-circle is based on the historical stages of european colonization @xcite . this distinction within the world englishes paradigm is meant to capture the perceived prestige differences of these dialects rather than to make a distinction between dialects and varieties as linguistic objects. for example, inner-circle populations tend to have a higher socio-economic status and better access to digital technologies, leading to their status as prestige varieties. both groups can be considered dialects. in some cases speakers of outer-circle varieties could be considered second-language learners; however, regardless of a distinction between native and non-native speakers, the production found in outer-circle varieties remains robust and predictable over time. thus, we treat both inner-circle and outer-circle varieties as dialects of equal standing but maintain the terminology from the world englishes paradigm in order to provide a bridge to work in sociolinguistics. we first train embeddings on each dialectspecific corpus and then measure variation across a lexicon that is annotated for concreteness, ageof-acquisition, semantic domain, part-of-speech, and frequency. we train five sets of embeddings for each dialect-specific corpus, each based on a random reshuffling of the corpus. this allows us to measure the difference between variation (across dialects) and baseline instability (within dialects). we work with skip-gram embeddings (sgns: @xcite as implemented in the fasttext framework @xcite . in particular, we use the skip-gram variant with negative sampling (@xmath0.05 and 100 dimensions. the character n-gram sizes range from 3 to 6, with a maximum of 2 million n-gram buckets allowed. because previous work has shown that different random seeds can cause instability (gonen et al., 2020), we control for such instability by using the same random seed for each set of embeddings. thus, variation caused by random seed and by training parameters is taken into account in this experimental set-up. several considerations support the use of noncontextual skip-gram embeddings for these experiments. in the first case, the focus here is on semantic variation rather than lexical or syntactic structures and the long-distance co-occurrences captured by the skip-gram task are taken as better representations for such semantic variation. in the second case, the inclusion of low-resource dialects like african english means that the amount of training data available is limited and insufficient for training robust contextual embeddings. given the dual goals of focusing on semantics while also including low-resource dialects, skip-grams provide the most practical type of embedding for answering these particular experimental questions. ## data the data used here represents different geographic locations which, in turn, represent different dialects. the data itself is drawn from two registers, web pages and tweets, both derived from the corpus of global language use @xcite . the experiments train character-based embeddings for these four different regional dialects, as shown in ## methods for each regional variety of english, we train embeddings using the fasttext framework with the parameter settings described above. previous work has shown that this family of embeddings can be unstable; in this context, instability means that the same training corpus could result in multiple sets of nearest neighbors over different iterations @xcite . we control for this by randomly shuffling each corpus and retraining the embeddings five times. because all comparisons are between two sets of embeddings, we thus obtain ten observations (unique comparisons) to represent each condition, as visualized in figure 1 . we use the same random seed and the same parameters across all sets of embeddings to control for other sources of variation. vocabulary features. the vocabulary for the embedding space is derived from semantic and psycholinguistic resources that provide categorizations for specific lexical items. this source of vocabulary allows us to compare stability and variation across different sub-sets of the lexicon. the first source of lexical annotations is a participant-based study of concreteness @xcite . this source provides concreteness ratings between 1 and 5 for each lexical item, with higher values reflecting more concrete and lower values reflecting more abstract judgements from participants. this source also provides the most common part-of-speech for each lexical item. the distribution of the vocabulary across concreteness ratings and parts-of-speech is shown in the next source of lexical annotations is the ucrel semantic analysis system @xcite which provides a high-level semantic domain for each vocabulary item. for example, there are 586 items belonging to the domain calculating overlap. the stability and similarity of word representations are calculated using the overlap of nearest neighbors @xcite . given two sets of embeddings (i.e., north america and europe) we iterate over each word in the lexicon. first, we retrieve the k nearest neighbors using cosine similarity. second, we calculate the overlap between the two sets of nearest neighbors. for example, if all ten out of ten words appear in both embeddings as nearest neighbors, the overlap is 100%. if only five words out of ten appear as neighbors, the overlap is 50% (five shared words out of 10 possible shared words). this provides a word-specific measure of overlap. this method aligns the vocabulary space rather than aligning the embedding spaces; this approach is taken because alignment methods have previously been shown to be unstable @xcite and thus less suitable for identifying variation across dialects. a selection of example levels of overlap is shown in we measure overlap with values for k of 5, 10, 25, and 50. the distribution of overlap values is shown in figure 2 for the european and north american model (on the right) and for the european and european model (on the left). thus, the distributions on the right are across dialects and those on the left are within the same dialect. the impact of k is shown in the plots, with @xmath1. smaller values of k lead to ragged distributions simply because the number of possible overlap values is limited. how much impact does the choice of k have on the results? we can see that higher values lead to finer estimates of the distribution of overlap, but overall the values are much the same once k is above 10. for example, there is a significant pearson correlation between overlap values at @xmath2.937 on the right and 0.879 on the left, in both cases with @xmath3.001). we use @xmath4, but the choice of k (above 10) has minimal impact on the results. we also see that the overlap within the same dialect (on the left) is greater than the overlap between different dialects (on the right). the figures for all distributions are available in the supplementary material. foot_0 ## overlap within vs between dialects the first experiment evaluates whether the variation between dialects remains meaningful when compared with baseline instability within a single dialect. the overlap measure described above compares the similarity between two sets of embeddings. we visualize the within vs between dialect condition in figure 3 for europe, with each type of comparison a separate violin plot. in blue we see the within-dialect overlap in which we compare european embeddings to other european embeddings. in orange we see between-dialect overlap for each of the other three regions. there is a clear distinction here between variation within the same dialect (baseline instability) and between different dialects (actual variation). while we measured overlap between ten unique pairs of embeddings for each condition, this figure shows only the first pair for each. the supplementary materials contain the figures for all comparisons. the conclusion remains the same: the variation in embeddings across dialects is not simply a result of instability alone. there is a clear visual distinction between within-dialect and betweendialect overlap in all cases. we test for significance using a paired t-test: for example, are the values for europe-europe comparisons actually different from the values for europe-africa comparisons? for each comparison, we randomly choose a single pair of embeddings to test (i.e., so that we compare europe and africa only once). in each case the difference is significant with @xmath5.001. thus, there is a visually clear and statistically significant difference between baseline instability and variation across dialects. we quantify the magnitude of this difference in ## lexical factors we have shown that there is a significant difference in embedding spaces depending on the dialect represented in the training data, a difference that is much greater than baseline instability within dialects (as simulated by shuffling and retraining on the same corpora). this section explores dialectal variation in embedding spaces further by focusing on the impact of the lexical factors described in section 5. we ask whether this kind of variation is distributed equally across the lexicon or whether it is concentrated in particular types of vocabulary. we model the relationship between lexical attributes and overlap using a linear mixed effects regression model, with one model for each dialect. within each model, the region of comparison is a fixed effect: for example, we model variation within the european embeddings using their overlap with north america, africa, and south asia as fixed effects. for random effects we include all lexical attributes. we represent each region using the average overlap across all ten pairs of embeddings, using @xmath7. the means of different regions are independent in the sense that each vocabulary item is modelled independently from corpora representing that region. the coefficients and p-values for each lexical attribute are shown in positive categorical factors are shown above and negative factors below. columns show results from the four dialect-specific models. while some factors are significant in one dialect but not another, no attributes have opposite effects across dialects (i.e., indicate more variation in one dialect but less variation in another). within semantic domains, vocabulary involving body and individual (e.g., pain and ache) are more stable across dialects, as are food and farming (e.g., celery and sushi) and science and technology (e.g., biologist and geologist). these terms are less socially-conditioned in the sense that they refer to tangible objects or to specially-defined fields (like biology) that transcend cultural boundaries. on the other hand, vocabulary from semantic domains home and housing (e.g., guest or pew), movement and travel (e.g., turnpike or curbside), and names and grammar (e.g., northwestern or roman) are subject to more variation. these words are more socially-conditioned in the sense that they presume socially-defined concepts: a guest requires a definition of family units and a pew is a part of the concept church. within parts-of-speech, function words (e.g., of or and) and adverbs (e.g., hardly and exactly) are much more stable. and named entities (e.g., flint) are much less stable. verbs are more important to the model than nouns. of the three scalar attributes, frequency has a significant effect but the coefficient is so small it is negligible. concreteness is significant in every region, with more abstract words (e.g., surreal and sanctimonious) being more stable while more concrete words (e.g., cookie and bug) are less stable. in this case, the specific instances (the referents) of these more concrete terms are likely to be quite different across dialects (cookies are different in different places). age-of-acquisition is significant in three out of four regions, but it has only a relatively small effect, with words acquired at a younger age being more stable. for instance, mother and grandmother (learned at age 2) are quite stable while ethos and polarization (learned at age 15) are subject to variation. the full regression results and the stability/variability values for the entire lexicon are available in the supplementary materials. ## discussion and conclusions these experiments have shown that embedding spaces are subject to variation according to the dialect represented by the training data. this variation is significantly greater than noise caused by baseline instability in the embeddings themselves. this finding confirms the importance of regional dialects in nlp: while previous work has shown the impact of dialect on lexical and syntactic representations, this paper shows that such variation also extends to semantic representations. previous work has focused on distinguishing between dialects or on directly modelling variation over space and time. this paper has taken a different approach by training otherwise comparable models on corpora representing different dialects, controlling for other sources of variation like pa-rameter settings and random seeds. the results show that the dialects represented in the training context have significant downstream impacts on common semantic representations (embeddings). these findings raise important questions for future work. first, is the influence of dialect consistent across languages or is this a result of the colonial history of a few languages like english? second, do contextual embeddings also manifest this type of variation or is it confined to non-contextual skipgram embeddings? third, would a larger inventory of dialect-specific embeddings change the distribution of variation within the lexicon or is this a stable effect? regardless of such further questions, these experiments show that dialect has a downstream effect on semantic representations, expanding previous work on lexical and syntactic representations.