Stability of Syntactic Dialect Classification over Space and Time (2022)

Abstract: This paper analyses the degree to which dialect classifiers based on syntactic representations remain stable over space and time. While previous work has shown that the combination of grammar induction and geospatial text classification produces robust dialect models, we do not know what influence both changing grammars and changing populations have on dialect models. This paper constructs a test set for 12 dialects of English that spans three years at monthly intervals with a fixed spatial distribution across 1,120 cities. Syntactic representations are formulated within the usage-based Construction Grammar paradigm (CxG). The decay rate of classification performance for each dialect over time allows us to identify regions undergoing syntactic change. And the distribution of classification accuracy within dialect regions allows us to identify the degree to which the grammar of a dialect is internally heterogeneous. The main contribution of this paper is to show that a rigorous evaluation of dialect classification models can be used to find both variation over space and change over time.

## geographic variation over time this paper experiments with the stability of dialect classification models over space and time in order to determine the degree to which they capture language variation and change. the assumption in previous work has been that a geo-referenced corpus @xcite @xcite captures the linguistic behaviour of specific populations. this paper experiments with the spatial and temporal stability of dialect models by systematically constructing monthly test sets spanning a three-year period. this allows us to evaluate the continuing effectiveness of dialect models over time, an important criteria for determining their validity. because different locations represent different populations, we use spatial sampling to construct test sets which represent different local populations within each country. this allows us to determine the degree to which a dialect like new zealand english adequately represents the varied populations within new zealand. dialect classification is the task of predicting the location of origin for the individual who produced a given sample @xcite @xcite . thus, dialect classification, by focusing on the latent properties of geo-referenced samples, differs from geolocation @xcite which focuses on predicting the location of the sample itself and from geo-characterization @xcite which focuses on predicting attributes of the location. while all three tasks rely on geographic information, dialect classification is unique in modelling variations in the linguistic system. beyond this, dialect classification is part of ensuring that nlp represents the world's population, including non-standard and non-western populations. the temporal evaluation (section 6) shows that most dialects share the same performance decay rate. this indicates a general effect of model decay rather than cases of change over time within individual dialects. the spatial evaluation, however, shows that prediction accuracy for all dialects is spatially-conditioned within countries (section 7). this indicates that, while dialect models capture proto-typical populations within each country, they do not equally describe all local populations. the experiments in this paper use construction grammar @xcite @xcite to represent syntactic structure for the purpose of observing dialectal variation. cxg is a usage-based approach to syntax, a bottom-up theory of language in which frequent exposure is hypothesized to lead to the emergence of grammatical units @xcite . the use of syntactic representations for dialect classification ensures that the model does not rely on extraneous information like place names or local topics of in-terest. from this perspective, a grammar is a set of constructions that together represent the structure of a language. a dialect model is a matrix of spatial weights in which the number of rows corresponds to the number of constructions in the grammar and the number of columns corresponds to the number of dialects. these weights, learned using a linear svm, support dialect classification and also represent spatial variation in the grammar. in order to undertake a spatio-temporal evaluation, we collect a balanced corpus of tweets to represent 12 varieties of english around the world. the basic experimental paradigm is to train models on a fixed period (july through december 2018) and then test those models at monthly intervals from 2019 to 2021. each monthly test set maintains the same geographic distribution as the training data, so that fluctuations in performance are not caused by changes in the locations represented. after considering related work on dialect classification and other geographic models (section 2), we consider the corpora used in these experiments (section 3). we then present the syntactic representations used (section 4) and the basic experimental methods (section 5). the performance of dialect models over time is presented in section 6 and the performance over space in section 7. the main contribution of this paper is to show that the performance of dialect classification models remains stable over time but that there is significant spatial variation in performance within dialect areas. ## related work early work showed that part-of-speech trigrams are able to distinguish between some regional dialects @xcite , a method that continues to appear in recent work @xcite . similar methods have been used for authorship analysis @xcite and for characterizing immigrant populations @xcite . in other contexts, non-syntactic features can out-perform syntactic features for modelling dialects @xcite , so that many approaches to distinguishing between dialects are similar to language identification models @xcite . more recent work has modelled geographic syntactic variation by combining grammar induction with geospatial text classification @xcite . the use of grammar induction to learn a syntactic feature space mitigates the fact that most grammars represent standard varieties @xcite , thus poorly representing many dialects around the world. in this paradigm, the learned grammar provides a feature space (c.f., section 4) and the frequency of grammatical constructions in each sample is used to model dialects: a bag-ofconstructions approach to text classification. most work on geographic variation is focused on lexical variation @xcite and change @xcite . recent work has shown a close correspondence between lexical variation in tweets and lexical variation in a dialect survey @xcite . this work is important for showing that digital usage mirrors face-to-face usage. other work has shown that geographic variation can be taken into account during language identification to ensure the inclusion of non-standard varieties @xcite . models of lexical variation have generally failed to account for polysemy, so that competition between senses is not captured @xcite , but more recent work has been able to account for polysemy in this context @xcite . a related line of work uses language data to model non-linguistic properties of populations and places. for example, the problem of geo-location is to predict the location of a user given properties of a document @xcite @xcite . this task differs from dialect classification in that named entities and topic features can provide significant information. a related task is to model the characteristics of a particular place rather than the population of that place @xcite @xcite . while there is a close connection between a place and its population, this line of work remains focused on characterizing non-linguistic attributes. this paper makes two main contributions: first, it experiments with geographic syntactic variation over time and within dialect regions, significantly expanding our understanding of geographic variation in syntax. second, from a more practical perspective, this paper evaluates the degree to which geographic models remain robust over space and time, an evaluation not previously available. ## geographic language data this paper draws on social media data from the corpus of global language use (cglu), using geo-referenced tweets that are identified for language using the idnet package @xcite collection method for social media in the cglu involves geographic searches from co-ordinates of individual cities. here we sample from 1,120 cities representing 12 countries and six regions, as shown in within each group we include three regions, each with two country-level varieties. as shown in the outer-circle group also contains three regions: africa (ghana and kenya), south asia (india and pakistan), and southeast asia (malaysia and the philippines). the collection of data from these countries is distributed across 553 cities, with a comparable sample of 8.37 million words for each month across the training and testing periods. to maintain a comparable geographic distribution over time, we maintain the same number of samples from each city. this means, for example, that the relative influence of brisbane and perth in australia remain constant over time. a sample for the purposes of this paper is an aggregation of individual tweets from the same place and time until the sample reaches 500 words. these larger samples provide more syntactic information for each dialect than do individual tweets. while previous work has used samples of 1,000 words @xcite , here we use smaller samples in order to increase the capacity for error analysis. as with many tasks, there is a trade-off between the higher accuracy provided by larger sample sizes and the flexibility provided by smaller sample sizes. the distribution of samples across cities is taken from the training period (2018). thus, the density of data by location across time is fixed to represent the density during the training period. this allows us to control for changes in the collection: for example, if wellington began to produce more data in 2021, this change in distribution within new zealand would appear to be syntactic variation while actually reflecting a change in the means of observation. data collection spans from 07-2018 until 12-2021, a period of 42 months. the training period is 2018 and the testing period is 2019 through 2021. the geographic distribution across countries, as shown in ## syntactic representations this section details the main ideas of construction grammar (cxg), including both (i) the grammar induction algorithm used to learn syntactic representations here and (ii) examples of constructions used in the dialect models. the basic approach here is, first, to use grammar induction to learn a grammar and, second, to use the frequency of the constructions in that grammar to undertake geospatial text classification @xcite . cxg can be distinguished from other approaches to syntax given its three core ideas: first, cxg posits a continuum between the lexicon and the grammar rather than a strict separation (for example, into a vocabulary and a set of phrase structure rules). this constructicon contains both lexical items and traditional syntactic structures. for example, a grammar-and-lexicon approach would analyze (a) below as an intransitive sentence by labelling the verb laugh as intransitive. the problem is that verb valency is quite fluid, as shown in (b) and (c). the cxg analysis of this fluidity is that (a) represents an intransitive construction into which laugh is merged and (b)/(c) represent a caused-motion construction into which laugh is merged. thus, the fluidity of the argument structure here is explained by an underlying construction, itself meaningful, which interacts with specific lexical items. (note that the grammar used for modelling dialects does not contain any individual lexical items as constructions). a third main idea in cxg is that constructions are constraint-based representations in which slotfillers are drawn from lexical, syntactic, and semantic categories. each unit in a construction is a slot, separated by dashes in (e)/(f)/(g) above. each slot is defined using a slot-constraint. for example, the intransitive construction in (e) can be represented using only syntactic constraints. in contrast, the caused-motion construction in (f) has two semantic constraints; these are labelled for purposes of exposition as object and location. the construction in (g) is item-specific and idiomatic, so that it can only be described using lexical constraints. the point, then, is that different levels of abstraction are captured in cxg using different types of slot-constraints. this paper draws on previous approaches to the unsupervised learning of constructions @xcite . the first challenge is to build the inventory of lexical, syntactic, and semantic constraints that constructions are built on. here we use the most frequent 100k words across the entire corpus of tweets as the lexicon. the syntactic constraints are drawn from the universal part-of-speech tagset @xcite as implemented by the ripple-down-rules tagger @xcite . the semantic constraints are drawn from fasttext embeddings @xcite clustered into discrete semantic domains using k-means. a complete inventory of these semantic domains is provided in the supplementary material; this approach ignores polysemy in lexical items when defining semantic constraints, using a single representation for each word-form. from the perspective of varying levels of abstractness, syntactic constraints are the most general because they are divided into the smallest inventory of labels (only 14). lexical constraints are the least general, with a lexicon of 100k words. and semantic constraints are in the middle, with an inventory of 1,000 domains. this parameter choice (i.e., using 1,000 semantic domains) results from the desired granularity in domains, falling between the very general syntactic constraints and the very specific lexical constraints. thus, constructions are a sequence of slots, each of which is defined by a slot-constraint. each type of slot-constraint (lexical, semantic, and syntactic) differ in their level of abstractness. for instance, lexically-defined constructions are more idiomatic and item-specific than syntactically-defined constructions. this work relies on a loss function based on minimum description length @xcite and a construction parser with a beam-search strategy @xcite ) that operates on top of a psychologically-plausible association measure, the ∆p @xcite . the contribution of this paper is to analyze syntactic variation across space over time using previous work on computational cxg; thus, we do not provide a fuller description of the framework here. previous work has shown that these grammars converge onto stable representations as the amount of training data is increased @xcite , that grammars of individuals are significantly different than grammars of groups of individuals @xcite , and that transformer-based language models can be fine-tuned using constructional information @xcite . the grammar used in these experiments is learned from the training period (2018) but includes a wider pool of 18 english-speaking countries in order to provide a global grammar of english. this larger training corpus for grammar induction contains 478 million words. the fasttext embeddings are trained on this same extended corpus, but covering the entire period in order to increase the amount of data available for training; this larger corpus contains 4.2 billion words. this results in a single grammar that contains 6,119 individual constructions, some of which are shown in (h) through (n) below. dialect models are learned by parsing each sample using this grammar, counting the frequency of each construction in each sample, and using the resulting feature space for dialect classification. the complete grammar, along with examples from the training data for each construction, is available in the supplementary material. the following examples illustrate the nature of constructions; both constructions like (h) and examples like (h1) are drawn from the grammar used in the experiments. each slot in (h) is separated by dashes and each slot-constraint is defined using lexical (lex), syntactic (syn), or semantic (sem) categories. lexical constraints are words given in italics; syntactic constraints are drawn from part-of-speech tags; and semantic constraints are formulated using numbers that refer to clustered embeddings, such @xmath0. for dialect classification, each construction (h) provides a feature and the frequency of that construction (h1 through h3) provides a sample-specific quantification. (h1) 'it is set' (h2) 'it was shut' (h3) 'it can go' the first example, in (h), shows a simple clause with an expletive it as subject and a variable auxiliary verb. the example in (i) is a lexicallyconstrained noun phrase with ability as the head of an infinitival verb. a further lexically-constrained noun phrase in (j) shows the importance of a tweetspecific grammar: ur replaces the more traditional your as the pronoun. the adposition phrase in (k) contains a semantic constraint on the complement noun, in this case a type of location. as an example of how constructions themselves can be meaningful, (l) shows a copula construction with an ending conjunction. but the construction as a whole marks a caveat on the evaluation that is expressed by the copula. the more complicated verb phrase in (m) contains a main verb, myself as a direct object, and an infinitival verb. this implicitly constrains the main verb to verbs of thinking like compare and tell, showing that implicit semantic constraints arise from interactions between slots. finally, the complex noun phrase in (n) reflects a specific template of np + adp. in this way, constructions capture grammatical units of varying size and abstractness. (m) [syn:v -lex:myself -lex:to -syn:v] (m1) 'allowing myself to hope' (m2) 'forcing myself to sleep' (m3) 'tell myself to stop' (n) [lex:the-syn:n -lex:of -syn:det -syn:n] (n1) 'the happiness of another person' (n2) 'the owner of the station' (n3) 'the masters of the game' this section has presented cxg as a paradigm for usage-based syntax and reviewed previous work on computational cxg. an unsupervised construction grammar is learned from the training period, providing an adaptable feature space that contains structures from many different dialects. as the discussed examples show, these learned constructions provide a rich syntactic feature space for modelling geographic variation in production over time. ## dialect models the task of dialect classification or identification is to predict the location of origin for the author of a sample given some set of linguistic features. the classification here predicts country-level dialects like new zealand english or australian english. from the perspective of linguistics, dialect classification allows us to study variation in a highdimensional space: variation across an entire grammar @xcite rather than variation in individual and independent features @xcite . from the perspective of nlp, dialect classification is part of the general problem of ensuring that language technology represents the world's population rather than privileged sub-sets of the world's population @xcite . because part of the goal is to model spatiotemporal variation in the grammar, a dialect model takes the form of a matrix in which each feature (a construction in the grammar) is a row and each dialect (a country-level label) is a column. this matrix represents the degree to which a given part of the grammar is subject to geographic variation. taken row-wise, this matrix provides a measure of whether a particular construction varies across space. and, taken column-wise, this matrix provides a description of each dialect that, for example, can be compared with every other dialect to determine which are the most similar. as discussed below, dialect models are implemented as linear svms that are trained using a bag-of-constructions approach in which the parser counts how many times each construction occurs in each sample. using the data from 2018 for training, we compare three models: first, a syntactic model based on the frequencies of the constructional features described above. second, a baseline model that uses the frequency of function words like of or was, a common baseline for problems in authorship analysis @xcite @xcite when content words need to be avoided. third, for the purpose of comparison, we include a unigram lexical model with tf-idf weighting and function words removed so that it contains no syntactic information. each of these models are implemented as a linear svm. within this task, svms remain competitive, as shown by recent shared tasks on romanian dialect identification @xcite and on identifying similar uralic languages @xcite . in each case, we use a development set to determine parameters. in each case, we train three models: inner contains only inner-circle varieties like american english; outer contains only outer-circle varieties like indian english; and all contains all 12 varieties. these are trained on the data from 2018 and tested on data from 2019, 2020, and 2021. the reason for maintaining separate models in some conditions is that inner-circle varieties have significantly more training and testing data available, which could lead to higher performance as an artifact. thus, for example, the inner-circle condition contains only training and testing data from the six countries listed as inner-circle in as an initial analysis, the f-scores of each of these three models over time is shown in figure 1 , with the y-axis indicating the weighted f-score and the x-axis indicating time. all three classifiers are well above the majority baseline. the lowest performing is the function word model, a weak approximation for syntactic variation. the highest performing is the lexical model. this hierarchy remains stable across the three year testing period. au ca ie nz australia canada ireland nz australian canadian irish zealand mate ontario dublin auckland melbourne trump cork jacinda sydney toronto limerick te abc vancouver galway kiwi brisbane trudeau lads liked labor km hurling lincoln nsw kpa county hamilton turnbull alberta final kph given the results in figure 1 , could we use the lexical model to examine dialects? the issue, as in previous work, is that the information contained in this model does not represent linguistic variation. ## syntactic variation over time we begin the analysis by looking at the weighted average f-score by model for the beginning of the test period (2019-01) and the end (2021-12), as shown in second, we notice in the decay rate represents the overall trend for a given dialect but it does not take into account the specific errors made. the confusion matrix for each dialect provides a monthly representation of the distribution of false positive errors. for example, in the cxg model that includes all dialects, canadian english has 1,488 false positives as american english in the first test period, but only 48 with india and 7 with pakistan. this distribution of false positive errors over time provides a more detailed view of the classifier's performance. because the classification model itself does not change after training, changes in the distribution of errors reflect changes that have arisen in a given dialect after training. the question here is whether the relationship be-tween dialects (geographic variation) changes over time. we model this using a vector error correction model (vecm: @xcite . this model checks for relationships between multiple time series, which in this case reflect changing error patterns between dialects. the data represents a non-stationary time series because the number of errors in all dialects increases over time (i.e., there is a decline in performance as shown in to partially control for the increase in errors over time, we examine the relative frequency of false positives by country by month. the vecm model allows us to determine if there is a significant longterm trend in the distribution of errors from a given dialect, robust to short-term variations. we examine the significant changes by country for the inner-circle and outer-circle models with cxg features in ## syntactic variation within countries this measure has values closer to 1 when the variable is highly spatially organized and closer to 0 when there is no spatial organization. given that there are different numbers of samples from each location, it is possible that a generic moran's i would view sparse locations as outliers; thus, we use the empirical bayes rate adjustment to control for the level of precision in each location as well @xcite . the table also shows the mean accuracy across cities and the min and max accuracy. these results show that there is an effect for location: the dialect models work well in some places and not so well in others. the moran's i determines whether this variation in performance is spatially structured. because different locations represent different populations, these are measures of how well the dialect models work for the entire population of a country. full maps and spatial results are available in the supplementary material. all countries have a significant spatial pattern to their accuracy distribution. within inner-circle countries, canada has the highest deviation, with a wide range in accuracy and a significant spatial structure to that variation. the us and uk have the highest accuracy, while nz performs much worse than other dialects, perhaps because of the change over time discussed above. to explore this further, we visualize the internal variation for nz, the innercircle dialect with the lowest performance and the most change over time, in figure 3 . each collection point is a dot and the shading in the surrounding radius represents the accuracy for that collection area. darker colors represent higher accuracy. the main cities (auckland, wellington, christchurch) have the most consistent performance. but areas with known distinct linguistic landscapes like northland (far north) and southland (far south) have much lower accuracy. more rural areas around the country have consistently lower accuracy as well. the main point in this spatial error analysis is that, because different locations represent different populations, the observed variations in accuracy show that these dialect models do not equally represent all populations within the country. ## conclusions this paper has shown that syntax-based dialect classifiers can reveal both spatial and temporal patterns in linguistic variation. we find that the models remain robust over time, with a fixed decay rate, with the exception of change observed in nz. this means that, while classification performance does decline, the rate of decline is predictable and evenly distributed. within dialect regions, however, there is a significant spatial effect on performance. this evaluation is important for establishing an understanding of how dialect models and other geographic models function in the face of on-going linguistic change and population change over space and time. here, even the best dialect models do not equally represent all speakers of a dialect.