Representations of Language Varieties Are Reliable Given Corpus Similarity Measures (2021)

Abstract: This paper measures similarity both within and between 84 language varieties across nine languages. These corpora are drawn from digital sources (the web and tweets), allowing us to evaluate whether such geo-referenced corpora are reliable for modelling linguistic variation. The basic idea is that, if each source adequately represents a single underlying language variety, then the similarity between these sources should be stable across all languages and countries. The paper shows that there is a consistent agreement between these sources using frequency-based corpus similarity measures. This provides further evidence that digital geo-referenced corpora consistently represent local language varieties.

## introduction this paper evaluates whether digital geo-referenced corpora are consistent across registers (web data vs. tweets). this question is a part of validating the use of digital data to represent local language usage in specific places around the world. in other words, recent work has taken samples of varieties like new zealand english from social media (twitter) and the web (common crawl). but how closely do these two sources actually match up, both with one another and with new zealand english in offline settings? the first question is about the reliability of geo-referenced corpora and the second is about their validity. the question of reliability and validity becomes important when we use geo-located digital corpora to model both lexical variation @xcite @xcite @xcite and the difference between language varieties @xcite @xcite @xcite . these computational approaches are quickly refining our un-derstanding of language variation and change. at the same time, though, both of these lines of work depend on large, reliable sources of geographic language data. this paper evaluates these data sources across nine languages and 84 language varieties in order to determine how reliable they are. the data can be explored further alongside demographic information at earthlings.io. ## reliability and validity if digital geo-referenced corpora are reliable, then both web corpora and twitter corpora from one location should exhibit similar patterns (as quantified by corpus similarity measures). in other words, no matter how many samples we observe of tweets or web pages from new zealand, we expect a reliable similarity among samples. if digital corpora are not reliable, then it follows that they are not suitable for modelling linguistic variation. this is because a model of a variety like new zealand english would depend on arbitrary differences in the specific samples being obeserved. the further question is whether digital georeferenced corpora are valid representations of an underlying population. in other words, if we observed lexical choices by interviewing speakers in new zealand, we would expect those same lexical choices to be observed in tweets. if these digital corpora are not reliable, then they are also not valid. but reliability itself does not guarantee a valid representation of the underlying population @xcite . for example, it is possible that register differences lead to different linguistic behaviour, so that lexical choices in written language may never correspond to lexical choices in spoken language. if this were the case, then it would of course never be possible to generalize from patterns in written language to patterns in spoken language. ## related work on validity there have been three distinct approaches to the broader problem of validating digital representations of dialects and varieties. first, a linguistic approach starts with survey-based dialect studies as a ground-truth. these dialect surveys provide a point of comparison for an underlying variety like american english, one which does not rely on digital language data at all. this line of work has shown that there is a strong correspondence between traditional studies of dialectal variation and both tweets @xcite and web-crawled corpora @xcite . this family of results shows that, if we accept dialect surveys as a gold-standard, there is significant justification for substituting digital corpora (like tweets) for much more expensive survey-based data. the correspondence between dialect surveys and tweets, however, is far from perfect. this raises a question: when is the relationship strong enough to justify a purely digital approach to language varieties? a second demographic approach compares digital corpora with ground-truth census data in order to triangulate the similarity between represented populations. for example, given a corpus of tweets from around the world, the geographic density should correspond to some degree with population density @xcite . otherwise, some locations would be over-represented in the data, thus given more influence. in the same way, a corpus drawn equally from around the world should represent local language use that corresponds to some degree with a census-based study of what languages are used in each country @xcite . the basic idea here is that geo-referenced corpora which correspond with census data are more likely to represent actual local populations, rather than representing non-local populations like tourists or short-term workers @xcite . this line of work has also shown a strong relationship between demographics and digital corpora. but, again, when is that relationship strong enough to justify using geo-referenced corpora alone to model language varieties? a third computational approach leverages some model of language variation to measure relationships between different partitions of linguistic data. for example, recent work has measured the linguistic similarity between varieties of english, like new zealand english vs. australian english @xcite . when expanded across aligned corpora, these models can be used to determine if there is a consistent pattern of variation: does new zealand english have the same distinctive lexical choices on the web that it has in tweets? the basic idea is that if two sources of data (the web and tweets) accurately represent new zealand english, both should produce similar models of that variety. on the one hand, even if the models agree (reliability) it remains possible that both sources of data are distorting the underlying population (validity). but it is more likely that such reliability confirms the validity of a purely digital approach to language varieties. this kind of work has also been systematically evaluated across different similarity measures, with a ground-truth derived from studies in perceptual dialectology @xcite . this paper expands on these three approaches by systematically experimenting with the reliability of geo-referenced data (i) across register boundaries, (ii) across geographic boundaries, and (iii) across language boundaries. the more reliable the relationship between web data and tweets, the more confidence we can have in computational models of language varieties and linguistic variation. ## measuring corpus similarity the approach in this paper is to measure where digital geo-located corpora diverge from one another, across languages and across countries. unlike previous work, this does not rely on a comparison with an external ground-truth (i.e., dialect surveys, census data, or pre-trained models of variation). first, we collect geo-referenced web data and tweets across 84 varieties of nine languages. each variety is represented using aligned corpora drawn from web pages and from tweets. these representations are distributed across many individual samples of 1 million words each. this design allows us to make a direct comparison between samples that have been drawn from the same varieties. thus, for example, we can measure the similarity of a single variety across register boundaries: new zealand english on the web and in tweets. and we can also measure the similarity between varieties within a single register: new zealand english and australian english in tweets. this is related, for example, to the problem of measuring similarity within a single language variety over time @xcite paper is to examine the reliability of digital representations of language varieties. we start by drawing on measures of corpus similarity, which calculate the distance between two corpora @xcite . previous work has shown that frequencybased measures out-perform more sophisticated approaches that make use of language models or topic models @xcite . the highestperforming frequency models work by ranking word or character frequencies and using either the spearman rho or the χ 2 measure between frequency ranks. the χ 2 performs slightly better for samples with the same number of features, but the spearman rho has more flexibility for dealing with unattested features while experiencing only a minimal impact on performance @xcite . based on these previous evaluations, we rely on two variants of the spearman frequency-based similarity measure: one variant uses unigram frequencies and one variant uses character trigram frequencies (motivated by related work on language identification). the experiments in this paper are performed using equal-sized samples, where each sample contains 1 million words. these two measures are labelled as word and char in the tables that follow. the inventory of features (i.e., words or character trigrams) consists of the most common 100k features across all samples from a given language. a higher spearman value indicates a higher similarity between samples. in the ideal case, there would be a consistent level of similarity across samples representing all languages and all countries. in other words, we would see a similar impact of register variation (web vs. tweets) regardless of which specific language variety is involved. a result like this would suggest that both sources of geo-located language are representing a single underlying linguistic object (i.e., offline new zealand english). the alternate outcome is that there is an arbitrary fluctuation between web usage and tweet usage. for example, it could be the case that new zealand english is very similar on the web and in tweets, but australian english shows a wide divergence between the two, with different lexical choices being represented. if this were the case, the digital representation of new zealand english would be more reliable than the representation of australian english. this work is related to previous explorations of noise in social media @xcite , but with a focus on the interaction between geographic variation and register variation. the question is: how consistent are language varieties when sampled from different digital sources? ## data and experimental design the experiments draw on existing geographic corpora from the web (ultimately derived from the common crawl data) and from tweets @xcite . both of these data sources are aligned by language and country: for example, each variety like cuban spanish or new zealand english is present from both sources of data. the experiments include nine languages: arabic (ara), german (deu), english (eng), french (fra), indonesian (ind), dutch (nld), portuguese (por), russian (rus), and spanish (spa). these languages each have a different number of varieties, as shown in each sample contains 1 million words. some varieties are very well represented, with hundreds of samples (and thus hundreds of millions of words); but other varieties are less well represented, with only a few samples. we restrict the maximum number of samples per variety per register to 20. as shown in the first experiment (section 6), focuses on the internal consistency of each variety-register combination: given similarity relationships between many samples of mexican spanish tweets, how stable is the internal similarity between samples? the basic idea is that any variety-register combination with a low internal similarity is less reliable. the second experiment (section 7) evaluates the accuracy of both word-frequency and characterfrequency measures. the idea in these experiments is to validate the underlying similarity measure to ensure that the later experiments are also valid. the third experiment (section 8) is the main part of the paper: measuring the difference between registers (web vs. tweets) for each language variety. our hypothesis is that only a significantly stable relationship supports the use of digital corpora for representing language varieties. the fourth experiment (section 9) looks for differing results by language. it may be the case, for instance, that varieties of english and spanish are adequately represented by digital corpora, but not varieties of dutch. the fifth experiment (section 10) creates an average frequency vector for each language across all varieties and registers. this average is used to situate each national variety according to its relative distance from that center, creating a rank of which varieties are the most divergent or the most unique. there is one rank for each register and the experiment tests where there is a significant relationship between ranks for each language. taken together, this set of experiments is designed to evaluate the suitability of digital geo-referenced corpora for representing language varieties, building on the validation work discussed in section 3. ## experiment 1: consistency across samples the first question is the degree to which each language variety-register combination (e.g., mexican spanish tweets) is internally consistent. we take 50 observations of unique pairs of samples, each representing the same variety-register combination. then we measure the similarity between the two samples and look at the distribution of similarity values for each variety-register combination. varieties with a high average similarity are more internally consistent than varieties with a low average similarity. the results are shown in first, this table shows that character representations are more consistent across languages, although the accuracy of each type of feature is not evaluated until section 7. in other words, the average internal similarity for tweets ranges from 0.82 to 0.85 for character features but from 0.63 to 0.71 for word features. at the same time, a smaller range of variation also could mean that character features are less able to discriminate between varieties. second, this table shows that tweets from a particular variety have a higher internal similarity than web data from that variety. this is true for all languages and for both word and character features. while this experiment does not indicate why this is the case, we can offer some potential reasons: (i) it could be that tweets are more similar in terms of discourse functions, while web data represents a range of different communication purposes; (ii) it could be that tweets represent a more restricted subset of local populations (e.g., twitter users), which have been shown to be more influenced by demographic attributes like per capita gdp. third, all of these spearman correlations are highly significant, which means that there is at least a strong relationship between representations of each variety drawn from the web and from tweets. at the same time, as discussed in section 3 in regards to alternate methods of evaluation, it is not clear what threshold for similarity is sufficient. this question is what the remainder of the experiments attempt to address. ## experiment 2: word vs character features the next question is two-fold: first, which frequency-based measure of similarity is more reliable in this setting: words or characters? previous in this experiment we look at the average accuracy for each language for each type of feature. for each variety, we take 100 pairs that come from the same register (tweets against tweets) and 100 pairs that come from different registers (tweets against web pages). if the similarity for a pair is closest to the average similarity for same-register pairs, we predict that each sample comes from the same register (and so on for cross-register pairs). this is a simple framework for evaluating each measure: the percent of correct predictions. this is similar to previous evaluation methods for corpus similarity measures @xcite . the results, shown in ## experiment 3: differences in register now that we have established that there is a significant relationship between registers across all varieties and that the character-based measure is generally more accurate, we proceed to the main question: how robust is the relationship between registers? there are two competing hypotheses figure 2: cross-register similarity (german) figure 3: cross-register similarity (english) moving on to german, in figure 2 , we see a similar pattern: the distribution (with somewhat higher scores) is centered within the same values across countries. what matters for the hypothesis in this paper is the range of values for cross-register similarity, here with averages ranging from 0.52 (czechia, cz) to 0.56 (germany, de). this constrained range indicates that there is a stable relationship between digital sources for these varieties. we next evaluate varieties of english (figure 3 ) and french (figure 4 ). both of these languages are more widely used in digital contexts, thus having a larger number of varieties. english ranges from an average of 0.43 (hong kong, hk) to 0.51 (nigeria, ng). french ranges from an average of 0.42 (algeria, or dz) to 0.52 (gabon, ga, and the democratic republic of congo, cd). in both languages we see a somewhat wider range, with outliers like hong kong english or algerian and tunisian french; but still the gap between registers remains largely consistent for most varieties. the results for russian (figure 5 ) and spanish (figure 6 ) continue the same pattern of relative consistency. the figures for dutch, indonesian, and portuguese are excluded because they each have a small number of varieties that largely overlap. the figures for russian and spanish show greater consistency across countries, although russian does have a few outliers for each country (i.e., the narrow tails). russian averages range from 0.49 (ukraine, ua) to 0.52 (estonia, ee). spanish shows a slightly larger range of averages, from 0.48 (argentina, ar) to 0.55 (el salvador, es). once again, however, the relationship between registers remains stable. the results in this section suggest that both web data and tweets provide reliable representations for a large number of language varieties, regardless of the specific language or country involved. the distance between samples from each source is stable across both categories (language and country). this pattern is what we expect if both sources adequately represent the underlying language variety. this reliability converges with the previous work discussed in section 3 to suggest that digital georeferenced corpora do in fact represent local language use. as with previous evaluations, however, it is difficult to define a threshold for reliability. ## experiment 4: differences by language our next set of experiments further explores the interaction between dialect variation (e.g., new zealand vs. australian english) and register variation (e.g., web data vs. tweets). as we compare a large number of samples for each language, how much of the variation comes from register-effects and how much from geographic-effects? the experiments shown in this geographic variation is a secondary finding, however, in the sense that much more precise methods of modelling geographic variation exist. the purpose here is simply to confirm that each language shows a significant geographic effect given this measure applied to these sources of data. ## experiment 5: relations between varieties our final experiment looks at relationships between national varieties or dialects. a final piece of evidence comes from ranks of varieties: if both the web and tweets make singapore english most similar to malaysian english, this confirms the reliability digital sources of geo-referenced data. first, we create an average representation across all samples for a given language using the mean feature frequency. second, we estimate the average feature frequency for each variety-register combination (like mexican spanish tweets). third, we compare the similarity between (i) the languagebased mean feature value and (ii) the average for a specific variety (like mexican spanish tweets). this provides a rank of distance from the average frequency for each variety. for example, what is the most unusual variety of english? we do this for both registers, tw and cc in the results for all languages taken together is highly significant, with a correlation of 0.453. taken individually, some languages are also significant: german, french, portuguese, and russian. given the small number of observations when constrained to individual languages, the lack of significance at that level is not particularly important. what this shows, however, is that even a simple similarity measure can produce, overall, significantly related ranks of varieties. the importance is not the specific ranks, because we know that better methods for modelling language varieties exist. the importance is that we have additional evidence for stability in the relationship between registers and varieties. this is precisely what we would see if both sources of digital data were valid representations of a single underlying linguistic object. ## geographic trends the focus so far has been on varieties of languages. a different way of framing the question of register similarity is by country: is there a geographic pattern to where the web and tweets are more or less similar? here we take cross-register similarity values as described above in section 8. when we bring all languages together, these experiments represent 66 countries. of these, 10 are represented by two or more languages; the map in figure 7 averages the values by country to show geographic trends in cross-register similarity. this kind of geographic analysis does not show larger trends. for example, one common trend in digital corpora is that wealthy western countries like the us, canada, western europe, and australia and new zealand pattern together. but this is not the case here, with canada and new zealand contrasting with the us and australia. thus, a geographic analysis further confirms the reliability of geo-referenced corpora in the sense that what little variation we do see does not correspond with expected geographic patterns. ## conclusions the experiments in this paper have shown that there is a consistent relationship between the representation of language varieties across two sources of geo-referenced digital corpora: the web and tweets. in other words, these digital corpora are reliable. this finding is important because it disproves the alternate hypothesis that one or both sources is a poor or arbitrary representation of the underlying language varieties. this kind of unreliability would lead to an inconsistent relationship between registers. reliability is not the same as validity. however, the reliability shown here converges with evidence from dialect surveys and census data and computational models of language varieties, all of which do evaluate the validity of digital corpora. this robust reliability gives us confidence in the use of digital corpora to study linguistic variation. and this evaluation of both reliability and validity is important because it justifies the use of computational modelling which, in turn, enables globalscale experiments that help us better understand language variation and change.