Measuring Linguistic Diversity During COVID -19 (2020)

Abstract: Computational measures of linguistic diversity help us understand the linguistic landscape using digital language data. The contribution of this paper is to calibrate measures of linguistic diversity using restrictions on international travel resulting from the COVID-19 pandemic. Previous work has mapped the distribution of languages using geo-referenced social media and web data. The goal, however, has been to describe these corpora themselves rather than to make inferences about underlying populations. This paper shows that a difference-in-differences method based on the Herfindahl-Hirschman Index can identify the bias in digital corpora that is introduced by non-local populations. These methods tell us where significant changes have taken place and whether this leads to increased or decreased diversity. This is an important step in aligning digital corpora like social media with the real-world populations that have produced them.

## data sources we draw on twitter data sampled globally from 10k cities over a 25-month period (july 2018 through august 2020). this city-based collection reduces production bias from the start (as opposed to collecting data by user or search term) because it forces non-central cities to be included. the cities are selected to represent the global population and all retweets are removed. this provides 623 million tweets, distributed across regions as shown in the idnet language identification package is used to provide language labels @xcite . any tweet under 40 characters (after cleaning urls and hashtags) is removed because of reduced identification accuracy below this threshold. the average tweets per month per country is visualized in figure 1. because we are looking at change over time by country, the data is binned into potentially small categories (e.g., nigeria in july 2019). both the table and the map show that countries in east asia are under-represented. thus, we use significance testing within countries when looking for change over time. ## demographics and language use the next question is the degree to which the production of this data is driven by underlying populations (potential production bias) and by demographic factors like gdp (potential selection bias). we start, in figure 3 , by looking at the relationship between each country's population and share of the corpus. this expands on the region aggregations in to better understand the factors influencing the geographic distribution of the data, we work with three variables: population, the number of people in each country; internet population, the number of internet users in each country; and gdp, a measure of each country's economic output @xcite . figure 3 shows three regression plots in which these variables (on the y axis) are compared with the average monthly data production per country (given in number of tweets per month on the x axis). in each case, there is a close relationship between data production and demographics, with several extreme outliers. for population, the outliers are china and india. both are highly populated countries with significantly lower than expected data production (especially china). both countries have relatively low rates of internet access: 38% for china and 11% for india; this lowers the total population in each country. thus, although the populations are quite large, most of the population is not able to produce digital language data. for the influence of gdp, the outliers are the us and china. for the us, in particular, the gdp is quite high: there seems to be a ceiling after which increased gdp is unlikely to influence digital behaviours. further, that gdp is not evenly distributed across the entire population. for the influence of internet access, the outliers are again china and the us. with a few notable exceptions there is a relatively close relationship between data production and the demographic factors of each country. with these three outliers removed (the us, china, india), there are very significant correlations between these three variables and the geographic distribution of the data: 0.46 (population), 0.61 (population with internet access), and 0.59 (gdp). this leaves some unexplained production factors. the most obvious missing factor here is social media platforms specific to given countries (e.g., sina weibo). these alternative platforms will siphon away enough users to distort the representation of a population given access only to other platforms. further, twitter is banned in china: because only some companies are allowed to use it through specific vpns, the text is not representative of language use in china. casual users of twitter will use a vpn through another country which would distort this method of data collection. regardless, this section has shown that we can explain a significant portion of the geographic distribution of the data. this is important because we want to describe populations by observing digital corpora. if there is no relationship between the two in terms of distribution, it is difficult to make such inferences. what we have seen, however, is that there is a very significant relationship. what is the required threshold for establishing a relationship like this? we should think about this as a metric for evaluating digital corpora: data with a stronger relationship to demographic variables are more representative. the next question is whether this relationship remains stable over time: can we depend on these demographic factors across the entire period? the next question is whether these production factors are stable over time. here we build a baseline for temporal variation: to what degree is the data subject to unrelated fluctuations that will reduce our ability to assign a cause-and-effect relationship to linguistic diversity during travel restrictions? although the same collection and processing methods are maintained over the two-year period, there is variation in the total number of observations (tweets) per month. there are many reasons why this might be the case. what matters to us, though, is the relative share of each country. in other words, the population does not change from month to month in the same way that the number of tweets changes. regardless of the total amount of data collected per month, is the geographic distribution consistent? figure 2 shows stability over time by representing the relative proportion of observations per region by month. western europe is removed for the sake of clarity, as it represents a significantly higher share (roughly 25%). the distribution of samples is consistent over time. the main exception is that, for a two-month period in 2018, there is much more data from oceania. we use a t-test to find out if the share of each region is stable over time. if the distribution changes significantly, then it may be hard to determine the cause of any individual change. none of the regions show a significant fluctuation; this is helpful because it shows that there is not random noise in the data that could interfere with measures of linguistic diversity. the difference-in-differences methods we use in section 8 would control for such noise, but this gives us further confidence. we use a t-test, rather than a time-specific test like dickey-fuller, because we are interested in consistency rather than in non-stationarity. these results show that, in the aggregate, the distribution of samples remains constant. but how much variation within individual countries does this region-based measure disguise? to answer this, we look at the same t-test approach by country: do individual countries vary widely in their relative production? no countries show a significant change. these findings show that we can largely focus on diversity, the distribution of languages within a country by month, rather than on the production of data over month as in section 3. there is natural variation in the data, of course, and this is taken into account in our later approaches. for example, if we compute the correlation between population and language production (as in section 3) for each month in isolation, there is no significant difference over time. this stability is important for creating a baseline against which to understand demographic changes during travel restrictions. because the relationship between demographics and the data set remains stable, we can focus specifically on changes in linguistic diversity. ## measuring linguistic diversity linguistic diversity is an important part of accurate language and population mapping. the goal is to have a single measure that can tell us how much language contact is taking place and which communities are multi-lingual. to do this we must generalize across specific languages: linguistic diversity in the us might involve english and spanish, but it might involve portuguese and spanish in brazil. we measure linguistic diversity as a probability distribution over languages for each country. drawing on previous work on short-sample language identification, this paper includes 464 languages across 157 countries. for each country, then, we have a relatively accurate identification of which languages are used on twitter. given this probability distribution for each country, we compare countries using the herfindahl-hirschman index (hhi) as shown in figure 4 . the hhi was developed in economics to measure market concentration: the more of a given industry is dominated by a small number of companies, the higher the hhi @xcite . the measure is derived using the sum of the square of shares, in this case the share of each language in each country. the higher the hhi (the darker red) for a country, the more one language dominates the linguistic landscape. thus, the hhi is higher when the distribution is centered around just a few languages. for example, in figure 4 shows linguistic diversity across the world: lighter countries (like israel) have a mix of languages while darker countries (like the us) are mostly monolingual. there are many linguistic landscapes around the world, ranging from multilingual to monolingual. this figure 4 is a baseline representation, averaged across the entire time period (july 2018 to august 2020). it is possible that this averaged representation disguises temporal fluctuations. we have already seen that there are only a few changes in the share of data per country per month, and no significant change in the relationship between the data set and demographic factors like gdp. the question here is whether there is arbitrary variation in the linguistic diversity per country per month. in other words, if israel becomes significantly more diverse every three months, it will be difficult to find out what is causing those changes. we use a a t-test for the mean of each country to determine if each country's diversity is actually a single group. there are no significant fluctuations across the period as a whole. ## finding non-local populations to what degree do countries change during travel restrictions resulting from covid-19? we have a measure of diversity (the hhi) and data collected by month. the basic approach is to create two groups of samples: first, months during the pandemic (march through august, 2020); second, months not during the pandemic (march through august, 2019). these two groups are aligned by month so that seasonal fluctuations are taken into account (e.g., tourism high season in february for new zealand and in july for italy). given these two groups of samples, we use a t-test for two independent samples to determine whether these groups are, in fact, different. if we reject the null hypothesis, it means that linguistic diversity during travel restrictions is significantly different than the seasonally-adjusted baseline. the results show that 70 countries have a changed linguistic landscape during the pandemic. this is visualized in figure 5 , with p-values classed into highly significant (under 0.001), very significant (under 0.01), and significant (under 0.05). we see, for example, that the us and canada undergo significant change, but not mexico and south america. there are clear geographic patterns in linguis-tic change: north but not central or south america; east africa but not west africa; south/east asia but not east asia; europe but not russia. we will examine in more detail how and why the linguistic landscape changes in section 7. these significant changes during international travel restrictions show that our measure (the hhi) and our data (tweets) offer a meaningful representation of underlying populations. if the data did not represent populations, we would not see the relationships examined in section 3. there are no random fluctuations in the distribution of the data across countries or in the distribution of languages within countries. at the same time, given a massive social change (i.e., the covid-19 pandemic), the measure clearly identifies changes in the linguistic landscape. thus, the measure is both precise (not disguised by noise) and accurate (observing change where we expect it). the key point is that the change in diversity during the covid-19 period is identifiable against the background noise. a country's linguistic landscape could change by becoming more diverse (i.e., with more languages) or by becoming less diverse (i.e., with fewer languages). which is causing the significant changes that we are observing? figure 6 distinguishes between countries with an increasing hhi (becoming more monolingual) and a decreasing hhi (becoming more multilingual). we can think about two contexts in which this change can take place: a country like india might look more multilingual because non-local tourists who speak english are no longer creating noise in the data; or, a country like south africa might look more monolingual ## identifying out-of-place populations our task now is to use these changes during travel restrictions to identify which populations are outof-place in ordinary times. in other words, if india has decreasing english use during the pandemic period, then we know that english is over-represented in the country as a result of non-local populations. we find these languages by repeating the comparison of pandemic vs. normal periods per country per month, but now we look at the share of individual languages rather than the hhi (in countries with a significant change). we are only interested in languages which account for at least 1% of a country's usage. less commonly used languages may be changing significantly but have less influence on a country's overall linguistic landscape. we start by looking at countries where the use of english falls dramatically during the pandemic period, in the influence of non-local populations returning home is shown for russian and arabic in ## sources of change this paper has shown that there is a significant change in the linguistic diversity of many countries during the travel restrictions caused by covid-19. but to what degree are these changes related to the travel restrictions themselves? for example, we could imagine a population that is changing over time which we just happen to observe in midchange. it could be the case that a country has been becoming less diverse over the past decade because of fewer incoming immigrants; the approach taken so far in this paper would misinterpret such macrotrends to be a direct result of covid-19. we use a difference-in-differences method @xcite to correct for this. the basic idea behind a difference-in-differences approach is to conduct a natural experiment with a control group (here, data from 2018) and an effect group (here, data from 2020) differentiated by time. we have three months (july, august, september) that are shared across 2018, 2019, and 2020. so, using the same methods described above, we find out which countries have a significant change between 2019 and 2020. this is the period that takes place during travel restrictions. if travel restrictions influence linguistic diversity, we would expect such influence to take place during this period. we then find out if the countries which show a significant change in 2020 also show a significant change from 2018 to 2019. this provides a baseline: removing any country whose linguistic diversity was already in the process of changing. over this three-month period (july through september), 58 countries show a change in linguistic diversity during the pandemic. this is a smaller number than the main results reported above for two reasons: (i) the time span is shorter, giving less robust results and (ii) this particular time span came after some travel had resumed. of these 58 countries that show a significant change in diversity, most @xcite show no difference at all in the baseline period before the pandemic. another eight show a much greater difference during the covid-19 period (e.g., p-values of 0.03 vs 0.004 for baseline and covid-19, respectively). this means that the pandemic has either created or has significantly contributed to 79.3% of the cases of changing linguistic diversity. the remaining 20.7% of changes, then, must have been created by macro-trends like immigration or changes in bilingual behaviour. the main conclusion from this difference-in-differences examination, however, is that most of these changes can be specifically connected to covid-19. ## conclusions the goal of this paper is to validate measures of linguistic diversity using changes in underlying populations during the covid-19 pandemic. we have shown that there is a significant relationship between our data and the underlying population. thus, what we are observing (tweets) can tell us about the people we want to study. at the same time, both the distribution of the data across countries and the distribution of languages within countries are stable. thus, the data does not have random fluctuations that will get in the way. using the hhi as a measure of diversity, there is a significant change in the linguistic landscape of 70 countries against a seasonally-adjusted baseline. this reflects non-local populations (e.g., the impact of tourists leaving a country or short-term visitors returning to their own countries). these results validate a measure of linguistic diversity that is based on digital language data and shows that we can correct for the bias introduced by non-local populations.