Dataset Geography: Mapping Language Data to Language Users (2022)

Abstract: As language technologies become more ubiquitous, there are increasing efforts towards expanding the language diversity and coverage of natural language processing (NLP) systems. Arguably, the most important factor influencing the quality of modern NLP systems is data availability. In this work, we study the geographical representativeness of NLP datasets, aiming to quantify if and by how much do NLP datasets match the expected needs of the language speakers. In doing so, we use entity recognition and linking systems, also making important observations about their cross-lingual consistency and giving suggestions for more robust evaluation. Last, we explore some geographical and economic factors that may explain the observed dataset distributions.

## introduction the lack of linguistic, typological, and geographical diversity in nlp research, authorship, and publications is by now widely acknowledged and documented @xcite @xcite . nevertheless, the advent of massively multilingual models presents opportunity and hope for the millions of speakers of under-represented languages that are currently under-served by language technologies. broadening up the nlp community's research efforts and scaling from a handful up to the almost 7000 languages of the world is no easy feat. in order for this effort to be efficient and successful, the community needs some necessary foundations to build upon. in seminal work, @xcite provide a clear overview of where we currently stand with respect to data availability for the world's languages and relate them to the languages' representation in nlp conferences. choudhury and edu/project/datasetmaps/. deshpande (2021) study how linguistically fair are multilingual language models, and provide a nuanced framework for evaluating multilingual models based on the principles of fairness in economics and social choice theory. last, @xcite provide a framework for relating nlp systems' performance on benchmark datasets to their downstream utility for users at a global scale, which can provide insights into development priorities; they also discuss academic incentives and socioeconomic factors that correlate with the current status of systematic cross-lingual inequalities they observe in language technologies performance. these works provide insights into current data availability and estimated utility that are paramount for making progress, as well as an evaluation framework for future work. however, there is one missing building block necessary for real progress: a way to estimate how representative of the underlying language speakers is the content of our datasets. any evaluation framework and any utility estimates we build can only be trustworthy as long as the evaluation data are representative. @xcite and @xcite recognize the importance of this information, including them in their proposed guidelines for "datasheets" and "data statements" respectively; but most datasets unfortunately lack such meta-information. to the best of our knowledge, marvl @xcite is the only dataset that is culturally-aware by design in terms of its content. 2 we propose a method to estimate a dataset's cultural representativeness by mapping it onto the physical space that language speakers occupy, producing visualizations such as figure 1 . our contributions are summarized below: ## mapping datasets to countries assumptions this work makes two assumptions: that (a) data locality matters, i.e., speakers of a language are more likely to talk about or refer to local news, events, entities, etc as opposed to ones from a different side of the world, and (b) that we can capture this locality by only focusing on entities. @xcite discuss these topical correlations that are present in datasets, 3 noting that they exist and that l1 language identification models tend to pick up on them, i.e. if a text mentions finland, a l1 langid model is probably going to predict that the speaker is finnish, because p(finland @xmath0. in that work @xcite make explicit effort to avoid learning such correlations because they are interested in building models for p(l1 text) (i.e. p(@xmath1. the mere fact they need to do this, though, confirms that realworld text has such topical confounds. as for our second assumption that we can capture these topical correlations by only looking at entities, one need only to take a look at approach for a given dataset, our method follows a simple recipe: 1. identify named entities present in the dataset. 2. perform entity linking to wikidata ids. 3. use wikidata to link entities to countries. we discuss each step below. ## dataset-country maps before delving into our case studies, we first list a set of statistics of interest that one could extract from our produced dataset-country maps, in order to gauge a dataset's representativeness. representativeness measures we will avoid providing a single metric, largely because the ideal metric to use will be very dataset-specific and related to the goals of the creators of the dataset and the socioeconomic correlates they are interested in (see discussion in section §3.3). as a first straightforward representativeness measure, we will compute the percentage of entities associated with countries where the language is largely spoken. for example, according to ethnologue @xcite , most swahili speakers foot_3 reside in tanzania, kenya, uganda, dr. congo, and rwanda. for a swahili dataset, then, we compute the percentage of all entities associated with this set of countries ("in-country"). notions of equity or fairness across countries could be measured by various fairness metrics, given the distribution of entities over countries in a dataset: from simply computing the standard deviation of the observations, foot_4 to treating countries as a population and computing fairness indices like the popular gini index @xcite or the indices proposed by @xcite . we will opt for a simpler, much more interpretable measure, the number of countries not represented in the dataset i.e. countries with associated entity count below a given threshold (we use zero for simplicity but higher values would also be reasonable for large datasets). last, especially for languages with significant amounts of speakers in more than one country, it is important to go deeper and measure the representativeness of this in-country portion. for a simple example, an english dataset with entities only from the uk is probably not representative of nigerian or jamaican english speakers. hence, we will create two distributions over the countries where the language is largely spoken: the distribution of speaker populations (as available from ethnologue and other public data), and the distribution of entities observed in the dataset. discrepancies between these two distributions will reveal potential issues. while one could easily compute some measure of distance between the two distributions (e.g. the bhattacharyya coefficient @xcite ), in this work we will rely on the interpretable advantages of the visualizations. measures of fairness could be computed for this portion of the dataset, similarly as discussed above. in the example dataset of the swahili portion of masakhaner in figure 1 , the utility of our method is apparent. through the visualization, a researcher can quickly confirm that the dataset seems to not reflect the users of the language to a large extent: only about 17% of the entities indeed correspond to tanzania, kenya, uganda, dr. congo, or rwanda (where swahili and its varieties are treated as a lingua franca, at least in portions of these countries). wealthy or populous countries like usa, france, and china, are well-represented, foot_5 as one would expect, while 156 countries and territories have no representation. at the same time, the visualization allows a researcher to identify gaps: ## bypassing ner for entity linking we use mgenre @xcite for the task of multilingual entity linking, a sequence to sequence system that predicts entities in an auto-regressive manner. it works particularly well in a zero-shot setting as it considers 100+ target languages as latent variables to marginalize over. typically, the input to mgenre can be informed by a ner model that provides the named entity span over the source. for instance, in the italian sentence "[start] einstein [end] era un fisico tedesco." (einstein was a german physicist.) the word einstein is enclosed within the entity span. mgenre is trained to use this information to return the most relevant wikidata entries. due to the plasticity of neural models and mge-bre's auto-regressive token generation fashion, we find that by simply enclosing the whole sentence in a span also yields meaningful results. in particular, for the previously discussed italian sentence now the input to mgenre is "[start] einstein era un fisico tedesco. [end]". the advantage of this approach is two-fold. first, one does not need a ner component. second, exactly because of bypassing the ner component, the el model is now less constrained in its output; in cases where the ner component made errors, there's a higher chance that the el model will return the correct result. ## conclusion we present a recipe for visualizing how representative nlp datasets are with respect to the underlying language speakers. we plan to further improve our tool 17 by making ner/el models more robustly handle low-resource languages. we will also expand our dataset and task coverage, to get a broader overview of the current utility of nlp systems.