Geographical Erasure in Language Generation (2023)

Abstract: Large language models (LLMs) encode vast amounts of world knowledge. However, since these models are trained on large swaths of internet data, they are at risk of inordinately capturing information about dominant groups. This imbalance can propagate into generated language. In this work, we study and operationalise a form of geographical erasure wherein language models underpredict certain countries. We demonstrate consistent instances of erasure across a range of LLMs. We discover that erasure strongly correlates with low frequencies of country mentions in the training corpus. Lastly, we mitigate erasure by finetuning using a custom objective.

## introduction large pretrained models serve as base models for many downstream nlp applications, including question-answering, dialogue, common-sense reasoning, classification, tagging, translation, summarisation, and generation @xcite @xcite . despite their increasing utility, there are concerns about how they reflect and amplify biases in the training data. for instance, unfiltered data originating from the internet is known to be rife with toxic, misogynistic, and stereotyping content. many studies highlight biases in model outputs, primarily concerning representational harms @xcite , where a section of society (e.g., women, lgbtq+ communities) are represented in poor light, or are ignored by the system @xcite @xcite tan and celis, 2019). while important, such studies predominantly examine biases related to race, gender, occupation and sexual orientation. an important-and often overlooked-aspect of inclusive model development is geographical inclusion. this is particularly important at a time when most large-scale model training efforts come from a small set of regions. further, these models are trained using internet data, whose access, in the first place, is unequally distributed @xcite . minimising cultural and geo-graphic identities is referred to as erasure @xcite and is studied by linguists and social scientists in the context of imperialism and colonialism, where "people are silenced in the historical record [...], their contemporary presence rendered invisible, and their existence written out of the future" @xcite . automated systems and their developers exclude certain groups unintentionally, but the risk of being "written out of the future" remains pressing: the produced content is fed back into the internet. in lifelong learning setups, the generated content becomes the training data of tomorrow's models, closing the vicious circle of reinforcing social hierarchies (see §4.5 of @xcite ). in this paper, we reveal instances of geographical erasure, wherein models underpredict geographical regions (see fig. 1 ). for instance, gpt2 assigns nine times higher likelihood to "i live in canada" than "i live in pakistan", whereas pakistan's english-speaking population is almost four times to that of canada. by comparing model outputs with population statistics, we operationalise geographical erasure ( §3). using this measure, we first demonstrate the existence of erasure for several countries across different prompt formulations ( §4.1). studying the consistency across a range of language models from the gpt and llama model families, we find that several countries -nigeria, pakistan, eswatini, uganda and madagascar -are affected by erasure under all models ( §4.2). following related work @xcite @xcite , we study the impact of model size on the extent of erasure, and find it to be small -erasure occurs across all sizes ( §4.3). to identify the causes of erasure, we compute the unigram frequencies of countries in the training corpus ( §4.4). they closely match our model predictions indicating that the composition of training data is a main source of erasure. lastly, we alleviate erasure via supervised finetuning. we study the impact of mitigation on generation quality as measured in perplexity on wikitext-2-v1. our finetuning strategy proves to be an effective mitigation mechanism which generalises and has small impact on generation quality ( §4.5). ## related work the literature on fairness in machine learning distinguishes between representational and allocational harms @xcite @xcite . allocational harms concern the unfair distribution of resources, e.g. when a group is denied bank loans disproportionally by an automated system. allocational harms tend to be more easily measured through standard fairness metrics like demographic parity @xcite and equality of opportunity @xcite . those do not directly apply to open-ended generation tasks, where we instead study representational harms, which arise when a system "represents some social groups in a less favourable light than others, demeans them, or fails to recognise their existence altogether" @xcite ; the last case being the focus of our work. fairness measures for language generation usually define bias as differences between demographic groups @xcite . for example, @xcite find that female pronouns are more likely to elicit positive text from an llm than male pronouns. similarly, @xcite compare different occupations, names and countries on produced sentiments. @xcite and @xcite compare the probability of stereotypical and non-stereotypical sentences under a model in order to measure whether it encodes stereotypes against different demographic groups. along the same lines, the winogender test @xcite measures gender biases in co-reference resolution tasks. taking a distributional view similar to our work, @xcite investigate biases in the context of occupation, however, they again compare predictions for different genders with each other. in general, such comparative bias tests are well-adapted by authors proposing new models @xcite @xcite . instead of comparing model predictions against each other, we compare model predictions to real world ground truth distributions in order to quantify bias. ground truth-based measures are not commonly used as a metric for fairness but important when evaluating a model's truthfulness. @xcite and @xcite provide datasets of real world facts against which to benchmark llms' knowledge. similar to our work, @xcite measure the frequency of country predictions, and how underprediction correlates with a country's gdp. contrary to their count-based approach we propose a more fine-grained metric for erasure and extend the analysis to auto-regressive models. unlike theirs, our erasure metric can be employed as a loss function for finetuning, to specifically miti-gates erasure. @xcite propose a similar metric for erasure in the domains of gender and race. like us, they compare model to ground truth distributions, though they measure a total variation distance where we use a kl-divergence based approach (see §3.3). the authors assume uniform ground truth whereas we construct a domain specific distribution (see §3.2). lastly, unlike ours, their analysis does not cover any mitigation efforts. similar in spirit, geographical representativeness has been studied for text-to-image generation models @xcite . ## method our goal is to measure, and later mitigate, the extent to which large pretrained models underpredict some countries when generating language. we formalise this notion here. note that while we are studying autoregressive models in this work, the methodology extends straightforwardly to masked models. similarly, we are interested in measuring and reducing geographical erasure, but the analysis can be applied to other attributes where ground truth is available. for example, one could measure erasure with respect to age, ethnicity, religion or gender using the same formalism. ## experiments in this section, we show the existence of geographical erasure across different llms and different prompt wordings ( §4.1). we highlight the consistency of erased countries across models ( §4.2) and investigate the impact of model size on erasure ( §4.3). we identify possible causes of erasure ( §4.4) and explore a mitigation strategy ( §4.5). the models under consideration are gpt2 @xcite , 117m, 345m, 774m and 1.6b weight versions, gpt-neo @xcite , 125m, 1.3b and 2.7b weight versions, gpt-neox, 20b weights @xcite and open source reproductions of the llama model @xcite @xcite , 3b and 7b weights. we obtain all implementations from huggingface. 6 he comes from ___ . his father hails from ___ . my nephew is a citizen of ___ . our uncle is a citizen of ___ . your roots are in ___ . 0.0 0.5 1.0 1.5 erasure er 3 prompts c avg. er 3 er 3 (p true , p agg_model ) er 3 (p true , p agg_uni ) northern mariana isl. saint kitts and nevis antigua and barbuda malaysia saint lucia st vincent micronesia iraq suriname marshall islands eswatini the philippines sri lanka papua new guinea bangladesh solomon islands nigeria cameroon pakistan uganda madagascar sierra leone 0 2 4 6 8 10 erased by how many models , green is the uniform aggregate er 3 (p true , p uni_agg ) (8) and red is the model-induced aggregate er 3 (p true , p model_agg ) (10). size of dots corresponds to the probability assigned to the respective prompt under the model. the gap between blue and red/green aggregations is explained in §3.5. right: of the @xmath0, 105 do not experience erasure at @xmath1. for the remaining 22, we plot model counts here. bars are coloured according to counts and sorted by gdp per capita (decreasing from left to right). we use aggregated predictions according to equation 10. ## conclusion we motivated the need for large language models to be more geographically inclusive-which remains to be an overlooked aspect of inclusive model development. specifically, we studied and formalised a notion of geographical erasure, which captures the countries that are underpredicted and the extent to which they are underpredicted. we discussed how our formulation captures many desirable properties. in our experiments, we found clear instances of geographical erasure, which were consistently observed across 10 different language models. perhaps unsurprisingly, the output probabilities of language models closely follow the frequencies of country mentions in the training corpora, a likely cause of erasure. we examined a finetuning-based mitigation strategy and found it to be effective in alleviating erasure. the language (of prompts) used to analyze erasure should be accounted for while collecting ground truth data: for instance, english speaking countries are expected to have higher probability conditioned on "i live in", and similarly spanish speaking countries conditioned on "vivo en" are likely to have higher probabilities. in our work, we factor this by considering english speaking populations as ground truth in §4 (and one would proceed accordingly for a model in a different language). difficulty in obtaining ground truth. language specific ground truth data is less reliable and harder to obtain than raw population counts. such statistics are often self-reported and the level of proficiency differs dramatically across regions, especially since the numbers include second language speakers. 2 since we only measure erasure for countries where p true i is available, the availability of language specific ground truth data is itself a biasing factor. this is evident from figure 1 , which depicts how the lack of ground truth data predominantly affects central african regions.