Richer Countries and Richer Representations (2022)

Abstract: We examine whether some countries are more richly represented in embedding space than others. We find that countries whose names occur with low frequency in training corpora are more likely to be tokenized into subwords, are less semantically distinct in embedding space, and are less likely to be correctly predicted: e.g., Ghana (the correct answer and in-vocabulary) is not predicted for, “The country producing the most cocoa is [MASK].”. Although these performance discrepancies and representational harms are due to frequency, we find that frequency is highly correlated with a country’s GDP; thus perpetuating historic power and wealth inequalities. We analyze the effectiveness of mitigation strategies; recommend that researchers report training word frequencies; and recommend future work for the community to define and design representational guarantees.

## introduction how similar are the words "brooklyn" and "queens"? to a new yorker, they evoke two very different places, cultures, and cuisines, but to a seattleite, they are quite similar, both being boroughs of new york city. 1 our perception of entities such as cities or countries is conditioned on our backgrounds. here, we ask if language models are also susceptible to representational biases. we suggest three criteria to characterize the quality of representations for particular entities or groups: consistency, distinctiveness, and recognizability. for consistency, are all entities of a certain type (such as all country names) represented with the same number of tokens in the lexicon? for distinctiveness, are entities of the same category seen as equally distinct in representational space? for recognizability, are models capable of generating all entities of a certain type in response to questions? and are the differences between entities confounded across lines of historical inequity (like wealth of countries)? focusing on bert (bert-base-cased foot_1 ) representations @xcite , we find that names of countries that appear less frequently in training data are less likely to be invocabulary, are less semantically distinct from other countries, and are less frequently predicted in the masked language modeling (mlm) task. disappointingly, we find similar behavior in bert-base-multilingual-cased and roberta-base. we identify these differences as intrinsic representational harms where low frequency countries are more likely to be conflated with one another and their existence less recognized. a more troubling result is that training data frequency is highly correlated with the gross domestic product of a country (gdp) (pearson's @xmath0.82). our training data and thus the representation of entities through our language models encodes wealth and power disparities and perpetuates representational harms. given these significant differences in representation, what could it look like to impose a minimum quality of representation for significant entities? we recommend that the community consider designing representational guarantees for language models. in summary we: 1) reveal multiple ways in which the bert representation of high gdp countries is systematically richer than that of low gdp countries; 2) study the effectiveness of potential mitigation efforts; and 3) propose the idea of representational guarantees as future work for the community. ## related work language technologies have long been studied for potential intrinsic and extrinsic harms @xcite . known intrinsic harms include misrepresentation of gender @xcite , race @xcite , and ability (hutchinson et al., 2020) -all types of representational harms @xcite @xcite . other intrinstic harms include forms of erasure through under-representation of lgbtq+ identity terms @xcite and racial groups @xcite . extrinsic harms are often found in downstream tasks and include disparities in quality of service among user groups @xcite such as african-american users (blodgett and o'connor, 2017; @xcite . however, low statistical power has also made it difficult to make conclusive claims about the presence or absence of bias @xcite . many of these representational harms have been linked to word frequency in static embeddings @xcite @xcite @xcite . low frequency words also differ geometrically from other words, with smaller inner products @xcite and lower variance @xcite . recent work has also shown how frequency impacts contextual embeddings such as the under-estimation of cosine similarity among high-frequency words @xcite and the discrepancies in representations of personal names @xcite . our work extends these lines of work via the examination of representational harms for country names. ## rich countries have their own tokens are poor and rich countries tokenized the same way? here, we focus on bert's tokenization process and measure the consistency (or rather inconsistency) in how names of countries are tokenized and then represented. our gdp data is retrieved from the united nations statistics division from 2019 34 . subwords freq gdp (m) example 1 (@xmath1,882 430,596 uzbekistan 2 (@xmath2,148 870,702 comoros 3 (@xmath3,711 34,896 grenada 4+ (@xmath4,309 14,980 eswatini the united nations members list, 134 of them are in-vocabulary, -the remaining 25 are outof-vocabulary (oov). in wordpiece tokenization, oov words are tokenized into in-vocabulary subwords (e.g. "andorra" becomes "and" and "##orra", see we used ordinary least squares regression to predict the number of subword tokens in each country name, using training data frequency of each country name as the feature (bert training data estimated from the march 1st, 2020 wikimedia download and bookcorpus) @xcite . 6 . we found that training data frequency explains 38% of the variance in number of subwords (@xmath5.01), despite the confounder of multi-word countries being considered oov ( ## richer countries are most distinct in embedding space nations or entire regions are subject to bias. the african continent, for example, is often treated journalistically as a single homogeneous entity (nothias, 2018), as if african countries are all substitutable for one another. we draw on this finding to ask whether historically disadvantaged countries are also conflated with one another in the embeddings of their names (i.e., seen as less distinct from each other than other countries). we measure the semantic similarity between pairs of the 134 in-vocabulary countries by creating word embeddings for each name (done by averaging the last four hidden layers of bert). we calculate the average in-group cosine similarity of country names as grouped by frequency (i.e., we take the countries in each decile of frequency and measure the average cosine similarity across the 14 2 pairs). we repeat this ten times and find that the 10% least frequent names have an average ingroup similarity of 0.610 compared to an average similarity of 0.582 for the 10% most frequent countries (mean δ = 0.028, permutation test @xmath7.01). we then calculate the average semantic similarity of a country's embedding to all other countries to measure a country's distinctiveness (averaged over ten trials). using ols to predict average cosine similarity, frequency explains 8% of the variance ( to ensure that the semantic similarity discrepancies are not simply a consequence of how these countries are written about in test examples, we run the same ols experiment on an artificial dataset where names of countries appear in identical contexts. the results are consistent: frequency explains 9% of the variance in average cosine similarity ( ## richer countries are more frequently predicted the lack of recognition of people and groups @xcite ; gehman et al., 2020) has often been cited as an representational harm. here, we use the masked language modeling task as a proxy for many downstream tasks that need to be able to predict the name of a country (e.g., as the answer to a question, or in a summary) to measure whether countries are all minimally predicted or whether instead we see representational harms such as erasure. we randomly sample sentences from wikipedia that contain the name of a country, replace the name with a bert mask token ([mask]), and use the masked-out country name as the gold label. we use 100 examples of each of the 134 in-vocabulary country names. 7 the model will only predict invocabulary words, which again illustrates the impact of inconsistent tokenization: the very task bert was trained on is unable to handle oov country names without modification. high frequency countries (75th percentile) have an average accuracy of 42% while low frequency (25th percentile) countries have an accuracy of 26% ( ## the limitations of mitigation efforts we analyze the effectiveness of two popular mitigation efforts. 1. could these frequency-based effects be mitigated with additional training data? we measure the performance of multilingual bert (bert-ml) which includes wikipedia articles from 104 other languages. we continue to find accuracy disparities between low and high gdp countries (15% vs 29%). additional training data fails to mitigate these harms, most likely because the additional data continues to amplify existing imbalances (i.e., german, french, and polish are the next three biggest languages of wikipedia articles). data augmentation as a mitigation technique is challenging as datasets could easily maintain existing or introduce new frequency imbalances. we also tested roberta (trained on over 160gb more data) on this task with similar results ( ## discussion and conclusion we find significant disparities in the quality of representation of country names and show how these differences result in representational harms that perpetuate existing wealth and power inequalities. we make two recommendations on paths forward. we recommend the release of training word frequencies to increase transparency and isolate current representational harms @xcite @xcite . practitioners who use these models in their systems and research should have access to the topic and entity distribution of our models given the potential for frequency-related harms. we also recommend the community consider designing representational guarantees for significant entities to mitigate these downstream harms. our work illustrates the potential harms that arise when entities such as country names do not have representational guarantees. we encourage the community to consider the following questions: how can we ensure entities such as names of country be in-vocabulary? how can we guarantee a minimum distinctness and ensure recognition of historically disadvantaged groups? such guarantees will likely be difficult to design and will require expertise from multiple domains but mitigating these representational harms is an important task that we cannot ignore. 6 8 10 12 14 16 0 200 400 600 800 cambodia china france india gross domestic product (logged) # of predictions