Should We Ban E nglish NLP for a Year? (2022)

Abstract: Around two thirds of NLP research at top venues is devoted exclusively to developing technology for speakers of English, most speech data comes from young urban speakers, and most texts used to train language models come from male writers. These biases feed into consumer technologies to widen existing inequality gaps, not only within, but also across, societies. Many have argued that it is almost impossible to mitigate inequality amplification. I argue that, on the contrary, it is quite simple to do so, and that counter-measures would have little-to-no negative impact, except for, perhaps, in the very short term.

## inequalities if nlp makes people richer and happier, e.g., by allowing them more free time @xcite , it is unfortunate that nlp predominantly serves the needs of the richest and happiest among us. by and large, nlp supports languages spoken in the world's wealthiest regions @xcite . performance disparities within languages and across demographics are also well-documented @xcite @xcite -and performance correlates strongly with income levels @xcite . disparities may reflect data imbalances: if training data contains less data from a group, predictions for that group will tend to be worse. but disparities can also result from outlier behavior or higher degrees of variance in groups. disparities in resources, variation, priority, performance, and turn-around go hand-in-hand to create vicious circles that widen existing equality gaps. take mobile assistants, for example. mobile assistants help us organize our calendars, place calls, remind us of meetings, etc. since they are typically speech-operated, their performance depends heavily on the performance of available speech recognizers. for most languages, speech recognizers were developed for young, urban subpopulations, seen as early adopters. as a consequence, while young urbans reap the benefits of mobile assistants, mobile assistants are often much less useful for other demographics @xcite . many have argued that it is almost impossible to mitigate inequality amplification @xcite . i argue that it is quite simple to do something, and that doing so would have little-to-no negative impact (except for, perhaps, in the very short term). i address two levels of inequality in nlp: inequality across languages and inequality across social groups. what languages and subgroups are favored is somewhat task-specific, but generally, nlp seems biased toward english and the tech-savvy: the dominance of english existing estimates of how much of top venue nlp research is devoted to english vary a bit, but typically lie in the range of 50-90%, averaging around two thirds. 1 importantly, numbers do not seem to have changed for the better over the last 10 years. the vast majority of publicly available nlp datasets are limited to english. for this reason, it is much easier for start-ups and other companies to roll out products for the english-speaking markets. naturally, this means that speakers of english have way more technologies at hand @xcite . 2 the dominance of the tech-savvy the penetration rate of mobile assistants with young urbans leads to performance disparities in speech recognition. similarly, chatbots are developed for an audience that more frequently interacts with dialogue systems. training data is sampled from end users, and user feedback is leveraged as learning signals. as a result, performance disparities across demographic groups gradually widen until they become a matter of night and day. see figure 1 for an illustration of this effect across four synthetic datasets. 3 outline i have argued how nlp favors english and the tech-savvy. this is not the only way in which nlp models are biased, but arguably the 1 see @xcite , @xcite , as well as https: //sjmielke.com/acl-language-diversity.htm 2 what explains the dominance of english in nlp research? prestige seems to be an important factor. what is considered state of the art, is what achieves best performance on english. consider, for example, the acl wiki's list of 'state-of-the-art' part-of-speech taggers https://aclweb.org/aclwiki/pos_tagging_(state_ of_the_art) this list is based on the performance of these models on the english penn treebank @xcite . it is the same for acl wiki lists on np chunking, question answering, and so on. such prominent rankings make it more important for researchers to perform well on english benchmarks. visibility and impact are determined by your doing so, not by your performance on other languages. @xcite show that papers focusing on languages other than english are cited less often. ## justifications can the inequalities of nlp research be justified? one high-level justification of inequalities falls out of the way we have come to define fairness in nlp research. nlp researchers have almost uniformly adopted american philosopher john rawls' definition of fairness @xcite @xcite @xcite . rawls' concern @xcite is with the absolute position of the least advantaged group rather than their relative position, even if we have to abandon strict equality of income and wealth. rawls justifies inequality up to that point and thereby introduces a loophole. my concern with this form of justification is that the loophole turns out to be easily exploited because the effects of both policy and technology are indirect and must be evaluated in the long term @xcite . proponents of a free market argue that lowering high-income taxes may result in longterm benefits for the least advantaged, but what if they eventually do not? similarly, can we be sure that our cutting edge research on english will have spill-over effects for other languages in the long run? it can-in econonomy, as well as in nlp-be difficult to predict what will get the wheels spinning for everyone, and what will create a downward spiral; see, again, figure 1 for an illustration of this. english resources are abundant. ## measures what measures have nlp researchers proposed to mitigate inequalities? @xcite argue that 'being able to build neural language models for other languages with the same quality as english is key for language equality', and that the stepping stone is collecting 'large amounts of publicly available corpora of good quality'. i think this is insufficient in my rebuttal of opportunity in §2: resources are often available. our excessive focus on english and the tech-savvy is not primarily driven by data scarcity. @xcite argue it is the economic prowess of the users of a language that drives the development of nlp technologies, but they do not present specific proposals for mitigating inequalities. they only refer to a need for global coordination. one common strategy for mitigating performance disparities across languages is to make models language-independent @xcite . multilingual models often still exhibit cross-language disparities @xcite , but can be augmented with an objective minimizing the loss of the worstoff language @xcite . similarly, many learning algorithms have been developed to maximize performance on the groups with the worst performance. examples include square root sampling (stickland and murray, 2019), adaptive scheduling @xcite , lossbalanced task weighting, @xcite , groupdistributional robust optimization @xcite , and worst-case-aware automated curriculum learning @xcite . this does not and will not bridge existing gaps: the algorithms are commonly thought to ensure equal performance but in fact, because they implement rawlsian fairness, they only prescribe inequality up to a point. lin and chen (forthcoming) highlight the chal-lenges of achieving fairness in the context of structurally unjust societies; see also @xcite . such considerations, as well as the urgency of the matter, has made me wonder what holds us back in adopting more radical measures. inspired by policies proposed to mitigate climate change, i briefly discuss the pros and cons of three possible pathways forward: nlp cap and trade under cap and trade @xcite , lawmakers establish a limit (or "cap") on the overall cost or risk, say, the amount of greenhouse gases. such caps can be negotiated from year to year, and are ideally supported by commonly agreed-upon objectives and scientific evidence. in nlp, this could be a cap on monolingual language models, a cap on technologies for or research publications on english, a cap on male annotators, etc. just like governments could initially auction off emission allowances to the highest bidder or allocate them evenly or in light of special needs, acl could distribute quota for english models, biased end user groups, or biased annotator pools. subsequent to the initial allocation, research labs could reduce their 'emissions' and sell excess allowances to other research labs for quota. nlp carbon tax a carbon tax @xcite is the obverse of cap and trade: rather than fixing the amount of allowable emissions, it specifies their price. in the same way, nlp researchers could incur a cost-by paying higher conference fees, subtracting from their reviewer scores, or by disqualifying them from paper awards-if they 'emit greenhouse gases' by, say, working on english or with a biased set of end users or annotators. nlp car-free sundays an alternative to the above is the equivalent of car-free sundays. carfree sundays produce significant mean carbon emission reductions and reduce overall traffic activity. while these effects are variable @xcite , car-free sundays also help to promote the cause of mitigating climate change. car-free sundays are also less intrusive and less bureaucratic than cap and trade or the equivalent of a carbon tax. examples of regulatory steps in nlp that would be comparable to car-free sundays, would include a one-year ban on english models, biased end user groups, biased annotator pools, etc. in practice, bans could, for example, be implemented by automatic desk rejection of all such papers submitted to acl 2023 or to all the main conferences of that year. it is easy to see the positive effects of such an initiative: acl 2022 accepted 702 papers. 702 papers on other languages than english and/or with annotator pools would be a big step toward course-correcting and mitigating existing biases. 6 ## conclusion this paper is a position paper, arguing that nlp is contributing to global inequalities through a digital language divide, and by implicitly favoring the tech-savvy. while many of us have promoted linguistic diversity in recent years, numbers suggest our field is still massively biased toward developing english nlp for tech-savvy demographics. maybe time is ripe to consider more concrete measures? to get the discussion off ground, i briefly considered three types of measures inspired by regulation advanced to mitigate climate change: cap-and-trade, carbon tax, and car-free sundays. i argue that the nlp community would quickly adapt to most such initiatives, course-correcting in little or no time. perhaps initially, the equivalent of car-free sundays is the least intrusive and bureaucratic form of regulation, but exactly what our next steps should look like, i leave up for community-wide discussion.