Systematic Inequalities in Language Technology Performance across the World’s Languages (2022)

Abstract: Natural language processing (NLP) systems have become a central technology in communication, education, medicine, artificial intelligence, and many other domains of research and development. While the performance of NLP methods has grown enormously over the last decade, this progress has been restricted to a minuscule subset of the world’s \approx 6,500 languages. We introduce a framework for estimating the global utility of language technologies as revealed in a comprehensive snapshot of recent publications in NLP. Our analyses involve the field at large, but also more in-depth studies on both user-facing technologies (machine translation, language understanding, question answering, text-to-speech synthesis) as well as foundational NLP tasks (dependency parsing, morphological inflection). In the process, we (1) quantify disparities in the current state of NLP research, (2) explore some of its associated societal and academic factors, and (3) produce tailored recommendations for evidence-based policy making aimed at promoting more global and equitable language technologies. Data and code to reproduce the findings discussed in this paper areavailable on GitHub ( https://github.com/neubig/globalutility ).

## introduction the past decade has seen a rapid advance in natural language processing (nlp); it has grown from a relatively technical niche to a fundamental tool in virtually all domains that involve language data in any shape or form. nlp is now instrumental to not only bread-and-butter applications such as translation and question answering, but also tasks as wide ranging as detection of neurodegenerative diseases @xcite , exposing widespread gender and ethnic biases in societies @xcite , and predicting large-scale trends in collective consumer behavior @xcite . because of this nlp has become a staple technology for 1 data and code to reproduce the findings discussed in this paper are available on github ( https://github.com/ neubig/globalutility ). these technological capabilities can be attributed to several developments over the last few decades: 1. the advent of deep learning methods, which allow for more effective creation of nlp systems from existing data @xcite , 2. the existence of standardized benchmark datasets and evaluation metrics @xcite , 3. the prestige afforded by the research community to researchers who improve upon these benchmarks, 4. the resulting large number of resources, be they computation, data, or ingenuity, that are poured into optimizing performance thereon. as both a theoretical and technical endeavor, nlp is experiencing an explosive increase: the annual conference of the association of computational linguistics (acl) received in 2000 less than 300 papers, growing in 2010 to slightly less than 1,000, to over more than 3,500 submissions in its 2020 edition. largely as a result of this expansion of research effort, state-of-the-art systems have also achieved evaluation benchmark scores on par with human performance on a variety of nlp tasks such as question answering on english @xcite , or on automatic translation of news from german, russian, and chinese to english @xcite . 2 these upward slanting curves on standard benchmarks fail to show how uneven this development has been for all potential nlp users. extensive research across nlp tasks have found systematic performance drops according to dimensions such as gender, racial identity, and language varieties, among others. the reasons for these biases are multifactorial and can be traced to virtually all stages in the process of nlp development, from the data used to train systems @xcite @xcite @xcite @xcite to the very algorithms involved @xcite @xcite . the growing awareness of these biases in nlp technologies brought by these studies, along with the development of novel metrics and tests to evaluate these disparities, have resulted in progressively more efficient and principled strategies to understand and mitigate them. however, similarly systematic approaches are still lacking in one fundamental dimension of variation across individuals: their languages. out of the over 6,500 languages spoken or signed in the world today @xcite , only a handful are systematically represented in academia and industry @xcite . in spite of the aforementioned near-human results on translation or understanding of languages from the world's economic and political superpowers, the experience of any nlp practitioner is that, for the vast majority of languages, they fall far below such standards. critically, the languages of the world showcase substantial variation in most domains of description, and in fact, the performance of language technologies has been shown to be sensitive to diverse aspects of the language under study, including morphology, word order, or phonological repertoire, as well as more mundane aspects like writing script or data availability @xcite @xcite . hence, the transfer of nlp developments from one language to another is far from trivial, as it often means that building highly functional language technologies for any particular language is a non-automatic, costly, and technically challenging task. taking all these considerations together, and given that even the consequences brought by unequal nlp technologies across (racial, gender, socioeconomic) groups within the same nominal language are already substantial, there is a pressing need for measuring and understanding nlp performance inequalities across the world's languages. here we develop novel estimates on how the utility afforded by nlp systems is distributed across individuals, languages, and tasks at an unprecedented global scale. these estimates allow us to identify which languages are systematically underserved by language technologies and could benefit the most individuals from focused technology development. we finally trace these inequalities to the societal, economic, and academic correlates of nlp systems' performance, shedding light on its latent causes, and indicate how our results favor specific evidence-based policies in research and development. ## discussion our study, covering diverse nlp tasks and types of evidence, makes apparent the immense inequality in the development of language technologies across the world's languages. after english, a handful of western european languages dominate the field -in particular german, french, and and spanish-as well as even fewer non-indo-european languages, primarily chinese, japanese, and arabic. our preliminary investigation suggests it is the economic prowess of the users of a language (rather than the sheer demographic demand) what drives the development of language technologies. in spite of this, for some tasks (such as innumber of publications according to number of language users and approximate gdp. point size and transparency scales with number of publications. eng: english, zho: chinese, deu: german, fra: french, spa: spanish, jpn: japanese, rus: russian, nld: dutch, ces: czech, por: portuguese, tur: turkish, swe: swedish, ita: italian, fin: finnish, ell: greek, lat: latin, hun: hungarian, ara: arabic, kor: korean, hin: hindi, pol: polish, dan: danish. however, these more esoteric tasks are tenuously associated with those that users are more likely to interact with, such as machine translation or speech synthesis. user-facing tasks all have in common a tight dependency on computational resources and large data, which in turn hinge on substantial financial means. in a context of pressing user needs across multiple populations and languages, we submit that future developments on policies aimed at furthering crosslinguistic technologies would benefit from clear (and possibly standardized) metrics that assist in streamlining complex decisions regarding resource allocation. our measures of global coverage fulfill that role, and help identifying large but currently under-served languages. while we do not attempt to supplement the necessary in-depth evaluation of the need of each individual group and language, they provide a common ground for coordinating global efforts across heterogeneous actors. in addition, we would like to reiterate that our work here has necessarily made a large number of simplifying assumptions to even attempt to quantify disparities in language technology utility on a global scale. these most notably involve simplifying assumptions regarding the measurement of demand (based on native-speaker population and/or economic indicators) and the measurement of utility (based on simple accuracy metrics). future work may further clarify these assumptions, making more accurate estimates of true user demand on a technology-by-technology level, or more accurately clarifying the relationship between standard accuracy metrics and the utility derived by users.