G lobal B ench: A Benchmark for Global Progress in Natural Language Processing (2023)

Abstract: Despite the major advances in NLP, significant disparities in NLP system performance across languages still exist. Arguably, these are due to uneven resource allocation and sub-optimal incentives to work on less resourced languages. To track and further incentivize the global development of equitable language technology, we introduce GlobalBench. Prior multilingual benchmarks are static and have focused on a limited number of tasks and languages. In contrast, GlobalBench is an ever-expanding collection that aims to dynamically track progress on all NLP datasets in all languages. Rather than solely measuring accuracy, GlobalBench also tracks the estimated per-speaker utility and equity of technology across all languages, providing a multi-faceted view of how language technology is serving people of the world. Furthermore, GlobalBench is designed to identify the most under-served languages, and rewards research efforts directed towards those languages. At present, the most under-served languages are the ones with a relatively high population, but nonetheless overlooked by composite multilingual benchmarks (like Punjabi, Portuguese, and Wu Chinese). Currently, GlobalBench covers 966 datasets in 190 languages, and has 1,128 system submissions spanning 62 languages.

## introduction advances in multilingual natural language processing (nlp) technologies @xcite have raised the enticing possibilities of nlp systems that benefit all people around the world. however, at the same time, studies into the state of multilingual nlp have demonstrated stark differences in the amount of resources available @xcite and performance of existing nlp systems @xcite @xcite . why do these disparities exist? the causes of these disparities are multifarious, but @xcite argue that one major factor is a problem of incentives and resource allocation. for instance, languages associated with larger economic might 1 globalbench is available at https://github.com/ neulab/globalbench . (as measured by gdp of the countries where they are spoken) see more research and resource development, leading to more performant systems. in this paper, we propose globalbench, a new benchmark and leaderboard that is designed to specifically incentivize the global development of equitable language technologies that serve speakers of all languages throughout the world. glob-albench follows the footsteps of other successful multilingual benchmarks such as xtreme @xcite and xglue @xcite , which aggregate results of systems across several tasks to provide a general idea of progress being made in the field of multilingual nlp. however, these benchmarks, by design, are static and lack the goal to be an all-inclusive, ever-expanding collection of datasets. additionally, they mainly focus on average accuracy over all languages in the dataset, and thus say little about the downstream utility and figure 2: globalbench's philosophy: first, we aim to inclusively gather datasets for all tasks and languages. second, we present a multi-faceted evaluation of systems, going beyond average accuracies across languages, to keep track of the utility and equity of these systems. third, globalbench ranks the most to least under-served languages on the leaderboard of each task, and rewards improvement in utility, which can be achieved through both dataset and model contributions. the lower the ranking of a language the better since a lower ranking indicates it is less under-served. in the above example, the addition of dataset 3 improves the measured utility of l 3 in the ner leaderboard, and the addition of system 2 improves the measured utility of l 2 and l 3 for the ner leaderboard. hence, in designing globalbench, we make a number of intentional design decisions to explicitly promote the improvement of language technology for all of the world's citizens: in the remainder of this paper, we detail glob-albench's design principles ( §2), how interested research community members can participate ( §3), the currently covered tasks and systems ( §4), analysis of the current state of nlp viewed through the lens of globalbench ( §5), related work ( §6) and our expectations for the path forward ( §7). all in all, we believe that improving the quality and equity of language technologies for all speakers in the world is one of the paramount challenges of nlp today. in the famous mantra from peter drucker, what you cannot measure, you cannot improve; globalbench is a first step in this direction. ## implementation and participation details for globalbench globalbench accepts open submissions of datasets and systems. for datasets that are already a part of global-bench, system results for them can be submitted to globalbench for evaluation. the submission process of system results is simple: one can submit system results to globalbench through github directly. participants don't need to separately submit anything else. in addition, if people want to submit new datasets to benchmark, one can follow the dataset submission process provided in our github repository. 3 after doing so, corresponding system results can be submitted as per the above instructions. ## datasets and tasks in globalbench, we currently support 17 tasks that fall into 10 distinct categories. these tasks represent a diverse set of nlp technologies, ranging from those that are highly applicative and userfacing (question answering, machine translation etc.), to those that aren't directly applied, but are nonetheless fundamental to nlp (language modeling, etc.). we briefly summarize the tasks and provide one example for each task in figure 3 , and we provide a more detailed description and a list of ## results and analysis globalbench allows us to conduct a series of analyses to assess global progress in nlp technologies. while globalbench is intended to be evolving to gauge the continued improvements in performance of nlp systems, we examine the current state of systems that have been submitted, to demonstrate how globalbench can guide and incentivize future participation and improvement in language technology. ## conclusion in this paper, we introduce globalbench, an everexpanding collection of datasets and models spanning all tasks and languages within nlp. our aim is for the community to move towards a common goal of building nlp technology that equitably serves all of the world's citizens. to achieve this, we design a leaderboard resting upon three foundational principles: i) inclusivity: we track progress across all tasks and datasets for 6671 languages; ii) multi-faceted evaluation: our evaluation measures the per-speaker utility and equity of submitted systems, and also maintains a list of most under-served languages; iii) reward improvement: we reward data and modeling efforts that help improve utility across languages, rather than simply maintaining the best-performing systems. analysing the 1,128 system outputs already integrated within global-bench, reveals that ner has the highest utility at present, but is also least equitable. the combination of high demographic and low linguistic utility underscores that efforts have been mostly limited to populous languages. finally, we identify that the most under-served languages vary across tasks, but are primarily the ones with relatively high speaker population, but nonetheless low coverage in our datasets. all in all, we believe that globalbench is one step towards measurable progress in improving the global quality and equity of languages technologies for all speakers in the world, and we hope the rest of the research community will join us in pursuit of this goal.