XTREME - UP : A User-Centric Scarce-Data Benchmark for Under-Represented Languages (2023)

Abstract: Data scarcity is a crucial issue for the development of highly multilingual NLP systems. Yet for many under-represented languages (ULs) — languages for which NLP research is particularly far behind in meeting user needs — it is feasible to annotate small amounts of data. Motivated by this, we propose XTREME-UP, a benchmark defined by: its focus on the scarce-data scenario rather than zero-shot; its focus on user-centric tasks — tasks with broad adoption by speakers of high-resource languages; and its focus on under-represented languages where this scarce-data scenario tends to be most realistic. XTREME-UP evaluates the capabilities of language models across 88 under-represented languages over 9 key user-centric technologies including ASR, OCR, MT, and information access tasks that are of general utility. We create new datasets for OCR, autocomplete, semantic parsing, and transliteration, and build on and refine existing datasets for other tasks. XTREME-UP provides methodology for evaluating many modeling scenarios including text only, multi-modal (vision, audio, and text), supervised parameter tuning, and in-context learning. We evaluate commonly used models on the benchmark. We release all code and scripts to train and evaluate models.

## introduction the development of natural language processing (nlp) technology that serves most of world's languages is hindered by the stark lack of data for most languages @xcite . while there is increasing interest in developing datasets and models for under-represented languages (uls), existing datasets are often informed by established research directions in the nlp community @xcite . while linguistic tasks such as syntactic parsing have become less practically relevant @xcite , other impactful capabilities such as question answering or virtual assistants @xcite , often depend on ancillary technologies such as language id, data filtering, automatic speech recognition (asr), or optical character recognition (ocr) that are typically underperforming or unavailable for uls @xcite @xcite @xcite . as a result, speakers of uls are unable to reap the benefits, even if the development of models is successful. in order to make progress on nlp for uls, we should thus focus on evaluating models on tasks that are most likely to benefit speakers of those languages. 3 to this end, we propose xtreme-up (under-represented and user-centric with paucal 4 data), a benchmark focusing on evaluation of multilingual models on user-centric tasks in a scarce-data setting. we focus on tasks that technology users encounter in their daily lives: i) information access tasks reflecting generally useful nlp capabilities; and ii) input/output tasks that enable other technologies. we show the corresponding tasks and their role in interactions with language technology in figure 1 . moving away from the cross-lingual zeroshot setting @xcite , we standardize multilingual in-language fine-tuning based on the amount of data that can realistically be annotated within 8h for a language. our results highlight the limitations of current models on uls, demonstrate the potential of language models (lms) to improve user-centric applications, and show the benefit of byte-based approaches. in this work, we contribute the first massivelymultilingual few-example benchmark including: a) newly created data for qa, ocr, autocomplete, semantic parsing, and sentence-level transliteration; b) new task setups for named entity recognition (ner) enabling evaluation on natural-rather than tokenized-text; and for qa and retrieval providing a more interesting setting than the gold passage (goldp) setup while offering a lower barrier-toentry than the full tydi qa @xcite or xor @xcite tasks; c) carefullydesigned experimental setups, standardizing inlanguage fine-tuning and in-context learning and focusing on the information access scenario for uls for asr and mt; d) baseline results with commonly used subword and byte-based models. ## recommendations use of splits xtreme-up offers a train, validation, and test split for each task. we recommend using the training split for learning the parameters of your model or as exemplars for in-context learning while iteratively checking your progress on the validation (i.e. development) split. the test split should not be used for iterative evaluation of your models or other sorts of hill-climbing; instead, it should be reserved for reporting your results and comparing after you have finished development on your models. experiments that follow this customary scientific rigor should expect to show better generalization and less overfitting to the test split. use of additional pre-training data one potential confounder for results along different pretrained models is the variation in pre-training data; where this data overlaps with the targets (outputs) in xtreme-up validation and test splits, results can be artificially inflated, providing a sense that results are better than they are in reality-if the validation or test data leaked into the pre-training data via contamination during large-scale data scraping, then it's unlikely that the system would truly perform as well for new unseen inputs. therefore, we recommend that when researchers modify the pre-training data for a model, they explicitly report overlap (contamination) between the targets of the xtreme-up validation/test splits and their pre-training corpus. 17 use of additional supervised data it is entirely possible that the community will find creative ways to improve models based on supervised data not included with xtreme-up. however, researchers should bear in mind how this might affect the comparability of their results with other models. the following axes should be considered: 1. any additional data from high resource languages is always allowed in the xtreme-up setting. 2. supervised data (e.g. parallel data for mt) avoid off-the-shelf mt systems data augmentation via automatically translating high-resource supervised data to languages with less supervised data has proven a very effective means of improving system quality. however, it is not necessarily realistic to use a pre-existing mt system (e.g. an api or an open-source model) since those systems have typically been trained on a large amount of parallel data-or at least unknown data. this means that additional supervised data would then be leaking into the experimental setup, which is otherwise intended to reflect the reality that most under-represented languages have very little supervised data. if data augmentation via translation is used, we encourage researchers to report the parallel data sources used and argue why this experimental setup is realistic-or to clearly point out such usage in their experiments as an unavoidable confound and discuss the limitations this sets on what conclusions can be drawn about how results will extrapolate to the breadth of under-represented languages. in all cases, researchers should rigorously report what additional data was used and how; each use case comes with its own considerations and, contaminated pre-training data is equivalent, better, or almost as good as some other system). note, this analysis only needs to be done once for each pre-training corpus (e.g., once for mc4) and it is very likely that organizations with enough compute to pre-train a new model on a new corpus would also have sufficient compute to calculate overlap. above all, researches should make a well-reasoned argument that their use of data (i) does not artificially inflate evaluation scores and (ii) reflects a real-world scenario of finding and applying data. ## conclusion we have presented xtreme-up, a multilingual benchmark distinguished by its being (i) scarcedata, (ii) user-centric, and (iii) focused on underrepresented languages. the benchmark contains input modalities of text, images, and audio while still allowing experimentation with text-only models. we hope this benchmark will be useful in accelerating research that is useful to speakers of under-represented languages and in highlighting both the progress and limitations of current models of language.