XTREME - R : Towards More Challenging and Nuanced Multilingual Evaluation (2021)

Abstract: Machine learning has brought striking advances in multilingual natural language processing capabilities over the past year. For example, the latest techniques have improved the state-of-the-art performance on the XTREME multilingual benchmark by more than 13 points. While a sizeable gap to human-level performance remains, improvements have been easier to achieve in some tasks than in others. This paper analyzes the current state of cross-lingual transfer learning and summarizes some lessons learned. In order to catalyze meaningful progress, we extend XTREME to XTREME-R, which consists of an improved set of ten natural language understanding tasks, including challenging language-agnostic retrieval tasks, and covers 50 typologically diverse languages. In addition, we provide a massively multilingual diagnostic suite and fine-grained multi-dataset evaluation capabilities through an interactive public leaderboard to gain a better understanding of such models.

## introduction most research in natural language processing (nlp) to date has focused on developing methods that work well for english and a small set of other high-resource languages @xcite . in contrast, methods for other languages can be vastly more beneficial as they enable access to language technology for more than three billion speakers of low-resource languages and prevent the nlp community from overfitting to english. motivated by these benefits, the area of multilingual nlp has attracted increasing interest recently. however, evaluating multilingual models is challenging as it requires assessing performance on a wide range of typologically distinct languages in the face of limited heterogeneous data sources. recently large-scale benchmarks such as xtreme @xcite and xglue @xcite have been introduced, consolidating existing multilingual tasks and covering tens of languages. when xtreme was released, the gap between the best-performing baseline, xlm-r large @xcite , and human-level performance was roughly 25. this has since shrunk to less than 12 points, a much smaller but still substantial gap compared to the difference from human-level performance observed in english transfer learning @xcite , which has recently been closed entirely on some evaluation suites @xcite . in order to examine the nature of this progress, we first perform an analysis of state-of-the-art multilingual models on xtreme. we observe that progress has not been uniform, but concentrated on cross-lingual retrieval tasks where fine-tuning on other tasks and pre-training with parallel data lead to large gains. on other task categories improvements are more modest. models still generally perform poorly on languages with limited data and non-latin scripts. fine-tuning on additional translated data generally leads to the best performance. based on this analysis, we propose xtreme-r (xtreme revisited), a new benchmark with the dual purpose of ensuring that research in multilingual nlp focuses on the most challenging problems and equipping researchers with a broader set of tools to better understand their models (see we also make several logistic improvements to improve xtreme-r's utility as a leaderboard. to make it easier to choose the best model for a use case, each submission is required to provide metadata such as the number of parameters and amount of pre-training data, which we make available via an interactive leaderboard. we also introduce task and language-specific sub-leaderboards to invite submissions of dedicated models. in sum, we make the following contributions: a) an analysis of progress in cross-lingual modeling; b) an improved benchmark covering 50 languages, including a newly created retrieval task (mewsli-x); c) a massively multilingual diagnostic suite; d) fine-grained evaluation capabilities; e) experiments and analyses of state-of-the-art models; and f) an interactive metadata-rich leaderboard. ## xtreme-r in order to encourage the nlp community to tackle challenging research directions in pursuit of better cross-lingual model generalization, we propose xtreme-r (xtreme revisited). xtreme-r shares its predecessor's core design principles for creating an accessible benchmark to evaluate crosslingual transfer but makes some key changes. first, xtreme-r focuses on the tasks that have proven to be hardest for current multilingual models. to this end, it drops xtreme's paws-x and bucc tasks since recent advances have left less room for further improvement, and they cover only a small number of less diverse languages. they for tasks that have training and dev sets in other languages, we only report the english numbers. we report the number of test examples per target language, the nature of the test sets (whether they are translations of english data or independently annotated), and the size of the intersection with our selected languages (the total number of languages is in brackets). for wikiann-ner and ud-pos, sizes are in sentences. xcopa includes training data of siqa @xcite and tatoeba uses squad v1.1 data for fine-tuning. we also make changes to the structured prediction tasks, ner and pos. instead of only providing examples as lists of tokens, xtreme-r always provides the full text of an input sentence, thus ensuring that the entire benchmark now supports research on models that operate directly from the raw input string @xcite . furthermore, xtreme-r adopts a more realistic version of the ner task in which no gold tokenization is provided at all, meaning that systems will either have to use model-predicted tokens or embrace tokenizationfree approaches. finally, xtreme-r provides a multilingual diagnostic and evaluation suite ( §3.4). ## experiments training and evaluation setup xtreme-r focuses on zero-shot cross-lingual transfer from en-glish. while recent work @xcite @xcite demonstrates the benefits of fine-tuning on in-language data, we believe the zero-shot scenario remains the most effective way to evaluate the amount of a priori multilingual knowledge a pre-trained model captures. due to variation in cross-lingual evaluation @xcite , we recommend researchers to use the validation set of a single target language for development @xcite . ## conclusions our analyses and experiments have shed light on important directions where scale alone is not sufficient such as "strong" alignment, syntactic transfer, fine-grained natural language understanding, and answering of abstract questions. we encourage the development of better inductive biases, pre-training objectives, and evaluation resources. we make our data, translations, evaluation resources, and interactive leaderboard supporting detailed comparative analyses available to help the community gain a better understanding of multilingual models.