XGLUE : A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation (2020)

Abstract: In this paper, we introduce XGLUE, a new benchmark dataset to train large-scale cross-lingual pre-trained models using multilingual and bilingual corpora, and evaluate their performance across a diverse set of cross-lingual tasks. Comparing to GLUE (Wang et al.,2019), which is labeled in English and includes natural language understanding tasks only, XGLUE has three main advantages: (1) it provides two corpora with different sizes for cross-lingual pre-training; (2) it provides 11 diversified tasks that cover both natural language understanding and generation scenarios; (3) for each task, it provides labeled data in multiple languages. We extend a recent cross-lingual pre-trained model Unicoder (Huang et al., 2019) to cover both understanding and generation tasks, which is evaluated on XGLUE as a strong baseline. We also evaluate the base versions (12-layer) of Multilingual BERT, XLM and XLM-R for comparison.

## introduction pre-training + fine-tuning has become a new nlp paradigm, where the general knowledge are firstly learnt from large-scale corpus by self-supervised learning and then transferred to downstream tasks by task-specific fine-tuning. three different types of pre-trained models are explored recently, including monolingual pre-trained models @xcite @xcite @xcite , multilingual and cross-lingual pre-trained models @xcite @xcite and multimodal pre-trained models @xcite @xcite . in this paper, we focus on the cross-lingual pretrained models, due to their importance to alleviating the low-resource issue among languages, where an nlp task often has rich training data in one language (such as english) but has few or no training data in other languages (such as french and german). in order to further advance the development of cross-lingual pre-trained models for various downstream tasks in different languages, this paper introduces xglue, a new benchmark dataset that can be used to: (i) train large-scale cross-lingual pre-trained models using multilingual and bilingual corpora, (ii) evaluate generalization capabilities of the cross-lingual pre-trained models across a diverse set of cross-lingual tasks. the contribution of xglue is two-fold. first, it provides 11 diversified cross-lingual tasks covering both understanding and generation scenarios. xtreme @xcite ) is a concurrent work of xglue. but it includes cross-lingual understanding tasks only. besides, xglue introduces 6 new tasks selected from search, ads and news scenarios,which makes xglue have more practical values. second, an extended version of unicoder @xcite is described and evaluated as a strong cross-lingual pre-trained model baseline on xglue for both understanding and generation tasks. we also evaluate the base versions (12-layer) of multilingual bert @xcite , xlm @xcite and xlm-r @xcite for comparison. ## pre-train unicoder for cross-lingual generation tasks the encoder-decoder architecture is employed to extend unicoder to generation tasks, where the bpe embeddings are shared between encoder and decoder. two separate generative tasks are proposed for unicoder pre-training: multilingual de-noising auto-encoding (xdae) and multilingual future n-gram prediction (xfnp). ## conclusion we present xglue as a new cross-lingual benchmark and conduct comprehensive evaluations with interesting findings observed. we thank stc-a nlp, bing answers, bing ads, bing relevance and microsoft news for providing the datasets.