CLUE : A C hinese Language Understanding Evaluation Benchmark (2020)

Abstract: The advent of natural language understanding (NLU) benchmarks for English, such as GLUE and SuperGLUE allows new NLU models to be evaluated across a diverse set of tasks. These comprehensive benchmarks have facilitated a broad range of research and applications in natural language processing (NLP). The problem, however, is that most such benchmarks are limited to English, which has made it difficult to replicate many of the successes in English NLU for other languages. To help remedy this issue, we introduce the first large-scale Chinese Language Understanding Evaluation (CLUE) benchmark. CLUE is an open-ended, community-driven project that brings together 9 tasks spanning several well-established single-sentence/sentence-pair classification tasks, as well as machine reading comprehension, all on original Chinese text. To establish results on these tasks, we report scores using an exhaustive set of current state-of-the-art pre-trained Chinese models (9 in total). We also introduce a number of supplementary datasets and additional tools to help facilitate further progress on Chinese NLU. Our benchmark is released at https://www.cluebenchmarks.com

## introduction full-network pre-training methods such as bert @xcite and their improved versions @xcite @xcite have led to significant performance boosts across many natural language understanding (nlu) tasks. one key driving force behind such improvements and rapid iterations of models is the general use of evaluation benchmarks. these benchmarks use a single metric to evaluate the performance of models across a wide range of tasks. however, existing language evaluation benchmarks are mostly in english, e.g., glue @xcite and superglue @xcite . to the best of our knowledge, there is no general language understanding evaluation benchmark for chinese, whose speakers account for one-fourth of the world's population. also, chinese is linguistically very different from english and other indo-european languages, which necessitates an evaluation benchmark specifically designed for chinese. without such a benchmark, it would be difficult for researchers in the field to check how good their chinese language understanding models are. to address this problem and facilitate studies in chinese language, we introduce a comprehensive chinese language understanding evaluation (clue) benchmark that contains a collection of nine different natural language understanding tasks (two of which are created by us), including semantic similarity, natural language inference, short text classification, long text classification with large number of classes, and different types of machine reading comprehension tasks. to better understand the challenges posed by these tasks, we evaluate them using several popular pre-trained language understanding models for chinese. overall, we find that these tasks display different levels of difficulty, manifest in different accuracies across models, as well as the comparison between human and machine performance. the size and quality of unlabeled corpora play an essential role in language model pre-training @xcite @xcite . there are already popular pre-training corpora such as wikipedia and the toronto book corpus @xcite in english. however, we are not aware of any large-scale open-source pre-training dataset in chinese. also chinese models are mainly trained on different and relatively small corpora. therefore, it is difficult to improve model performance and compare them across model architectures. this difficulty motivates us to construct and release a standard clue pre-training dataset: a corpus with over 214 gb raw text and roughly 76 billion chinese words. we also introduce a diagnostic dataset hand-crafted by linguists. similar to glue, this dataset is designed to highlight linguistic and common knowledge and logical operators that we expect models to handle well. overall, we present in this paper: (1) a chinese natural language understanding benchmark that covers a variety of sentence classification and machine reading comprehension tasks, at different levels of difficulty, in different sizes and forms. (2) a large-scale raw corpus for general-purpose pre-training in chinese so that the comparisons across different model architectures are as meaningful as possible. ## related work it has been a common practice to evaluate language representations on different intrinsic and downstream nlp tasks. for example, @xcite measure word embeddings through a semantic analogy task and a syntactic analogy task. @xcite further expands the testing set to include other word similarity and named entity recognition tasks. similar evaluation procedures are also used for sentence representations @xcite . however, as different researchers use different evaluation pipelines on different datasets, results reported in the papers are not always fully comparable, especially in the case where the datasets are small, where a minor change in evaluation can lead to big differences in outcomes. senteval @xcite addresses the above problem by introducing a standard evaluation pipeline using a set of popular sentence embedding evaluation datasets. glue @xcite and superglue @xcite further improve senteval by providing benchmarks for natural language understanding tasks, ensuring that results from different models are consistent and comparable. they introduce a set of more difficult datasets and a model-agnostic evaluation pipeline. along with other reading comprehension tasks like squad @xcite and race @xcite , glue and superglue have become standard testing benchmarks for pre-training methods such as bert @xcite and albert @xcite . we believe a similar problem exists in chinese language understanding evaluation. although more and more chinese linguistic tasks @xcite have been proposed, there is still a need for a standard evaluation pipeline and an evaluation benchmark with a set of diverse and difficult language understanding tasks. ## clue overview clue consists of 1) nine language understanding tasks in chinese, 2) a large-scale raw dataset for pre-training and a small hand-crafted diagnostic dataset for linguistic analysis, and 3) a ranking system, a leaderboard and a toolkit. ## pre-training dataset large-scale language data is the prerequisite for model pre-training. corpora of various sizes have been compiled and utilized in english, e.g., the wikipedia corpus, the bookscorpus @xcite , and more recent c4 corpus @xcite . for chinese, however, existing public pre-training datasets are much smaller than the english datasets. for example, the wikipedia dataset in chinese only contains around 1.1 gb raw text. we thus collect a large-scale clean crawled chinese corpus to fill this gap. a total of 214 gb raw corpus with around 76 billion words are collected, consisting of three different corpora: cluecorpus2020-small, cluecorpus2020, and clueoscar. three models in this paper are pre-trained on the combined clue pre-training corpus (two albert models and roberta-large). cluecorpus2020-small it contains 14 gb of chinese text, with the following genres: ## conclusions and future work in this paper, we present a chinese language understanding evaluation (clue) benchmark, which consists of 9 natural language understanding tasks and a linguistically motivated diagnostic dataset, along with an online leaderboard for model evaluation. in addition, we release a large clean crawled raw text corpus that can be directly used for pre-training chinese models. to the best of our knowledge, clue is the first comprehensive language understanding benchmark developed for chinese. we evaluate several latest language representation models on clue and analyze their results. an analysis is conducted on the diagnostic dataset created by chinese linguists, which illustrates the limited ability of state-of-the-art models to handle some chinese linguistic phenomena. in contrast to the english benchmarks such as glue and superglue, where model performance is already at human performance, we can see that chinese nlu still has considerable room for improvement (i.e., models are ∼10% below our estimates of human performance), meaning that we expect that our benchmark will facilitate building better models in the short-term. once models have reached human performance, however, we believe that extending our benchmark to newer tasks, or newer forms of evaluation (e.g., taking into account performance as a function of model size as in @xcite ), could be a step forward. in this sense, we view clue, which is an entirely community-driven project, to be open-ended in that our current set of tasks serve as a first step in more comprehensively evaluating chinese nlu.