R ussian S uper GLUE : A R ussian Language Understanding Evaluation Benchmark (2020)

Abstract: In this paper, we introduce an advanced Russian general language understanding evaluation benchmark – Russian SuperGLUE. Recent advances in the field of universal language models and transformers require the development of a methodology for their broad diagnostics and testing for general intellectual skills - detection of natural language inference, commonsense reasoning, ability to perform simple logical operations regardless of text subject or lexicon. For the first time, a benchmark of nine tasks, collected and organized analogically to the SuperGLUE methodology, was developed from scratch for the Russian language. We also provide baselines, human level evaluation, open-source framework for evaluating models, and an overall leaderboard of transformer models for the Russian language. Besides, we present the first results of comparing multilingual models in the translated diagnostic test set and offer the first steps to further expanding or assessing State-of-the-art models independently of language.

## introduction with the development of technologies for text processing and then deep learning methods for obtaining better text representation, language models went through the increasingly advanced stages of natural language modelling. modern scientific methodology is beginning to gradually explore universal transformers as an independent object of study -furthermore, such models show the ability to extract causal relationships in texts (natural language inference), common sense and world knowledge and logic (textual entailment), to generate coherent and correct texts. an actively developing field of model interpretation develops testing procedures comparing their performance to a human level and even the ability to reproduce some mechanisms of human brain functions. nlp is gradually absorbing all the new areas responsible for the mechanisms of thinking and the theory of artificial intelligence. benchmark approaches are being developed, testing general intellectual "abilities" in a text format, including complex input content, but having a simple output format. most of these benchmarks (for more details see section 2) make the development of machine intelligence anglo-centric, while other, less widespread languages, in particular russian, have other characteristic linguistic categories to be tested. in this paper, we expand the linguistic diversity of the testing methodology and present the first benchmark for evaluating universal language models and transformers for the russian language, together with a portable methodology for collecting and filtering the data for other languages. the contribution of russianglue is two-fold. first, it provides nine novel datasets for the russian language covering a wide scope of nlu tasks. the choice of the tasks are justified by the design of prior nlu benchmarks @xcite . second, we evaluate two widely used deep models to establish baselines. the remainder is structured as follows. we overview multiple prior works on developing nlu benchmarks, including those designed for languages other than english, in section 2. section 3.1 lists the tasks and novel datasets, proposed for the russian nlu. section 4 presents with the baselines, established for the tasks, including a human level baseline. we overview compare achieved results in section 2 to the current state of english nlu. we discuss future work directions and emphasize the importance of nlu benchmarks for languages other than english in section 6. section 7 concludes. ## related work several benchmarks have been developed to evaluate and analyze word and sentence embeddings over the past few years. senteval @xcite ) is one of the first frameworks intended to evaluate the quality of sentence embeddings. a twofold set of transfer tasks is used to assess the generalization power of sentence embedding models. the transfer tasks comprise downstream tasks, in which the sentence embedding is used as a feature vector, and probing tasks, which are aimed to evaluate the capability of sentence embeddings to encode linguistic properties. the choice of the downstream tasks is limited to sentiment classification, natural language inference, paraphrase detection and image captioning tasks. the probing tasks are meant to analyse morphological, syntactical and semantical information encoded in sentence embeddings. the general language understanding evaluation (glue) @xcite benchmark is a collection of tools for evaluating the performance of language models across a diverse set of existing natural language understanding (nlu) tasks, adopted from different sources. these tasks are divided into two parts: single sentence classification tasks and sentence pair classifications tasks subdivided further into similarity and inference tasks. glue also includes a hand-crafted diagnostic test, which probes for complex linguistic phenomena, such as the ability of the model to express lexical semantics and predicate-argument structure, to pose logical apparatus and knowledge representation. glue is recognized as a de-facto standard benchmark to evaluate transformer-derived language models. last but not least glue informs on human baselines for the tasks, so that not only submitted models are compared to the baseline, but also to the human performance. the superglue @xcite follows glue paradigm for language model evaluation based on nlu tasks, providing with more complex tasks, of which some require reasoning capabilities and some are aimed at detecting ethical biases. a few recent projects reveal that glue tasks may be not sophisticated enough and do not require much tasks-specific linguistic knowledge @xcite . thus superglue benchmark, being more challenging, becomes much more preferable for evaluation of language models. decanlp @xcite widens the scope for language model evaluation by introducing ten disparate natural language tasks. these tasks comprise not only text classification problems, but sequence tagging and sequence transformation problems. the latter include machine translation and text summarization, while the former include semantic parsing and semantic role labelling. although decanlp along with the associated research direction focuses on multi-task learning as a form of question answering, it supports zero-shot evaluation. to evaluate models for languages other than english, several monolingual benchmarks were developed, such as flue @xcite and clue @xcite , being french and chinese versions of glue. these benchmarks include a variety of tasks, ranging from part-of-speech tagging and syntax parsing to machine reading comprehension and natural language inference. to the best of our knowledge, linspector @xcite is a first multi-lingual benchmark for evaluating the performance of language models. linspector offers 22 probing tasks to analyse for a single linguistic feature such as case marking, gender, person, or tense for 52 languages. a part of these 22 probing tasks are static, i.e. are aimed at evaluation of word embeddings, and the rest are contextual and should be used to evaluate language models. released in early 2020 two multilingual benchmarks, @xcite and xtreme @xcite , aim at evaluation of cross-lingual models. xglue includes 11 tasks, which cover both language understanding and language generation problems, for 19 languages. xglue provides with several multilingual and bilingual corpora that allow of cross-lingual model training. as for the russian language, xglue provides with four datasets for pos tagging, a part of xnli @xcite and two datasets, crawled from commercial news website, used for news classification and news headline generation. xtreme consists of nine tasks which cover classification, sequence labelling, question answering and retrieval problems for 40 languages. almost a half of the datasets were translated from english to the target languages with the help of professional translators. xtreme offers for the russian language five datasets, including ner and two question-answering datasets. both xglue and xtreme offer tasks that are much simpler than superglue and are aimed at evaluation of cross-lingual models rather than at comparison of mono-lingual models in similar setups. thus the need for novel datasets targeted at mono-lingual model evaluation for languages other than english is still not eliminated. ## russianglue overview we have intenooed to have the same task set in the framework as one in the superglue. there is no one-to-one mapping, but the corpora we use could be considered close to the specified tasks in the superglue framework. we divided the tasks into six groups, covering the general diagnostics of language models and different core tasks: common sense understanding, natural language inference, reasoning, machine reading and world knowledge. ## results the analysis of we hope that our benchmark will help to excel the performance of models for the russian language in the future, and will favour achieving comparably high results. can the results of a multilingual bert on russian and english data be considered analo-gous? based on the results of the assessment, multibert in english gets an overall score of 60.8 6 , while on russianglue task set an overall score of 54.2 is achieved-6% lower, but noting that the english benchmark includes additionally winograd gender parity @xcite dataset, giving sota models from 90 to 93% of accuracy added to the overall assessment. in the next section, a detailed comparison of the multilingual model performance is provided. ## discussion we hope that our project will give a start to new research in the application of universal language models and transformers, including multilingual ones. our example of an analysis of translated diagnostics shows that even in languages of the same european family (which russian and english belong to), significant differences in the influence of linguistic categories on model performance are possible. one of the directions of the next studies, we consider detailed experiments on the influence of model parameters and language categories in data on the quality of the model in different languages. an independent problem for the english original leaderboard is that a gradual improvement in the quality of models allows us to exceed the human performance level in individual tasks, as happened with the t5 @xcite model. we expect that a similar situation will soon happen on russian data, which means that when releasing straight off with complex superglue tasks, we will still be focused on adding tasks of a higher level of complexity in the future. such tasks can become those that are obviously inaccessible to models for the "understanding" of long texts and documents, seq2seq tasks, tasks that require knowledge graphs. in the further development of our leaderboard, we also see the possibility of adding an industrial assessment of models: for fair ranking and ease of use, all models could receive an estimate of the required memory resources, an estimate of performance, and so on. ## conclusion in this paper we present the first benchmark on general language understanding evaluation for the russian language. the benchmark including nine task sets is aimed to test bert-like models for their ability to perform entailment recognition, commonsense reasoning and machine reading while denoising various linguistic features added on the level of semantics, logical and syntactic structure. we invite developers, researchers, and ai experts to join our project. further development of the benchmark includes areas such as evaluation of industrial performance of models on the leaderboard and multilingual diagnostics.