Read and Reason with M u S e RC and R u C o S : Datasets for Machine Reading Comprehension for R ussian (2020)

Abstract: The paper introduces two Russian machine reading comprehension (MRC) datasets, called MuSeRC and RuCoS, which require reasoning over multiple sentences and commonsense knowledge to infer the answer. The former follows the design of MultiRC, while the latter is a counterpart of the ReCoRD dataset. The datasets are included in RussianSuperGLUE, the Russian general language understanding benchmark. We provide a comparative analysis and demonstrate that the proposed tasks are relatively more complex as compared to the original ones for English. Besides, performance results of human solvers and BERT-based models show that MuSeRC and RuCoS represent a challenge for recent advanced neural models. We thus hope to facilitate research in the field of MRC for Russian and prompt the study of multi-hop reasoning in a cross-lingual scenario.

## introduction machine reading comprehension (mrc) is a central task in natural language processing that simulates a human ability to read a text and provide the correct answer for a given question. therefore, it requires general language understanding, knowledge about the world, and interpretive reasoning. the task has been widely explored for the english language targeting different aspects of reading comprehension @xcite @xcite @xcite . recently, the paradigm has shifted towards a more complex setting, where the model is tested to infer the answer based on interpretive analysis of text @xcite , reasoning over multiple documents @xcite or joint natural language inference and commonsense reasoning @xcite . however, mrc in languages other than english, specifically russian, has not been well-addressed primarily due to the lack of high-quality and large-scale datasets. recognizing this need, several cross-lingual machine reading datasets have been constructed @xcite @xcite @xcite with a few of them being a part of cross-lingual benchmarks such as xglue @xcite and xtreme @xcite . though allowing to evaluate the current state of language transferring methods, these cover only a small number of languages, use a back-translation technique for the assembly or combine data from different annotation schemes. surprisingly, little prior research has been devoted to the task of mrc for russian, and no attempts have been made to explore the subject in multi-hop and commonsense reasoning scenarios. sberquad @xcite is the only one russian mrc dataset designed as a competition challenge analogous to squad @xcite . the task is formalized as an extractive reading comprehension, where the answer to a natural language question is a text span from the corresponding wikipedia passage. still, a significant portion of the questions can be answered by matching the language patterns between the question and the passage segment that contains the answer. to this end, we introduce two novel russian mrc datasets, called rucos and muserc, which require commonsense knowledge and reasoning over multiple sentences. the datasets are publicly available and included in the general language understanding benchmark russiansuperglue foot_0 analogous to super-glue for english @xcite . the datasets follow the design of superglue tasks, namely record @xcite and multirc @xcite . we provide a thorough comparative analysis of the russian and english datasets and demonstrate that in some aspects the proposed datasets are more complex as opposed to the original tasks, particularly due to the language specifics. the remainder is organized as follows. section 2 and section 3 outline the description of rucos and muserc datasets, particularly task abstraction, dataset collection procedure, annotation procedure and comparative analysis. section 4 describes tf-idf & bert-based baselines, evaluation of human performance, and comparison to the corresponding superglue tasks. we highlight the main contributions of this work and conclude in section 6. ## experiments in this section, we describe a two-step baseline approach and human performance on each task. we first used tf-idf method as a naive baseline (see section 4.1.1) and three bert-based models as advanced baselines (see section 4.1.2). we outline the design of the human benchmark in section 4.2 and provide more details in the appendices. ## results the results of the baseline models and human solvers are presented in it is worth mentioning that recent state-of-the-art language models for english, specifically t5 model @xcite , have outperformed the human results on both record and multirc datasets. the model achieved 94.1% of f1 score and 93.4% of em score for record. the following results are obtained for multirc: 88.1% of f1a score and 63.3% of em score. t5 demonstrates impressive results, which we hope can be achieved for russian as well. in future work, we are going to explore the language patterns in questions of different reasoning types for both english and russian mrc datasets. this may be useful when studying the linguistic properties of the language models, specifically multilingual ones. another line of research is to analyze top-k best leader-board models in a cross-lingual scenario. particularly, we suppose that the training objectives of language models such as text infilling or sentence shuffling as in t5 and bart @xcite ) may play a big role for the outstanding performance. ## conclusion this work is devoted to the assembly of rucos and muserc, two novel machine reading comprehension datasets for russian. the datasets are publicly available foot_12 and included in the evaluation suite of russiansuperglue, the russian general language understanding benchmark. the tasks require reasoning over multiple sentences, commonsense knowledge, and advanced natural language understanding. we hope to provide a detailed description of the construction procedure, as well as a comparative analysis of the proposed datasets, and their analogous tasks for english. due to the language specifics, rucos and muserc tend to be relatively more complicated in some aspects as opposed to record and mul-tirc. our baselines, including recent state-of-the-art bert-based models for russian, can not compete with human solvers falling beyond their performance. we hope that rucos and muserc will spur more research in the field of russian reading comprehension, and prompt the study of multi-hop reasoning in a cross-lingual scenario.