D u R eader_robust: A C hinese Dataset Towards Evaluating Robustness and Generalization of Machine Reading Comprehension in Real-World Applications (2021)

Abstract: Machine reading comprehension (MRC) is a crucial task in natural language processing and has achieved remarkable advancements. However, most of the neural MRC models are still far from robust and fail to generalize well in real-world applications. In order to comprehensively verify the robustness and generalization of MRC models, we introduce a real-world Chinese dataset – DuReader_robust . It is designed to evaluate the MRC models from three aspects: over-sensitivity, over-stability and generalization. Comparing to previous work, the instances in DuReader_robust are natural texts, rather than the altered unnatural texts. It presents the challenges when applying MRC models to real-world applications. The experimental results show that MRC models do not perform well on the challenge test set. Moreover, we analyze the behavior of existing models on the challenge test set, which may provide suggestions for future model development. The dataset and codes are publicly available at https://github.com/baidu/DuReader .

## introduction machine reading comprehension (mrc) requires machines to comprehend text and answer questions about it. with the development of deep learning, the recent studies of mrc have achieved remarkable advancements @xcite @xcite @xcite . however, previous studies show that most of the neural models are not robust enough @xcite @xcite and fail to generalize well @xcite . to further promote the studies of robust and well generalized mrc, we construct a chinese dataset -dureader robust which comprises natural questions and documents. in this paper, we focus on evaluating the robustness and generalization from the following aspects, where robustness consists of over-sensitivity and over-stability: in previous work, the above issues have been studied separately. in this paper, we aim to create a dataset namely dureader robust to comprehensively evaluate the three issues of neural mrc models. previous work mainly studies these issues by altering the questions or the documents. @xcite ; @xcite ; @xcite evaluate the over-sensitivity issue via paraphrase questions generated by rules or generative models. @xcite liang (2017 in recent years, with the rise of the amber market, the price of amber keeps going up. the price of opaque amber is generally 2-4 times the price of clear amber ... ## conclusion in this paper, we create a chinese dataset -dureader robust and use it to evaluate both the robustness and generalization of the mrc models. its questions and documents are natural texts from baidu search. this presents the robustness and generalization challenges in the real-world applications. our experiments show that the mrc models based on the pre-trained lms do not perform well on dureader robust challenge set. we also conduct extensive experiments to examine the behaviors of the mrc models on the dataset and provide insights for future model development.