D u QM : A C hinese Dataset of Linguistically Perturbed Natural Questions for Evaluating the Robustness of Question Matching Models (2022)

Abstract: In this paper, we focus on the robustness evaluation of Chinese Question Matching (QM) models. Most of the previous work on analyzing robustness issues focus on just one or a few types of artificial adversarial examples. Instead, we argue that a comprehensive evaluation should be conducted on natural texts, which takes into account the fine-grained linguistic capabilities of QM models. For this purpose, we create a Chinese dataset namely DuQM which contains natural questions with linguistic perturbations to evaluate the robustness of QM models. DuQM contains 3 categories and 13 subcategories with 32 linguistic perturbations. The extensive experiments demonstrate that DuQM has a better ability to distinguish different models. Importantly, the detailed breakdown of evaluation by the linguistic phenomena in DuQM helps us easily diagnose the strength and weakness of different models. Additionally, our experiment results show that the effect of artificial adversarial examples does not work on natural texts. Our baseline codes and a leaderboard are now publicly available.

## introduction the task of question matching (qm) aims to identify the question pairs that have the same meaning, and it has been widely used in many applications, e.g., community question answering and intelligent customer services, etc. though neural qm models have shown compelling performance on various datasets, including quora question pairs (qqp) @xcite , lcqmc @xcite , bq @xcite and afqmc 2 , neu- † equal contribution. the work was done when hongyu zhu was doing internship at baidu. recently, it attracts a lot of attentions from the research community to deal with the robustness issues of neural models on various nlp tasks, such as question matching, natural language inference and machine reading comprehension. early works examine the robustness of neural models by creating certain types of artificial adversarial examples @xcite @xcite , and involving human-and-model-in-the-loop to create dynamic adversarial examples @xcite . further studies discover that a few types of superficial cues (i.e. shortcuts) in the training data, are learned by the models and hence affect the model robustness @xcite @xcite . besides, several studies try to improve the robustness of the neural models by adversarial data augmentation @xcite and data filtering @xcite . all these efforts motivate us to better find and fix the robustness issues. however, there are several limitations in previous studies. firstly, the analysis and evaluation in previous work focus on just one or a few types of adversarial examples or shortcuts, but we need normative evaluation @xcite @xcite . in the normative evaluation (linzen, 2020), the objective is not to fool the system by exploiting its particular weaknesses, but to comprehensively evaluate the basic linguistic capabilities of the models with a variety of systemically controlled datasets. checklist @xcite , quail @xcite and textflint @xcite are great attempts of normative evalua-tions. however, it is not clear that if the artificial adversarial training is effective on natural texts from real-world applications @xcite . some other works manually perturb the examples to construct natural examples, but the manual perturbation is time consuming and costly @xcite . moreover, to the best of our knowledge, there are few chinese datasets for qm robustness evaluation. towards this end, we create an open-domain chinese dataset namely duqm containing natural questions with linguistic perturbation for evaluating the robustness of qm models. (1) by linguistic, we mean that the dataset provides a detailed breakdown of evaluation by linguistic phenomenon. as shown in tab. 1, there are 3 categories and 13 subcategories with 32 linguistic perturbations in duqm, which enables us to evaluate the model performance by each category instead of just a single metric. the contributions of this paper can be summarized as follows: the remaining of this paper is organized as follows. sec. 2 describes the 3 categories and 13 subcategories with 32 linguistic perturbations in duqm. sec. 3 gives the construction process of duqm. in sec. 4, we conduct experiments to demonstrate 3 characteristics of duqm. we conclude our work in sec. 5. ## linguistic perturbations in duqm the design of duqm is aimed at normative evaluation, that contains a detailed breakdown of evaluation by linguistic phenomenon. hence, we create duqm by introducing a set of linguistic features that we believe are important for model diagnosis in terms of linguistic capabilities. basically, 3 categories of linguistic features are used to build duqm, i.e., lexical features (see sec. 2.1), syntactic features (see sec. 2.2), and pragmatic features (see sec. 2.3). we list 3 categories, 13 subcategories with 32 operations of perturbation in tab. 1. the detailed descriptions of all categories are given in this section. ## construction we design duqm as a diverse and natural corpus. the construction process of duqm is divided into 4 steps and illustrated in fig. 1 . firstly, we preprocess the source questions to obtain their linguistic knowledge, which will be used to perturb the source texts. then we pair the source and perturbed question as an example. the examples' naturalness is reviewed manually. at last, the examples are annotated manually and duqm is finally constructed. we will introduce the construction details in this section. ## experiments in this section, we conduct experiments to discuss three characteristics (char.) of duqm. in sec. 4.1, we provide the experimental setup and the evaluation metrics. in sec. 4.2 ~4.4, we give the experimental results and discussions. ## conclusion in this work, we create a chinese dataset namely duqm which contains linguistically perturbed natural questions for evaluating the robustness of qm models. duqm is designed to be fine-grained and natural. specifically, duqm has 3 categories and 13 subcategories with 32 linguistic perturbations. we conduct extensive experiments with duqm and the results demonstrate that duqm has 3 characteristics: 1) duqm is challenging and has better discrimination ability; 2) the fine-grained design of duqm helps to diagnose the strengths and weakness of models, and enables us to evaluate the models in diverse ways; 3) artificial adversarial training fails in the natural texts of duqm.