Learning Invariant Representation Improves Robustness for MRC Models (2022)

Abstract: The prosperity of Pretrained Language Models(PLM) has greatly promoted the development of Machine Reading Comprehension (MRC). However, these models are vulnerable and not robust to adversarial examples. In this paper, we propose Stable and Contrastive Question Answering (SCQA) to improve invariance of representation to alleviate these robustness issues. Specifically, we first construct positive example pairs which have same answer through data augmentation. Then SCQA learns enhanced representations with better alignment between positive pairs by introducing stability and contrastive loss. Experimental results show that our approach can boost the robustness of QA models cross different MRC tasks and attack sets significantly and consistently.

## introduction machine reading comprehension (mrc) @xcite ) aims to answer the question based on a passage as context. it has experienced rapid development due to the evolution of deep neural networks @xcite and release of large-scale and high-quality datasets @xcite . by introducing different kinds of attack sets, however, a considerable amount of literature @xcite @xcite has shown that the result on in-domain test set tends to overestimate the models' performance. for example, @xcite exposed models' over-stability issue by adding one distracting sentence, which suggests the models' inability to distinguish a sentence that actually answers the question from one that merely share sufficient words with it but semantically changed. @xcite showed that a question paraphrased in a slightly different but semantically similar way can mislead the model to output a wrong answer. up to now, researchers have proposed several solutions to alleviate these robustness issue, including utilizing external knowledge to create adversarial examples to enrich training data @xcite and adversarial training based methods @xcite @xcite . however, recent literature @xcite suggests that models trained with specific augmented data are still easily attacked by other unseen perturbations. adversarial training can improve models' robustness under general attacks without requiring any explicit adversarial examples, but at the cost of iterative training schedule. to be capable of handling more general attacks rather than just a certain type attack without laborious iterative schedule, we propose scqa to addresses the above robustness issue by learning invariant representations of similar examples inspired by @xcite . in detail, the data augmentation module first constructs an example similar to the input example to form a positive pair. then scqa utilizes stability loss to scale down the change of probability distribution caused by small label-preserving perturbations. in addition, scqa introduces contrastive loss to pull semantically close pairs together to further improve the alignment property in the representation space. in the experimental part, we have organized different mrc tasks and several attack tests as a benchmark for mrc robustness, including spanbased extractive, multiple choice and yes/no mrc. the results show that scqa with dropout noise as implicit data augmentation can reduce the distance between embeddings of paired examples, and therefore improve the robustness of the mrc models over different types of adversarial perturbations consistently and significantly. moreover, it is worth to note that our approach can further boost the models' performance with specific explicit data augmentation strategies. ## experimental settings datasets following previous research @xcite @xcite , we organize a benchmark for mrc models' robustness by integrating different forms of reading comprehension tasks and different adversarial attacks. the datasets contain span-based extractive squad 1.1 @xcite and squad 2.0 @xcite , multiple choice race @xcite and reclor @xcite , and yes/no boolq @xcite and np-boolq @xcite . types of adversarial attacks include charswap, paraques, addsent and other dataset-specific attacks. we choose addsentdiverse @xcite and training set of np-boolq @xcite as explicit data augmentation to demonstrate that our scqa approach can further boost the models' performance against specific attack. the details of datasets, attacks and augmented adversarial examples are presented in appendix a.2. ## limitations current data augmentation strategy just contains creating positive training pairs to defense labelpreserving perturbations. scqa pays little attention to the construction of high-quality negative sample pairs, which need to be further explored to fully utilize the effect of contrastive learning.