Towards Interpreting and Mitigating Shortcut Learning Behavior of NLU models (2021)

Abstract: Recent studies indicate that NLU models are prone to rely on shortcut features for prediction, without achieving true language understanding. As a result, these models fail to generalize to real-world out-of-distribution data. In this work, we show that the words in the NLU training set can be modeled as a long-tailed distribution. There are two findings: 1) NLU models have strong preference for features located at the head of the long-tailed distribution, and 2) Shortcut features are picked up during very early few iterations of the model training. These two observations are further employed to formulate a measurement which can quantify the shortcut degree of each training sample. Based on this shortcut measurement, we propose a shortcut mitigation framework LGTR, to suppress the model from making overconfident predictions for samples with large shortcut degree. Experimental results on three NLU benchmarks demonstrate that our long-tailed distribution explanation accurately reflects the shortcut learning behavior of NLU models. Experimental analysis further indicates that LGTR can improve the generalization accuracy on OOD data, while preserving the accuracy on in-distribution data.

## introduction pre-trained language models, such as bert @xcite , have demonstrated substantial gains on many nlu (natural language understanding) benchmarks. however, recent studies show that these models tend to exploit dataset biases as shortcuts to make predictions, rather than learn the semantic understanding and reasoning @xcite . here we focus on the lexical bias, where nlu models rely on spurious correlations between shortcut words and * most of the work was done while the first author was an intern at adobe research. labels. this eventually results in their low generalizability on out-of-distribution (ood) samples and low adversarial robustness @xcite . in this work, we show that the shortcut learning behavior of nlu models can be explained by the long-tailed phenomenon. previous empirical analysis indicates that the performance of bert-like models for nli task could be mainly explained by the reliance of spurious statistical cues such as unigrams "not", "do", "is" and bigrams "will not" @xcite . here we generalize these hypotheses using the long-tailed phenomenon. specifically, the features in training set could be modeled using a long-tailed distribution via using local mutual information @xcite as a measurement. by utilizing an interpretation method to analyze model behavior, we observe that these nlu models concentrate mainly on information on the head of the distribution, which usually corresponds to nongeneralizable shortcut features. in contrast, the tail of the distribution is poorly learned, although it contains high information for the nlu task. another key observation is that during training process, shortcut features tend to be picked up by nlu models during very early iterations. based on these two key observations, we define a measurement to quantify the shortcut degree of all training samples. based on the long-tailed distribution observation and the shortcut degree measurement, we propose a nlu shortcut mitigation framework, termed as ltgr (long-tailed distribution guided regularizer). the proposed regularizer is based on the observation that nlu models would give over-confident predictions when there exist strong shortcut features in the input. this is because nlu models over-associate the shortcut features with certain class labels. ltgr is implemented using the knowledge distillation framework, to penalize the nlu model from outputting overconfident pre- input x teacher model softmax softmax ground truth y smoothed softmax distill loss student loss overconfident prediction long-tailed distribution example of model paying high attention to features on the head (a) long-tailed observation (b) mitigation framework shortcut degree student model head long tail ## proposed mitigation framework equipped with the observation of long-tailed phenomenon and the shortcut degree measurement b i obtained from the last section, we propose a shortcut mitigation solution, called ltgr (long-tailed distribution guided regularizer). ltgr is implemented based on self knowledge distillation @xcite (see fig. 1(b) ). the proposed distillation loss is based on the observation that nlu models would give over-confident predictions when there exist strong shortcut features in the input. this is because nlu models over-associate the shortcut features with certain class labels. the proposed distillation loss aims to suppress the nlu models from giving overconfident predictions for samples with strong shortcut features. it forces the model to down-weight its reliance on shortcut features and implicitly encourages the model to shift its attention to more task relevant features. ## while second stage do 5 initialize the student network fs(x); 6 calculate shortcut degree bi and softmax σ(z t i ) for each training sample {(xi)} n @xmath0; use eq. 5 to train the student network fs(x) output: discard ft (x). use fs(x) for prediction. ## experiments in this section, we aim to answer the following research questions: 1) does the long-tailed phenomenon explanation accurately reflect the shortcut learning behavior of nlu models? 2) does the proposed ltgr outperform alternative approaches, and what is the source of the improved generalization? 3) how do components and hyperparameters affect ltgr's generalization performance? ## related work we briefly review two lines of research that are most relevant to our work: shortcut learning demonstration and shortcut mitigation. shortcut learning phenomena. recently, the community has revealed the shortcut learning phenomenon for different kinds of language and vision tasks, such as nli @xcite , question answering @xcite , reading comprehension @xcite , vqa @xcite , and deepfake detection @xcite . this is typically achieved with the help of adversarial test set @xcite and dnn explainability @xcite @xcite . these analysis indicates that dnns are prone to capture lowlevel superficial patterns (including lexical bias, overlap bias, etc), rather than high-level task relevant features. we focus on lexical bias in this work. motivated by the high-frequency preference for cnns, i.e., the texture bias @xcite @xcite @xcite , we propose to use the long-tailed distribution to explain the shortcut learning behavior of nlu models. shortcut mitigation. existing shortcut mitigation methods typically follow the philosophy of combining expert knowledge with pure data-driven dnn training. the most representative format is to construct a bias-only teacher network @xcite , guided by the domain knowledge what in general the shortcut should look like. for instance, a hypothesis-only model @xcite or bag of words model @xcite for the nli task, and a questiononly model for vqa task @xcite are regarded as bias-only model. then a debiased model can be trained, either by combining debiased model and bias-only model in the product of expert manner @xcite , or encouraging debiased model to learn orthogonal representation as the bias-only model @xcite . other representative methods include re-weighting @xcite , data augmentation @xcite , explanation regularization @xcite , and adversarial training @xcite @xcite . nevertheless, most existing mitigation methods need to know the bias type as a priori @xcite . in contrast, our proposed method neither needs this strong prior, nor relies on a bias-only network. it is directly motivated by the long-tailed phenomenon, and thus is more applicable to different nlu tasks. ## conclusions and future work in this work, we observe that the training set features for nlu tasks could be modeled as a longtailed distribution, and nlu models concentrate mainly on the head of the distribution. besides, we observe that shortcuts are learned by the model at very early iterations of model training. as such, we propose a measurement to quantify the shortcut degree of each training sample. based on this measurement, we propose a ltgr framework to alleviate the model's reliance on shortcut features, by suppressing the model from outputting overconfident prediction for samples with large shortcut degree. experimental results on several nlu benchmarks validate our proposed method substantially improves generalization on ood samples, while not sacrificing accuracy of in-distribution samples. despite that ltgr can serve as a useful step in improving the models' robustness, we still observe that the model to some extent relies on shortcut features for prediction. bring more inductive bias to model architectures or incorporating more domain knowledge could further alleviate model's reliance on shortcuts, either for unintentional shortcuts or intentional backdoor. this is a challenging topic, and would be explored in our future research.