Mitigating Shortcuts in Language Models with Soft Label Encoding (2024)

Abstract: Recent research has shown that large language models rely on spurious correlations in the data for natural language understanding (NLU) tasks. In this work, we aim to answer the following research question: Can we reduce spurious correlations by modifying the ground truth labels of the training data? Specifically, we propose a simple yet effective debiasing framework, named Soft Label Encoding (SoftLE). First, we train a teacher model to quantify each sample’s degree of relying on shortcuts. Then, we encode this shortcut degree into a dummy class and use it to smooth the original ground truth labels, generating soft labels. These soft labels are used to train a more robust student model that reduces spurious correlations between shortcut features and certain classes. Extensive experiments on two NLU benchmark tasks via two language models demonstrate that SoftLE significantly improves out-of-distribution generalization while maintaining satisfactory in-distribution accuracy. Our code is available at https://github.com/ZiruiHE99/sle

## while inference stage do 10 ignore the dummy class and make the prediction based on values of the first k categories: ŷi = argmax @xmath0,••• ,k] pi,j. for training samples with a large shortcut degree, more smoothed new labels will be obtained. in contrast, we preserve original hard labels for samples with a low shortcut degree.