Look to the Right: Mitigating Relative Position Bias in Extractive Question Answering (2022)

Abstract: Extractive question answering (QA) models tend to exploit spurious correlations to make predictions when a training set has unintended biases. This tendency results in models not being generalizable to examples where the correlations do not hold. Determining the spurious correlations QA models can exploit is crucial in building generalizable QA models in real-world applications; moreover, a method needs to be developed that prevents these models from learning the spurious correlations even when a training set is biased. In this study, we discovered that the relative position of an answer, which is defined as the relative distance from an answer span to the closest question-context overlap word, can be exploited by QA models as superficial cues for making predictions. Specifically, we find that when the relative positions in a training set are biased, the performance on examples with relative positions unseen during training is significantly degraded. To mitigate the performance degradation for unseen relative positions, we propose an ensemble-based debiasing method that does not require prior knowledge about the distribution of relative positions. We demonstrate that the proposed method mitigates the models’ reliance on relative positions using the biased and full SQuAD dataset. We hope that this study can help enhance the generalization ability of QA models in real-world applications.

## introduction deep learning-based natural language understanding (nlu) models are prone to use spurious correlations in the training set. this tendency results in models' poor generalization ability to out-ofdistribution test sets @xcite , which is a significant challenge in the field. question answering (qa) models trained on intentionally biased training sets are more likely 1 our codes are available at https://github.com/ kazutoshishinoda/relativepositionbias . bold spans are the answers to the questions. in both the examples, answers are found by looking to the right from the overlapping words. see §2.1 for the definition of the relative position. to learn solutions based on spurious correlations rather than on causal relationships between inputs and labels. for example, qa models can learn question-answer type matching heuristics @xcite , and absolute-positional correlations @xcite , particularly when a training set is biased toward examples with corresponding spurious correlations. collecting a fully unbiased dataset is challenging. therefore, it is vital to discover possible dataset biases that can degrade the generalization and develop debiasing methods to learn generalizable solutions even when training on unintentionally biased datasets. in extractive qa (e.g., @xcite , in which answers to questions are spans in textual contexts, we find that the relative position of an answer, which is defined as the relative distance from an answer span to the closest word that appears in both a context and a question, can be exploited as superficial cues by qa models. see we aim to develop a method for mitigating the performance degradation on subsets with unseen relative positions while maintaining the scores on subsets with seen relative positions, even when the training set is biased with respect to relative positions. to this end, we propose debiasing methods based on an ensemble @xcite of intentionally biased and main models. the biased model makes predictions relying on relative positions, which promotes the main model not depending solely on relative positions. our experiments on squad @xcite using bertbase @xcite as the main model show that the proposed methods improved the scores for unseen relative positions by 0∼10 points. we demonstrate that the proposed method is effective in four settings where the training set is differently filtered to be biased with respect to relative positions. furthermore, when applied to the full training set, our method improves the generalization to examples where questions and contexts have no lexical overlap. ## related work recent studies have found that nlu models tend to learn shortcut solutions specific to the distribution of the training sets. in natural language inference, models can use spurious correlations regarding lexical items @xcite and lexical overlap @xcite . in extractive qa, existing studies have shown that models can learn several kinds of shortcut solutions. @xcite showed that a substantial number of questions could be answered only by matching the types of questions and answers. @xcite demonstrated that only using partial inputs is sufficient for finding correct answers in most cases. @xcite indicated that the absolute position bias of answers could severely degrade the generalization. @xcite showed that question generation models can amplify lexical overlap bias when used for data augmentation in qa. sampling training and test sets from different distributions @xcite @xcite is one of the most effective frameworks to assess whether models can learn generalizable solutions. in this study, we also employed the framework to analyze the generalization capability from a new perspective of relative position bias, and developed effective debiasing methods for this bias. specifically, we used the same loss functions as @xcite , and proposed new biased models in this study. ## conclusion we showed that an extractive qa model tends to exploit relative position bias in training sets, causing the performance degradation for relative positions unseen during training. to mitigate this problem, we proposed new biased models that perform well with existing debiasing algorithms on intentionally filtered training sets. furthermore, when applied to the full training set, our method improved the generalization to examples where questions and contexts have no common words. future work includes refining the definition of relative position bias that is more learnable for qa models, mitigating the trade-off between the accuracies for seen and unseen relative positions, and increasing the size of the test set with no lexical overlap between question and context.