On the Transferability of Minimal Prediction Preserving Inputs in Question Answering (2021)

Abstract: Recent work (Feng et al., 2018) establishes the presence of short, uninterpretable input fragments that yield high confidence and accuracy in neural models. We refer to these as Minimal Prediction Preserving Inputs (MPPIs). In the context of question answering, we investigate competing hypotheses for the existence of MPPIs, including poor posterior calibration of neural models, lack of pretraining, and “dataset bias” (where a model learns to attend to spurious, non-generalizable cues in the training data). We discover a perplexing invariance of MPPIs to random training seed, model architecture, pretraining, and training domain. MPPIs demonstrate remarkable transferability across domains achieving significantly higher performance than comparably short queries. Additionally, penalizing over-confidence on MPPIs fails to improve either generalization or adversarial robustness. These results suggest the interpretability of MPPIs is insufficient to characterize generalization capacity of these models. We hope this focused investigation encourages more systematic analysis of model behavior outside of the human interpretable distribution of examples.

## experimental methodology all models trained, including drqa @xcite , bert @xcite , and xlnet @xcite , employ setup and parameter choices from @xcite . 2 we generate mppis by iteratively removing the least important word from the question, while keeping the original prediction unchanged. the least important word is given as that for which the model's confidence in its prediction remains highest in its absence. 3 to examine how mppis transfer across question answering domains we employ 6 diverse qa training sets and 12 evaluation sets. 4 the datasets were selected for annotation variety, differing on: question type, document source, annotation instructions, whether the question was collected independently of the passage, and skills required to answer the question. this set represents a realistic spectrum of domains for evaluating generalization. we set aside 2k examples from each domain's validation sets in order to generate mppis for model evaluation. for each experiment we also generate a set of randomly shortened queries to compare against the mppis -we refer to this as the "random mppi" baseline. for each of the original examples, we generate this baseline by randomly removing words until the length matches that of the corresponding mppi. ## discussion in squad, the most common mppi is the empty string (40%). among non-empty strings, the most common mppi tokens are: "what", "?", "who", "how", "when". despite the pattern of interrogative words, these tokens are already among the most frequent in squad questions, so it's challenging to measure the unique information they convey. a more direct approach to understand the informative signal of mppis is to measure their "human insufficiency" property directly. we conduct a grading task, comparing human ability to answer real, mppi, and random mppi queries. we find no evidence that mppis are explained by poorly calibrated neural models, lack of pretraining knowledge, or dataset-specific bias. alternatively they may relate, at least in part, to useful and transferable signals. while practitioners, especially in model debiasing tasks, have focused on human understandable and generalizable features, this work would encourage them to also consider the presence of generalizable features which are not human interpretable. this observation closely relates to prior work in computer vision suggesting human uninterpretable, adversarial examples can be the result of "features", not "bugs", in which @xcite observe "a misalignment between the (human-specified) notion of robustness and the inherent geometry of the data." we hope this work provides a framework to rigorously evaluate the impact of bias mitigation methods on robustness and generalization, and encourages ml practitioners to examine assumptions regarding unexpected model behaviour on out-of-distribution inputs. ## conclusion we empirically verify the surprising invariance of mppis to random seed, model architecture, and pretraining, as well as their wide transferability across domains. these results suggest that mppis may not be best explained by poorly calibrated neural estimates of confidence or dataset-specific bias. examining their relationship to generalization and adversarial robustness, we highlight the ability to maintain in-domain performance but significantly alter out-domain performance and robustness. we hope our results encourage a more systematic analysis of hypotheses regarding model behavior outside the human interpretable distribution of examples. for completeness, we describe our method of computing span confidence for question answering, given that there are many variations. let @xmath0, both of sequence length n . for every combination of i, @xmath1, n ] where @xmath2, n ), and @xmath3, we compute the confidence for that span of answer text as the sum of their respective logits s i + e j . the final confidence probability p i j for a given span is as shown in equation 2 . the model, on the other hand, can still make the same prediction as it did on the full input, and with a similar degree of confidence.