Overconfidence in the Face of Ambiguity with Adversarial Data (2022)

Abstract: Adversarial data collection has shown promise as a method for building models which are more robust to the spurious correlations that generally appear in naturalistic data. However, adversarially-collected data may itself be subject to biases, particularly with regard to ambiguous or arguable labeling judgments. Searching for examples where an annotator disagrees with a model might over-sample ambiguous inputs, and filtering the results for high inter-annotator agreement may under-sample them. In either case, training a model on such data may produce predictable and unwanted biases. In this work, we investigate whether models trained on adversarially-collected data are miscalibrated with respect to the ambiguity of their inputs. Using Natural Language Inference models as a testbed, we find no clear difference in accuracy between naturalistically and adversarially trained models, but our model trained only on adversarially-sourced data is considerably more overconfident of its predictions and demonstrates worse calibration, especially on ambiguous inputs. This effect is mitigated, however, when naturalistic and adversarial training data are combined.

## introduction end-to-end neural network models have had widespread success on standard benchmarks in nlp @xcite @xcite . however, models trained with maximum-likelihood objectives under the standard empirical risk minimization paradigm are liable to succeed in these settings by fitting to features or correlations in the data which are ultimately not representative of the underlying task and fail to generalize out of distribution, e.g., under domain shift or adversarial perturbation @xcite . one promising method to overcome this difficulty is to however, the process of developing adversarial data is imperfect, and adversarial data may itself not resemble naturalistic distributions. for example, @xcite find that the aflite adversarial filtering algorithm @xcite , designed to find challenging examples in existing datasets, disproportionately favors contentious examples with annotator disagreement. this is suggestive that adversarially collected datasets, where humans actively try to fool a model, may be subject to these same biases indeed, phang et al. also show that adversarially-collected datasets may disproportionately penalize models that are similar to the one used during data collection. the qualitative properties of adversariallycollected data also vary depending on the adversary used during data collection, as shown by @xcite for the adversarial nli dataset @xcite . for these reasons, it is not clear what a model's performance under adversarial evaluation implies about its performance characteristics on naturalistic distributions, nor is it clear how training on adversarial data aids a model's perfor-mance in natural settings. in this work, we focus on the interplay of adversarial learning and evaluation with ambiguity, or annotator disagreement. just as adversarial filtering may over-sample ambiguous inputs @xcite , adversarial annotators may produce strange, ambiguous, or disputable inputs as they employ tricks to fool a model in the adversarial setting. to preempt this issue and ensure data quality, adversarial data collection methods filter out examples with low human agreement @xcite , but it's possible that this approach could over-correct for the issue and under-sample such inputs in comparison to naturalistic data. for this reason, it is plausible that models trained on adversarially-collected data may be miscalibrated against the ambiguity of their inputs, forming a predictable blind spot. we investigate this issue by training models on naturalistically and adversarially collected datasets, then comparing their performance with respect to gold annotator distributions. as a testbed, we use natural language inference, an nlp benchmark task with already-available adversarial data @xcite and full annotator distributions @xcite . we find no clear difference in accuracy between naturalistically and adversarially trained models, but our model trained only on adversarially-sourced data is considerably more overconfident of its predictions and demonstrates worse calibration, especially on ambiguous inputs. on the other hand, including both naturalistic data in training as well -as is standard practice @xcite ) -mitigates these issues. while our results do not raise alarms about standard practices with adversarial data, they suggest that we should keep in mind the importance of including naturalistic data in training regimes moving forward. foot_0 2 background: robustness and adversarial data suppose we are interested in learning a conditional probability distribution p(y | x). the classical machine learning approach of empirical risk minimization does so with the use of input data drawn from a distribution d: one complication that arises when sourcing adversarial data is with ambiguous or arguable examples. suppose θ perfectly models p(y | x). plugging this into formula 2 yields max @xmath0, where d is concentrated on the inputs x which maximize the entropy of y . in this context, high entropy in the conditional distribution of y corresponds to high annotator disagreement. 2 when a human searches for an adversarial example, they are looking for a disagreement between themselves and the model. in this setting, there may be competition for inclusion in these adversarial tasks between ambiguous examples on which the model is close to the gold (annotator) distribution and less ambiguous examples where the model is further from gold. thus an adversarial data generation process may be biased towards input examples which are ambiguous but unhelpful for training. formally, a simple way to think about counteracting this may be to explicitly subtract the gold entropy from the loss being minimized: ## experimental setup task setting we use natural language inference @xcite as our underlying task, as there exist adversarial annotations for this task @xcite and annotator disagreement has been well studied @xcite @xcite . ## results all results in this section are reported on the snli and multinli development set portions of the chaosnli data. in all graphs, we provide smoothed kernel density estimates of the distributions over x and y values in the margins where appropriate. shaded areas around the lines represent 95% confidence intervals. ## discussion in our experiments using snli, multinli, and anli, we find that training only on adversariallycollected data produces similar accuracies across all regimes of ambiguity, but worse calibration at high confidence, and more overconfidence on ambiguous examples. this suggests that the adversarial data collection process may bias the model by favoring less ambiguous examples, but there are other potential interpretations of our results. in particular, the observed miscalibration of ad-versarial may the result of a more general domain shift between snli/multinli and anli. this could explain why adding snli and multinli to training, as in the all model, eliminates the effect. however, one might also expect to see a clear difference in accuracy as well if this were the issue. it's also worth noting that the snli and mnli training sets are larger than anli's (see some notable trends hold for all models we test. first, they all perform worse on ambiguous examples (figure 1 , figure 2 ). this may be in part due to the relative scarcity of such examples in the training data or the relative difficulty of learning to model them. second, they all demonstrate overconfidence, with model perplexity growing slower than human perplexity (figure 3 ) and relatively poor calibration at high confidence levels (figure 6 ). even though augmenting training with adversarially-collected data has been shown to improve robustness in some settings @xcite , our results do not yet show any benefits to calibration on ambiguous examples in existing data. finally, while we hypothesize that the overconfidence issue with training on adversarial data arises from filtering for annotator agreement, it is also possible that for anli, the adversarial annotators found examples that were less ambiguous in the first place (as annotators might, for example, want to focus on sure-fire model mistakes). @xcite found that about 5% of examples in anli "could reasonably be given multiple correct labels," suggesting a low level of ambiguity, but this was by the judgment of a single expert and may not correspond to the full variation in label assignment seen with crowdsourced annotators (which could potentially be investigated using the original unfiltered anli data). measuring, controlling, managing, or representing ambiguity in adversarial annotation should be an interesting direction for future work, perhaps incorporating insights from recent work about construal @xcite , explicit disambiguation @xcite , model training dynam-ics @xcite , and other model-in-the-loop adversarial data collection efforts @xcite @xcite . ## conclusion we have shown that training only on adversariallycollected data, at least in the case of the adversarial nli (anli) dataset, can produce undesirable performance characteristics in the resulting models. in particular, when tested on snli and multinli data, these models produce output distributions that are much further from annotator distributions and fail to accurately convey annotator uncertainty, with highly confident predictions even on highly ambiguous examples. it is also possible that adversarial training in this setting could produce lower prediction accuracy in regimes of low human agreement, but baseline accuracy is already so low for our models and data, and there are so few examples in the extremely-ambiguous regime, that such an effect is hard to find. in our results, if a large amount of naturalistic data is also included in training (as in the all model) -as is standard practice -the overconfidence problem is mitigated. this is encouraging, as any adversarially-collected data must start with some naturalistic data to construct the initial adversary. however, it remains an open question how little naturalistic data is sufficient; a large enough seed corpus may be beneficial for avoiding such issues in a setting of dynamic adversarial data collection @xcite . future work can investigate this question, as well as how using full annotator distributions at training time @xcite or model calibration techniques may further help models deal with ambiguous inputs.