Whispers of Doubt Amidst Echoes of Triumph in NLP Robustness (2024)

Abstract: *Do larger and more performant models resolve NLP’s longstanding robustness issues?* We investigate this question using over 20 models of different sizes spanning different architectural choices and pretraining objectives. We conduct evaluations using (a) out-of-domain and challenge test sets, (b) behavioral testing with CheckLists, (c) contrast sets, and (d) adversarial inputs. Our analysis reveals that not all out-of-domain tests provide insight into robustness. Evaluating with CheckLists and contrast sets shows significant gaps in model performance; merely scaling models does not make them adequately robust. Finally, we point out that current approaches for adversarial evaluations of models are themselves problematic: they can be easily thwarted, and in their current forms, do not represent a sufficiently deep probe of model robustness. We conclude that not only is the question of robustness in NLP as yet unresolved, but even some of the approaches to measure robustness need to be reassessed.

## introduction the versatility and consumer growth of commercial llms like chatgpt give the impression that robustness evaluations such as out-of-domain (ood) and stress testing are no longer relevant. we argue that they remain important. many applications do not need the broad range of skills offered by general-purpose models, from writing clinical notes to drawing l a t e x unicorns @xcite . specializing moderate-scale models with finetuning still works better when a model must perform a specific task. 1 some nlp research embraces this view, but takes it to an extreme. bowman (2022) highlights the continued use of the 2018 bert-base model as a baseline, even though larger and better-pretrained models can be finetuned on a single gpu today. the rush to innovate upon general-purpose models and the disregard of stronger baselines make it unclear where the field stands in terms of established robustness evaluations. we seek to address this gap. we first point to popular experimental setups that, while of interest, may be outdated for robustness studies. specifically, we record the train-test splits of 177 acl publications that include keywords in figure 1 . after filtering, 101 splits remain that we use to finetune and evaluate 19 models that differ in (i) transformer type (encoder-/decoderonly, encoder-decoder), (ii) model size (60m to 13b), and (iii) pretraining objectives (lm/mlm only, additional multitask pretraining). we show that for 14 train/ood-test pairs, a finetuned model that is over 90% accuracy on the standard test set does not break on the ood test set. further research with these data splits is less likely to advance our understanding of ood robustness. we also show that a few challenge sets continue to break nli and reading comprehension models, but sentiment classifiers are robust to stress tests. through behavioral testing with the checklist methodology @xcite , we show that highly accurate models still struggle with the most basic task phenomena. larger models help, but do not fully resolve the issue; there is scant evidence that increasing size further would be beneficial. next, we employ contrast set evaluations @xcite , which measure model accuracy in a set of subtly different and differently labeled examples. these evaluations continue to expose major model weaknesses, raising the question of why they are not more widely used. finally, we stress a crucial finding -some robustness evaluations can themselves be fragile. we demonstrate this by re-figure 1 : finetuning and evaluation datasets determined by analyzing train-test splits in *acl/emnlp publications from 2020-2022 ( §2). individual traintest splits are reported in ## better evaluation paradigms exist evaluating with sets of mutually dependent examples can uncover model fragility. yet, models' continue to be assessed by their performance on benchmarks such as mmlu @xcite that consist solely of i.i.d. instances. are evaluations that go beyond the i.i.d. assumption unnecessary, or simply overlooked? background. motivated by the ongoing challenge of creating artifact-free nlp benchmarks, gardner et al. (2020) propose to evaluate with sets of examples that are minimally different from each other. they report contrast set consistency, a measure of how often a model correctly addresses every example in a set. models that succeed on standard benchmarks by exploiting data shortcuts can do this rarely. gardner et al. (2020) create contrast sets for 10 datasets, one of which is multimodal. ravichander et al. (2022) introduce condaqa, another contrastive dataset. experimental setup. instead of training 10 (unimodal datasets) ˆ19 (models) ˆ3 (seeds) = 570 models, we reexamine contrast set evaluation with flan-t5-11b @xcite whose instruction finetuning includes training data of all datasets in question except condaqa. we prompt flan-t5 with an instruction and 8 demonstrations (tables 18 19 20 21 22 23 24 25 , appendix). since flan models are instruction finetuned with chain-of-thought (cot) prompting @xcite and self-consistency @xcite , we also try these reasoning-boosting addons to get the highest consistency possible. 6 q1: is there still a gap between original test sets and their contrastive counterparts? yes, figure 4 shows that flan-t5-11b's performance on only the original instances in contrast sets is much higher than its consistency for each contrast 6 explanations are written by one of the authors. set. 7 surprisingly, the benefits of including explanations are negligible -relative to the standard greedy decoding, chain-of-thoughts do not improve consistency in a single case, and self-consistency improves it only for mctaco and matres. gardner et al. ( 2020 ) demonstrate the feasibility of creating a high-quality contrast set with 1k examples from an existing dataset in just a week's work by an expert. given the confirmed benefits of contrast datasets in robust reasoning evaluation, the development of contrastive versions of popular benchmarks such as bbh @xcite and mmlu seems beneficial. mindful of the challenges in evaluating across multiple benchmarks, each with dozens of sub-tasks, we deem it is more strategic to direct a portion of resources to evaluations that extend beyond benchmarks i.i.d. with test sets than having more of the same. q2: has consistency improved despite gaps? only in some cases. the consistency values for ropes, quoref, mctaco, and imdb show large gains over @xcite . yet, they remain low across the board (except for imdb). moreover, although flan-t5-11b's standard task performance exceeds the results reported by gardner et al. (2020) across almost all tasks, these improvements do not uniformly translate into increased consistency (e.g., boolq and drop). we also experimented with more recent llms like llama-2 @xcite and @xcite , but found them, on average, to be less consistent than flan-t5 (see appendix c.1 for results). in summary, building models that are consistently robust in local neighborhoods remains a challenge. ## conclusions we thoroughly investigate robustness of finetuned models up to 13b parameters from four perspectives: generalization under distributional shifts, checklist-based behavioral analysis, contrast set consistency, and robustness to adversarial attacks. our results collectively show that the current state of nlp robustness is multifaceted: in §3, we show that larger models generally exhibit more basic necessary task skills, and in §4, that contrast sets consistency has improved since their inception in 2020. we caution against robustness research that relies solely on bert-era models as baselines.