Measuring and Improving Attentiveness to Partial Inputs with Counterfactuals (2024)

Abstract: The inevitable appearance of spurious correlations in training datasets hurts the generalization of NLP models on unseen data. Previous work has found that datasets with paired inputs are prone to correlations between a specific part of the input (e.g., the hypothesis in NLI) and the label; consequently, models trained only on those outperform chance. Are these correlations picked up by models trained on the full input data? To address this question, we propose a new evaluation method, Counterfactual Attentiveness Test (CAT). CAT uses counterfactuals by replacing part of the input with its counterpart from a different example (subject to some restrictions), expecting an attentive model to change its prediction. Using CAT, we systematically investigate established supervised and in-context learning models on ten datasets spanning four tasks: natural language inference, reading comprehension, paraphrase detection, and visual & language reasoning. CAT reveals that reliance on such correlations is mainly data-dependent. Surprisingly, we find that GPT3 becomes less attentive with an increased number of demonstrations, while its accuracy on the test data improves. Our results demonstrate that augmenting training or demonstration data with counterfactuals is effective in improving models’ attentiveness. We show that models’ attentiveness measured by CAT reveals different conclusions from solely measuring correlations in data.

## introduction reliance on spurious correlations compromises nlp models' abilities to generalize across different domains and tasks @xcite . intuitively, spurious correlations are features that are useful in the training data but are cat: counterfactual attentiveness test ❓ figure 1 : we are interested in quantifying model's attentiveness to part of the input. we propose cat, an evaluation that replaces the premise p with a counterfactual p ′ . a change in behavior to the counterfactual input indicates the model is attentive to the premise, otherwise, the model relies solely on the hypothesis to make a prediction. spurious correlations in datasets are very common. for instance, the hypothesis-only baseline @xcite was designed to measure spurious correlations in natural language inference (nli) datasets by training a model only on one part of the input (e.g., the hypothesis), effectively discarding information deemed critical for performing the task. surprisingly, it performs significantly better than random guessing, despite never seeing the premise. these results suggest that the hypothesis-only baseline has taken the shortcuts between the hypotheses and the labels during training. however, they should not be interpreted as evidence that models trained on the full inputs (full-input models) also take these shortcuts. the question that then arises is how to quantify the reliance of full-input models on the spurious correlations present in the hypotheses. we propose a method for measuring models' reliance on partial input: counterfactual at-tentiveness test, (cat, §3). we present an overview of the approach in figure 1 . cat is simple, intuitive, and is based on the idea of counterfactuals from the causal inference literature @xcite . importantly, it only requires a labeled dataset for the task and black box model predictions. to test a model that performs a tasks with paired-inputs (tasks that consist of at least two components, e.g., nli), we query the model, and replace one part of the input (e.g., the premise) with that part from another instance randomly drawn from the same data split. intuitively, this perturbation results in the default label for that task (typically a label that signifies no relation between the two inputs, e.g. neutral in nli). as such, we focus on the subset of instances where the model predicts non-neutral. 2 if the prediction changes, it suggests that the model is attentive to the perturbed input. conversely, a model that keeps the prediction unchanged is likely to be not attentive, as it insensitive to the counterfactual. following this intuition, our metric measures attentiveness by calculating the percentage of predictions that changed from the non-neutral labels on the original instances. we conduct extensive experiments on four different tasks, ten datasets, and 15 different models, using two setups: supervised and in-context learning ( §4). we first extend previous work to obtain new results using the partial input baseline-a model trained only on partial inputs, that quantifies spurious correlations in the data. then, using our attentiveness metric, we show that even in datasets with spurious correlations between parts of the inputs and the labels, models do not always rely on them ( §5). finally, we study whether data augmentation using our counterfactuals improves models' attentiveness and find that is often the case ( §6). our results indicate that the appearance of -often unavoidable -spurious correlations in the data do not indicate models rely on them, and propose a simple and easy-to-use test for measuring it. ## background & related work in this section we review the hypothesis-only baseline in nli and its extensions to other tasks: the partial input baseline. we then discuss a related work by @xcite and highlight the similarities and differences to our work. ## quantifying attentiveness we seek a method that measures model attentiveness to a part of the input in paired-input tasks, such as the premise or hypothesis in nli. formally, assume an input x contains two parts: x : (x 1 , x 2 ), and let m @xmath0. we are interested in measuring the causal effect of x 1 on model m. if such effect is prominent, we conclude that the model relies on x 1 for inference, foot_1 otherwise it ignores it. by measuring such effect on a dataset we quantify the extent by which the model relies on x 1 . cases where such effect is large are especially interesting when the underlying training data contains correlations between x 2 and y, since the model does not rely on such correlations during training. we describe the partial input baseline and a naive approach for measuring attentiveness and explain why they do not fit as such metric. we then describe our method, cat, in §3.3. we summarize these approaches in ## experimental setup we consider four diverse tasks and ten english datasets. we also consider two learning strategies: supervised learning through fine-tuning using 12 models from six model families, and in-context learning (icl) using nine models from three model families. ## results standard evaluation we begin by training models on their corresponding datasets. for example, deberta-large achieves 87.73%, 91.28%, and 76.72% accuracy on rte, mnli, and wanli, respectively. 11 the full results are in appendix a. ## cat augmentation considering the simplicity and automatic nature of cat, we reuse the same sampling process for data augmentation to improve non attentive models. we follow a similar procedure as in the evaluation to augment the training data: apart from the regular training set, for every x 1 , x 2 pair, where the label y is different than the default label (e.g. neutral for nli), we sample x ′ 1 from the training data, assigned the default label. 13 the following experiments study the models that perform poorly on our metric in the standard setup. we focus on mnli since models are the least attentive on this dataset, and experiment once with t5large in the supervised setup, and with gpt-3 in the icl setup, using the original data (or demonstrations), and add the counterfactual instances. the labels of counterfactuals match the default label -neutral for mnli. we report both accuracy on the standard development set that verifies the augmentation does not degrade model performance and attentiveness. for each dataset, we report the performance of the vanilla model (those trained in §5 and used for cat in §5.1) and the augmented model. the results are in for gpt3, we expeirment with two baselines, one with 3 demonstration tuples with balanced labels, and the other with four, half of which having the neural label. the latter aims to control for the label distribution shift invited by the data augmentation process. the trend is different than the finetuend t5-large: gpt-3 with data augmentation sees decreased accuracy and increased attentiveness, but still far from perfect attentiveness. these results suggest that this simple augmentation procedure can be used as a method for improving model's robustness, as measured by cat. ## conclusions we studied models' reliance on partial inputs and proposed a method to quantify it. we applied our method, counterfactual attentiveness test (cat) on multiple tasks, datasets, and setups including fine-tuning and in-context learning. we found that the spurious correlations between a part of the inputs and the labels are not always indicative of the full model's reliance on such heuristics in the supervised setup. for instance, we found that while partial inputs in squad2.0 are correlated with the labels, models trained on it are attentive to both parts of the input. this is in contrast to mnli, whose hypotheses and labels are highly correlated, and models trained on it are also inattentive to the premises. in addition, in icl, we found that more demonstrations lead to decreased attentiveness in models such as gpt-3.