An Investigation of the (In)effectiveness of Counterfactually Augmented Data (2022)

Abstract: While pretrained language models achieve excellent performance on natural language understanding benchmarks, they tend to rely on spurious correlations and generalize poorly to out-of-distribution (OOD) data. Recent work has explored using counterfactually-augmented data (CAD)—data generated by minimally perturbing examples to flip the ground-truth label—to identify robust features that are invariant under distribution shift. However, empirical results using CAD during training for OOD generalization have been mixed. To explain this discrepancy, through a toy theoretical example and empirical analysis on two crowdsourced CAD datasets, we show that: (a) while features perturbed in CAD are indeed robust features, it may prevent the model from learning unperturbed robust features; and (b) CAD may exacerbate existing spurious correlations in the data. Our results thus show that the lack of perturbation diversity limits CAD’s effectiveness on OOD generalization, calling for innovative crowdsourcing procedures to elicit diverse perturbation of examples.

## introduction large-scale datasets have enabled tremendous progress in natural language understanding (nlu) @xcite with the rise of pretrained language models @xcite . despite this progress, there have been numerous works showing that models rely on spurious correlations in the datasets, i.e. heuristics that are effective on a specific dataset but do not hold in general @xcite @xcite . for example, bert @xcite trained on mnli @xcite learns the spurious correlation between world overlap and entailment label. a recent promising direction is to collect counterfactually-augmented data (cad) by asking humans to minimally edit examples to flip their ground-truth label @xcite ). figure 1 shows example edits for natural language inference (nli). given interventions on robust features that "cause" the label to change, the model is expected to learn to disentangle the spurious and robust features. despite recent attempt to explain the efficacy of cad by analyzing the underlying causal structure of the data @xcite , empirical results on out-of-distribution (ood) generalization using cad are mixed. specifically, @xcite show that cad does not improve ood generalization for nli; @xcite find that for question answering, cad is helpful only when it is much cheaper to create than standard examples -but @xcite report that the cost is actually similar per example. in this work, we take a step towards bridging this gap between what theory suggests and what we observe in practice in regards to cad. an intuitive example to illustrate our key observation is shown in figure 1 (a) , where the verb 'eating' is changed to 'drinking' to flip the label. while there are many other words that could have been changed to flip the label, given only these two examples, the model learns to use only the verbs (e.g. using a naive bayes model, all other words would have zero weights). as a result, this model would fail when evaluated on examples such as those in (b) where the quantifier 'two' is changed to 'three', while a model trained on the unaugmented data may learn to use the quantifiers. first, we use a toy theoretical setting to formalize counterfactual augmentation, and demonstrate that with cad, the model can learn to ignore the spurious features without explicitly intervening on them. however, we find that without perturbing all robust features to generate cad, perturbations of one robust feature can prevent the model from learning other unperturbed robust features. motivated by this, we set up an empirical analysis on two crowdsourced cad datasets collected for nli and question answering (qa). in the empirical analysis, we identify the robust features by categorizing the edits into different perturbation types @xcite (e.g. negating a sentence or changing the quantifiers), and show that models do not generalize well to unseen perturbation types, sometimes even performing worse than models trained on unaugmented data. our analysis of the relation between perturbation types and generalization can help explain other observations such as cad being more beneficial in the low-data regime. with increasing data size, improvement from using cad plateaus compared to unaugmented data, suggesting that the number of perturbation types in existing cad datasets does not keep increasing. another consequence of the lack of diversity in edits is annotation artifacts, which may produce spurious correlations similar to what happens in standard crowdsourcing procedures. while cad is intended to debias the dataset, surprisingly, we find that crowdsourced cad for nli exacerbates word overlap bias @xcite and negation bias @xcite observed in existing benchmarks. in sum, we show that while cad can help the model ignore spurious feature, its effectiveness in current cad datasets is limited by the set of robust features that are perturbed. furthermore, cad may exacerbate spurious correlations in existing benchmarks. our results highlight the importance of increasing the diversity of counterfactual perturbations during crowdsourcing: we need to elicit more diverse edits of examples that make models more robust to the complexity of language. ## toy example: analysis of a linear model in this section, we use a toy setting with a linear gaussian model and squared loss to formalize counterfactual augmentation and discuss the conditions required for it's effectiveness. the toy example serves to motivate our empirical analysis in section 3. ## diversity and generalization in cad in this section, we test the following hypothesis based on the above analysis: models trained on cad are limited to the specific robust features that are perturbed and may not learn other unperturbed robust features. we empirically analyze how augmenting counterfactual examples by perturbing one robust feature affects the performance on examples generated by perturbing other robust features. ## cad exacerbates existing spurious correlation an artifact of underdiverse perturbations is the newly introduced spurious correlations. as an example, in the extreme case where all entailment examples are flipped to non-entailment by the negation operation in ## conclusion and future directions in this work, we first analyzed cad theoretically using a linear model and showed that models do not generalize to unperturbed robust features. we then empirically demonstrated this issue in two cad datasets, where models do not generalize well to unseen perturbation types. we also showed that cad amplifies existing spurious correlations, pointing out another concern. given these results, a natural question is: how can we fix these problems and make cad more useful for ood generalization? we discuss a few directions which we think could be helpful: we hope that this analysis spurs future work on cad, making them more useful for ood generalization.