A uto CAD : Automatically Generate Counterfactuals for Mitigating Shortcut Learning (2022)

Abstract: Recent studies have shown the impressive efficacy of counterfactually augmented data (CAD) for reducing NLU models’ reliance on spurious features and improving their generalizability. However, current methods still heavily rely on human efforts or task-specific designs to generate counterfactuals, thereby impeding CAD’s applicability to a broad range of NLU tasks. In this paper, we present AutoCAD, a fully automatic and task-agnostic CAD generation framework. AutoCAD first leverages a classifier to unsupervisedly identify rationales as spans to be intervened, which disentangles spurious and causal features. Then, AutoCAD performs controllable generation enhanced by unlikelihood training to produce diverse counterfactuals. Extensive evaluations on multiple out-of-domain and challenge benchmarks demonstrate that AutoCAD consistently and significantly boosts the out-of-distribution performance of powerful pre-trained models across different NLU tasks, which is comparable or even better than previous state-of-the-art human-in-the-loop or task-specific CAD methods.

## introduction state-of-the-art nlu models have achieved impressive in-distribution performance, even surpassing humans on many benchmarks such as glue @xcite and superglue @xcite . however, these apparently powerful nlu models are known to suffer from shortcut learning, i.e., learning spurious features in datasets instead of actually solving the underlying tasks, thereby leading to unsatisfactory generalizability @xcite . for example, in natural language inference, a negation word can be a strong indicator ## conclusion we propose autocad, a fully automatic and taskagnostic counterfactually augmented data generation framework. autocad combines effective rationale extraction methods and a controllable generative model enhanced by unlikelihood training, which can generate diverse counterfactuals. extensive experiments on multiple out-of-domain and challenge benchmarks demonstrate that auto-cad consistently and significantly improves the out-of-distribution performance of powerful pretrained nlu models. more importantly, autocad achieves comparable or even better performance than previous state-of-the-art human-in-the-loop or task-specific methods. we believe this work has broad interests in various nlu tasks.