Counterfactual examples have been shown to be useful for many applications, including calibrating, evaluating, and explaining model decision boundaries. However, previous methods for generating such counterfactual examples have been tightly tailored to a specific application, used a limited range of linguistic patterns, or are hard to scale. We propose to disentangle counterfactual generation from its use cases, i.e., gather general-purpose counterfactuals first, and then select them for specific applications. We frame the automated counterfactual generation as text generation, and finetune GPT-2 into a generator , Polyjuice, which produces fluent and diverse counterfactuals. Our method also allows control over where perturbations happen and what they do . We show Polyjuice supports multiple use cases: by generating diverse counterfactuals for humans to label, Polyjuice helps produce high-quality datasets for model training and evaluation , requiring 40\% less human effort . When used to generate explanations, Polyjuice helps augment feature attribution methods to reveal models' erroneous behaviors .
While counterfactual examples are useful for analysis and training of NLP models, current generation methods either rely on manual labor to create very few counterfactuals, or only instantiate limited types of perturbations such as paraphrases or word substitutions. We present Polyjuice, a general-purpose counterfactual generator that allows for control over perturbation types and locations, trained by finetuning GPT-2 on multiple datasets of paired sentences . We show that Polyjuice produces diverse sets of realistic counterfactuals, which in turn are useful in various distinct applications: improving training and evaluation on three different tasks (with around 70\% less annotation effort than manual generation), augmenting state-of-the-art explanation techniques, and supporting systematic counterfactual error analysis by revealing behaviors easily missed by human experts .
R
Counterfactual examples have been shown to be useful for many applications, including calibrating, evaluating, and explaining model decision boundaries. However, previous methods for generating such counterfactual examples have been tightly tailored to a specific application, used a limited range of linguistic patterns, or are hard to scale. We propose to disentangle counterfactual generation from its use cases, i.e., gather
While counterfactual examples are useful for analysis and training of NLP models, current generation methods either rely on manual labor to create very few counterfactuals, or only instantiate limited types of perturbations such as paraphrases or word substitutions. We present Polyjuice, a
R
counterfactuals first, and then select them for specific applications. We frame the automated counterfactual generation as text generation, and finetune
counterfactual generator that allows for control over perturbation types and locations, trained by finetuning
R
into a generator , Polyjuice, which produces fluent and diverse counterfactuals. Our method also allows control over where perturbations happen and what they do
on multiple datasets of paired sentences
R
Polyjuice supports multiple use cases: by generating diverse counterfactuals for humans to label, Polyjuice helps produce high-quality datasets for model
that Polyjuice produces diverse sets of realistic counterfactuals, which in turn are useful in various distinct applications: improving
R
, requiring 40\% less human effort . When used to generate explanations, Polyjuice helps augment feature attribution methods to reveal models' erroneous behaviors
on three different tasks (with around 70\% less annotation effort than manual generation), augmenting state-of-the-art explanation techniques, and supporting systematic counterfactual error analysis by revealing behaviors easily missed by human experts