Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models (2021)

Abstract: While counterfactual examples are useful for analysis and training of NLP models, current generation methods either rely on manual labor to create very few counterfactuals, or only instantiate limited types of perturbations such as paraphrases or word substitutions. We present Polyjuice, a general-purpose counterfactual generator that allows for control over perturbation types and locations, trained by finetuning GPT-2 on multiple datasets of paired sentences. We show that Polyjuice produces diverse sets of realistic counterfactuals, which in turn are useful in various distinct applications: improving training and evaluation on three different tasks (with around 70% less annotation effort than manual generation), augmenting state-of-the-art explanation techniques, and supporting systematic counterfactual error analysis by revealing behaviors easily missed by human experts.

## introduction counterfactual reasoning -mentally simulating what would have happened if conditions were different -is a common tool for making causality assessments @xcite , which in turn are crucial for model evaluation, error analysis, and explanation @xcite . for example, in figure 1 , "it is great for kids" is perturbed into multiple variations, each providing unique insights by simulating what would have happened if the sentence was different. applications of counterfactual reasoning to nlp generally specify the relationship x ) x, and then create x according to the relationship. as a result, prior work has tailored counterfactual generators for different applications, only collecting subsets of x that are useful for the specific task. for example, to support model training and evaluation, human annotators create counterfactuals it is great for kids. it is great for @xmath0. it is @xmath1. ## counterfactual evaluation & training we ask crowdworkers to label polyjuice-generated counterfactuals for sentiment, nli, and qqp, for the purposes of evaluation and training. 5 in each labeling round, the worker is presented with an original x and its label, and asked to annotate the groundtruth for three x, rejecting non-fluent ones (details and interface in appendix b.1). we use a simple heuristic to select which counterfactuals are presented for labeling, aimed at increasing diversity. representing each x by its token changes, control code, and dependency tree structure, we greedily select the ones that are least similar to those already selected for labeling. this avoids redundancy in the labeling set, e.g., common perturbation patterns such as black ) white. ## counterfactual explanations a popular way of explaining nlp models is to attribute importance weights to the input tokens, either using attention scores @xcite or by summarizing the model behavior on perturbed instances (e.g., lime @xcite and shap (lundberg and lee, 2017)). though ubiquitous, token scores may not always reflect their real importance @xcite . popular packages like lime or shap estimate scores by masking words, and therefore may not reflect model behavior on natural counterfactual cases. for example, the token "friend" in figure 3a is not considered important even though a natural substitution in figure 3b flips the prediction. the opposite happens to "in depression," where a significant change makes no difference to the model's prediction (figure 3c ). even perfect importance scores may be too abstract for users to gain real understanding @xcite , e.g., users may not grasp the significance of a low importance score for the token "help" without concrete examples such as the one in figure 3d . since presenting a large number of concrete counterfactuals would be overwhelming, we propose a hybrid approach, displaying feature attributions as a high-level summary, together with a judicious selection of polyjuice counterfactuals that make behaviors concrete and highlight potential limitations. following @xcite 's observation that people look for explanations revealing unexpected behavior, we select surprising counterfactuals. 8 that is, we estimate the expected change in prediction with feature attributions, and select counterfactuals that violate these expectations, i.e., examples where the real change in prediction is large even though importance scores are low (figure 3b ), and examples where the change is small but importance scores are high (figure 3c ). of course, users can also view additional counterfactuals that perturb tokens of particular interest, a technique that we explore in the next section. user evaluation. we study the scenario where an expert has access to a model and local explanations, and evaluate the additional benefit of showing counterfactuals, i.e., whether they bring new insights. we evaluate three ways of generating counterfactuals: (1) polyjuice-random, a baseline where we show random polyjuice counterfactuals, (2) expert-surprise, where two graduate students (non-participants) were given access to the model and instructed to create counterfactuals that are surprising given the associated shap scores, and (3) polyjuice-surprise, which uses the selection procedure described in the previous paragraph. we recruited 13 participants (graduate students with experience in model explanation), and had them analyze the aforementioned qqp model. in each round, users were shown an example, the model prediction, and a shap explanation, as in figure 3a . users were instructed to create up to 10 counterfactuals in order to better understand model behavior around the example, for which model predictions were given (users created 6 on average). finally, users simulated what the model would do on six counterfactuals @xcite , two from each condition (in random order). counterfactuals where users make mistakes are prefer- as shown in figure 4 , humans simulated model behavior on polyjuice-surprise counterfactuals only slightly better than random guessing (45% ± 6%), i.e., these examples display model behavior that is surprising to users even after seeing explanations and creating their own counterfactuals. expert-surprise also had a high error rate, but at a much higher cost: generating these for just 20 original instances took 1.5-2 hours of expert labor. while high error rates could be achieved with unrelated or nonsensical examples, all counterfactuals under evaluation were close to the original examples, when measured by syntactic tree edit (≈1.0) or levenshtein distance (≈0.2), polyjuicesurprise being the closest on both. an independent rater labeled 95% of polyjuice-surprise counterfactuals as "likely written by a native speaker," in contrast to 85% for expert-surprise, indicating that experts sometimes resorted to ungrammatical or nonsensical sentences to find surprising behaviors. qualitatively, the study participants tended to create counterfactuals by perturbing the token with the highest weights (84% of their x perturbed tokens in the top 15% quantile of weights), not gaining a real understanding of how the other tokens impact predictions. participants also made a significant number of mistakes even for tokens they had inspected, e.g., a participant perturbed the example in figure 3a by replacing help ) play with, yielding a non-duplicate model prediction. when faced with help ) find in figure 3d , they incorrectly assumed the behavior would be the same. these results indicate that polyjuice counterfactuals complement feature attribution explanations by displaying information that users often miss, even after they have manually explored the model behavior beyond explanations. moreover, polyjuice counterfactuals for this application were more surprising and fluent than expert-surprise, despite being computed automatically. ## interactive analysis while our use of polyjuice has so far relied on automatic selection of counterfactuals, we show in this section how an analyst can benefit from multiple counterfactuals per x, make use of controlled generation for more advanced analysis, and extract general patterns from individual observations. our use case is counterfactual error analysis @xcite of roberta finetuned on nli (used in §3.1), although the techniques are generally applicable. there is a known correlation between the label contradiction and hypotheses with negation in nli datasets @xcite , which may cause models to fail on non-contradiction negations. we explore this in figure 5a by generating counterfactual hypotheses for a random neutral instance, conditioning only on the original x and the negation control code. while the first two counterfactuals display this failure mode, there is a surprising inconsistency in model behavior between "not" and "n't". we note that manual analysis may not explore these three negation forms, and thus not surface this puzzling behavior. to verify if the pattern is widespread, we generate counterfactuals with the negation control code for a random set of instances correctly predicted as neutral (@xmath2. to generalize individual changes into patterns, we extract frequent counterfactual templates with tempura @xcite (details in appendix d.2), shown in figure 5b . the top templates (in bold) show that the model flips its prediction from neutral to contradiction with roughly the same frequency (≈43%) whether the negation word is "not" or "n't", but flips much more frequently with a different negation pattern where a determiner is replaced with "no" (92.8%). while these behaviors may be correct in some instances, they often are not (e.g., figure 5a ), and thus would warrant further exploration, and potential mitigation strategies (e.g., counterfactual training, §3). tangentially, the impact of det ) no might lead the analyst to explore the impact of perturbing the subject of hypotheses, which we do in figure 6 by placing a [blank] on the subject rather than using a control code. this leads to the discovery of unstable and erroneous behaviors regarding quantifiers, which we analyze in more detail in appendix d.1. discussion. polyjuice is a powerful tool for interactive analysis. generating multiple counterfactuals per instance leads to insights that might be missed by manual analysis, and the steering provided by control codes and [blank]s allow for analyses that would be non-trivial to do manually @xcite or with masked language models (e.g., figure 5b places negations in various parts of sentences, and figure 6 replaces spans with other spans of varying lengths). besides error analysis, an analogous interactive use of polyjuice may be suitable for test creation @xcite and forms of data augmentation that are more controlled than what we presented in §3. ## related work some prior work in training and evaluation relies on humans to generate counterfactuals from scratch @xcite @xcite . our experiments in §3 indicate that asking humans to label polyjuice counterfactuals yields similar or better results at a lower cost, which motivates an exploration of a mixture of manual and semi-automated generation. similarly, prior work on analysis relies on experts to create individual counterfactuals or perturbation functions @xcite . in §5, we show that polyjuice enhances current practice by generating multiple counterfactuals that might have been overlooked, and by providing abstractions that allow for new kinds of analyses. prior work on automatically generating counterfactuals typically has a narrower scope in terms of the relationships x ) x. for example, adversarial generators aim to maintain semantics while changing model predictions @xcite @xcite , whereas concurrent work to our own @xcite automatically generates x that change predictions for explanation or analysis, with no constraints on semantics. however, as shown in §3- §5, a mix of label-preserving and label-flipping counterfactuals generated by polyjuice is quite useful for training, evaluation, explanation, and analysis. further, general-purpose counterfactuals may lead to serendipitous discoveries ( §5), especially as polyjuice is not fine-tuned to the target domain (and thus less liable to merely replicate what is already there). finally, by allowing control through control codes and [blank]s, polyjuice supports humangenerator collaboration, where a person specifies desired changes (e.g., perturb the sentence subject). such collaboration is hard to imagine using automatic generators with no control, or with coarser control through predefined style attributes or labels @xcite . to our knowledge, prior work on controlled generation @xcite does not address counterfactual generation. ## conclusion and future work we propose polyjuice, a general-purpose generator that produces fluent and diverse counterfactuals, allowing for control over the kinds and locations of perturbations. with simple, task-specific selection heuristics, polyjuice supports various downstream tasks on different domains, including counterfactual data augmentation, contrast set generation, counterfactual explanation, and error analysis. while polyjuice is broadly applicable, it is not bias-free: control codes are pre-defined and certainly not exhaustive, and the model is fine-tuned on a collection of paired datasets where certain perturbations are more or less likely (e.g., we observe that words with negative sentiment tend to be slightly more likely than positive ones in some contexts). collecting naturally occurring counterfactuals is an important area of future research, as is the development of generators that allow for control even without a-priori control codes. besides improving the generators, further work is needed to improve the value of counterfactuals. for example, while polyjuice shows consistent gains across tasks in data augmentation, the improvements on some datasets are not as significant. this aligns with observations in prior work that even manual counterfactuals can be marginally beneficial @xcite , possibly because the original data is already diverse enough, or the perturbed signal in counterfactuals is too subtle to affect the model (e.g., when only a single word is changed in a long sentence.) we hope to perform more thorough experiments on tuning the amount and the distribution of counterfactual augmentation, as well as other ways of incorporating counterfactuals, such as having explicit terms in the loss function for contrasting counterfactuals with original data @xcite , or other forms of contrastive learning. although our applications all involved people, the human-polyjuice collaboration in labeling and explanations could benefit from richer interaction mechanisms. we believe polyjuice motivates future research on more expressive forms of counterfactual training, where users generate counterfactuals together with polyjuice, and label counterfactual patterns rather than individual instances. similarly, interactive explanations and analysis are exciting directions, especially as we develop new ways of selecting, presenting, and aggregating counterfactuals for various analysis objectives. having noted these opportunities, we believe polyjuice is already a powerful tool for counterfactual reasoning, in particular for tasks where people are directly involved. polyjuice is opensource, and available at https://github.com/tongshuangwu/polyjuice .