CE val: A Benchmark for Evaluating Counterfactual Text Generation (2024)

Abstract: Counterfactual text generation aims to minimally change a text, such that it is classified differently. Assessing progress in method development for counterfactual text generation is hindered by a non-uniform usage of data sets and metrics in related work. We propose CEval, a benchmark for comparing counterfactual text generation methods. CEval unifies counterfactual and text quality metrics, includes common counterfactual datasets with human annotations, standard baselines (MICE, GDBA, CREST) and the open-source language model LLAMA-2. Our experiments found no perfect method for generating counterfactual text. Methods that excel at counterfactual metrics often produce lower-quality text while LLMs with simple prompts generate high-quality text but struggle with counterfactual criteria. By making CEval available as an open-source Python library, we encourage the community to contribute additional methods and maintain consistent evaluation in future work.

## introduction the rise of deep learning and complex "black-box" models has created a critical need for interpretability. as @xcite notes, explanations often involve counterfactuals to understand why event p occurred instead of q. ideally, these explanations show how minimal changes in an instance could lead to different outcomes. for example, to explain why the review "the film has funny moments and talented actors, but it feels long." is negative rather than positive, a counterfactual like "the film has funny moments and talented actors, yet feels a bit long." can be used (see fig. 1 for more counterfactual examples generated by different methods on the same original instance). this explanation highlights specific words to change and modifications if you haven't seen this, it's terrible a masterpiece. it is pure trash brilliance. i saw this about 17 years ago, and i'm still in screwed up awe from it. if you haven't seen this, it's terrible pretty. it is pure trash genius. i saw this about 17 years ago, and i'm still screwed up from it. if you haven't seen this, it's terrible complicated. it is pure trash the magic. i saw it about 17 30 years ago, and i'm still screwed reeling up from it. if you haven't seen this movie , it's terrible definitely worth seeing. it is pure trash 's great. i saw it about 17 years ago, and i'm still screwed up from it. if you haven't seen this, it's terrible. it is pure trash. i saw this about 17 years ago, and i'm still screwed up from it. ## related work terms like "counterfactual" and "contrastive" generation are often used interchangeably in literature @xcite and our work adopts an inclusive definition. we define counterfactual generation as a process of generating a new instance x ′ , from the original instance x, that results in a different model prediction y ′ with minimal changes. this definition includes counterfactual, contrastive generation, and adversarial attacks. primarily, adversarial attacks focused on changing the label without considering text quality. recent work like gbda @xcite focuses on producing adversarial text that is more natural by adding fluency and semantic similarity losses. hence, we include gbda in our benchmark. technically, counterfactual generation methods for text fall into three categories: masking and filling methods (mf): these methods perform two steps: (1) identifying important words for masking by various techniques, such as selecting words with the highest gradient or training a separate rationalizer for the masking process and (2) replacing the masked words using a pretrained language model with fill-in-the-blank capability. in step (1), mice @xcite and autocad @xcite use the gradient of the classifier. docogen @xcite identifies all domain-specific terms by calculating a masking score for n-grams (where @xmath0. meanwhile, crest @xcite trains spectra (guerreiro and martins, 2021) as a separate rationalizer to detect which phrases or words to mask. in step (2), each of these methods fine-tunes t5 to fill in the blanks created during masking. additionally, polyjuice @xcite ) takes text with user-specified manual masking as input and fine-tunes a roberta-based model to fill in the blanks using control codes. conditional distribution methods (cd): methods like gbda @xcite and cf-gan @xcite learn a conditional distribution for counterfactuals. the counterfactuals are obtained by sampling from this distribution based on a target label. counterfactual generation with large language models: recently, there has been a trend towards using large language models (llms) for counterfactual generation. approaches like core @xcite , disco @xcite and flare @xcite optimize prompts fed into llms to generate the desired counterfactuals. this trend is driven by the versatile capabilities of llms in various tasks @xcite . despite the diverse approaches proposed in generating counterfactuals across various studies, the common objective remains to generate high-quality counterfactuals. however, previous studies employed different metrics, baselines, and datasets, as illustrated in ## benchmark design we focus on counterfactual generation for textual data, which involves editing given text with minimal modifications to produce new text that increases the probability of a predefined target label with respect to a black-box classifier. this process aims to generate a counterfactual, denoted as x ′ , that changes the classifier's predictions compared to the original text x. formally, given a fixed classifier f and a dataset with n samples (x 1 , x 2 , . . . , x n ), x @xmath1, z 2 , . . . , z n ) represents a sequence of n tokens. the original prediction is denoted as f (x) = y, while the counterfactual prediction is y ′ ̸ = y. the counterfactual generation process is represented by a method e : (z 1 , . . . , z n ) → (z ′ 1 , . . . , z ′ m ), ensuring that f (e(x)) = y ′ . the resulting counterfactual example is x ′ = (z ′ 1 , . . . , z ′ m ) with m tokens. a valid counterfactual instance should satisfy the following criteria @xcite : predictive probability: a counterfactual instance x ′ should closely produce the predefined prediction y ′ . in other words, the counterfactual text should obtain the desired target label. textual similarity: a counterfactual x ′ should maintain as much similarity as possible to the original instance x in terms of text distance. this ensures that the generated text remains coherent and contextually aligned with the original. likelihood in feature space: a counterfactual should exhibit feature values that resemble realworld text, indicating that x ′ remains close to a common distribution for text. this criterion ensures that the generated text is plausible, realistic and consistent with typical language patterns. diversity: when an explanation is ineffective, humans can offer alternatives. similarly, if a counterfactual is unrealistic or not actionable, it is beneficial to modify the original instance differently to provide diverse options @xcite . therefore, an effective counterfactual method should present multiple ways to change a text instance to obtain the target label. diversity is measures for a set of counterfactual instances. ## counterfactual methods selection in this section, we briefly describe the counterfactual generation methods we evaluate with our benchmark. we selected at least one representative for each of the categories masking and filling (mf), conditional distribution (cd) and large language models (llms) (cf. section 2) based on the following criteria: we used the criteria reproducible code and problem agnostic as hard filters and access to the target classifier as soft filter. a problem agnostic method is versatile enough to generate counterfactuals for various types of classification problems (whereas methods like polyjuice @xcite or tailor @xcite require control codes, which limits their flexibility). methods without access to the target classifier are at disadvantage, as they have no information about the internals of the target classifier. hence, wherever available, we opted for a method with access to the target classifier. the selection based on these criteria (cf. details in appendix, gbda @xcite is a gradient-based adversarial attack that uses a novel adversarial distribution for end-to-end optimization of adversarial loss and fluency constraints via gradient descent. similar to mice, this approach needs access to the classifier for internal label verification. this method represents the adversarial attack domain. crest @xcite follows a similar approach as mice in first masking tokens that should be changed. instead of using the highest gradient tokens to find the masks, the authors train a rationalizer using spectra (guerreiro and martins, 2021). then, they fill the blanks with t5 same as mice. given the popularity of the mask and filling type, we chose this method for a more comprehensive comparison. llama-2 @xcite : large language models have shown good performance on many tasks with only simple prompts @xcite . therefore, in this study, we use llama-2 with simple one-shot learning as a baseline that is not specifically designed for counterfactual generation, but has strong language generation capabilities. the choice for llama-2 as an opensource model is made in contrast to other studies that used closed-source llms. the hyperparameters of each selected method can significantly impact the results, particularly for mice @xcite and crest @xcite . the percentage of masked tokens in both methods, representing the upper bound of changed tokens, directly influences the token distance and indirectly affects the flip rate: a lower percentage allows fewer tokens to change, resulting in a smaller distance but potentially a lower flip rate. in our experiments, we maintain the hyperparameters as specified in the original papers of each method. in case of llama-2, the temperature of llms affects word sampling: lower temperatures yield more deterministic results, while higher temperatures enhance creativity. for the comparison with other methods, we use a temperature of 1.0 and analyze the impact of varying temperatures at the end of the next section. ## results we evaluate all counterfactual generation methods against human crowd-sourced and human expert generations. note that mice and gbda have access to the prediction model during generation, while crest employs a pre-trained t5 model for internal label verification and transfers its prediction to the target bert model. in contrast, llama-2 and both human evaluation groups (crowd and expert) generate counterfactual examples solely based on the provided text and prompt. we start with an example to illustrate the methods' varying characteristics before discussing our observations from the quantitative results. fig. 1 shows the shortest example in the imdb dataset where all methods, including human edits, change the label of the original sentence on the generated counterfactual. for this simple instance, all methods and human groups agree on replacing negative words like terrible and trash with positive words, even though they differ in their choice of positive words. gdba is the only exception, its replacements do not always convey a positive sentiment, which reduces text quality. similarly, mice and crest fail to detect the negative phrase screwed up , which renders the text less cohesive and fluent than the text generated by llama-2 and humans, who adapt this negative phrase as well. besides correctly identifying important words, gdba also replaces irrelevant words like 17 30 , resulting in a larger edit distance. for a more complex example with higher variation of edits and generated text, see ## comparison of llms for text quality evaluation evaluating text quality with chatgpt has been shown to be effective @xcite . however, such evaluations come at high costs, limited control and customization constraints, and lack transparency. therefore, we investigate an open-source llm, mistral-7b @xcite as an evaluation proxy. ## conclusion we propose ceval to standardize the evaluation of counterfactual text generation, emphasizing the importance of both counterfactual metrics and text quality. our benchmark facilitates standardized comparisons and analyzes the strengths and weaknesses of individual methods. initial results show that counterfactual methods excel in counterfactual metrics but produce lower-quality text, while llms generate high-quality text but struggle to reliably flip labels. combining these approaches could guide future research, such as using target classifier supervision to enhance llm outputs. the diversity in method performance highlights the need for robustness analyses of target classifiers with mul-tiple methods. our findings also suggest that the target classifier may be poorly calibrated, warranting further investigation. finally, we demonstrate that text quality evaluation using llms is robust to temperature changes. additionally, we show that open-source llms, like mistral, can serve as alternatives to closed-source models, such as chatgpt, for evaluating text quality, thereby overcoming weaknesses of closed-source models, such as api deprecation or high costs. this leads to ceval being a fully open-source python library, encouraging the community to contribute additional methods and to ensure that future work follows the same standards. for future work, we plan to integrate llms specifically designed for evaluation, such as prometheus @xcite , as an option for assessing text quality. furthermore, instead of only considering the difference between instances to measure diversity, the diversity metric can be expanded to incorporate the particular types of changes, such as negation and word replacements.