LLM s for Generating and Evaluating Counterfactuals: A Comprehensive Study (2024)

Abstract: As NLP models become more complex, understanding their decisions becomes more crucial. Counterfactuals (CFs), where minimal changes to inputs flip a model’s prediction, offer a way to explain these models. While Large Language Models (LLMs) have shown remarkable performance in NLP tasks, their efficacy in generating high-quality CFs remains uncertain. This work fills this gap by investigating how well LLMs generate CFs for three tasks. We conduct a comprehensive comparison of several common LLMs, and evaluate their CFs, assessing both intrinsic metrics, and the impact of these CFs on data augmentation. Moreover, we analyze differences between human and LLM-generated CFs, providing insights for future research directions. Our results show that LLMs generate fluent CFs, but struggle to keep the induced changes minimal. Generating CFs for Sentiment Analysis (SA) is less challenging than NLI and Hate Speech (HS) where LLMs show weaknesses in generating CFs that flip the original label. This also reflects on the data augmentation performance, where we observe a large gap between augmenting with human and LLM CFs. Furthermore, we evaluate LLMs’ ability to assess CFs in a mislabelled data setting, and show that they have a strong bias towards agreeing with the provided labels. GPT4 is more robust against this bias, but it shows strong preference to its own generations. Our analysis suggests that safety training is causing GPT4 to prefer its generations, since these generations do not contain harmful content. Our findings reveal several limitations and point to potential future work directions.

## introduction the growing popularity of artificial intelligence (ai) and increasingly complex "black-box" models have triggered a critical need for interpretability. as @xcite seek to understand why an event p occurred instead of an alternative q. ideally, explanations should demonstrate how minimal changes to an instance could have led to different outcomes. in the context of textual data, this translates to introducing minimal modifications to the text through word additions, replacements, or deletions, to flip the label assigned by a given classifier. counterfactual generation in nlp aims to foster an understanding of models, thereby facilitating their improvement @xcite , debugging @xcite , or rectification @xcite . in the field of nlp, llms have consistently demonstrated remarkable performance across diverse tasks. however, despite significant advancements in counterfactual generation methods, the efficacy of llms in producing high-quality counterfactuals (cfs) remains an open question. our study bridges this gap by rigorously assessing the inherent capability of llms to generate cfs and identifying the most effective ones. we conduct a comprehensive comparison of several common llms, spanning different sizes and accessibility levels, evaluating their performance specifically on the counterfactual generation task. our assessment encompasses standard metrics for cfs quality, as well as an in-depth evaluation of language fluency tailored to the context of counterfactual generation. furthermore, we extend our analysis to data augmentation. we consider generating cfs for 3 tasks in this study: sentiment analysis (sa), natural language inference (nli), and hate speech (hs). our analysis demonstrates that llms are able to generate fluent text. however, they have difficulties in inducing minimal changes. generating cfs for sa is less challenging than nli and hs, where llms exhibit weaknesses in generating cfs that flip the labels. for data augmentation, sa cfs from llms can be an alternative to human cfs, as they are able to achieve similar performance, while on nli and hs further improvements are needed. furthermore, we show a positive correlation between keeping minimal changes and data augmentation performance. this suggests a new direction to generate improved data for augmentation, potentially leading to more efficient augmentation approaches. we further assess the ability of llms to act as evaluators of cfs. we show a sample of cfs from different llms with the corresponding scores in figure 1 . by conducting controlled experiments, we show that llms have a strong bias to agree with the given labels, even if these labels are incorrect. gpt4 demonstrates strong preference to its own generations. our analysis suggests that one reason for this preference is safety training, i.e., gpt4 prefers its own generations, because these generations do not contain any harmful content. finally, to facilitate further research, we contribute a new dataset of cfs generated by various llms. 1 ## evaluation methodology we conduct a multi-faceted evaluation, considering several use cases where cfs could be beneficial. ## conclusion in this work, we investigated the use of various llms for cfs generation. our results show that llms generate fluent cfs, but struggle to keep the induced changes minimal. generating cfs for sa is less challenging than nli and hs, where llms show weaknesses in generating cfs that change the original label. cfs from llms can replace human cfs for the purpose of data augmentation on sa and achieve similar performance, while on nli and hs further improvements are needed. further, our results suggest that cfs with minimal changes are essential for data augmentation. we also showed that when asked to asses cfs, llms exhibit a strong bias towards agreeing with the provided label even if this label is incorrect. gpt4 appears to be more robust than gpt3.5 against this bias. furthermore, we showed that gpt4 scores its own generations higher and that safety training might be one reason for this preference, i.e., gpt4 prefers its own generations, because they do not contain any harmful content. future work should focus on (i) leveraging llms for higher quality nli and hs cfs, which correctly change the label and keep changes minimal, (ii) assessing the evaluation abilities of llms in mislabeled data settings, and (iii) investigating the effects of safety training on llms as evaluators.