Unsupervised argument reframing with a counterfactual-based approach (2023)

Abstract: Framing is an important mechanism in argumentation, as participants in a debate tend to emphasize those aspects or dimensions of the issue under debate that support their standpoint. The task of reframing an argument, that is changing the underlying framing, has received increasing attention recently. We propose a novel unsupervised approach to argument reframing that takes inspiration from counterfactual explanation generation approaches in the field of eXplainable AI (XAI). We formalize the task as a mask-and-replace approach in which an LLM is tasked to replace masked tokens associated with a set of frames to be eliminated by other tokens related to a set of target frames to be added. Our method relies on two key mechanisms: framed decoding and reranking based on a number of metrics similar to those used in XAI to search for a suitable counterfactual. We evaluate our approach on three topics using the dataset by Ruckdeschel and Wiedemann (2022). We show that our two key mechanisms outperform an unguided LLM as a baseline by increasing the ratio of successfully reframed arguments by almost an order of magnitude.

## introduction framing is an important mechanism in argumentation, as participants in a debate tend to emphasize those aspects or dimensions of the topic under debate that support their standpoint @xcite . in this context, reframing is a task that has recently received increased attention, consisting in switching the underlying framing of an argument @xcite . in our conceptualization of the problem, there are frames to be deleted, d, and frames to be added, a, to an argument. our approach essentially masks the tokens that belong to frame d and uses a language model to regenerate the tokens so that ideally they belong to a. instead of rewriting complete sentences as in previous work @xcite , our approach aims to maximize the change in framing by minimal precise and controlled intervention into the argument. this "mask-and-replace" approach circumvents the need to fine-tune a language model for the specific task and is thus unsupervised. consider the following argument debating "nuclear energy" that emphasizes aspects related to safety: "while geothermal, solar, and wind are safe, nuclear energy is not". a minimal change to the argument that changes the frame from focusing on safety aspects towards emphasizing economic aspects could yield the following argument: "while geothermal, solar, and wind are affordable, nuclear energy is not". in this paper, we draw inspiration from current explainable ai (xai) approaches to propose a novel reframing approach that is based on counterfactual explanation generation to explain the decision of a classifier @xcite . a counterfactual can plainly speaking be seen as an answer to the question: how would an example have to be different to belong to a different class? we transfer this idea to the task of reframing arguments, coming up with a "counterfactual" that answers the question: how would the argument need to be changed to have a different frame? counterfactual generation can be seen as a search in the space of possible changes to a given example or argument that switches the class or respective frame. different metrics have been proposed to constrain and guide the search in the space of possible counterfactuals. as two examples, approaches have used the following metrics: proximity, which measures the similarity of the generated instance to the initial instance, and data manifold closeness, which measures how well the generated counterfactual fits within the target data distribution @xcite . our approach in particular works on the token level, assuming that each frame-relevant token of an argument is assigned to a frame class. the task of frame classification on the token level of an argument has been proposed by @xcite . given the feasibility of this task, we build on this representation and use the models by @xcite as a starting point for our reframing approach. our contributions are: the manual annotations, spanning over 600 reframed arguments as well as our code are available on github foot_0 . ## related work the automatic analysis of frames in texts has been pioneered by @xcite and @xcite , who applied it to the analysis of newspaper articles. frames help to organize and structure text and arguments but are also used to bias discussions @xcite or tailor arguments to specific audiences @xcite @xcite . the task of reframing arguments, as we consider, has been tackled before by @xcite , who used the generic frame classes defined by @xcite and relied on fine-tuned language models to rewrite complete sentences, using two surrounding sentences as context. similar to our goal of minimal changes, @xcite extended this approach to generate a reframed argument that is closely related the original one. they propose an approach that first identifies parts of the original argument to be replaced and then relies on a fine-tuned bart model to generate replacement candidates, picking the candidate that has the highest score of being entailed by the original argument according to an entailment model. in contrast to the above-mentioned previous work on reframing that relies on models fine-tuned for the task, our approach is unsupervised. beyond the inventory of 15 generic frames proposed by @xcite , recent work has made a strong case for more fine-granular and topicspecific framesets. @xcite have for example explored an approach by which frame labels can be derived bottom-up by clustering, and @xcite have demonstrated that the transferability of frames across topics is limited. @xcite have made the case that arguments rarely only evoke one frame and that often multiple aspects are emphasized. in alignment with this observation, @xcite have operationalized the assignment of frames as a span extraction task rather than as a document classification task. following up on this, @xcite present a dataset with topic-specific frame classes annotated on token-level. we directly build on the work of @xcite and @xcite as a starting point and rely on an argument in which each token is labeled with a corresponding topicspecific frame. this allows us to select the token/spans that have to be modified to switch the frame. our proposed approach is inspired by research in xai, which uses counterfactuals to explain classifier decisions. in the context of xai, counterfactuals are explanations rooted in counterfactual reasoning. this process entails pinpointing the specific features that, if altered, would result in different outcomes or predictions @xcite . given this, applying counterfactual approaches to reframing feels intuitive, since changing the frame is conceptually similar to changing a classifier's prediction. from the literature on counterfactuals, we adopt the idea that suitable metrics can be used to guide the search in the space of potential counterfactuals. common metrics for selecting an appropriate counterfactual are validity and proximity. a recent paper catalogued up to eight such metrics from contemporary xai research on generating and evaluating counterfactuals @xcite . in our work, we reuse the metrics related to validity, proximity, and data manifold closeness that are explained in section 3.2. counterfactual methods in natural language processing have primarily been used to explain and evaluate sentiment classifiers @xcite or to uncover dataset artifacts @xcite . to our knowledge, their application in the context of reframing is novel, marking a primary contribution of our paper. ## methodology we model the task of reframing as a generative mask-and-replace approach. given an argument and its frameset s, called source frameset, and a target frameset t , the task is to shift the aspects covered by the argument towards this target frameset t by rewriting it. hence, the goal is to remove n d frames contained in set d to be deleted and add n a new frames in a set a that are not contained in s. the frameset of the rewritten argument is thus expected to be identical with the target frameset @xmath0. our unsupervised approach is described in figure 1. in particular, given an argument to be reframed, we apply a sequence tagging model to classify each token into its corresponding frame, relying on the approach proposed by @xcite . we then mask each token that has been assigned a frame label that is in the set d. for each masked span, a language model generates an alternative text span which is placed in the corresponding spot, resulting in a new text that is a mixture of original text spans and newly generated text spans. in order to guide the replacement of a masked span by a span related to set a, we rely on two strategies to increase the ratio of successfully re-framed arguments: framed decoding and various output reranking strategies based on the field of counterfactual explanations. we explain these strategies in more detail in what follows. ## results we analyze the appropriateness of our reframed arguments along different dimensions. first of all, we evaluate the reframing success of our approach, that is the success of fitting the target frameset t . for this, we compare the frames covered by the reframed argument (p) with the target frames t . note that p is predicted by the model of @xcite in the case of the automatic evaluation and annotated by humans in the case of the manual evaluation. we present results with respect to the frames towards three criteria: i) fit measuring the target-set-fit ratio, that is the ratio of instances where @xmath1, ii) rem measures the ratio of instances where the unwanted frames are successfully removed, i.e. p ∩ @xmath2, and iii) add measures the ratio of successfully added frames a ⊆ p. these three metrics allow us to judge the reframing validity of our approach. the automatic results covering three topics are presented in section 5.1. in section 5.2 we broaden the perspective by also including further metrics that measure other relevant aspects of the reframed argument beyond frame-validity, considering the other automatic metrics introduced in section 3.2. in addition, we present the results of our manual study in section 5.3, adding the criteria of meaning preservation and plausibility, evaluating the impact of reranking and framed decoding as well as the impact of the model size of the text-generating model (section 5.3.1) and the impact of the edit distance between the source frameset s and the target frameset t (section 5.3.2). we conducted further experiments regarding the impact of the number of rewritings per argument in appendix a. ## conclusion we have proposed an unsupervised approach to argument reframing, which takes inspiration from approaches to counterfactual explanation generation in the sense that we transfer and adapt metrics used in counterfactual generation to implement a reranking strategy for reframed arguments. we use an llm to replace text spans that were tagged by a token classifier with a frame to be deleted by tokens that are associated with the frame to be added. our automatic and manual evaluation demonstrates that the combination of framed decoding and reranking, utilizing metrics such as frame-validity, proximity, data manifold closeness, and grammaticality, outperforms a vanilla llm baseline by nearly an order of magnitude in terms of reframing success. furthermore, by showing a tradeoff between tailoring the rewritten argument to the target frameset and yielding a plausible and grammatically correct argument, we identified a sweet spot in the strength of framed decoding yielding across two different language generation model sizes.