Tree-of-Counterfactual Prompting for Zero-Shot Stance Detection (2024)

Abstract: Stance detection enables the inference of attitudes from human communications. Automatic stance identification was mostly cast as a classification problem. However, stance decisions involve complex judgments, which can be nowadays generated by prompting Large Language Models (LLMs). In this paper we present a new method for stance identification which (1) relies on a new prompting framework, called Tree-of-Counterfactual prompting; (2) operates not only on textual communications, but also on images; (3) allows more than one stance object type; and (4) requires no examples of stance attribution, thus it is a “Tabula Rasa” Zero-Shot Stance Detection (TR-ZSSD) method. Our experiments indicate surprisingly promising results, outperforming fine-tuned stance detection systems.

## introduction stance was defined by @xcite as the expression of an author's standpoint and judgment towards a given proposition. as such, stance is used for discovering attitudes from various sources, ranging from social media, to debates or news articles, cf. @xcite . regardless of the context in which it is used, stance always has a subject and an object. the subject of stance can be the speaker in a conversation or the author of a social media posting (smp). the stance object, as reported in @xcite ; @xcite , can be sometimes a topic, while other times it can be a frame of communication. topics typically correspond to the name of a controversial political figure, e.g. hillary clinton, donald trump, or a noun phrase, e.g. "gun control" or "abortion". automatically inferring the stance towards a topic entails modeling complex judgments that sometimes use commonsense knowledge about the topic, as shown in the example illustrated in figure 1 example implies that it is best to give birth -even at the cost of pain. this is a direct quote from mother teresa, in a famous speech concerning abortion at a national prayer breakfast in washington, d.c., on february 5, 1994. therefore, the smp's author has a stance of reject towards the topic of "abortion". frames of communication (focs), the other possible objects of stance that we consider in this paper, highlight problems of interest by addressing what causes those problems, and determining moral judgments, as defined in @xcite . the foc illustrated in figure 1 (b) highlights the problem of collective responsibility during vaccination campaigns, explaining why people should make their own vaccination decisions. the smp from this example contains both a text and an image. at first look, the foc is contradicted by the text of the smp, but interestingly, the image also contributes to the stance decision. the image represents the terrifying nazi character of hans landa from the movie inglourious basterds, and the text superimposed on the image implies that he is looking for unvaccinated people to punish them. a closer inspection of the entire smp indicates that the quoted text in fact is sarcastic, and combined with the image, it accounts for the inference of an accept stance. the examples illustrated in figure 1 showcase some of the complex forms of reasoning required when inferring stance values. this motivated us to find whether existing large language models (llms) and large multimedia models (lmms) can be used to access their knowledge, and, more importantly, to use their reasoning capabilities to infer the stance of communications involving both text and images. in this paper we focused on social media platforms where millions of users express their opinions and attitudes in conversations about issues of their interest, therefore we explored stance detection from smps. previous stance detection methods @xcite @xcite @xcite mostly used topics as stance objects, casting stance detection as a classification problem, trained on texts addressing a certain number of topics, and tested on texts addressing the same topics. however, as reported in @xcite ; @xcite , stance detection methods designed in this way are limited in their ability to generalize to a wide variety of topics. to address this limitation, zero-shot stance detection (zssd) was considered in @xcite ; @xcite ; @xcite ; @xcite , in which the stance topics from the test data are not seen in the training data. to take this a step further, in this paper we address the problem of "tabula rasa" zero-shot stance detection (tr-zssd), in which no example of stance inference towards any topic is needed, and moreover, the stance of smps towards topics as well as focs is identified. instead of classifying the stance value of an smp towards a topic or an foc, we considered a spe-cial form of counterfactual reasoning, in which we prompt an llm or an lmm to generate an explanation if the stance s of an smp p towards the stance object o is considered to take a value v. since the value v is not known, we consider counterfactuals, i.e. all alternative possibilities of stance values, given the observed p and o. this entails that for each pair (p, o) we obtain (1) an explanation e a , providing the rationale for assigning the value accept to s; (2) an explanation e r , providing the rationale of the reject value; and (3) an explanation e n , providing the rationale of the no stance value. for each explanation, the llm (or lmm) follows different reasoning paths, thus it generates a tree-of-thoughts, inspired by @xcite . finally, the llm (or lmm) verifies which of the three explanations is most likely, using chain-of-verification prompting @xcite , leading to the final inference of the stance value v. in all this reasoning process, based on multiple prompting of an llm (or lmm), no need for prior examples of stance value assignments is required. we call this method tree-of-counterfactual (toc) prompting for tabula rasa zero-shot stance detection (toc-tr-zssd), making the following contributions: ¡1£ we introduce the first tabula rasa zero-shot stance detection (tr-zssd) method, that operates without any prior example of stance value assignment. tr-zssd is a step forward from previous zssd, in which examples of stance value assignment are provided, but new topics are not available at training time. ¡2£ our tr-zssd method operates not only on texts, but also on smps that contain images. moreover, the method allows the stance to consider different objects, e.g. topics or focs. ¡3£ we introduce the first stance detection method that prompts llms or lmms to generate explanations of possible stance values and to verify the explanations, selecting the most rational stance value. ¡4£ our toc prompting for stance detection refocuses the problem of stance identification on reasoning from current approaches relying on classification, enabling an explainable approach to stance identification. we make available all code, prompts, and experiments on github foot_0 . ## related work stance detection on social media attracted plenty of interest, spearheaded by task 6 of semeval-2016 @xcite , where multiple neural systems @xcite @xcite were evaluated. the dataset used in semeval-2016 considered tweets discussing several controversial topics (e.g., "abortion"), annotated with the stance of each tweet's text towards these topics. continued interest was sparked by the release of tweeteval @xcite , a commonly utilized benchmark for stance detection @xcite @xcite . additional research interest formed around the zero-shot stance detection (zssd) setting, which focused on identifying the stance of novel topics, not present in the training data @xcite . cross-topic zssd systems typically focus on stance datasets with hundreds or thousands of topics, available from vast @xcite ) and wt-wt @xcite . this formulation of cross-topic zssd stands in contrast with the tabula rasa zssd employed in this work, in which no training examples are provided for the toc-tr-zssd method. toc-tr-zssd also contrasts with prior unsupervised stance detection research, which focused on identifying stance via the structural properties of the speaker graph @xcite . furthermore, stance detection research has considered incorporating external knowledge, such as @xcite , which complement the introduction of toc-tr-zssd. counterfactual reasoning considers alternatives to facts, events, or states, making inferences about what might have happened or what may have been possible. for example, if you trip and fall due to not tying your shoes, you can easily imagine a counterfactual reality where you tied your shoes and therefore did not trip. counterfactual reasoning is known to play an important role in human learning and decision-making. in philosophical and ai literature @xcite @xcite , counterfactuals were considered to give us reliable insights into the specifics of causality. however, recently, counterfactual reasoning has been integrated into nlp systems to enhance understanding and generation capabilities by considering altering narratives or decisions based on hypothetical scenar-ios. for example, @xcite utilize counterfactual reasoning for story re-writing, while zeng et al. (2020) incorporated counterfactual reasoning to improve named entity recognition (ner) by generating counterfactual examples. counterfactual reasoning also yields insights into the explainability of nlp systems, with @xcite focusing on identifying and altering features in input data with counterfactuals that change model predictions, thereby providing insights into model behavior, and @xcite where classification predictions are explained with counterfactuals. recently, counterfactuals have been utilized to improve prompt design for llms @xcite , yielding improved few-shot performance on vision and language tasks by discovering more generalizable prompts. however, none of these llm prompting techniques directly utilized llms to generate, compare, and explain counterfactual rationales, as we do in the toc-tr-zssd method. ## the method tree-of-counterfactual (toc) prompting of large language models (llms) foot_1 for stance detection is structured into three distinct phases. 2(a) , involves constructing a tree of counterfactuals, considering each possible stance value. for instance, an accept branch is constructed for the smp p i and the stance object o j when we assume the stance value assigned to s i,j is accept. similarly, counterfactual reject and no stance branches are constructed. note that each branch is always constructed, regardless of the content of p i or o j . however, only one of the possible stance values will hold -but we need to proceed to the next phases to find out which stance value is the most rational for p i towards o j . therefore phase a sets the stage for the following phases to perform reasoning that allows s i,j to be inferred. 2 (b), prompts the llm to explain the reason why a certain stance value s i,j is attributed to the pair (p i , o j ), for each of the possible counterfactual stance values. the prompting approach seeks to generate post hoc rationalizations for why each counterfactual stance value is held by smp p i towards stance object o j . ## conclusion in this paper we introduce the task of tabula rasa zero-shot stance detection (tr-zssd). we demonstrate that large language models (llms) and large multimodal models (lmms) are capable of performing high-quality tr-zssd with tree-of-counterfactual (toc) prompting, a novel prompting methodology relying on counterfactual reasoning. stance detection experiments were performed over several varied stance datasets, and numerous strong baseline systems were considered for comparison. toc prompting with llms, such as gpt-4, outperformed fine-tuned baselines and alternative prompting methodologies for text-based tr-zssd. furthermore, toc prompting with lmms, such as gpt-4v, demonstrated competitive performance with fine-tuned baselines for multimodal tr-zssd. our toc-tr-zssd methodology required zero labeled examples, successfully generalized across different objects of stance (i.e. topics or frames of communication), improved performance across both text-only and multimodal stance detection, and generated explanatory rationales, enabling llm and lmm interpretability.