Chain-of-Thought Embeddings for Stance Detection on Social Media (2023)

Abstract: Stance detection on social media is challenging for Large Language Models (LLMs), as emerging slang and colloquial language in online conversations often contain deeply implicit stance labels. Chain-of-Thought (COT) prompting has recently been shown to improve performance on stance detection tasks — alleviating some of these issues. However, COT prompting still struggles with implicit stance identification. This challenge arises because many samples are initially challenging to comprehend before a model becomes familiar with the slang and evolving knowledge related to different topics, all of which need to be acquired through the training data. In this study, we address this problem by introducing COT Embeddings which improve COT performance on stance detection tasks by embedding COT reasonings and integrating them into a traditional RoBERTa-based stance detection pipeline. Our analysis demonstrates that 1) text encoders can leverage COT reasonings with minor errors or hallucinations that would otherwise distort the COT output label. 2) Text encoders can overlook misleading COT reasoning when a sample’s prediction heavily depends on domain-specific patterns. Our model achieves SOTA performance on multiple stance detection datasets collected from social media.

## introduction detecting the stance of a text with respect to a certain topic is vital to many nlp tasks @xcite . detecting stances on social media platforms like twitter poses unique challenges, as emerging knowledge and colloquial language patterns can make it difficult to detect stances without additional context. for example, consider the top tweet shown in figure 1 . this tweet contains no direct mention of donald trump, and is thus difficult to classify without further context -such as how trump supporters on twitter widely supported voter fraud propaganda. such emerging knowledge is difficult for llms with knowledge cut-offs to understand and may only be discernible by observing similarly labeled samples in the training set. one way to solve this problem is by employing models with extensive world knowledge. for example, recent works have shown that using chatgpt on stance detection can provide significant performance increases @xcite . unfortunately, llms (e.g., chatgpt, llama) still have many issues understanding complex stance relationships from twitter data. in this study, we highlight two issues with the state-of-the-art chainof-thought (cot) prompting approach to stance detection. 1) implicit stance confusion: as shown in figure 1 , llms continue to struggle with understanding implicit tweet stance, even when employing advanced prompting strategies like cot reasoning @xcite . 2) stance label hallucination: llms are prone to hallucinations, which cause them to output sound reasonings, but for the wrong stance topic (see figure 1 example). even when llms analyze the correct topic, they are also prone to using the provided label space incorrectly, producing accurate but ill-structured outputs. in this study, we mitigate these two problems by introducing chain-of-thought (cot) embeddings. our approach feeds the cot reasoning text to a transformer encoder to be used as an additional feature in a traditional stance detection pipeline. the intuition behind this approach is three-fold: (i) text encoders are robust to stance label hallucinations if the cot reasoning is correct. this can make incorrect cot predictions useful in a text classification pipeline. (ii) text encoders can choose to ignore certain signals as needed. thus, when a sample is too implicit to be understood by llms, the model may choose to focus on how similar tweets were classified. (iii) cot reasonings can inject world knowledge into a text encoder. that is, cot texts often contain reasonings and justifications grounded in world knowledge not available in the tweet. we find that, by using this approach, we can achieve state-ofthe-art results on multiple stance detection datasets. a summary of our contributions is as follows: stance detection: this task is a well-explored research problem, where early studies employed various machine learning and deep learning techniques @xcite . the emergence of large language models has further pushed the stateof-the-art performance on many stance detection datasets @xcite . many stance detection problems require domain-specific solutions with models which explicitly inject world knowledge into stance detection systems @xcite ; llms for stance detection recently, few works have used chatgpt for stance detection directly @xcite . in @xcite , the authors achieve superior performance on several stance detection datasets by prompting chatgpt to do chain-of-thought inference. in this study, we use a similar prompting strategy to perform stance detection, but show the benefits of embedding these cot reasoning texts and using them as a feature in a stance detection pipeline. ## methods we employ a 1-shot cot prompt for each tweet in each dataset, aiming to determine the stance of the tweet in relation to a specific topic foot_0 . we specifically ask the models to provide a cot reasoning and to include its predicted label in brackets (e.g. ## conclusion in this study, we have shown that embedding chainof-thought reasoning extracted from llms (e.g., chatgpt, lllama) can boost the performance of stance detection models. specifically, we highlight how we can outperform vanilla cot by augmenting text encoders with cot embedding. our analysis highlights how text encoders are robust to llm hallucinations and aid in the prediction of deeply implicit stance labels. we encourage future works to consider embedding cot reasoning for stance detection and similar tasks using social media data.