Is External Information Useful for Stance Detection with LLM s? (2025)

Abstract: In the stance detection task, a text is classified as either favorable, opposing, or neutral towards a target. Prior work suggests that the use of external information, e.g., excerpts from Wikipedia, improves stance detection performance. However, whether or not such information can benefit large language models (LLMs) remains an unanswered question, despite their wide adoption in many reasoning tasks. In this study, we conduct a systematic evaluation on how Wikipedia and web search external information can affect stance detection across eight LLMs and in three datasets with 12 targets. Surprisingly, we find that such information degrades performance in most cases, with macro F1 scores dropping by up to 27.9%. We explain this through experiments showing LLMs’ tendency to align their predictions with the stance and sentiment of the provided information rather than the ground truth stance of the given text. We also find that performance degradation persists with chain-of-thought prompting, while fine-tuning mitigates but does not fully eliminate it. Our findings, in contrast to previous literature on BERT-based systems which suggests that external information enhances performance, highlight the risks of information biases in LLM-based stance classifiers.

## introduction stance detection is a task that determines whether a given content supports, opposes, or remains neutral toward a target. when the content assumes implicit information about the target, stance detection systems can benefit from external information, such as wikipedia excerpts, regarding the target. accordingly, recent research has explored incorporating such information to improve stance detection, highlighting its benefits @xcite @xcite . on the other hand, large language models (llms) have demonstrated remarkable capabilities across various reasoning tasks, including mathematical reasoning @xcite , coding @xcite , and language understanding @xcite . given these advances, recent research has begun exploring the potential of llms for stance detection @xcite . with these parallel trends, an important question arises: can external information enhance llms in stance detection? in this paper, we systematically evaluate how external information about targets impacts the performance of a diverse set of llms across a wide range of datasets and targets. surprisingly, we find that such information, from wikipedia or web search, tends to compromise stance detection performance. to explain this phenomenon, we show that model predictions often adopt external information stance and sentiment. despite llms' known sensitivity to prompt variations, we show consistent results through experiments on different chain-of-thought prompting methods. finally, we find that fine-tuning mitigates but does not fully eliminate performance degradation. our research serves as a caution against the use of external information without proper bias consideration for llms in stance detection and natural language reasoning at large. ## related work stance detection with external information. a key line of related work investigates leveraging external information, often from wikipedia, to enhance stance detection. @xcite fine-tuned bert models which take wikipedia excerpts, in addition to given texts and targets, as inputs and report significantly improved stance detection performance. subsequent works in the literature either utilized external information in a different formu-lation of stance detection @xcite or introduced new knowledge organization and filtering schemes for such information @xcite . while these works have primarily focused on fine-tuning smaller, bertlike models for stance detection, we extend this research to llms, which possess emergent reasoning abilities but require significantly more resources for fine-tuning. stance detection with llms. relatedly, another stream of works examines how llms can be applied to stance detection. @xcite and @xcite proposed prompting schemes where reasoning on stance is organized as ensembles or multi-agent discussions. meanwhile, @xcite introduced a calibration network which serves to mitigate internal biases of llms. orthogonal yet complementary to these efforts, our work provides a foundational analysis of how external information influences their decision-making, uncovering unintended effects and offering insights to guide future research. ## conclusion we investigated the question of whether external information can benefit llms for stance detection. contradicting previous literature on bertbased stance detection with external information, our experiments indicated that such information can actually harm the performance of llms. we also verified that this phenomenon is partly caused by llms being biased by the stance or sentiment they perceive in external information. furthermore, cot prompting is of little benefit, while fine-tuning lessens but does not completely alleviate this problem. given such observations, we call for more consideration of bias factors in llm stance detection and natural language reasoning at large.