Large Language Models as Reader for Bias Detection (2025)

Abstract: Detecting bias in media content is crucial for maintaining information integrity and promoting inclusivity. Traditional methods analyze text from the writer’s perspective, which analyzes textual features directly from the writer’s intent, leaving the reader’s perspective underexplored. This paper investigates whether Large Language Models (LLMs) can be leveraged as readers for bias detection by generating reader-perspective comments. Experiments are conducted on the BASIL (news bias) and BeyondGender (gender bias) datasets with LLMs Gemma-7B, Phi-3-3.8B, Llama3.1-8B, Llama3.1-70B, and GPT4. The results demonstrate the effectiveness of reader-perspective comments for open-source LLMs, achieving performance comparable to GPT4’s. The findings highlight the significance of emotion-related comments, which are generally more beneficial than value-related ones in bias detection. In addition, experiments on Llamas show that comment selection ensures consistent performance regardless of model sizes and comment combinations. This study is particularly beneficial for small-size open-source LLMs.

## introduction the rapid expansion of digital media has intensified concerns regarding biased content, characterized by deviations from objective representation that favor particular viewpoints, groups, or outcomes, whether introduced intentionally or unintentionally. identifying such biased language (bias detection) in media content, such as news articles and social media posts, has become a critical challenge @xcite @xcite . traditional methods adopt the writer's perspective to analyze textual features directly tied to the author's intent. they assume that bias originates from the writer's language and * corresponding author framing, implicitly adopting hall's "encoding" perspective @xcite . 1 however, existing methods neglect the "decoding" process where readers actively interpret and construct meaning @xcite . large language models (llms) have shown remarkable capabilities in text understanding and generation @xcite , often being used for data synthesis and reasoning explanation. while they have been used for bias detection, their potential as a reader, observing data and generating rational or emotional comments instead of from the writer's perspective, remains underexplored. on the other hand, bias detection datasets are annotated by the human audience (readers) rather than the content producer (writer). therefore, we intu-itively utilize llms to align with human annotation through this process, shifting the lens from what the writer says to how readers perceive them. in addition, inspired by the fact that human perceptions of bias can be influenced by user comments @xcite @xcite , we wonder whether llms can be leveraged as peerreaders to simulate this dynamic and enhance bias detection capability. the research questions are as follows: rq1) are reader-perspective comments effective in bias detection? rq2) can llms be influenced by peer readers ' comments? rq3) what kind of comment generation policies are most / more beneficial? rq4) whether a trained selector deciding to add the comment or not improves the performance? experiments are designed as follows: initially, we utilize an llm to generate comments that capture diverse viewpoints or express emotions evoked by the content. then, llms make the inference with these comments combined with the original content. we evaluate on the news bias dataset basil @xcite and the gender bias dataset beyondgender @xcite with different llm backbones: gpt, phi, gemma, and llama, primarily focusing on small-sized llms. the main contributions and findings are as follows: 1) a novel perspective utilizing llms as reader to generate comments for bias detection, which is effective on news bias and gender bias detection (rq1), 2) findings that small-size llms' performance is significantly improved by the influence of peer-reader's comments (rq2), resourceefficient for computing. 3) findings that emotionrelated comments are generally more beneficial than value-related ones and that comments vary with the reader's gender (rq3), providing insights into how to utilize comments effectively in biased content analysis. 4) findings that comment selection may be helpful, yet the positive effects depend on the backbone, requiring further analysis. (rq4). ## related work traditional methods for bias detection often rely on supervised learning, focusing on identifying the appropriate contextual information for training (van den berg and markert, 2020; @xcite and training data augmentation through rule-based alterations or translation @xcite . recent advancements in large language models @xcite have simplified data augmentation @xcite and also brought new possibilities for bias detection @xcite . for instance, @xcite explores the potential of llms in news bias detection using prompt-based techniques, while @xcite leverages multi-agent llm interactions to detect gender bias. however, existing studies primarily analyze text from the writer's perspective. on the other hand, research in psychology and social science has discovered the importance of external perspectives in bias perception and that human perception of bias is often influenced by cognitive frameworks, social norms, and individual perspectives @xcite @xcite . drawing inspiration from this, we utilize llms as readers to generate reader-perspective comments, providing additional signals for bias detection. ## experiment design the workflow is illustrated in figure 1 . by default, we employ a greedy strategy, where the best policy comments are appended to the original data (section 5.1 and 5.2). 2 further, we explore the comment selection setting (fig. 4 in appx. b), where a selector evaluates the usefulness of each comment and decides whether it should be appended with. (section 5.3). ## conclusion in this work, we explore leveraging llms as readers to generate reader-perspective comments for bias detection. through the design of comment generation policies and experiments on various llms, the results demonstrate significant effectiveness and robustness in detecting bias for both news and gender bias. the findings highlight the potential of utilizing large-sized llms as dynamic readers in various roles and small-sized llms as efficient detectors for other content analysis. additionally, although our findings highlight the significance of emotion-related comments in bias detection, the exact nature of this relationship remains unclear and warrants further investigation. our educated guess is that llms might identify subtle nuances amplified by the comments as biased, leading to a decrease in performance. this phenomenon may be attributable to factors such as the subjective nature of bias perception and llms' tendency for overcorrection. we also observe that comments are particularly beneficial when the baseline performance is suboptimal. in contrast, for large closed-source models like gpt-4, which already exhibit strong bias detection capabilities, the impact of comment augmentation is less pronounced. since our focus is the small-sized open-source llms, few large-sized and closed-source models are evaluated.