Media Bias Detection Across Families of Language Models (2024)

Abstract: Bias in reporting can influence the public’s opinion on relevant societal issues. Examples include informational bias (selective presentation of content) and lexical bias (specific framing of content through linguistic choices). The recognition of media bias is arguably an area where NLP can contribute to the “social good”. Traditional NLP models have shown good performance in classifying media bias, but require careful model design and extensive tuning. In this paper, we ask how well prompting of large language models can recognize media bias. Through an extensive empirical study including a wide selection of pre-trained models, we find that prompt-based techniques can deliver comparable performance to traditional models with greatly reduced effort and that, similar to traditional models, the availability of context substantially improves results. We further show that larger models can leverage different kinds of context simultaneously, obtaining further performance improvements.

## introduction both mass media and social media are potent channels for expressing viewpoints and shaping decisions. news outlets stand out, exerting a pivotal influence on altering and molding individual and collective perspectives @xcite . clearly, biased media has the ability to influence people, for instance, in a study by @xcite , it is revealed that there is a negative association between trust in the news media and online news participation. @xcite highlight that distinctive contributions can be made by computer scientists to study bias. we follow them in defining bias as subjective standpoints manifested variously in, for example, word choice, framing, intentional omission or misrepresentation of specific details @xcite @xcite . in nlp, media bias has been scrutinized in different ways using different names @xcite . an important step towards a common perspective was the development of the bias annotation spans on the informational level (basil) dataset, which established the specific concepts of information bias and lexical bias @xcite . informational bias refers to selective presentation of content in a factual manner to sway opinion of readers @xcite , whereas lexical bias refers to linguistic attributes like word selection and syntax @xcite @xcite . these types of bias have been studied with supervised learning methods @xcite @xcite @xcite . however, so far the task proves challenging and often results in unsatisfactory performance when trained with a limited set of labeled examples. this can be due to the expensive nature of data annotation and extensive variety of domains, languages, and tasks @xcite . this is particularly true given the importance of context in the identification of bias (van den berg and markert, 2020; @xcite @xcite . for example, in the basil dataset, where political bias is identified at the sentence-level, it has been shown that informational bias depends fundamentally on the context of the sentence @xcite , arises from manipulation of information or selective presentation of content in a factual way, e.g., use of quotes, to evoke reader's emotions towards news entities @xcite . thus, recent work on automatic bias detection in news has focused on identifying the right context to present during training. context ranges from whole articles (van den berg and markert, 2020), to just sentences surrounding the target sentence @xcite , and other sentences discussing the same entity sampled from the article @xcite . although these approaches lead to substantial improvements, all of them have so far focused on model fine-tuning, which limits their applicability in broader scenarios. this paper investigates whether bias detection can profit from current developments on large language models (llms), which can achieve excellent performance on a wide variety of downstream tasks utilizing zero-shot or few-shot approaches @xcite , i.e., without finetuning. previous work @xcite @xcite shows the significance of in-context learning for a large variety of tasks. our work continues in this direction, and we propose to test prompting strategies for bias detection using llms. our prompt design is motivated to a large extent by its flexibility as an experimental platform: we can investigate the interaction between various experimental variables, the most important of which are: (a) the style of the prompt (concise vs. detailed); (b) the amount of document context presented to the model; (c) the amount of supervision afforded to the model (zero-shot vs. few-shot). we carry out experiments regarding these variables on a range of current llms, empirically evaluating zero-shot and few-shot learning approaches using task-specific prompts to detect political bias across a wide range of llms. to the best of our knowledge, our work is the first to attempt this kind of investigation. our work shows the importance of the specificity and detail in prompt engineering, which is a highly task-dependent and a potentially cumbersome process. we additionally show that providing an llm with more appropriate and consistent bias contexts, either in the form of examples, or of related sentences, can approximate conventional supervised learning approaches, setting up a research direction for the future. ## related work the societal consequence of misinformation in various fields are substantial. news articles can contain biased opinion, resulting in misleading views @xcite . political scientists have determined that bias in news reporting can be defined by the choices in content selection and organization of information within articles @xcite . according to @xcite , bias can also be introduced when less information or facts on news articles is provided because for journalists who try to appear neutral, avoids conveying ample information. in another study, @xcite show a negative association between trust in the news media and online news participation. since news media advance their interests by devoting resources to control reporting @xcite , the nlp field has devoted attention to predicting the political leaning and trustworthiness of news media outlets @xcite . regarding bias in news, the introduction of the basil dataset @xcite marks the onset of interesting paradigms. within basil, sentences are annotated with their corresponding bias type, a designated target (the primary entity), and several other labels. prior studies on basil have shown that the incorporation of contextual information enhances supervised learning models: van den berg and markert (2020) utilized article and event-level context; @xcite ; @xcite integrate three levels of context, i.e., article, event, and adjacent sentences, and chen et al. (2020) utilized second order bias features to detect article-level bias. generative models are outlined as complex systems by @xcite due to their tendency to exhibit emergent behaviors, for example, the ability of llms to perform in-context learning @xcite . along these lines, instructing llms with prompt in zero or few-shot settings enables models to perform tasks with no or minimal task-specific training, which is evidence of the llms' generalization capability @xcite , and has put llms at the center of recent progress in nlp @xcite @xcite @xcite . in essence, llms offer a broad spectrum of beneficial applications that extend beyond their generative functionalities. task division into multiple concise steps and sequentially introducing them to the large language model yields improved performance across an array of reasoning tasks, spanning word problems, arithmetic operations, and code execution @xcite . notably, gpt models have achieved success in diverse language-related endeavors, showcasing their capacity to produce text resembling human-written content @xcite . ## method recent studies have shown that fine-tuning of llms to instruction-style prompts is a common approach to achieve gains in performance @xcite @xcite . however, language model fine-tuning is also computationally very expensive. consequently, based on our comprehensive analysis on how models for language representation have been incorporating broader contextual scopes into their predictive processes, for example, models have evolved to consider neighboring words @xcite , sentences @xcite @xcite , paragraphs @xcite , and even articles (van den berg and markert, 2020; guo and zhu, 2022a), we align analogous contexts with prompts to learn the capacity of context dependence across different large language models. our basic setting adopts a zero-shot approach with prompts shown in zero-shot in this approach, using zero-shot means we do not give the model any knowledge of question-answer pairs as to how a certain sentence gets classified as a particular label, nor fine-tuned the language models in any aspect. context-augmented (+ctx) given the contextsensitive of the task, we find that a large portion of previous work in detection of political bias on basil has focused on introducing contextual information into classification @xcite @xcite , for example, by mixing contexts of informational and lexical bias at both the article-level (entire article encompassing target sentence) and event-level (triplet of articles discussing the same event). in this context, recent work by @xcite proposed a comprehensive framework to generate more consistent and similar bias contexts to improve performance when fine-tuning models. in this paper, we propose to adapt these techniques for the task of zero-shot and few-shot political bias detection. concretely, we take advantage of three context-augmented techniques: bias-aware neighborhood (banc) extends the target sentence with neighboring spans, i.e., combining former and next sentence with target sentence. article-based target-aware (abta) extends this idea by extending the target sentence with spans from the same article which share both the same bias label (bias type of basil, i.e., informational and lexical) and the same target, i.e., main entity or topic of the target sentence already annotated in basil (cf. figure 1a ). event-based target-aware (ebta) extends the abta approach by gathering spans for the same bias label and target across articles, i.e., triplets of basil (cf. figure 1b ). utilizing annotated bias spans gathered through these approaches are added to the input prompt, which effectively mix relevant local and global contexts of informational and lexical bias at the article and event levels. few-shot (+k-shots) our experiments with few-shot learning are parallel to the zero-shot case, except that we present the model with questionanswer (i.e., sentence-label) pairs. since the available data is limited, the choice of parameter k (the number of examples provided) is dependent on the task. more details about this are given in §4. ## results on our first set of experiments, we use the con-cise prompt, which we test on regular baseline (zero-shot) and also in the ctx setting. results are shown in overall, it is observed that llms benefits the most from informational context examples, i.e., inf bias over non-bias oth i.e., inf/oth task. furthermore, we note that performance improvement of detailed prompt over concise prompt is more prominent in instruction-tuned models including flan-t5 variants, flan-ul2, gpt-3.5-turbo and gpt-4, whereas the concise prompt holds more significance when presented to gpt-neo and opt models. we think this is because smaller language models tend to struggle when presented with long contexts, being essentially unable to parse the information provided, even if it is potentially use-ful for the task at hand. finally, we also find that gpt-4 outperforms gpt-3.5-turbo. ## conclusion this paper aims at advancing our understanding of prompt-based identification of media bias by llms, a crucial and challenging task to the research community, news media, and social media companies. we establish a framework for examining prompt-based models in a zero-shot and fewshot configuration to classify sentences that manifest bias. our approach demonstrates the utility of using context-augmented informational and lexical bias from prior work, an area that we think has received inadequate attention. we provide insights into the dynamics between llms, tasks, and prompts, discerning their individual capacities in the detection of bias. notably, we find considerable performance improvements when adding contextaugmented information to both small and largeparameter llm models, indicating that in-context learning can approach the performance levels of traditional models in classifying media bias. while our study demonstrates that llms are applicable to the area of misinformation detection, they do not always represent a huge improvement. formulating prompts that strike a balance between the simple and the (over)-elaborate remains a challenge, as is particularly evident in gpt-neo and opt. we hope our work inspires further research in bias detection as a means to gain insight into enhancing llms. future work includes exploring additional aspects of bias, e.g., bias related to culture, race, or age, as well as exploring the understanding and limitations of llms. we also think other similar bias-detection tasks could be grouped together in our prompting setting, following @xcite .