I ndi V ec: An Exploration of Leveraging Large Language Models for Media Bias Detection with Fine-Grained Bias Indicators (2024)

Abstract: This study focuses on media bias detection, crucial in today’s era of influential social media platforms shaping individual attitudes and opinions. In contrast to prior work that primarily relies on training specific models tailored to particular datasets, resulting in limited adaptability and subpar performance on out-of-domain data, we introduce a general bias detection framework, IndiVec, built upon large language models. IndiVec begins by constructing a fine-grained media bias database, leveraging the robust instruction-following capabilities of large language models and vector database techniques. When confronted with new input for bias detection, our framework automatically selects the most relevant indicator from the vector database and employs majority voting to determine the input’s bias label. IndiVec excels compared to previous methods due to its adaptability (demonstrating consistent performance across diverse datasets from various sources) and explainability (providing explicit top-k indicators to interpret bias predictions). Experimental results on four political bias datasets highlight IndiVec’s significant superiority over baselines. Furthermore, additional experiments and analysis provide profound insights into the framework’s effectiveness.

## introduction the widespread expansion of digital media platforms has introduced an era characterized by unparalleled accessibility to news and information. in today's digital era, misinformation and disinformation frequently gains traction on social media, thereby exerting a significant influence on public perception and decision-making. given the critical impact of media bias on shaping attitudes and opinions, there exists a pressing need for the development of effective tools designed for detecting bias in media content. to this end, extensive efforts have been dedicated to social media bias detection @xcite @xcite , with the primary objective being the prediction of whether a given input (e.g., an article, a paragraph, or a sentence) exhibits bias or not. however, most of previous research focus on fine-tuning models specific to particular datasets @xcite and subsequently testing them on corresponding test sets. we argue that such trained models lack adaptability and provide predictions that are essentially blackbox, lacking in explainability. in this work, we propose a novel bias detection framework based on a comprehensive bias indicator database. the term bias indicator in this context refers to a concise, descriptive label or tag designed to represent the presence or nature of media bias. diverging from the coarse-grained framing concept proposed in previous works @xcite @xcite , which cannot be directly applied to bias prediction, our media bias indicators are fine-grained, offering direct insight into the bias exhibited by a given input. to provide a clearer distinction between framing and our fine-grained media bias indicators, we present several illustrative examples in in contrast to much of prior research, which often relies on fine-tuning methods or the training of specific models tailored to particular datasets, leading to limited adaptability and potential performance issues when confronted with out-of-domain data, our indivec framework displays notable versatility in bias detection across a wide spectrum of previously unencountered datasets sourced from various origins. our approach begins with the construction of a bias indicator set, followed by the construction of a vector database based on llm api. leveraging the created bias vector database, when processing new text inputs that may contain bias, our bias prediction framework initially extracts or summarizes descriptors based on the given input. subsequently, these descriptors are matched with indicators stored in the database. the bias label associated with the top-matched indicators dictates the final bias label assigned to the input in question. we conduct explorations on various political leaning prediction datasets with different bias levels (i.e., sentence-and article levels), initially constructing our indicator database based on a single dataset (i.e., flipbias @xcite ). the findings demonstrate that our indivec method significantly outperforms the chatgpt baseline on four distinct political leaning datasets (i.e., flip-bias @xcite , basil @xcite , babe @xcite , mfc @xcite ) with different sources. furthermore, our indivec framework shows superiority in explainability. when tasked with detecting bias in a new article or sentence, our framework matches the top-k indicators from the indicator database to represent the bias inclination within the given input based on the distance with bias descriptors if given input. the majority label among these top-k indicators is subsequently employed to classify the input. importantly, these top-k matched indicators can be interpreted as explanations for the bias prediction. they can also function as a valuable tool for aiding humans in annotating bias data, showing the high degree of explainability of our framework. in brief, the main contributions of this paper are: • we propose a novel bias prediction framework, called indivec, which is based on finegrained media bias indicators and a matching and voting process that departs from conventional classification-based methods. political bias. it refers to a text's political leaning or ideology, potentially influencing the reader's political opinion and, ultimately, their voting behavior @xcite . political bias detection has been done at different granularity levels: single sentence @xcite and article @xcite level. in this work, we conduct experiments on both sentence-and article-level political bias datasets. framing. framing refers to emphasizing desired aspects of an issue to promote and amplify a particular perspective @xcite . framing in news media and social networks has been studied to analyze political polarization @xcite @xcite . @xcite propose a multitask learning model that jointly learns to embed sentence framing language and predict political bias. however, the frames studied in kim and @xcite are still limited and in the form of topic, which lacks of fine-grained semantics and could not be adopted directly to predict bias label. and the multi-task joint learning's promotion is limited and lack adaptability compared to our indivec framework. recommendation. although the bias detection task is typically considered a classification task, our indivec solution aims to address bias detection from the perspective of a recommendation task. for instance, in the quotation recommendation task @xcite @xcite , it is common and fundamental to match quotation candidates with the current query based on the learned representations of both candidates and the query. in this context, indivec endeavors to solve a classification task using a recommendation-oriented approach. ## methodology in this section, we first present the construction of the media bias indicator dataset in §3.1. then, we discuss the challenges associated with indicatorbased bias prediction and introduce our method of adopting indicators for bias prediction in §3.2. ## experimental setup datasets. though our media bias prediction framework is applicable for various types of bias, we primarily conducted experiments on political bias datasets due to their higher visibility and greater abundance. in our main experiments, we established a bias indicator vector database based on the flipbias dataset @xcite . this dataset was sourced from the news aggregation platform allsides.com in 2018, comprising a total of 2,781 events and each event is represented with sufficient text from different political leanings, dataset bias level source bias label paired # of instances avg length % of biased instances flipbias (chen et al., 2018) article new york times, huffington post, fox news and townhall left, center, right yes 6,447 909 76.5 % basil (fan et al., 2019) sentence huffington post, fox news, and new york times lexical bias, informational bias yes 7,984 24.1 19.6% babe (spinde et al., 2022) sentence fox news, breitbart, alternet and so on biased, non-biased no 3,674 32.6 49.3% mfc (v2) (card et al., 2015) article lexis-nexis (database) pro, neutral, anti no 37,623 260 84.5% ## conclusion this work introduces indivec, a novel bias prediction framework. indivec leverages fine-grained media bias indicators and employs a unique matching and voting process. we also contribute a bias indicator dataset, encompassing over 20,000 indicators. our comprehensive experiments and analyses further confirm the effectiveness, adaptability, and explainability of the indivec framework, highlighting its potential as a valuable tool for bias detection in media content.