SAFARI : Cross-lingual Bias and Factuality Detection in News Media and News Articles (2024)

Abstract: In an era where information is quickly shared across many cultural and language contexts, the neutrality and integrity of news media are essential. Ensuring that media content remains unbiased and factual is crucial for maintaining public trust. With this in mind, we introduce SAFARI (CroSs-lingual BiAs and Factuality Detection in News MediA and News ARtIcles), a novel corpus of news media and articles for predicting political bias and the factuality of reporting in a multilingual and cross-lingual setup. To the best of our knowledge, this corpus is unprecedented in its collection and introduces a dataset for political bias and factuality for three tasks: (i) media-level, (ii) article-level, and (iii) joint modeling at the article-level. At the media and article levels, we evaluate the cross-lingual ability of the models; however, in joint modeling, we evaluate on English data. Our frameworks set a new benchmark in the cross-lingual evaluation of political bias and factuality. This is achieved through the use of various Multilingual Pre-trained Language Models (MPLMs) and Large Language Models (LLMs) coupled with ensemble learning methods.

## introduction the integrity and objectivity of the news media are crucial in an age where information is rapidly disseminated across diverse cultural and linguistic landscapes @xcite . as observed @xcite , misleading information or "fake news," spreads six times faster than the truth and reaches a much larger audience. this underscores the need for comprehensive data to assess political bias and factuality in news media and articles, particularly in a cross-lingual context, which remains a significant challenge @xcite . thus, we introduce a novel corpus safari specifically designed for the cross-lingual analysis of political bias and factuality in news media and articles. our work in developing this corpus is motivated by the absence of cross-lingual resources for detecting political bias and factuality in media and articles analysis. to address this issue, we offer a dataset for ten languages: (i) at media-level political bias, we have slightly less than 2k, and for factuality marginally over 2.6k media, (ii) at article-level we collect around 190k for political bias and around 190k of articles for factuality, and (iii) for joint modeling at article-level we have moderately less than 100k of english articles. furthermore, the methodology behind our study incorporates the use of mplms and llms to assess dataset tasks. our approach enables mplms and llms to provide an evaluation of political bias and factuality across languages at the source and article levels and in joint modeling. moreover, mplms and llms (using zero-shot learning) are coupled with ensemble learning methods for evaluation. our contributions are as follows: in section 2, we provide a review of previous and recent studies that focus on political bias, factuality, and joint modeling analysis. in section 3, the data collection process and the subsequent examination are elaborated. in section 4, the research tasks are defined and the statistics of the dataset are introduced, together with the frameworks and techniques. section 5 delineates our analysis and the results obtained using multiple mplms with ensemble learning. section 6 explores our investigation of zero-shot learning for our tasks using llms and nuances of distant supervision vs. expert annotated data. finally, section 7 summarizes our findings and suggests potential future directions. political bias understanding political bias is a nuanced exploration with varying definitions, including uneven coverage or favoritism @xcite and systematic preferences for candidates or ideas @xcite . @xcite employed pre-trained bert @xcite models to detect linguistic political bias in news articles. @xcite expanded the concept of media bias, considering dimensions such as selection and presentation of political bias influenced by the choice of newsmakers @xcite . the study by @xcite used annotated media from @xcite , analyzing 300 nyt, fox, and hpo articles for bias, similar to our distant supervision approach to capture diverse ideological perspectives. research on selection political bias requires huge databases, with studies using commercial @xcite @xcite and public datasets @xcite using multi-source approaches @xcite . various methods measure the political bias of news media, including linking news outlets with politicians, analyzing shared audiences @xcite , and identifying the intricate linguistic techniques used to shape readers' opinions and emotions @xcite . alternately, political bias is assessed through twitter interactions @xcite @xcite . predictions extend to political bias at the media, article and sentence levels, often using distant supervision with small datasets only in english @xcite @xcite @xcite @xcite @xcite . factuality veracity of information is examined at various levels: claim-level (e.g., "factchecking"), article-level (e.g., "fake news" detection), user-level (e.g., identifying trolls), and medium-level (e.g., source reliability estimation). claim-level efforts focus on fact-checking and rumor detection using social media interactions @xcite @xcite @xcite @xcite @xcite @xcite , focusing on the stance and reliability of the source. early work estimated source reliability based on a medium's stance towards true/false claims using an english dataset @xcite @xcite @xcite . recent approaches, such as @xcite , used gold labels and various english information sources, which are relatively small compared to our work. @xcite and @xcite used graph-based frameworks to profile news media outlets, focusing on relationships and audience overlap. llms (e.g., chatgpt) are used for the estimation of source reliability, as demonstrated by @xcite , correlated with human expert ratings, and @xcite introduced a framework that combined graph-based models, llms and human expertise for the profile of news media, effectively identifying fake news with minimal joint modeling joint modeling of factuality and political bias remains underexplored, with an attempt by @xcite using a small english dataset using multi-task ordinal regression. understanding the relationship between factuality and political bias in the news media, especially when outlets exhibit different behaviors on these aspects, presents a significant challenge. ## dataset construction our methodology encompasses two levels of data collection: media-level and article-level, both using the distant supervision technique @xcite for article collection. we use a two-step criterion: (i) we exclusively used sources expertly annotated by media bias/fact check 1 . (ii) we select active media outlets. in media-level, we gather sources from media bias/fact check (mbfc) and collect up to 30 front-page articles from each web-1 www.mediabiasfactcheck.com site, labeled according to their sources. similarly, in article-level, we apply distant supervision by assigning labels to articles from media annotated by mbfc, and collect expert-annotated data for political bias from allsides foot_0 to compare performance with distant supervision data. in addition, during the data scraping process, we specifically targeted sections that focused on political, economic, and social issues. with this in mind, we used the ebkmeans @xcite clustering to analyze our entire dataset and identified 15 clusters. furthermore, we validate our choice with the silhouette score, confirming the quality and separation of the clusters. the percentage distribution of data points was calculated across the clusters and visualized in figure 1 . ## discussion in this section, our analysis focuses on the latest llms, specifically mistral 7b and llama2 7b , examining their capabilities in zero-shot learning coupled with ensemble using hard voting. furthermore, we explore why models tested on distant supervision data exhibit higher performance levels compared to those tested on expert-annotated data, specifically regarding the detection of political bias at the article-level in the english language. ## conclusion & future work in this article, we introduce safari, a new largescale corpus for cross-lingual evaluation at the media and article levels, specifically designed for the detection of political bias and factuality of reporting, along with our data construction pipeline. furthermore, we present an exclusive english dataset for joint modeling at the article-level. we also compare the performance of distant supervision vs. human-annotated data for political bias at the article-level. moreover, our corpus is evaluated using mplms, and we implement hard and soft ensemble learning voting for all tasks. lastly, we experimented with llms using hard voting. in future work, our aim is to gather a larger multilingual corpus and conduct a more fine-grained analysis of political bias and factuality. acknowledging that the u.s.-centric left/center/right political spectrum is not universally applicable, we plan to model biases that are more relevant to different regions and cultures. we also intend to collaborate with experts, seek alternative data sources, and expand the date ranges of news outlets to reduce data imbalance and create a larger and more diverse dataset. furthermore, we plan to perform a multimodal analysis of political bias and factuality in news media and articles. we will also deepen our error analysis, breaking it down by language to improve performance. additionally, we will conduct experiments to study cross-lingual abilities in detail, focusing on discrepancies in factuality and political bias for articles on the same topic across different languages, and stratify results based on topic distribution. finally, we plan to investigate political bias and factuality using fine-tuned llms, potentially leveraging techniques such as lora @xcite and qlora @xcite .