Quantifying Generative Media Bias with a Corpus of Real-world and Generated News Articles (2024)

Abstract: Large language models (LLMs) are increasingly being utilised across a range of tasks and domains, with a burgeoning interest in their application within the field of journalism. This trend raises concerns due to our limited understanding of LLM behaviour in this domain, especially with respect to political bias. Existing studies predominantly focus on LLMs undertaking political questionnaires, which offers only limited insights into their biases and operational nuances. To address this gap, our study establishes a new curated dataset that contains 2,100 human-written articles and utilises their descriptions to generate 56,700 synthetic articles using nine LLMs. This enables us to analyse shifts in properties between human-authored and machine-generated articles, with this study focusing on political bias, detecting it using both supervised models and LLMs. Our findings reveal significant disparities between base and instruction-tuned LLMs, with instruction-tuned models exhibiting consistent political bias. Furthermore, we are able to study how LLMs behave as classifiers, observing their display of political bias even in this role. Overall, for the first time within the journalistic domain, this study outlines a framework and provides a structured dataset for quantifiable experiments, serving as a foundation for further research into LLM political bias and its implications.

## introduction the current generation of llms has emerged as an important factor in the ongoing digital transformation @xcite . chatgpt, in particular, has become the fastest adopted technology as of 2024 @xcite . these models have shown a clear impact across various fields, such as software development @xcite and academia @xcite . the specific sector we aim to focus on is journalism. it has, in recent years, experienced a trans-formative period @xcite , moving from printed to digital news and grappling with the increased importance of social media and a subsequent rise of disinformation @xcite . recent reports @xcite suggest that generative ai will bring the next significant shift, with 28% of publishers reportedly using ai in their processes in 2023, and openai expressing interest in this field. 1 the integration of llms into journalism promises new avenues for content creation and dissemination @xcite . however, it comes with a set of challenges and considerations, which are particularly pressing given the importance of quality journalism to the functioning of a free and democratic society @xcite . llms notably face a multitude of problems, from style alignment @xcite to bias management @xcite . nevertheless, when it comes to utilising llms in journalistic processes, one of the most critical yet limitedly understood text properties is political bias. unchecked, it can significantly impact how people consume information-even with regard to verifiable facts-and form opinions on them @xcite . its presence in media has been shown to enhance polarisation in society and a rise in extremism @xcite . therefore, this study seeks to investigate the occurrence of political bias directly in generated news articles, an aspect not covered in existing literature. by examining the extent to which current llms exhibit political bias in their generated content, in what direction this bias leans, and under what conditions it manifests, we can gain a concrete understanding of this issue and better assess the risks it poses. to answer these questions, this work delivers: ## article generation our aim is to generate synthetic news articles related to human-authored articles using their summaries in the prompts. to broaden our experiment, we will not only explore generation under unbiased conditions but also investigate deliberately biased settings. in these, models are prompted to emulate either a left-wing or a right-wing news style. ## experiment with all the data generated, the final dataset consists of 2,100 human-written articles and 56,700 ai-generated articles, each ai-generated article corresponding to a human-written counterpart. the primary aim of the entire experiment is to investigate the political bias of llms quantitatively. this involves measuring the political bias shift. to calculate this, we first determine the political alignment score for each article. this score is computed based on the probabilities that an article exhibits right-wing (p right ) or left-wing (p left ) tendencies. the formula for the political alignment score is: we then calculate the alignment scores for both the ai-generated (pa generated ) and human-written (pa human ) articles. with human-written articles acting as a baseline, the political bias shift (∆pa) is calculated as follows: the resulting value of ∆pa reflects the change in political alignment between the human-written and synthetic article. a positive ∆pa indicates a shift towards a right-wing bias in the ai-generated article, while a negative ∆pa suggests a left-wing bias. overall, this approach enables us to study biases within llms by focusing solely on the shifts, allowing for comparative and quantitative analysis. ## discussion through this analysis, we have identified several patterns that offer insights into the behaviour of specific llms in generating news content. ## conclusion this study introduces a curated dataset of paired human-written and machine-generated news articles, offering resources and a framework for the quantitative exploration of stylistic and semantic shifts within a journalistic context. we employed this dataset to investigate political bias in llms, conducting several experiments with both supervised models and llm classifiers. our results reveal significant political bias in instruction-tuned models, raising concerns about their broader application in journalism. furthermore, the llms exhibited a much stronger tendency towards leftwing bias and a greater susceptibility to generating it when prompted, in contrast to right-wing bias. lastly, we found that llms exhibit political bias even in classification tasks, which, alongside findings from other studies, confirms that bias permeates various applications of llms. consequently, we recognise the need for further research before llms can be safely integrated into the media industry. we encourage subsequent examinations of llm behaviour to build on the provided dataset.