The COVID That Wasn’t: Counterfactual Journalism Using GPT (2022)

Abstract: In this paper, we explore the use of large language models to assess human interpretations of real world events. To do so, we use a language model trained prior to 2020 to artificially generate news articles concerning COVID-19 given the headlines of actual articles written during the pandemic. We then compare stylistic qualities of our artificially generated corpus with a news corpus, in this case 5,082 articles produced by CBC News between January 23 and May 5, 2020. We find our artificially generated articles exhibits a considerably more negative attitude towards COVID and a significantly lower reliance on geopolitical framing. Our methods and results hold importance for researchers seeking to simulate large scale cultural processes via recent breakthroughs in text generation.

## introduction the rush to cover new covid-19 developments as the virus spread across the world over the first half of 2020 induced a variety of editorial mandates from public broadcasters. chief among these was a desire to mitigate shock from a public unaccustomed to large-scale public health emergencies of the calibre covid-19 presented. this desire translated into systematic underreporting together with a reluctance to portray covid-19 as the danger it was @xcite . broadcasters in the united states @xcite , the united kingdom (garland and @xcite , and italy @xcite all exhibited this phenomenon. although many studies have verified the above effects, few if any studies to date have considered alternative approaches the media could have taken in their portrayal of covid-19. evaluating these alternatives is critical given the close relationship between media framing, public opinion, and government policy @xcite . in this paper we present a novel method of simulating media coverage of real world events using large language models (llms) as a means of interpreting news industry biases. llms have been used in a variety of settings to generate text for real-world applications @xcite . to our knowledge, they have not yet been used as a tool for critically understanding the interpretation of events through media coverage or other forms of cultural framing. to do so, we use generative pre-trained transformer 2 (gpt-2), which was trained on text produced prior to the onset of covid-19, to explore how the canadian broadcasting corporation (cbc) covered covid and how else they might have reported on these breaking events. by generating thousands of simulated articles, we show how such "counterfactual journalism" can be used as a tool for evaluating real-world texts. ## background the covid-19 pandemic has given researchers a variety of opportunities to study human behavior in response to a major public health crisis. one core dimension of this experience is reflected in the changing role that the media has played in communicating information to the public in a quickly changing health environment @xcite . times of crises enshrine the media as a valuable mediator between the public and government. this changing role registers itself in the editorial policies at news corporations across the world. research published over the past two years has confirmed that public news broadcasters in australia, sweden, and the united kingdom all significantly altered their editorial style in response to both societal and governmental pressures @xcite @xcite during . what this research has so far lacked is the ability to infer what could have been communicated, i.e. what losses were entailed in these editorial shifts. while a great deal of recent work has studied the biases intrinsic to large language models foot_0 , no work to date has used llms to study the biases of human generated text. reporting on real-world events inevitably requires complex choices of selection and evaluation, i.e. which events and which actors to focus on along with modes of valuation surrounding those choices. simulating textual production given similar prompts such as headlines can provide a means of better understanding the editorial choices made by news agencies. in using a language model as a simulative mechanism, we draw on a long research tradition of using simulation to understand real-world processes. simulation has proven a boon for those working in the sciences, including climate science and physics @xcite , and for those working in the social sciences, where agent-based social modelling has led to advances in understanding complex social phenomena @xcite . we seek to bring these techniques to the study of cultural behavior, where simulation has historically seen less of an uptake @xcite . ## method our project consists of the following principal steps: ## discussion in this section we identify three noteworthy discrepancies between the behavior of our models and the real-world cbc corpus and discuss their potential implications. ## conclusion the aim of our paper has been to develop a framework for using the text-generation affordances of large language models to better understand the interpretive perspectives of the news media when covering major social events. we rely on a simulative process whereby the generation of thousands of alternative views of a real world event can provide a framework for understanding the interpretive perspectives employed by news organizations. given that language models can approximate human discourse @xcite , they can be used to generate a distribution of possible responses to an event to better understand the actual selection mechanisms used by real-world actors. key to this process is validating the extent to which the qualities of artificially generated text are a function of model parameters or the process of fine-tuning, i.e. an effect of the real-world event we aim to simulate. our aim in doing so is to illustrate how language models can be used as diagnostic tools for human behavior. given no prior knowledge of a major event, what would a language model say? and what might this tell us about our own human reactions? based on the results we have obtained here, we see the following possible avenues for further research using llms for textual simulation: further domain exploration. what other scenarios might llms be analytically useful for? in this paper, we have explored llms as a tool to assess media coverage, but future work will want to observe how they behave in other domains. news is a particularly well-structured form of textual communication and thus we expect llms to perform more adequately in this domain given prior research @xcite . we await future work exploring other textual domains.