Reap the Wild Wind: Detecting Media Storms in Large-Scale News Corpora (2024)

Abstract: Media storms, dramatic outbursts of attention to a story, are central components of media dynamics and the attention landscape. Despite their importance, there has been little systematic and empirical research on this concept due to issues of measurement and operationalization. We introduce an iterative human-in-the-loop method to identify media storms in a large-scale corpus of news articles. The text is first transformed into signals of dispersion based on several textual characteristics. In each iteration, we apply unsupervised anomaly detection to these signals; each anomaly is then validated by an expert to confirm the presence of a storm, and those results are then used to tune the anomaly detection in the next iteration. We make available the resulting media storm dataset. Both the method and dataset provide a basis for comprehensive empirical study of media storms.

## introduction media storms -dramatic increases in media attention to a specific issue or story for a short period of time @xcite -are central components of media dynamics. such outbursts include, for example, news reports on acts of terrorism, public scandals, or major political decisions. they usually begin with a specific trigger event (e.g., @xcite , and then surge to disproportionate levels of coverage -hype (e.g., @xcite . storms intensify nearly all media-related effects (e.g., @xcite . in addition, being pivotal moments in the public agenda, storms can be critical junctures for political actors @xcite . however, we still lack a systematic and comprehensive understanding of such outbursts of media attention. one reason is that it is not clear how to operationalize this concept into a concrete measurable object @xcite . essentially, previous researchers are left devising "arbitrary" thresholds for their studies @xcite . in addition to this amorphousness, an additional challenge is that media storms are relatively sporadic phenomena. @xcite approximate that they consist about 11% of all media coverage, a finding that was later corroborated by @xcite . these properties make it extremely difficult to create a gold-labeled data-set to train a model, or to even begin reading the raw articles to identify media storms directly, necessitating the development of a different strategy to solve this challenging task. traditionally, communication researchers employed manual content analysis to label and measure issue attention over short periods (e.g., @xcite . recent computational work has utilized topic modeling @xcite and keyword analysis @xcite for the task. however, the drawback of these approaches is their sensitivity to research designthe keyword choice or delineation of topics. a researcher might choose a model with broad topics -hampering the ability to recognize deviances of specific outburst. conversely, an overly complex model might cause a media storm to be dispersed across several topics, diluting attention peaks. this could make significant media events less discernible. meanwhile, focusing on keywords may obfuscate the actual story behind the tokens. another approach adopted in recent computational communication research has been to focus on news story chains. such methods utilize clustering to identify news events -articles describing the same event or story (e.g., @xcite . these techniques 'uncover' the stories occurring in the corpus -groups of documents discussing the same, specific event. recently, @xcite fine-tuned a model to generate document embeddings for the clustering. while these methods identify media stories, they do not encompass the theoretical concept of media storms. rather than capturing prominent stories, we are aiming for periods where the media coverage is not structured normally @xcite , but rather characterized by 'hype'-dramatic and anomalous levels of coverage of a story @xcite . however, it is impossible to determine hype when only taking into account the structure of a single story at a single time-step without noting long-term trends and cycles as baselines. tellingly, such methods tend to identify many more instances than we do in our experiments here. for example, @xcite identify 98 cases over nearly two years, while we detect 221 over a 20 year period. with these points in mind, we sought a different approach we believe better reflects the theoretical conception of storms. we return to the basic definitive property of media storms: a dramatic, temporary spike in attention to an issue (above the norm). in other words, storms are anomalies in news coverage, so we turn to anomaly detection to identify them. we create several signals representing the daily dispersion of texts across the time frame. these signals are the basis for a two-step procedure. first, an unsupervised anomaly detection model identifies media storm candidates-anomalous periods of news convergence. then, a domain expert labels true media storms from these candidates. this human-in-the-loop process iterates until convergence, uncovering media storms over the period. our approach offers several advantages. first, methodologically speaking, it integrates the temporal features of topic-or keyword-based outlier detection described above, without relying on or being limited by idiosyncrasies of researcher design. additionally, the utilization of unsupervised anomaly detection allows us to overcome the huge quantities of data, presenting experts with a small set of candidates to focus on in determining the existence of media storms. furthermore, our approach attempts to bypass inherent amorphousness by offering a solution that is not based on predefined statistical thresholds designed for specific texts, but rather relies on the overall dynamics of news coverage for any given period. the use of unsupervised anomaly detection allows media dynamics to reveal themselves in the data. our expert input comes into play in validating these patterns, confirming they correspond to the theoretical concept. this expert input in the choice of seeds and dispersion signals can also allow researchers to integrate their own research perspective within the process -an advantage when dealing with an inherently amorphous concept.thus, we are able to uncover additional, more diverse media storms than in previous studies. we utilize a large-scale corpus of news articles spanning 20 years of media coverage to demonstrate our method. we employ two distinct experimental setups, addressing a broad spectrum of potential research applications. the first setup utilizes a seed list of media storms to uncover additional occurrences within the same time frame. the second setup utilizes an analyzed time frame to detect media storms in a new, unlabelled target period. we conclude with a preliminary analysis of our findings from both setups, underscoring the efficacy of our method and its potential for media storm research. we then test the capability of a generative large language model (llm) to perform the expert validation. the results justify the human-inthe-loop approach, while pointing to the possibility of further automation in the future. finally, we contribute these findings as a media storms dataset for the years 1996-2016. we believe that this dataset opens up a wide array of exciting research avenues. while the concept of media storms holds great significance to social actors, politicians and social scientists from various fields, empirical exploration has been limited. as the classification of storms within large-scale news coverage data improves, we can enhance our understanding of how these news hypes unfold from a single story or event to a cascade of public interest. in an era marked by heightened concern over the media's impact on the information landscape -highlighted by issues like polarization, the spread of misinformation, and the prominence of social media -such insights into these significant elements should offer important contributions. ## data 2.1 news articles to track the media coverage, we assembled a corpus of 1,187,607 news articles taken from three major news outlets -the new york times, the los media outlet articles tokens tokens/article (avg.) new york times 520,648 373,980,075 718.30 washington post 360,788 293,024,961 812.18 los angeles times 306,171 240,119,545 784.27 total 1,187,607 907,124,581 763.83 ## automated validation the second step in our proposed procedure (section 3.3) involves manual validation of media storm candidates by an expert. in order to estimate the possibility to automate this step, we performed an experiment designed to test the capability of a generative llm to perform this task. we assembled all the media storms candidates produced in the first step in the procedure, during both experimental setups described in section 3.4, resulting in a set of 320 unique candidates. for each candidate, we prompted the gpt-4 model @xcite to decide whether or not it constitutes a media storm, providing it with a sample of 75 news articles from the relevant dates as well as their pairwise cosine-similarities (see appendix c for full details). notably, the expert and gpt-4 were in agreement about 45% of the storm candidates. among these, they agreed on the storm's label in 74% of the cases. however, gpt-4 failed to identify a large number of media storms found by the expert. these include some clear cases, such as the british petroleum oil spill in the gulf of mexico (2010), the shooting of u.s. representative giffords in arizona (2011), and the ebola outbreak (2014). while these results justify the human-in-the-loop approach, they merit further exploration into the possibility of utilizing computational models in performing (or at least aiding in) the validation step. this analysis further offers a unique opportunity to explore possible false-negatives by the expert (media storms they had missed). a total of 32 candidates were validated as media storms by the gpt-4 model but not by the expert. after reviewing these, five were determined to qualify as media storms by our expert: one new event, the khobar tower bombing (1996), and four cases of additional peaks in coverage surrounding media storms previously validated as such by the expert. ## conclusion & future work in this paper, we offer several contributions. first, we present a human-in-the-loop method to detect media storms in a large corpus of news texts. we describe a two-step iterative procedure, combining unsupervised anomaly detection and expert validation, to identify these rare events within a larger dataset. significantly, whereas previous studies build upon 'arbitrary' statistical thresholds, we utilize an unsupervised anomaly detection algorithm to allow the media dynamics to reveal themselves in the data. our expert input comes into play in validating these patterns, confirming they correspond to the theoretical concept. consequently, we are able to uncover additional, more nuanced media storms than in previous studies. by incorporating expert validation, we can set the granularity or type of the storms which we seek to identify; researchers can express their research agenda to decide what types of media storms they are interested in detecting. additionally, we performed a comparison between the expert and gpt-4, demon-strating that while not fully capable of replacing a human expert, there is some potential in utilizing a generative llm during the validation process. second, our method offers a procedure that can be applied in various research scenarios, over diverse and large corpora, while leveraging expert knowledge for validation. within the realm of this paper, we included three english-language newspapers for a specific time-frame. however, the method could plausibly be applied on any news corpora in any language, provided the necessary techniques could be utilized (e.g., entitydetection, sentence transformers). additionally, researchers might be able to use this approach on non-mainstream media sources as well, including identifying periods of textual convergence in social media platforms and digital news. third, through the two experimental setups, we collected a comprehensive list of media storms. this time frame we chose to focus on is of particular significance for media scholars. between 1996 and 2016, the media landscape underwent dramatic transformations, with the rise of 24-hour news cycles, the interactivity of social media and the fragmentation of the attention landscape @xcite . these validated storms provide opportunities to examine intriguing theoretical questions, including how the volatility of the media landscape has evolved, changes in the events triggering storms, and perhaps developing predictive capabilities regarding storm outbursts and durations. thus we use the results of this study to provide a dataset consisting of media storms with their start and end dates, which will be made publicly available to researchers together with the dispersion signals extracted from the corpus.