Conflicts, Villains, Resolutions: Towards models of Narrative Media Framing (2023)

Abstract: Despite increasing interest in the automatic detection of media frames in NLP, the problem is typically simplified as single-label classification and adopts a topic-like view on frames, evading modelling the broader document-level narrative. In this work, we revisit a widely used conceptualization of framing from the communication sciences which explicitly captures elements of narratives, including conflict and its resolution, and integrate it with the narrative framing of key entities in the story as heroes, victims or villains. We adapt an effective annotation paradigm that breaks a complex annotation task into a series of simpler binary questions, and present an annotated data set of English news articles, and a case study on the framing of climate change in articles from news outlets across the political spectrum. Finally, we explore automatic multi-label prediction of our frames with supervised and semi-supervised approaches, and present a novel retrieval-based method which is both effective and transparent in its predictions. We conclude with a discussion of opportunities and challenges for future work on document-level models of narrative framing.

## introduction media discourse around contested issues is often biased by experiences or interests of the news outlets and different stakeholders they give voice to. news framing by the media has been formalized and examined on many levels in communication and social sciences, ranging from selection of information @xcite over discoursecentric @xcite and entity focused approaches @xcite . while a growing body of work in nlp attempts to automatically detect framing in the news or social media, most work adopts well-defined yet oversimplifying approaches like topic modeling or simple 1 code and data are available at https://github.com/ phenixace/narrative-framing . resolution: solution/alleviation of the issue conflict: disagreements between individuals, groups, institutions, countries, etc. human interest: emotionalization and dramatization of an issue through the lens of affected individuals moral: moral or religious references economic: economic consequences for individuals, groups, institutions, countries, etc. hero: entity contributing to/responsible for issue resolution villain: entity contributing to/responsible for issue cause victim: entity suffering the consequences of an issue this paper addresses the above shortcomings by considering framing through the lens of narratives. we adopt a small set of high-level framing devices established in the communication literature @xcite , and integrate them with narrative roles assigned to key actors (entities) in the discourse. table @xcite summarizes the frames used in our work. our framework departs from existing nlp approaches in two ways: first, we adopt a multilabel classification paradigm, allowing for a more nuanced analysis and avoiding the oversimplification of framing to a single label per item. secondly, our framework emphasizes the narrative structure of the full article: frames such as conflict, resolution or human interest are central building blocks of narratives, and are dominant in news coverage of contested issues @xcite . building on components of the narrative policy framework @xcite , we identify the key entities responsible for the issue (villains), those who are affected (victims) and those who can resolve the issue (hero); see we apply our framework to the issue of climate change, a pressing global challenge with widereaching impacts @xcite , which remains politically contested in terms of the understanding of its urgency, causes, and possible solutions @xcite . importantly, studies show that climate 'skeptics' (and those lacking scientific backing) are cited almost twice as often in the mainstream news media as those calling for climate action @xcite , rendering the examination of media framing and its effects on public support for climate change mitigation a pressing goal. while a substantive body of work on climate change framing emerged in the social and communication sciences @xcite , the issue has attracted surprisingly little attention in the nlp community to date. exceptions include work on stance detection in news @xcite or social media @xcite or models of scepticism detection @xcite , whereas we focus on the narrative framing of the issue across political leanings. to recap, in this paper we present: ## background media framing refers to the deliberate presentation of information in order to elicit a desired response or shift in reader's attitude. we introduce into nlp five high-level frames ( to identify each frame, @xcite proposed a set of binary indicator questions which improved annotation quality and portability of the framework across studies. we construct the first publicly available data set annotated with this framework, cover a larger and more diverse set of news articles than prior work, and link the frames with narrative roles assigned to key entities appearing in the story. media framing may manifest through the narrative roles -hero, villain or victim -assigned to key entities in a document, and this phenomenon has been widely studied in the communication sciences in general @xcite and in the context of climate change in particular @xcite . we draw on this work as well as work which identified key stakeholder categories in the climate change discourse @xcite @xcite to analyze the framing of entities in news articles along the political spectrum. nlp studies on framing have predominantly focused emphasis framing, the strategic inclusion or omission of aspects of an issue, such as legality or public opinion @xcite or, to a lesser extent, on equivalence framing as different expressions of identical concepts ("alien" vs "immigrant", @xcite al. (2022); ziems et al. (2022a) ). both perspectives focus on local, lexical signals. emphasis framing is typically formalized as a single-label prediction of the most dominant frame in a news article @xcite , headline @xcite , or a social media post @xcite . the media frames corpus @xcite is one of the most comprehensive frame-labeled data sets comprising several thousand news articles across five contested issues. while the data includes spanlevel labels which could be used for multi-label classification, work using the mfc predominantly attempts document-level prediction of a single "primary" article frame, disregarding span labels @xcite . 2 more broadly, our work complements emphasis frames by considering more abstract frames around conflict, resolution, and personal, moral and economic impacts. both formalizations of framing have a strong foundation in the communication literature, and studying their interaction at scale with nlp methodology is an interesting avenue for future work. @xcite consider a variety of framing strategies in the context of tweets, but with less of a focus on story structure due to the short document lengths. our framework complements their work in three ways: i) we approach framing as multi-label classification, relaxing the assumption of a single frame per article; ii) we present a set of frames that are abstract with evidence distributed across a document, requiring higher-level document model; and iii) we link frames with narratives via entity roles in a unified annotation framework consisting of a series of binary indicator questions, allowing us to study the interplay of framing and narratives. ## the narrative frames corpus we identified 17.9k english-language news articles on climate change published in 2017-2019 in the uk and in the us by matching a set of climate change-specific keywords in articles from the nela corpora @xcite @xcite . see appendix a for more details. for each article, nela provides 2 although see @xcite for an exception. metadata about its publication date, media outlet, and its associated political leaning as identified by the media bias fact check (mbfc) website. 3 we manually annotated a subset of 428 articles of this data set, balanced across the three years and the four most dominant mbfc categories: left, centerleft, right and questionable source. 4 we recruited four on-site annotators, all english native speakers with a background in the social/political sciences. the annotators went through an extensive training phase including several rounds of feedback. details of annotator remuneration can be found in the ethics statement. frame annotations we adapted @xcite 's frame indicator questions. we added a pre-screening question to confirm that an article is predominantly (>70%) about climate change, removed one question about visual information (as we focus on text only), and changed wording specific to the 'government' to 'any entity' to align with our broad definition of stakeholder entities, discussed below. the full questionnaire is shown in appendix b. annotators were presented with the full article text together with the questionnaire, but no explicit meta information such as outlet name or date of publication. the raw annotations provided answers to a list of binary indicator questions. we verified that the mapping (i.e., the factor structure) between the five frames in in a multi-label data set with articles covering zero (12%), one (39%), two (32%) three (15%) or four (3%) frames. see appendix f for additional data set statistics. entity annotations entities were annotated as part of the binary indicator questionnaire introduced above. three indicator questions assessed whether an article contained an entity that could alleviate the problem (hero); was responsible for the problem (villain) or was negatively affected by the issue (victim). if an annotator answered 'yes' to any of these questions, they were asked to identify the most appropriate entity in the text. an entity was meant to be selected only if the article was explicit about that entity's role (e.g., a politician was depicted as "the only person who could save the planet") and strictly based on the entity's presentation in the article, rather than the annotator's opinion about that entity. 7 we included all entities extracted by our annotators as part of our published data set. annotator agreement krippendorff's α across four annotators and 13 frame indicator questions is 0.52, indicating fair agreement as expected for a complex task like frame annotation. average pairwise agreement without chance-correction is 0.78 (@xmath2.75, @xmath3.81). a total of 2,185 entities were extracted across all narrative roles. average pairwise agreement on existence of a role in an article was 0.59 (krippendorff's α = 0.40). to assess agreement on the identity of entities for roles which were attested by at least two annotators, we computed the exact string match of associated entities, after basic text normalization. entities match exactly 41% of the time. we also computed more lenient metrics based on token overlap (average rouge-@xmath4.45) and embedding similarity (average. @xmath5.91) between pairs of extracted entities. the agreement for both role detection and entityrole assignment was low overall, suggesting that the task is challenging. in this paper, we use the narrative role labels in the exploratory analysis in section 4 and discuss future work on computational modeling of narrative roles in section 6. ## narrative framing of climate change we conduct an exploratory analysis on the framing of climate change in media outlets with different political leaning, as well as the interplay of frames, narrative roles and stakeholder categories. framing and political leaning figure 2 shows the proportion of articles mentioning each frame by the media outlets' political leaning. resolution (re) is more prevalent in the leftleaning outlets (left, left_center), while the opposite is true for human interest (hi): right-leaning (and questionable) outlets are more likely to refer to personal stories and use language evoking empathy. these findings are partially consistent with prior work, e.g, @xcite show that both economic and conflict are more likely to be used in conservative outlets, while we find conflict prevalent across the board. however, @xcite only included three major us news papers, in contrast to 41 in our analysis. frames, roles and stakeholders figure 1 illustrates the association of narrative roles with different frames (1a) and stakeholders (1b). unsurprisingly, the hero, an entity presented with the ability to fix or alleviate the issue under discussion, is the most prevalent role in the resolution frame. the villain dominates most other frames, except for the human interest frame where victim is equally dominant. this aligns with a well-known "negativity bias" in news reporting, i.e., a dominance of negative content with a focus on problems, conflicts and their causes and victims @xcite . we explore the distribution of roles across stakeholder categories in figure 1b . overall, governments & politicians are the most dominant stakeholder category, typically depicted as the villain (of all stakeholder categories they are also most likely to be depicted as the hero pointing to ambivalent attitudes toward this category). the environment and the general public dominate the victim role, somewhat unexpectedly followed by governments & politicians and industry & emissions. we explain this phenomenon next by disentangling the labels by political leaning. figure 3 reveals how the framing of a particular stakeholder category can vary with political leaning of the source. right-leaning media are more likely to depict environmental activists & organisations, and legislation as the villain and industry and emissions as either the hero or the victim in the context of climate change news. conversely, left-leaning media are more likely to frame legislation as a hero, cover environmental activists less frequently overall, and predominantly frame the industry as a villain. ## narrative frame prediction predicting narrative frames automatically and with high quality would open new possibilities for scaling media framing analyses to larger data sets, longer time spans or more languages. given the political sensitivity of automated media analysis, models should not only be reliable but also transparent in their predictions. to this end, we present retrieval-based frame prediction (rbf), which incorporates an embedding-based retrieval module into supervised classifiers. we compare rbf against a range of neural classifiers on multi-label frame prediction. rbf not only outperforms off-the-shelf fine-tuned transformers on this task, but also increases interpretability by predicting frames for a given article together with the most relevant article sentences for the frame as evidence. section 6 discusses additional modelling tasks supported by our data set to be addressed in future work. ## discussion what are the recurring narratives that frame the public discourse about contested issues like climate change? existing nlp approaches to frame prediction fall short of answering this question due to a focus on localized signals. drawing on theories from the social and communicative sciences, we introduced a set of narrative framing devices to nlp, and integrated them with narrative roles assigned to the central entities in the news articles. we applied our framework to the issue of climate change, and @xmath6. our exploratory analysis demonstrated how our framework can be utilized to study multiple levels of framing, including differences across outlets; co-occurrence of frames and narrative roles; and assignment of narrative roles to stakeholder categories. with the ultimate goal of scaling such analyses to larger, unlabeled data sets, we introduced rbf, an effective and interpretable retrieval-based frame classifier. the 'explicit attention' module of rbf not only improved performance over its backbone, the vanilla longformer, but also naturally provides evidence for its predictions as a list of relevanceranked article sentences. our work addresses a disconnect between the complexity of framing acknowledged in the communication science literature, and models of framing in nlp. as recently surveyed by @xcite , nlp approaches to framing predominantly focus on topic models or frequency-based methods, leaning heavily on local lexical signals as indicators for the presence or absence of a single frame per unit of analysis. however, the framing of a news article typically emerges from indicators spread throughout the text; frames can co-exist and interact with each other within a single news story. this paper takes one step towards such an integrated notion of framing in nlp in considering narrative frames and roles at the article level and adopting multi-label task formalization. our work and results suggest many avenues for future research. the narrative frames corpus supports research on joint models of framing and entity roles: the presence of an entity with a specific role (e.g., the hero) should render the presence of certain frames (e.g., resolution) more likely. conversely, frames like conflict impact the probability of the existence of the number and kind of different roles (e.g., the hero and the villain). a joint model of frames and narrative roles could incorporate role labels with soft confidence weights as latent signal into a frame classification model. annotator (dis)agreement and aggregation of answers to indicator questions into frame tags provide fertile grounds for future work. we echo a line of recent work on acknowledging label variation as a signal of genuine complexity rather than noise. this holds true particularly for complex tasks like frame annotation which inevitably retain a level of subjective variation @xcite . in this paper we aggregated indicator labels into a hard frame label by voting, however we release the raw annotations as part of our data set. future work could explore soft aggregation methods, delineate genuine variation from noise, and adopt disagreement-aware models and evaluation metrics. the comparatively small size of the narrative frames corpus and the competitive semisupervised snippext suggest further exploration of semi-supervised approaches. integrating rbf with a snippext-inspired semi-supervised framework, most simply by soft labeling articles as a function of their retrieved sentences and rbf relevance scores, would allow to leverage large unlabeled data sets while retaining rbf's interpretability. alternatively, one could adapt models from different domains, for instance by drawing on the literature of modeling narrative roles in folk tales @xcite . ## limitations we acknowledge a range of limitations of our work. as discussed in sections 3 and 6, overall annotator agreement ranged from fair (frame annotations) to low (entity role annotations). we do not view this as a limitation per se, again pointing to the recent literature on the value of human label variance pointing at a potential loss of valuable information if we overly focus on arriving at a single gold label per instance, with high confidence @xcite . future modeling work involving entity labels should, however, carefully inspect the role label variation, and potentially remove or aggregate selected annotations, before incorporating the labels as signal into predictive models. we explicitly refrained from training model in this paper to avoid the risk of training a predictor on an unfavorable noise-to-signal ratio. our data set focuses on english-language news reports, sampled from 2017 to 2019 in mainstream media outlets in the us and uk, and as such focuses on cultures and communities which are already well-resourced and well studied. with climate change being a global challenge, broadening data sets, annotations and models to more languages is an important direction for future work. we explicitly caution against projecting annotations across languages without careful validation as we expect the manifestation of framing, views on entities (or sheer set of dominant entities) to vary widely across countries and communities. even within our english study, we acknowledge that the size of annotated data is small for nlp scales, and an extension in the future is desirable. a related current limitation is the focus on just a single issue (climate change) and validation of our narrative framing framework for other issues is an important direction for the future. finally, the annotation process was slow and costly, relying on trained, highly educated annotators with constant monitoring, rendering larger scale annotations challenging, on the one hand. on the other hand, we will release upon acceptance our annotation procedure including the full codebook with instructions, which have been optimized over several rounds of annotations and we hope can support more efficient annotation in the future.