Connecting the Dots in News Analysis: Bridging the Cross-Disciplinary Disparities in Media Bias and Framing (2024)

Abstract: The manifestation and effect of bias in news reporting have been central topics in the social sciences for decades, and have received increasing attention in the NLP community recently. While NLP can help to scale up analyses or contribute automatic procedures to investigate the impact of biased news in society, we argue that methodologies that are currently dominant fall short of capturing the complex questions and effects addressed in theoretical media studies. This is problematic because it diminishes the validity and safety of the resulting tools and applications. Here, we review and critically compare task formulations, methods and evaluation schemes in the social sciences and NLP. We discuss open questions and suggest possible directions to close identified gaps between theory and predictive models, and their evaluation. These include model transparency, considering document-external information, and cross-document reasoning.

## introduction the depiction of complex issues in the media strongly impacts public opinion, politics, and policies @xcite . because a handful of global corporations own an increasing proportion of news outlets, the reach and impact of biased reporting are amplified @xcite . although perfect neutrality is neither realistic nor desirable, media bias turns into an issue when it becomes systematic. if the public is unaware of the presence of bias, this can lead to dangerous consequences, including intolerance and ideological segregation @xcite . figure 1 illustrates the concepts of framing and media bias adopted in this paper, using the passing of the respect for marriage act as an example. framing refers to the emphasis of selected facts with the goal of eliciting a desired interpretation or reaction in the reader @xcite . the leftleaning article in figure 1 leads with an uplifiting picture of a wedding and emphasizes bill support, evoking a positive framing of new opportunities for same-sex couples; while the right-leaning article focuses on disputes in both image and text, framing the issue in a more negative light. political bias refers to partisan slanted news stories, or the "tendency to deviate from an accurate, neutral, balanced, and impartial representation of 'reality' of events and social world" (mcquail and deuze, 2020), which can be a result of a selected framing. in figure 1 , each document was flagged as far-left and far-right ideological leaning, respectively, on the basis of their publishing media outlets. political bias is typically deliberate @xcite while framing may be inadvertent and caused by external pressures such as space limitations. framing and media bias have been under active research in different subfields of the social sciences. angles of study include the manifestation of frames in the mass media and their effects on public opinion (communication sciences); the impact of frames in groups' and individuals' sense-making of the world (social psychology; sociology) or on their observable behaviour (economics and political science). we focus on the first notion: systematic analyses of framing bias in the mass media, through manual coding, or with nlp technology. in this paper, we will collectively refer to the studies of communication and mass media as social sciences. with the increasing pace and almost complete digitisation of news reporting there is a need and opportunity to scale the analysis of media bias @xcite . besides, evidence suggests that exposing media bias promotes healthy public debate, aids journalists to increase thoroughness and objectivity, and promotes critical news consumption @xcite . we discuss the specific role of nlp in this context in section 5. ## background: framing and media bias framing and politically biased news reporting are two strategies to systematically promote specific perspectives on contested issues. we note that not every presence of framing is political bias and not all political bias is represented as framing but their intersection can reinforce each other's impact. they are overlapping concepts which have been addressed jointly or with similar methods in nlp. as such, we include both strategies in this survey. framing has been conceptualised variously in different social science disciplines. prevalent notions of framing include equivalence framing -presenting the same logical information in different forms @xcite -and emphasis framing -highlighting particular aspects of an issue to promote a particular interpretation @xcite . additionally, framing has been conceptualised as a process @xcite @xcite , a communication tool @xcite , or a political strategy @xcite . frames have been conceptualised within different dichotomies. de @xcite distinguishes issue-specific and issue-generic frames which apply to only a single or across several issues, respectively. scheufele (1999) differentiates between media frames, as embedded in the political discourse, and audience frames, as the reader's interpretation of an issue. finally, @xcite defines episodic framing as portraying an issue with an individual example compared to thematic framing, which takes broader context into account. here, we cover both issue-specific and issue-generic frames and attach to @xcite 's notion of emphasis framing. while framing is a priori detached from partisan views, political bias refers to an explicit association of an article or media outlet with a specific political leaning. both concepts result in biased news reporting, and correspondingly nlp researchers have attempted to address them jointly, either by investigating political framing @xcite or by identifying correlations between framing and partisan slanted articles @xcite . nlp studies have attempted automatic media bias identification under several names, including: hyper-partisan news detection @xcite , media bias detection @xcite , identification of biased terms @xcite , and political ideology detection @xcite . their common goal is to detect and classify the bias of a data sample towards a particular political ideology. many of these approaches naturally relate to investigating how the story is told (i.e., framing). ## three disconnects to illustrate the disconnects between the social sciences and nlp, we use a representative study of media bias from the communication sciences (hernández, 2018) which investigates the framing of domestic violence in the south china morning post. the author formulates two research questions: 1. framing functions: are femicides recognized as a problem of domestic violence? what are their causes, and the solutions proposed? 2. frame narratives: what are the main narratives? and which sources are cited in support? the first research question considers the local written aspects within each news article. specifically, it studies the causes and solutions presented, grounded in @xcite 's conceptualisation of framing in terms of a problem, its cause, and its solution. the second research question relates these local aspects to a global (cross-document) view by contrasting narratives that present domestic violence as isolated incidents with those that treat it as a societal problem. it further connects the articles to extrinsic variables, including the sources used and cultural contexts of the story (e.g. whether the article refers the role of women in the chinese family or understands domestic violence through the lens of the confucian philosophy). furthermore, the study considers articles over an extended period, capturing the temporal development of framing and bias. in contrast, current nlp approaches to frame prediction have predominantly adopted a single-label prediction approach per unit of analysis @xcite @xcite , rather than treating frames as structures which could decompose into aspects like cause vs. solution (but see @xcite ; @xcite for recent exceptions). current approaches furthermore treat units of analysis (sentences, articles) as independent without considering links across documents, across time, or to document-external context. the multi-level and dynamic understanding of bias and framing is fundamental in the social science studies. in sum, we identify three fundamental properties of bias and framing that underpin social science research on bias and framing, and we also visually represent them in figure 2 : framing/bias is local and global it is local, because an article can contain several frames, and it is global because understanding the framing of an article may require to aggregate local frames and link them with information such as cited (or omitted) sources, or the outlets' political leaning. framing/bias is dynamic frames change across time, outlets, countries, and communities. understanding the dynamics of framing can shed light on trends and the impact of a sustained exposure to biased reporting on readers' opinions. framing/bias as a comparative task media bias and framing are most apparent when directly contrasting articles from different perspectives, places or times (cf., figure 1 ). formulating our task in a comparative way -rather than predicting instance labels in isolation -may improve the quality, reliability and interpretability of predictions. only 14.3% of our surveyed papers (@xmath0, 9.5% (@xmath1, and 1.6% (@xmath2. the full list of papers and their categorisation can be found in appendix a. the remainder of this article links these fundamental disconnects to the more practical research design decisions that arise across both disciplines: data, methods and evaluation. ## a critical review of current practices in nlp and social science to increase its potential for impact, nlp research needs to reconsider framing and political bias across the entire research pipeline. this includes refining benchmarks, methodologies, and evaluation strategies. in this section, we make recommendations for each component: exploring new aspects of existing datasets, moving beyond single-label classification and incorporating linguistic features as well as external information, and providing transparent and reliable evaluation outputs with error analyses. we critically compare approaches across nlp and the social sciences, pointing out discrepancies together with practical suggestions for future work. ## discussion harmonising depth and scale the differences in data sets and evaluation between the disciplines naturally follow from their respective goals. framing studies in the social sciences aim to uncover the principles underlying framing and its effects through careful, manual analysis of limited amounts of data, typically grounded in theoretical constructs. the primary goal of nlp in the space of media analysis is automation and scalability. complex annotation of large training data sets as required for supervised approaches is infeasible. besides, the required structured annotation paradigms would result in sparse observations of label co-occurrence which in turn would require even larger labelled corpora -and exploding annotation costs. harmonising the goal of scalability with depth and theoretical rigour is a difficult problem (that is not specific to the domain of framing and media bias). one approach towards addressing this problem is the use of semi-or unsupervised approaches, which limit the annotations to evaluation sets of more manageable size. incorporating small amounts of labelled data with powerful pre-trained models is an obvious methodological approach, however, ensuring the validity of predictions and interplay of biases encoded in these models with the target task at hand is an open and important research problem -particularly in a sensitive domain like media bias analysis. feasible yet valid annotation how can we obtain ecologically valid annotations in an efficient way and sufficient quantity? we suggest to follow a common strategy in the social sciences: break articles into self-contained segments, on the event or argument level @xcite . while recent work on argumentation in online debates has followed a similar approach of segmenting contributions into arguments and annotating frames on the argument level @xcite , it has not been applied in the news media context. localised rather than article-level annotations have three advantages: (1) a cognitively easier task for annotators; (2) interpretability through the possibility to provide local, extractive evidence for frame predictions; and (3) a richer document-model of framing that goes beyond the single most likely frame. cross-disciplinary expertise for documentexternal grounding section 3 pointed to a need for multi-level bias analysis, incorporating local, cross-document and broader cultural contexts. most nlp work models individual articles without integrating external information or other articles in the collection. a few exceptions exist, includ-ing @xcite who incorporate readership demographics from twitter and publisher information from wikipedia; and @xcite who incorporate article link structure into their models. both works still model data points in isolation, and fall short of incorporating the more subtle cultural, political or societal contexts that inevitably interact with news framing. we argue for a strong role of cross-disciplinarity and human oversight when incorporating those factors, involving domain experts at every step from formulating research questions to model design, transparency, robustness, and evaluation. cross-disciplinary projects would guide nlp researchers to develop novel methods that are valid and useful for studying the fundamentals of framing and media bias, and equip social scientists with enlarged data sets of high quality and relevance to enrich their research. open data nlp has a strong culture of sharing code and annotated data sets to encourage collaboration and reproducibility. this is less common in the humanities. sharing this data more explicitly through cross-disciplinary dialogue could provide critical assessment and feedback from domain experts. it could drive research into combining large (and potentially noisier) data with small-scale (but high-quality) data sets from the social sciences, to address increasingly complex questions on the emergence and effects of media biases and framing. the role of nlp in media bias analysis despite a surge in data sets and models for automatic analysis of frames and media bias, the ultimate goal of these works receives surprisingly little attention. with the broader adoption of nlp methods diverse applications emerge -from supporting social scientists in scaling their research to larger data samples, to tools that highlight (or even edit) biased news to general public news consumers to expose slanted reporting. an explicit notion of goals and applications (and corresponding statement in research papers) will inform model evaluation, risks and ethical concerns to be discussed in the paper. a mandatory adoption of model cards @xcite is one step in this direction. irrespective of the final application of nlp research, we argue that nlp can contribute safe and valuable tools and methods only if it recognises the complexity of bias an framing both in its data sets and annotations as well as in its evaluation procedures. ## conclusion we surveyed recent work in nlp on framing and media bias, and identified disconnects and synergies in datasets, methodologies, and validation techniques to research practices in the social sciences. despite the opportunities for nlp to support and scale social science scholarship on media bias, a current oversimplification in conceptualisation, modelling, and evaluation limits the validity and reliability of contributions. we have teased out three disconnects and proposed directions for future work, including: (1) analysing news articles from a local and global perspective, incorporating external non-textual features; (2) taking into account the dynamics of framing and bias across documents, cultures or over time; and (3) tackling the issue of media bias as a comparative task, defining frames on the basis of systematic differences between articles whose origins differ on pre-defined characteristics. this would allow for a more complex characterisation of bias than the currently dominant approach of single-label classification.