Narratives at Conflict: Computational Analysis of News Framing in Multilingual Disinformation Campaigns (2024)

Abstract: Any report frames issues to favor a particular interpretation by highlighting or excluding certain aspects of a story. Despite the widespread use of framing in disinformation, framing properties and detection methods remain underexplored outside the English-speaking world. We explore how multilingual framing of the same issue differs systematically. We use eight years of Russia-backed disinformation campaigns, spanning 8k news articles in 4 languages targeting 15 countries. We find that disinformation campaigns consistently and intentionally favor specific framing, depending on the target language of the audience. We further discover how Russian-language articles consistently highlight selected frames depending on the region of the media coverage. We find that the two most prominent models for automatic frame analysis underperform and show high disagreement, highlighting the need for further research.

## introduction framing is a phenomenon grounded in political and social sciences, which specifies how specific topics are presented by the media. it can manifest in loaded vocabularies, like the war on terror, or broader phrases with implicit assumptions. framing has long been studied as an instrument for creating a specific political image or favoring a particular point of view. while it is natural for any non-trivial argument to be framed by the presenter, its intentional (mis)use can create persistent associations and sway opinions on political issues. many works explore framing as an instrument of propaganda and misinformation spread @xcite @xcite . combined with the increased velocity of disinformation in today's media landscape, it highlights an acute need for a detection tool of persistent framing patterns. however, while natural language processing (nlp) is the most logical place for this tool, most advances in frame identification are based on english-speaking environments, in particular in the political context of the us @xcite . no single method has established itself as the state-of-the-art for multilingual data. the few existing methods vary in the best model choice and present conflicting views on the role of the target, non-english language. however, especially in international contexts (and conflicts), (national) language (and relatedly the political position of the presenter) plays an important role in framing. russian media present a prominent example of intentional media manipulation through framing and disinformation spread. several studies have already examined the framing of narratives directed inside the country @xcite . we compare the domestic messaging to the one spread abroad and observe how the same events receive very different framing depending on the language of the target country. contributions: this paper contributes to the growing body of framing research in two ways. 1) we compare two prominent (english-based) frame identification approaches on a novel multilingual dataset. we establish their strengths and weaknesses, and expose the underlying assumptions. 2) by applying the best method to the newly collected data, we contribute to the body of work on framing outside of the english-speaking context. for the languages in our data, we outline the salient topics in recent disinformation campaigns. 1 ## data identifying disinformation remains a matter of expert opinion and careful manual annotations, which makes it a scarce resource outside of the englishspeaking world. faced with the span and size limitations of labeled datasets on disinformation in russian @xcite , we decided to assemble a corpus of disinformation articles guided by the expert opinion on the subject. euvsdisinfo emerged as one such source, part of the eu's diplomatic service led by the eu's high representative, which publishes weekly reports on news articles containing pro-kremlin disinformation. the database includes articles in 15 languages from various news outlets, and more than 15k articles have been reviewed since 2015 to date. even though euvsdisinfo does not assume partial or complete ownership of the media outlets by the state, it is stated that the source articles contain "partial, distorted, or false depiction of reality and spread key pro-kremlin messages." the euvsdisinfo reporting is organized by a disinformation narrative, where a specific event or topic is at the center of the report, supported by links to source articles that reiterate the misinforming narrative. for the target corpus, we crawled all the source articles in russian, french, spanish, and italian for the reporting period from 06/01/2015 to 23/05/2023. we removed any short-form pieces, articles originating from social media platforms, and any news pieces shorter than 300 characters. ## related work the most common approaches to identifying frames treat the task as a variation of sentiment analysis or probabilistic topic modeling @xcite @xcite . while a standardized approach, sentiment, or stance analysis presents limitations to frame identification: most articles employ multiple frames at the same time with various concentrations. additionally, topic modeling doesn't facilitate the comparison of different corpora because of its corpus-specificity and difficulty of interpretation. the more advanced but still traditional approach is creating issue-specific manually annotated handbooks. annotation books, though more formalized, remain a labor-intensive and issue-specific approach, which presents little opportunity for automatic text analysis and frame identification. a more common quantitative approach to frame detection, started with the work of @xcite , is assembling a list of generic frames applicable across issues. beginning with the development of policy frames codebook @xcite and the media frames corpus @xcite , a growing body of work is concerned with automating frame identification at scale. while topic modeling is a versatile approach that can be used with any language, framing analysis with policy frames codebooks, and particularly mfc, relies on data written and annotated in english. this makes the state-of-the-art nlp approaches to frame identification, including the most recent findings of @xcite , english-centric with no apparent transition to other languages. so far, no method has established itself as a standard practice in multilingual frame identifications. two works emerge as the most prominent approaches to multilingual framing analysis. the earlier one is presented in the work of @xcite , which projects english framing onto russian through a lexicon-based, distantly supervised approach. their work focuses on expanding and localizing mfc annotations lexicon by creating language-specific lexicons using an extensive background corpus in the target language. the second approach, presented by @xcite , is based on translating original articles to english and evaluating them with a classifier based on large pre-trained language models. this approach emphasizes the target language less but claims to scale to low-resource languages without needing annotated material. it is advantageous when training data is insufficient, or the computations of training an entire model are prohibitively expensive. to date, these two works present the most prominent approaches to analyzing all frames in a text across languages. ## conclusions we compare two approaches for frame identification on a novel dataset. the formal comparison of the two approaches brought to light a more nuanced result than expected. while the lexicon-based method produced a higher overall competence in estimating framing on multilingual pairs, the results appear mixed depending on the presentation of the data. we suspect distinct reasons for each method's low performance. for the lexicon-based approach, the unexpected drop in performance could reflect the insufficient lexicon for specific frames. for the transformer-based approach, the poor performance on the frames overrepresented in the mfc could be either a consequence of choices in model fine-tuning setup or a direct result of heterogeneity of texts in the mfc itself. the latter point should be investigated in the future, as the mfc data sampling decisions translate directly or indirectly into the approaches' performance. as both approaches present mixed performance, nuanced by language context and specific frames, we cannot conclude unequivocally the most accurate approach to be one method or the other. further seeing low inter-method agreement scores and the range of disagreement across languages and frames, we conclude that both approaches are highly nuanced and context-sensitive, even when based on the same pre-training on mfc. thus, neither of the prominent multilingual methods can guarantee performance in a new context, especially in low-resource languages. applied to our multilingual disinformation pairs, the joint decision of both methods produced various salient frames depending on the languages of the article, as we expected in the hypothesis. our findings confirm that in disinformation campaigns, articles presenting the same event or topic focus on different aspects of the issue, depending on which audience the campaign targets. we confirm this hypothesis for four languages in the dataset and a subset of regions that are targeted with articles in the russian language. we recognize that, while the timespan for which we collected the disinformation articles (2015-2023) provides invaluable insights into the russia-backed disinformation campaigns, it does not allow us to generalize into an analysis of the best methods for frame absence/presence at a sentence level. a more task-focused approach, that considers aspect and the most recent studies in frame presence/absence methods is a point of future research.