Challenges and Opportunities in Information Manipulation Detection: An Examination of Wartime R ussian Media (2022)

Abstract: NLP research on public opinion manipulation campaigns has primarily focused on detecting overt strategies such as fake news and disinformation. However, information manipulation in the ongoing Russia-Ukraine war exemplifies how governments and media also employ more nuanced strategies. We release a new dataset, VoynaSlov, containing 38M+ posts from Russian media outlets on Twitter and VKontakte, as well as public activity and responses, immediately preceding and during the 2022 Russia-Ukraine war. We apply standard and recently-developed NLP models on VoynaSlov to examine agenda setting, framing, and priming, several strategies underlying information manipulation, and reveal variation across media outlet control, social media platform, and time. Our examination of these media effects and extensive discussion of current approaches’ limitations encourage further development of NLP models for understanding information manipulation in emerging crises, as well as other real-world and interdisciplinary tasks.

## introduction on february 24, 2022, russia began an open military invasion of ukraine. at the time of writing, this ongoing conflict has killed thousands of people and displaced millions. 1 the conflict has also manifested in ongoing information warfare, as russian, ukrainian, and ally forces attempt to shape online narratives of the war. 2 even before these events, russian-backed entities have aimed to influence opinions outside @xcite and inside russia @xcite @xcite . more broadly, researchers consider the manipulation of public opinion over social media as "a critical threat to democracy" and have identified computational propaganda in over 80 countries @xcite . because these campaigns often rely on textbased news and social media content, nlp can be a valuable tool in combating them. in this work, we examine the usability of nlp approaches for combating information manipulation campaigns through the release of an in-progress data set, voy-naslov, focused on the 2022 russia-ukraine war ( §2). this dataset itself addresses one challenge for research in this space: the need for real-world data. in contrast to contemporaneous twitter data sets @xcite , our corpus is explicitly designed to capture russiangovernment-backed information manipulation; we collect posts by state-affiliated and independent russian media outlets and reactions to them on twitter, which is more dominant in europe and the u.s., and vkontakte (vk), one of the most widelyused social media platforms in russia @xcite . 3 unlike approaches that crowd-source data to study a specific task @xcite , we derive research tasks directly from real-world data. the dominant nlp research paradigm in information campaigns has focused on automated factchecking or propaganda and fake news detection @xcite @xcite @xcite . however, this work typically involves supervised approaches and pre-annotated data, which are not available in emerging situations, and only captures one extreme form of media manipulation. instead, we draw on a common paradigm of information manipulation from communications research and examine signs of agenda setting, framing, and priming in voynaslov ( §3). for each media effect, we investigate the utility of the most common and recently developed nlp approaches. our analysis first reveals evidence of manipulation tactics in our data, showing that voy-naslov presents an avenue for studying them, and second, exposes open challenges in current nlp approaches towards uncovering, analyzing, and mitigating information manipulation campaigns. we conclude by highlighting broader limitations of extant nlp approaches, discuss why model performance advancements have not yet translated to deployable technology in crises, and propose directions for future work to close this gap ( §4). our contributions, visualized in figure 1 , are a new data set of russian media activity, which we use to analyze media effects, and an in-depth discussion of challenges and opportunities in nlp research on information manipulation campaigns. we hope to facilitate research on information warfare and ultimately enable reduction and prevention of disinformation and opinion manipulation. ## voynaslov voynaslov contains posts from russian news outlets on vk and twitter, which primarily feature breaking news or summaries of original articles. here, we describe data collection and statistics. list of news outlets we identified russian media outlets and their twitter and vk handles start- voynaslov-twitter we similarly collect tweets and metadata such as like and retweet count from russian media accounts. we capture public reaction with the twitter search api and iteratively craft search terms. starting from a small seed list, we collect an initial set of tweets. we augment our seed list with frequent terms from this initial set judged to be relevant to the war. after several rounds, our final list contains 264 terms and hashtags (appendix a). since the twitter api only supports search with a 7-day limit, we collect 17m public tweets from 24 feb -31 may 2022. data statistics voynaslov suggests that state-affiliated media dominates vk, but independent media dominates twitter. on average, state-affiliated vk accounts have 26k posts, over twice as much as independent media. this pattern is reversed on twitter, where independent accounts are slightly more active. vk's publicly-available data includes view counts, presenting a unique opportunity to study exposure @xcite . not only are state-affiliated outlets more active on vk, but their content reaches a larger audience (18k vs 10k views per post). popularity cues, e.g., likes, comments, and retweets, can serve as indicators of the success of the media's opinion manipulation strategies (see §3.3). independent media posts receive more engagement on twitter, but state-affiliated posts receive more engagement than independent posts on vk. however, independent posts on vk still have engagement rates if we account for their smaller audiences. as discussed in §1, vk is more widely used in russia and enables analyzing internal russian information manipulation campaigns as well as reactions of people likely to be in russia. our data enables comparisons between vk and twitter and could reveal differences in strategies used by state-affiliated russian media when targeting domestic and international audiences. ## facilitation of nlp research on media opinion manipulation we demonstrate how voynaslov can facilitate research on media opinion manipulation by focusing military ukraine 0.00 0.01 0.02 0.03 0.04 avg. topic proportion state-affiliated independent figure 2 : war-related topic proportions for stateaffiliated and independent media outlets, as learned by a 30-topic ctm. we display the two topics we identified as most war-related: "military" (top words: округа:area, авиации:aviation, военного:military, военно:military, флота:navy, учения:military exercise, су:sukhoi su, военнослужащие:military personnel, противника:enemy, сил:forces), and "ukraine" (top words: днр:dpr, лнр:lpr, мариуполя:mariupol, мирных:peaceful, жителеи:residents, украинские:ukrainian, народнои:folk, новости:news, донбасса:donbass, мариуполе:mariupol). we report all topics in figure 10 . on three media effects: agenda-setting ( §3.1), framing ( §3.2), and priming ( §3.3), though we note disputes over the distinctness and convergence of these concepts @xcite @xcite . for each media effect, we first provide background and review existing nlp approaches. we then apply current state-of-the-art models from the most dominant nlp paradigms on voynaslov, presenting evidence that this data can support examinations of these effects. we conclude each subsection with open challenges exposed by our analyses. ## discussion there are numerous opportunities for nlp research to have positive impacts in identifying and mitigating information manipulation campaigns. as a first step, we release voynaslov, focused on russian media activity on social media before and during the 2022 russia-ukraine war. grounded in political communication research on media effects, we apply both traditional and state-of-the-art nlp techniques to analyze the language underpinning information manipulation. indeed, we uncover variations in agenda setting and framing strategies across time (pre-war or wartime), social media platforms (vkontakte or twitter), and media control (state-affiliated or independent). we encourage future work to continue to explore the plethora of social science theories and nlp techniques to analyze the data in voynaslov. furthermore, we hope that voynaslov will aid future efforts in using nlp to address not only the 2022 russia-ukraine war, but emerging crisis situations more broadly. through both our data collection and analysis, we learn that vast improvements in model performance on core nlp tasks have not yet translated into deployable technology capable of addressing information manipulation campaigns. our work suggests several reasons for this discrepancy, and we foreground such limitations in order to set forth concrete directions for future work. first, much prior work has focused on developing supervised models to detect fake news and propaganda in iso-lated media texts, including the establishment of shared tasks and standardized data sets @xcite @xcite . while there are settings where these approaches can be useful, they typically require carefully-labeled data that are unavailable in emerging settings, and we did not identify them as applicable in our analysis of voynaslov. similarly, nlp advancements in model-pretraining @xcite are difficult to deploy over emerging data: most pre-training data is in english, and it is difficult to disentangle patterns in the target data from ones learned during pre-training @xcite . preannotated data and pre-trained models can be immensely valuable for analyzing the past but have limited utility in understanding ongoing events. second, even technological approaches that aim to target ongoing events, just as systems for aiding human fact-checkers @xcite typically consider only the most extreme and overt form of opinion manipulation: disinformation and fake news. however, our results build upon existing communications studies, demonstrating that media manipulation often constitutes more subtle strategies, such as selectively covering (or avoiding) issues ( §3.1) and changing minor word choices to influence audiences ( §3.2). little nlp work has examined agenda setting and priming at all. while substantial work has focused on framing ( §3.2), it disproportionally focuses on u.s. politics, with few applications to non-english languages, other social contexts, or information warfare. third, nlp research on media manipulation has primarily examined isolated news texts without additional context, neglecting the larger hybrid media ecosystem comprised of intricate interactions between journalists, media organizations, political actors, social media platforms, and civilians @xcite . voynaslov attempts to facilitate research in this area with content from a specific context and includes multiple outlets and platforms. nevertheless, it remains challenging to truly comprehend the media's motivations for how they present the news and their desired effects on public opinion, which then enables specific and nuanced analyses of manipulation strategies. although we emphasize limitations in existing nlp approaches, we conclude by asserting that nlp has a unique opportunity to uncover information manipulation campaigns and contribute to social science research. media effects have been rigorously studied within social science, but common approaches, including focused analyses of small sets of articles and human experiments with constructed stimuli in highly-contrived settings, are insufficient for assessing the scale and societal impact of media manipulation. computational methods can be representative of the full media environment and capture more realistic audience responses to news content shared on social media. this work aims to shift the paradigm for research on automated opinion manipulation to encompass broader tactics, have grounding in social science theory, and incorporate emerging context. we believe these expansions will enable nlp to have positive impact during, rather than after, ongoing crisis situations. we hope that our release of voynaslov, our analysis of media effects, and our discussion of open nlp challenges facilitate the detection and ultimately the prevention of information manipulation. ## limitations our work includes the release of a new data set, data analysis using state-of-the-art nlp models, and a discussion of open challenges in this space. the comprehensiveness of our data is limited by decisions about the data collection process, including which news outlets to focus on and which keywords and hashtags to use when collecting tweets. while we take steps to broaden the coverage of our data, such as multiple rounds of identifying news outlets and relevant terms, collection biases could reduce the reliability of any analyses conducting with this data. our data set cannot be considered to capture all relevant content from this time period. throughout §3 we focus on highlighting the limitations of current nlp approaches in this setting, and we refer to this section for details. we acknowledge that our discussing of challenges and limitations is itself limited by the discussion framework. structuring our analysis using different social science theories could lead to different results. we additionally focus on entirely text analysis and do not discuss limitations related to other types of media, such as images or video (beskow and carley, 2019). finally, our discussion of limitations is based on our choice of nlp methodology to use over our data. while we attempt to select state-of-the-art models for the most dominant nlp research paradigms for each media effect, other methods and paradigms that reduce our discussed limitations may exist. ## ethical considerations given the ongoing war and the limitations on free speech in russia, including the recently passed law that punishes spreading "false information" with up to 15 years in prison, it is possible that our data set contains content that could have physical and legal ramifications for individual users or media outlets. even in our initial data collection, some vk data was flagged as deleted by moderators. we take several steps to mitigate the impact our work may have on the risk to individuals or media outlets. all of the data collected in this work is publicly available and we do not make any attempt to uncover nonpublic data. while we do include posts by general users on twitter, we primarily focus on posts from media outlets and replies to them, where we can assume a lower expectation of privacy. in order to preserve users' ability to delete content, we do not release any raw text data and instead only release post ids, which other researchers can use to recollect raw data, if it has not been removed. we further note that all data was collected in accordance with social media platforms' terms of service. throughout this work, we also avoid using specific examples from the data or referring to individual users. we encourage future work on this data to exercise similar caution, and we do not condone any research that attempts to deanonymize or profile users or identify narratives that could result in individuals being targeted. we refer to @xcite and @xcite for a more in-depth discussion of ethical considerations of research using social media data. we also primarily focus on news content posted by russian media outlets, which we suggest provides avenues for studying disinformation, because of prior work on russian information manipulation strategies and because russia is the aggressor in this conflict. however, we note that independent reports have also found evidence of misinformation perpetuating pro-ukranian narratives. 14 more generally, the authors of this work are situated in the u.s. and our assumptions in this work (e.g. that russia is the aggressor) reflect this context, but we note that this viewpoint is not universal.