Wartime Media Monitor ( W ar MM -2022): A Study of Information Manipulation on R ussian Social Media during the R ussia- U kraine War (2023)

Abstract: This study relies on natural language processing to explore the nature of online communication in Russia during the war on Ukraine in 2022. The analysis of a large corpus of publications in traditional media and on social media identifies massive state interventions aimed at manipulating public opinion. The study relies on expertise in media studies and political science to trace the major themes and strategies of the propagandist narratives on three major Russian social media platforms over several months as well as their perception by the users. Distributions of several keyworded pro-war and anti-war topics are examined to reveal the cross-platform specificity of social media audiences. We release WarMM-2022, a 1.7M posts corpus. This corpus includes publications related to the Russia-Ukraine war, which appeared in Russian mass media and on social networks between February and September 2022. The corpus can be useful for the development of NLP approaches to propaganda detection and subsequent studies of propaganda campaigns in social sciences in addition to traditional methods, such as content analysis, focus groups, surveys, and experiments.

## introduction contemporary autocracies rely on media manipulation more than violent dictatorships of the past @xcite . as citizens might recognise manipulative intent @xcite , authoritarian governments attempt to veneer the propaganda messages via state-sponsored social networks. these online "astroturfing" campaigns @xcite appear as a genuine grassroots support for the regime and artificially inflate the visibility of pro-regime messages. in this paper, we document the presence of such a campaign in russia in 2022 and explore its key characteristics, using a large corpus of online messages from russian social media about the russian-ukrainian war @xcite . our data-driven approach can provide a more realistic picture of audience response to political information in the context of war than traditional methods of communication research, such as surveys. an important outcome of this media monitoring project is a corpus of online publications on the russian-ukrainian war which appeared the websites of russian newspapers and tv channels between february and september 2022 and on a number of social media platforms between july and september 2022. the corpus includes temporal, spatial, and some socio-demographic metadata, which can be used to develop nlp approaches to the detection of various forms of propaganda. to the best of our knowledge, there is only one published dataset, @xcite , which is specifically designed to capture media coverage of, and public reaction to, content related to the russia-ukraine war. voynaslov includes posts from a limited pre-defined number of news outlets (42 in total) published on either vkontakte or twitter. the data from these platforms have been sampled following dissimilar approaches: the twitter subset includes posts with war-related hashtags while there is no such filter for vkontakte. in terms of the amount of textual data, voynaslov includes 597k documents published on vkontakte and 219k on twitter. our dataset has a much broader coverage with regard to the sources of information and includes only the publications about the war, although our time frame is more limited. our dataset presents a realistic snapshot of the online information environment experienced by russian internet users in real-time during the war, and we hope that this resource can be useful not only for the nlp community but also for communication scholars and political scientists. ## corpus description the warmm-2022 corpus represents online political discourse produced in russia for and by do-mestic audiences. the corpus is composed of two parts: a subcorpus of publications by traditional mass media (press and tv) and public posts from social networks. the full list of sources includes 415 websites of media outlets, 25 websites of tv channels, and 85 social media platforms. the distribution of publications is very uneven across the sources of each type. most active media websites are by gazeta.ru, ura.news, ren.tv, vz.ru, russian.rt.com, iz.ru. the content of tv programmes related the russian-ukrainian war is captured by the respective transcripts published by tv channels on their websites. in our collection, the transcripts most often come from channel one, ren tv, channel 5, and russia 24. importantly, warmm-2022 includes the regional affiliations of information sources allowing one to trace the specifics of ukraine-related news coverage across russia. by the number of collected documents, the most represented social media platforms are vkontakte, odnoklassniki, telegram, twitter, facebook, live-journal, youtube (in decreasing order). the full list of mass-media, tv and social-media sources is released with the corpus. the textual data and associated metadata, including public reactions, such as the number of views, likes, re-posts and comments, were obtained with the technical support of scan interfax and brand analytics media monitoring systems. the parameters of data collection were configured to meet the requirements of the current project. media sources were limited by their availability to russian audiences, i.e. the crawled webpages were directly accessible in russia at the time of collection. social media sources were limited to posts from accounts that were registered in russia and published in russian, where possible. we aim to produce a realistic snapshot of the online information environment regarding the war and during the war. the data collection began in july 2022 in a monitoring mode. we were aggregating publications that appeared on mass media websites and on social media daily until the end of september 2022. a separate subcorpus of publications on mass media and tv websites for the preceding months (february to june) was collected in a onetime retrospective crawling effort in mid-july 2022. the corpus was built using a list of eight general context keywords to filter in publications related to the war in ukraine. these eight terms were war, special operation, military operation, svo (special military operation), special operation, military operation, denazification, and demilitarization (in russian). this list was developed as a result of iterative filter-setting experiments and manual analysis of daily crawls in the first two weeks preceding the start of the data collection. section 5 reports a cross-platform study focusing on the three most popular social media platforms in russia (at the backdrop of traditional media content): odnoklassniki (ok), telegram (tg) and vkontakte (vk). ## methodology the analysis of news topics includes statistical and unsupervised methods. the findings are interpreted in the context of external unfolding events. in this project, we constructed expert-curated lists of topical keywords, and their normalised frequencies were used to compare messaging across social media platforms (ok, vk, tg) in a time series fashion. general frequency analysis setup. we focused on frequency of individual keywords and aggregated frequencies of pre-defined terms that marked a particular topic. in total, we explored 20 thematic aspects of the publications and extracted the frequencies of over 250 words and phrases. the full list of search items in their nonlemmatised version in russian for each topic is available in the corpus documentation. it is designed to include topics that are typical for prowar and anti-war discourses as well as shared between the two. appendix a lists the topics and subtopics. for example, we traced "dehumanisation", the topic marked by the use of derogatory names for the ukrainians (e.g. ukrop) and numerous derivatives with ukro-and nazi-prefixes (e.g. ukronazist, ukrofascist, banderovetz, nazbat) as well as loaded ideological terms used to describe ukraine (e.g. kyiv regime, sneaky, guileful, hypocrite). the frequency of each item was based on a lemmatised version of the corpus to account for possible grammatical forms, which is important for morphologically-rich languages like russian. the raw texts went through minimum preprocessing before lemmatisation, including symbol unification, discarding .png/.jpg and url to reduce noise. the lemmatised version of the corpus was obtained from morpho-syntactic annotation produced using udpipe (v1, @xcite , a parser within the universal dependencies framework. all frequencies were normalised to the size of the respective subcorpora within a given time series and subcorpus, with the normalisation base of 100,000 words. this made possible the comparison of frequencies across subcorpora of various sizes directly, including using them in graphs based on the same scale. time series. as we were interested in fluctuations of topical content, we constructed time series using three-day intervals as our default setting, i.e. most results in this study reflect the frequencies of search items in the documents published within successive 3-day periods. whenever we wanted to explore a specific timespan in more detail or have a more aerial perspective, we analysed daily or monthly frequencies, respectively. unique publications vs repetitive content. taking into account the anticipated repetitiveness of publications, we compared the frequencies of selected keywords before and after deleting duplicate posts. duplicate posts were identified by matching the first 20 words in the raw text. the ratio of repeated texts (excluding the first occurrence) amounts to 47.98% on social media, with about 23.4% being exact unmodified copies of the original publication, often repeated many times. user attitude studies. several analytical approaches were employed in an attempt to reveal the users' attitudes to the topics discussed online. it is a challenging task as russia's information environment is heavily censored and populated with bots and trolls masquerading as real citizens; risks of legal prosecution make it difficult for users to state their positions publicly. in an attempt to overcome these limitations, we analysed (i) publications by users with different levels of publication activity and (ii) publications with the highest engagement scores. user groups by publication activity. this analysis was based on social media subcorpus only. the total number of unique users in this subcorpus is 263,665. users produced 1,544,918 posts in three months. in particular, we distinguish between professional users (more likely accounts of established information agencies) who publish more than 20 posts about ukraine a week (over 260 in three months) and the general public, i.e. users with one or fewer posts a week across 13 weeks. other users include an intermediate group of active users with over 13 but less than 260 posts per week. the parameters of each group and their contribution to the production of content on the analysed social networks can be found in activity patterns varied across the groups, with the professional users capable of generating surplus texts during some periods of time and being less active during others. these groups also demonstrated different patterns in the use of keywords from a range of analysed topics (see section 5). engagement scores. the posts on social media were analysed from the point of view of public reactions they generated. we calculated the engagement index as a sum of likes, re-posts, and comments to each post and built a subcorpus of the most popular posts, which included 5% of all posts sorted by the involvement score. this subcorpus included 85,317 posts (out of 1,706,343 in the entire social media corpus across three months). the engagement scores in this subcorpus ranged from 110,528 to 39, with a mean of 444.1. furthermore, to estimate the level of support for the pro-and anti-war messages and assess their visibility and influence in public discussion, we selected the top-1000 posts with the highest engagement score reactions and identified their sources. using external knowledge, we classified these 215 authors as pro-war, anti-war, or neutral. finally, we counted the number of posts by these authors and their overall engagement scores. ## results: cross-platform analysis this section applies the described methodology to explore users' participation in online discussions, their attitudes and reactions to media content. to capture the astroturfing campaign, we use a range of selected topics (pro-war and anti-war), which also help us to reveal political attitudes prevailing on the selected social media platforms. generally, we noticed that social media had deviated from the press and tv in the frequency patterns of pro-war topics. the usage of keywords was very volatile with several regular peaks. we identified these peaks as massive infusions of almost identical messages. in figure 1 these peaks are smoothed out by removing duplicate publications. in combination with observations for other topics, this can be a sign of information manipulation. as discussed above, the major russian social media platforms (odnolassniki, vkontakte, telegram) vary according to socio-demographic parameters. to disentangle the effect of repetitive publications observed in the entire social media subcorpus, shown in figure 1 , we looked at the frequencies of each topic by network. first, the frequencies of denazification and demilitarisation -the key terms justifying russian invasion -on tg and vk are both steady and low in comparison to abnormal spiky patterns registered in ok publications (the extreme frequencies ranging between 45 and 60 per 100k words vs over 140 on ok). note that demilitarisation, which disappeared from state-controlled media accounts towards the end of summer 2022, fell into disuse on ok, too. removing identical posts levels off the spikes in the use of justifications on ok. with duplicates removed, the three platforms demonstrate similar frequencies. the same pattern across platforms is observed for dehumanising language. there is a significant amount of anti-ukrainian derogatory terms on all three platforms. markers of this language are more frequent on ok, where every month there are significant spikes that exceed the volume of vk and tg by the factors from two (late july and late august) to four (early september), and six (early august). the patterns on vk and tg are stable and do not suggest any artificial inflation. nevertheless, all social media platforms remain plagued with hate speech toward ukrainians. to double-check that the observed peaks are artificial, aggregated frequencies based on the entire corpus after removing duplicates were compared. the amount of dehumanising vocabulary decreased manifold. nonetheless, even without identical messages, ok remains the most pro-war platform followed by vk and tg. these observations suggest that the kremlin disproportionately targets ok with pro-war online astroturfing. the function of state-controlled trolls cannot be reduced to producing identical content. paid users are often instructed to improvise and can produce original messages different from each other. hence, identical messages alone do not represent the scale of online astroturfing accurately. however, as identical messages are unlikely to be attributed to anything else but artificial content, removing it can make us underestimate the scale of online astroturfing, not to overestimate it. in other words, identical messages are a conservative estimate of the scale of kremlin-related astroturfing. one might argue that the messages we identified as astroturf were viral and resonated with the online public. while we cannot exclude such a pos-sibility, the short life-span of these messages (they disappeared completely from communications in 1-2 days) and a poor quality of its content (we found nothing sensational or novel in terms of production for several manually selected entries) indicate that the traction with the general public was limited at best. to further investigate differences in ideological spin and the scale of online astroturfing, we focus on anti-war vocabulary. figure 2 (on the left) shows the aggregated frequencies of keywords typical for kremlin opponents, such as russian aggression, annexation, occupation of ukrainian territories, russian invasion, occupation of donbas/crimea, russian occupants, etc. the graphs reflect the fluctuations in the use of anti-war language across platforms from july to september. it shows that tg is the most anti-war platform among the three. despite the presence of many pro-kremlin channels, the independent media flourish, too. anti-war vocabulary is the least present on ok. removing duplicates (the right panel in figure 2 ) does not change the observed pattern significantly. if anything, the absence of repetitive content makes anti-war stance on vk more visible (notice the upward shift of the orange line in the right-hand panel). this might suggest that spamming the information space with duplicates makes sense as it creates additional noise and makes it more difficult for users to hear other voices. with identical messages removed, tg remains the most anti-war platform. patterns on vk and ok resemble each other implying that they are more similar in terms of ideological spin. additionally, we explored topics related to timespecific external events, such as the ukraine's advances on the battlefield in late august -early september (e.g. advances of ukrainian army, ukraine's military success, retreat of russian troops, successful counterattack of ukrainian forces, russian defeat, etc). despite censorship on official press and tv, the news about the failures of the russian military percolated to social media, especially tg. the frequencies capturing this topic were very low but genuine: removing duplicates did not affect the counts. we conclude that ok was disproportionately targeted by the regime. the presence of astroturfing can also be confirmed by looking at the phrases from the temniki -the guidelines issued by the kremlin to cover politically sensitive topics in the media. unlike the other two platforms, ok demonstrated a growing scale of occurrences for this vocabulary from early august onwards suggesting that this platform was the primary target. public perception. the analysis of publications by professional users, who produce over 20 publications a week about ukraine, and regular users, who might represent the general public, showed that the latter were less eager to portray ukrainians as the enemy. in figure 3 , the flat orange line represents relatively low and stable frequencies for dehumanising vocabulary (which are still very high in comparison with visible anti-war stance in other graphs). in a subcorpus built from the top 5% of publications based on engagement scores, anti-war topics (e.g. ukrainian military success, framing russian actions in ukraine as war, occupation or invasion) are more frequent, while pro-war rhetoric is noticeably less dense than in the entire corpus. out of 215 authors who produced the publications with the highest engagement score, 80 were classified as anti-war (479 out of 1000 most popular posts), 24 authors as neutral with (27 posts), 111 authors were classified as pro-war (494 posts). however, in the top 50 most popular posts, we found only two posts by pro-war authors. the first 38 posts by the level of engagement and 48 posts in total (out of 50) were written by anti-war authors. as a result, the level of involvement for anti-war posts is 1.5 times higher than pro-war posts (8.5 mln reactions vs 5.7 mln reactions). ## discussion and consolidation by tracing the frequencies of pro-war and antiwar topics across social media platforms over time, linking them to external events and checking for repetitive content, we were able to reveal signs of information manipulation aimed at shaping public opinion by controlling the agenda and framing the concepts to fit the current ideology. as we demonstrate, some of the major themes artificially promoted by the state include: (i) the ideas of denazification and demilitarisaion and (ii) dehu-figure 3 : anti-ukrainian hate speech is publications by most prolific accounts and regular users manization of ukrainians. based on this dataset, we also identify several other promoted ideas, such as (iii) the existential threat posed by nato and (iv) framing ukraine as controlled by the hostile 'collective west'. however, we do not include them in the analysis due to limited space. the strategies of public opinion manipulation can include the attempts to undermine trust in mass media, frame any source of information except state-controlled as spreading 'fakes', and appropriate opponents' vocabulary and diluting its meaning. in this analysis, we demonstrate one of these strategies: imitation of popular support for promoted ideas on social media. our findings point at possible mechanisms behind the kremlin's digital war propaganda. instead of attempting to reach war opponents or users without clear preferences, the regime's astroturf communication seems to flourish in a predominantly pro-war environment. in line with both classical research on media effects @xcite and contemporary research on the effects of propaganda in authoritarian russia @xcite , these findings suggest that the main strategy of the regime's astroturf online communication might be similar to the one of authoritarian propaganda: to reinforce beliefs of those who are already proregime rather than to win new supporters. our cross-platform analysis indicates that discussions on ok are largely influenced by astroturfing. tg remains a relatively free space devoid of official rhetoric, while vk users exhibit a mild tendency to re-produce official narratives about the war. aiming at revealing the level of support for pro-moted ideas and the effectiveness of the said strategies, our analyses based on user group activity and reactions on social media demonstrated that many of these narratives fall flat on domestic audiences. modest numbers of russians participating in public discussions show that a lot of communication online is one-way, with people withdrawing from the public space. the public reactions that are available in warmm-2022 demonstrate that the extent of support for the promoted ideas is rather limited. ## conclusion this study reports details of textual data collection and analysis in the interests of social sciences. we release warmm-2022, a corpus of public online communications collected from a large number of mass media websites and social media platforms, which was used to obtain the results reported in this paper. our analysis relies on expert-curated lists of words and phrases which are used to cross-examine topical content in posts from a wide range of russian mass media and most popular social media, published in july-september 2022. informed by the previous work on data-driven propaganda detection, we aimed to assess the scale and societal impact of media manipulation in wartime russia. in particular, we were interested in the distribution of, and support for, selected topics reflecting opposite viewpoints on the events in ukraine. we revealed that the distribution of topics in social media (unlike traditional media, including tv) was largely affected by state-controlled interventions that varied in scale across the three social media compared in this work. the patterned nature of these interventions and their alignment with the kremlin's intentions expressed in recommendations for the press suggest that these are signs of "networked authoritarianism", a system of measures to exert control over the internet. the study was focused on the pro-war and anti-war themes in social networks and revealed a considerable amount of "astroturfing" (imitation of public support online). our results support the idea that the kremlin employs a digital propaganda ecosystem including networks of state-controlled accounts -bots and paid influencers -across russia's main social media platforms. this ecosystem is engaged in an organised manner to shape public opinion on current or forthcoming events. the frequency patterns of topics related to the russia-ukraine war reveal the artificial nature of online communication on russian social media in july-september 2022 and help us to identify the key messages infused by the astroturfing campaigns, such as the existential threat posed by nato, the need for patriotic unity against the hostile west and dehumanisation of the ukrainians. although anti-war voices are largely silenced by censorship and the threat of persecution, these opinions are heard and get more public attention than any propagandist content. at the same time, the number of russians who get publicly involved in online participation as authors is ridiculously small. as users, russians have to navigate an increasingly volatile, noisy, and restrictive environment infused with highly repetitive pro-war content.