Analysing State-Backed Propaganda Websites: a New Dataset and Linguistic Study (2023)

Abstract: This paper analyses two hitherto unstudied sites sharing state-backed disinformation, Reliable Recent News (rrn.world) and WarOnFakes (waronfakes.com), which publish content in Arabic, Chinese, English, French, German, and Spanish. We describe our content acquisition methodology and perform cross-site unsupervised topic clustering on the resulting multilingual dataset. We also perform linguistic and temporal analysis of the web page translations and topics over time, and investigate articles with false publication dates. We make publicly available this new dataset of 14,053 articles, annotated with each language version, and additional metadata such as links and images. The main contribution of this paper for the NLP community is in the novel dataset which enables studies of disinformation networks, and the training of NLP tools for disinformation detection.

## introduction coordinated, state-backed disinformation operations have become an increasing problem in recent years, particularly surrounding the war in ukraine @xcite . in september 2022, a sophisticated network of doppelganger websites (impersonating genuine news sites from across europe) was discovered by eudisinfolab @xcite and later expanded on in a report from meta @xcite . among these was also a small number of conventional false news sites. the focus of this study is on two related disinformation sites in particular: reliable recent news 1 (rrn) and war on fakes 2 (wof). both sites are multilingual, publishing in arabic, chinese, english, french, german, and spanish, and rrn additionally in italian 3 . they have been promoted by russian government sources, including being shared by russian embassies @xcite 1 https://rrn.world , formerly called reliable russia news using rrussianews.com. hanley et al. (2022) included selected articles from waronfakes and nine other disinformation websites in an analysis of narratives spread on reddit. in contrast, our dataset includes all waron-fakes posts and extracts the full article content. propaganda is defined as content that intentionally influences opinion to advance its creators' goals @xcite . numerous propaganda datasets have previously been created, with both document-level @xcite and span-level @xcite technique annotations, using articles collected from multiple disinformation sites. at article-level, classifiers using combinations of multiple linguistic representations based on style and readability outperform content representation @xcite , whereas content-based transformer models such as bert have seen use at span-level @xcite . detectors are often evaluated on single datasets, prompting concerns on generalisation @xcite . we are not aware of any prior work including rrn, nor of any work which has released a com-plete dataset of a disinformation operation, including a detailed linguistic analysis. thus the contributions of this paper are: i) a new publicly available foot_1 dataset of content from two state-backed disinformation websites; ii) a linguistic, topic, and temporal analysis of their articles; and iii) our open-source toolkit for processing site data and extraction of translations foot_2 . ## future work there is much additional work which could be performed on this dataset. although we identify the subject of articles via topic clustering and n-grams, we do not attempt to identify stance towards it. more complex topic analysis, such as identifying commonly co-occuring topics, would also be possible. given the mixture of true and false posts on the sites, this dataset may be a useful resource for automated fact-checking, although this would require human annotation and ground-truth may be difficult to establish in the complex information environment of the war in ukraine. ## conclusion this paper presented an analysis of the russian disinformation sites recent reliable news and waronfakes, including an analysis of the articles' topics, publication times, and linguistic properties. we show that the sites cover a diverse range of topics, and that their linguistic properties differ from those of reputable media. we analysed the presence of cyrillic characters due to site operator errors, and their practice of backdating articles, showing that a significant proportion of translations are falsely dated. this new multilingual dataset will facilitate further research in disinformation analysis and promote repeatability.