Check News in One Click: NLP -Empowered Pro-Kremlin Propaganda Detection (2024)

Abstract: Many European citizens become targets of the Kremlin propaganda campaigns, aiming to minimise public support for Ukraine, foster a climate of mistrust and disunity, and shape elections (Meister, 2022). To address this challenge, we developed “Check News in 1 Click”, the first NLP-empowered pro-Kremlin propaganda detection application available in 7 languages, which provides the lay user with feedback on their news, and explains manipulative linguistic features and keywords. We conducted a user study, analysed user entries and models’ behaviour paired with questionnaire answers, and investigated the advantages and disadvantages of the proposed interpretative solution.

## introduction evidence that we are living through a global crisis of trust in news is substantial, which inspired many a debate concerning the measures needed to rebuild it @xcite ). an increasing number of people are getting their news online, particularly the younger generation, while many have started avoiding the news, first those concerning the covid-19 pandemic and now those about the russian war in ukraine, majorly due to low credibility and negativity. @xcite . at the same time, digital platforms are viewed more sceptically, than traditional news, especially political ones as they are believed to be agenda-driven and contain propaganda @xcite @xcite . state-sponsored pro-kremlin propaganda became a major issue, as reports claim that only a small per cent of russian bots are being uncovered and detected @xcite . @xcite showed that twitter's (now x's) activity supporting russia generated nearly 1 million likes, about 14.4 million followers and a substantial proportion of pro-russian messages that went viral. to address this issue, we created an accessible online user interface to check news in terms of pro-kremlin propaganda, general manipulation and non-neutrality in 7 languages. it receives users' news and offers the model's verdict, its probability, as well as an explanation of manipulative keywords, linguistic strategies and indicators, shown to be associated with pro-kremlin news. in addition to the models from our previous study @xcite , we trained new ones for italian and german languages, exploring the usefulness of the data-augmentation strategy through translation, as well as multi-language versus language-specific pre-trained transformer models for this task. here, we present our system architecture and the user study we conducted, quantifying user satisfaction and desirable features and analysing user entries. ## results do you think the output is accurate? would you continue using this app? did you like the keywords explanation? did you like the linguistic explanation? would you recommend the app to someone? is the app overall usefull? in which form would you use it? figure 4 : the figure shows the results of the user study questionnaire. german, with one american and spanish user. ukrainian was named as the main language only 55% of the time though, while 20% searched news in english, 13% in german and 10% in russian. the full pull of users showed further language variety: almost 1/3 of all news entered into the app were actually in english, 1/3 in russian and a slightly smaller percentage in ukrainian. apart from 10 entries detected in german, other languages included french, spanish, slovenian and mandarin. as for the political views of the respondents, 41% self-identified as centrists, 24% as moderate right or left, while only 7% and 3.5% were left or right respectively (see figure 3 ). ## discussion and conclusion two major factors could explain the discrepancies between the user labels and the bert predictions: either the user was wrong or the model, and here both tendencies seem to be present. as illustrated by figure 5 , we compared the distribution of the text lengths of the conventional news (which are rather large ∼407 tokens), telegram news (which are rather short ∼32 tokens) in the training set and the news offered by the user (∼205). we could see that the latter distribution with all quartiles falls perfectly in between the 2 training set constituents. generally, the news can be even larger than the ones in our training set. for instance, the average article length of the new york times is 622 words and 516 for the washington post (menendez-alarcon, 2012). a brief qualitative analysis shows that while many inputs are indeed news, they are also majorly reddit comments, tweets, and user-generated words and sentences. we implemented the opportunity for the user to provide us with the link and not only copy-paste a text, which then we scrape using newspaper library 12 . some inserted a link to elon musk's tweets, and while x cannot be scraped. on very long entries, the model did not once predict 'propaganda' and coincided in this prediction with the user. it had at least 15% better matching with user labels on very short samples, similar to telegram posts in length, proving that length can indeed be a reason for some miss-classifications, when the user was correct. however, the length is only the surface description of the underlying genre missing from the training material: the users are not as interested in conventional news checking, as in flagging and quick discovery of bots and malicious actors in social media comments and tweets. a high number of ukrainian participants and a high number of certain responses concerning the tool's accuracy also showed that users predominantly were sure of their ability to recognize propaganda, but were interested in ways of quickly eliminating it from the informational eco-sphere. the indicators and keywords provided an important addition to the main model's verdict. not only did the constraints we introduced on the main model help mitigate strong language-related biases, but they also appeared to be more reliable, as they do not mismatch as often with user annotations. only in 16% of cases where there were more pro-russian propaganda features found and 8%, where no pro-western features were reported at all, would the user consider it a 'no propaganda' sample. with the user label being 'russian propaganda', there was only 12% with more pro-western than pro-kremlin indicators identified, and 7% where no pro-kremlin associated features were offered to the user. the strong performance of the indicators may have had a positive influence on the overall user evaluation of feedback's accuracy. users also often underestimate their knowledge of propaganda or are not very attentive when providing the label. while we received a lot of negative labels, the linguistic features indicate that most of the news pieces are not neutral. 37% of the news which was strongly not neutral were attributed to the 'no propaganda' label by the users. only 6% of truly neutral entries were rightfully annotated as such, and 4% of them were called propaganda. overall, the results of the user survey, however limited in number, are positive. both accuracy, recommendation, and interest in continuing to use the app are majorly high and both keywords and linguistic explanations were appreciated. in the free form, where we asked the users what they would like to change, it was even suggested to put more stress on the explanations and take away the overall verdict, showing the percentage of propaganda present. apart from minor front-end suggestions, such as more visual support and instructions, some users were indicating that there was news with a pro-western stance which were citing the president of russia, which contained propaganda, and such cases may have to be dealt with separately. for the same reasons, the field of fact-checking is moving from the direct text-to-label classification towards more fine-grained and multi-featured infosphere-based prediction @xcite . the need to introduce many constraints for the main model through other models in our study is also a reflection of this trend. including the layer user and human moderators in the research should become standard practice, as it helps better understand the needs of the community and tailor future solutions accordingly.