HQP : A Human-Annotated Dataset for Detecting Online Propaganda (2024)

Abstract: Online propaganda poses a severe threat to the integrity of societies. However, existing datasets for detecting online propaganda have a key limitation: they were annotated using weak labels that can be noisy and even incorrect. To address this limitation, our work makes the following contributions: (1) We present HQP: a novel dataset (N=30000) for detecting online propaganda with high-quality labels. To the best of our knowledge, HQP is the first large-scale dataset for detecting online propaganda that was created through human annotation. (2) We show empirically that state-of-the-art language models fail in detecting online propaganda when trained with weak labels (AUC: 64.03). In contrast, state-of-the-art language models can accurately detect online propaganda when trained with our high-quality labels (AUC: 92.25), which is an improvement of 44%. (3) We show that prompt-based learning using a small sample of high-quality labels can still achieve a reasonable performance (AUC: 80.27) while significantly reducing the cost of labeling. (4) We extend HQP to HQP+ to test how well propaganda across different contexts can be detected. Crucially, our work highlights the importance of high-quality labels for sensitive NLP tasks such as propaganda detection.

## introduction propaganda is used to influence, persuade, or manipulate public opinions @xcite . nowadays, propaganda is widely shared on social media as a practice of modern warfare (e.g., in the ongoing russo-ukrainian war) and thus poses a significant threat to the integrity of societies @xcite . existing nlp works for propaganda detection (e.g., @xcite @xcite 2020; @xcite rely upon datasets that were exclusively annotated using weak labels and were, therefore, not validated by humans. because of this, labels can be noisy and even incorrect. we later provide empirical support for this claim and show that the overlap between weak labels and human annotations is only ∼41%. to fill this gap, we develop hqp: a novel dataset with high-quality labels for detecting online propaganda. hqp consists of @xmath0,000 posts in english from the russo-ukrainian war ( our main contributions are as follows: foot_0 1. we construct hqp, a novel dataset with highquality labels for online propaganda detection using human-annotated labels. detecting harmful content: prior literature in nlp has aimed to detect a broad spectrum of harmful content such as hate speech (e.g., @xcite @xcite , rumors (e.g., @xcite @xcite , and fake news (e.g., @xcite @xcite . further, claim detection has been studied, for example, in the context of the russo-ukrainian war @xcite . overall, literature for detecting harmful content makes widespread use of datasets that were created through human annotations (e.g., @xcite @xcite . detecting propaganda content: previous works for propaganda detection can be loosely grouped by the underlying content, namely (1) official news and (2) social media. few-shot learning in nlp: generally, constructing large-scale datasets with high-quality labels in nlp is costly. hence, there is a growing interest in few-shot learning. common methods typically leverage prompting, where the downstream task is reformulated to resemble the masked language modeling task the plm was trained on (e.g., @xcite @xcite @xcite . prompting has been highly successful in few-shot learning, e.g., for rumor detection @xcite or humor detection @xcite . however, to the best of our knowledge, no work has so far adapted few-shot learning to detect propaganda. ## dataset construction we construct a human-annotated dataset of english social media content with propaganda (hqp). for this, we construct a corpus of posts with russian propaganda from the 2022 invasion of ukraine. we collect posts from february 2021 until october 2022, i.e., our timeframe starts one year before the invasion due to the widespread opinion that the invasion was planned far in advance. 2 we intentionally choose the russo-ukrainian war due to its significance for global politics @xcite and the size of the propaganda campaign @xcite . our methodology for constructing hqp (and hqp+) follows best practices for human annotation @xcite . specifically, we follow a three-step process: (1) data collection, (2) sampling, and (3) human annotation. ## methods in our experiments, we follow state-of-the-art methods from the literature (see sec. 2) to ensure the comparability of our results. to this end, we use a binary classification task (@xmath1, @xmath2. results are reported as the average performance over five separate runs. in each run, we divide the dataset into train (70%), val (10%), and test (20%) using a stratified shuffle split. ## discussion we introduce hqp: the first large-scale dataset for online propaganda detection with human annotations. our experiments have direct implications for the nlp community. ## limitations as with other works, ours is not free of limitation. first, there is no universal rule to identify propaganda. hence, the perception may vary across individuals. we address this by having our dataset annotated through multiple raters and showing raters a task description that includes a widely accepted definition of propaganda (see smith, 2022). in addition, we carefully selected our inclusion criteria for data collection to ensure a broad and diverse set of posts. second, we are further aware that plms may embed biases that are populated in downstream tasks. hence, we call for careful use when deploying our methods in practice. third, narratives that fall under the scope of propaganda may change over time. hence, we recommend that both the dataset construction and the plm fine-tuning is repeated regularly. to this end, we provide a cost-effective approach through few-shot learning. ## ethics statement our dataset will benefit research on improving social media integrity. the construction of our dataset follows best-practice for ethical research @xcite . the dataset construction and usage were approved as ethically unproblematic by the ethics commission of the faculty of mathematics, informatics and statistics at lmu munich (ethics approval number: ek-mis-2023-160). in particular, our dataset contains only publicly available information. the privacy policy of twitter/x warns users that their content can be viewed by the general public. further, we respect the privacy of users and only report aggregate results throughout our paper. although we believe the intended use of this work is largely positive, there exists potential for misuse (e.g., by propaganda campaigns to run adversarial attacks and develop techniques to avoid detection). to this end, we call for meaningful research by the nlp community to further improve social media integrity. finally, we encourage careful use of our dataset, as it contains potentially offensive language and manipulative content, which lies in the nature of the task. hqp (and hqp+) was annotated using prolific. the workers were paid 11.40 usd per hour, which is above the federal minimum wage.