MIPD : Exploring Manipulation and Intention In a Novel Corpus of P olish Disinformation (2024)

Abstract: This study presents a novel corpus of 15,356 Polish web articles, including articles identified as containing disinformation. Our dataset enables a multifaceted understanding of disinformation. We present a distinctive multilayered methodology for annotating disinformation in texts. What sets our corpus apart is its focus on uncovering hidden intent and manipulation in disinformative content. A team of experts annotated each article with multiple labels indicating both disinformation creators’ intents and the manipulation techniques employed. Additionally, we set new baselines for binary disinformation detection and two multiclass multilabel classification tasks: manipulation techniques and intention types classification.

## introduction mitigating the spread of disinformation on the web has become an important social challenge. numerous significant events, including the covid-19 pandemic and the russo-ukrainian conflict, highlight disinformation's negative impact on individuals and society @xcite . the high-level group of experts (hleg) set up by the european commission defines disinformation as "false, inaccurate, or misleading information designed, presented, and promoted to intentionally cause public harm or for profit" (de cock buning, 2018). there are two significant aspects in this definition: intention types ("the why") and misleading manipulations ("the how"). however, to our knowledge, no study in the literature examines intention types and manipulation in disinformation together, possibly due to a lack of quality annotated data. therefore, we share with the research community the manipulation and intention in polish corpus of disinformative web articles: the mipd dataset. the mipd dataset sheds light on the authors' intention and manipulation techniques in disinformation. our highquality open corpus, annotated by five professional fact-checkers and debunkers, will provide a multifaceted understanding of disinformation. initially, we focus on polish, the 5th most spoken language in the european union @xcite . we chose this language because it is the largest of the v4 countries (slovak republic, czech republic, poland, and hungary), which have been particularly vulnerable to disinformation in recent years due to the russo-ukrainian conflict @xcite . here are the main contributions of this work: • we introduce the largest dataset foot_0 of online articles in polish, annotated with intents, manipulation techniques and whether they are disinformative. to the best of our knowledge, this is the first corpus of its kind. ## annotation methodology and guidelines in order to ensure high-quality annotations, our annotation guidelines and methodology were created by fact-checking and debunking experts. we employed five polish native-speaker experts with at least three years of fact-checking and debunking experience (on a one-year competitive salary). all debunking experts working on the project were previously employed in debunking organizations with the accreditation of the international fact-checking network 2 . the same experts used the methodology to annotate the articles in mipd. the methodology described here is also an educational tool for students who wish to learn how to detect disinformation. the methodology is divided into five main steps: 1. determining the article's thematic category. 2. evaluating the credibility of the article's source and author (if known). 3. determining the article's main class: credible, disinformation, misinformation or hard-to-say. 4. for disinformation, evaluating of manipulation. 5. for disinformation, evaluating of intention types and narratives. experts could return to previous steps and typically re-evaluate the main class after a detailed investigation of an article suspected to contain disinformation. ## mipd corpus mipd is a novel dataset that includes 15,356 web articles in polish. in addition to the article content, the data we publish contains four annotations: (i) whether an article is disinformative or credible; ## experiments our experiments aim to test the data quality further and provide a baseline upon which future works can build. ## gpt and disinformation classification in addition, we decided to explore the efficacy of generative models in disinformation classification. specifically, we experimented with two openai generative models that are accessible via their apis: gpt-3.5 and gpt-4 8 . our objective was to assess the ability of these models to classify articles as containing disinformation using a zero-shot classification approach. we employed two zero-shot 8 details on the models used: we utilized a snapshot of gpt-4 from june 13th, 2023, named gpt-4-0613, and gpt-3.5-turbo-instruct, which has capabilities similar to gpt-3 era models. the last access to these models was on 28th may 2024. first, we randomly drew a sample of 10% of the articles from our entire dataset. then, we used a prompt to classify articles with generative models (our prompts are available in appendix f). we repeated these steps for two approaches: (i) zeroshot classification with a disinformation definition included in the prompt; (ii) and zero-shot classification without a definition. finally, we calculated various evaluation metrics. our investigation reveals that polish bertbased models fine-tuned on the mipd dataset significantly outperform chosen generative models: gpt-4 and gpt-3.5. the gpt models, when applied in a zero-shot approach without a definition of disinformation, achieved weighted f 1 scores of 0.84 for gpt-4 and 0.61 for gpt-3.5, respectively. in the zero-shot approach with the given definition of disinformation, both the gpt-4 and gpt-3.5 models improved their results. nevertheless, these results are inferior to any polish bert-based models. our findings highlight the effectiveness of herbert and polish roberta in handling polish disinformation, likely due to their specialized training and fine-tuning utilizing polish datasets. in contrast, the results of gpt models suggest that generative models may require domain-specific fine-tuning to reach the performance of language-specific bert variants below we discuss previous work related to each of the three classification tasks we present. ## conclusion and future work our study introduces a novel multifaceted polish mipd dataset, enabling a comprehensive approach to understand disinformation. the mipd dataset includes a rich set of annotations, such as article topics and disinformation vs. credible information annotation. beyond the binary annotation of disinformation, we explore manipulation techniques and malicious intention types utilized by creators of disinformation. in order to ensure high-quality annotations, our annotation guidelines and methodology were created by fact-checking and debunking experts. moreover, our dataset was annotated by experts with at least three years of experience in debunking organizations with the accreditation of the international fact-checking network. we publish our annotation guidelines and methodology, which are not language-specific and may be used in any other language. utilizing our dataset, we established new baselines for disinformation classification and two multilabel multiclass tasks: manipulation techniques and intention types classification. in our experiments, we used four different polish bert-based models. additionally, for disinformation classification, we also present a preliminary analysis utilizing two llms, namely gpt-4 and gpt-3.5. our analysis reveals that polish bert-based models fine-tuned on the mipd dataset significantly outperform chosen generative models. annotating articles by experts is costly and timeconsuming. employing a comprehensive annotation methodology makes this process even more expensive. accordingly, we plan to focus our future efforts on developing cost-effective methods using a semi-weakly supervised approach to detect misleading content. these methods could be invaluable for professional fact-checkers and debunkers and reduce manual efforts. moreover, we plan to collect and annotate a dataset containing more languages and cultural contexts.