S ahara Pioneers at FIGNEWS 2024 Shared Task: Data Annotation Guidelines for Propaganda Detection in News Items (2024)

Abstract: In today’s digital age, the spread of propaganda through news channels has become a pressing concern. To address this issue, the research community has organized a shared task on detecting propaganda in news posts. This paper aims to present the work carried out at the University of Tripoli for the development and implementation of data annotation guidelines by a team of five annotators. The guidelines were used to annotate 2600 news articles. Each article is labeled as “propaganda”, “Not propaganda”, “Not Applicable”, or “Not clear”. The shared task results put our efforts in the third position among 6 participating teams in the consistency track.

## introduction propaganda is a powerful tool often used to influence people's opinions and beliefs, especially during times of war and political contention. in the digital age, the spread of propaganda through online news and social media has become a major concern. identifying propaganda is challenging as differentiating propaganda from other types of persuasion techniques is a difficult task. this is where artificial intelligence (ai) can play a valuable role. the ai research community has developed numerous tools and techniques to help detect propaganda in news articles and other online content. a major trend in the development of aibased propaganda detection tools relies heavily on the use of advanced machine learning (ml) models where the availability of training data is crucial. the study of @xcite focused on binary-class propaganda detection at the news article level. their model evaluates the level of propaganda based on the presence of keywords, the style of writing, and legibility. they also experimented on tshp-17 and qprop corpora, where for the tshp-17 corpus, they binarized the labels: propaganda vs. nonpropaganda. @xcite focused on detecting various classes at a document-level where propaganda was one of four classes in their annotated data: satire, hoax, trusted, and propaganda. they adopted an analytical method to the language of news media in the context of political fact-checking and fake news detection and the articles were collected from the english gigaword corpus and from seven other unreliable news sources. they trained their model by using word n-gram representation with logistic regression and reported that the model performed well only on articles from sources that the system was trained on. a more fine-grained propaganda study was proposed by @xcite , who developed a corpus of news articles annotated with 18 propaganda methods which was used in two shared tasks: semeval-2020 @xcite and nlp4if-2020 @xcite . besides that, improved models such as the prta system @xcite , were proposed to address the limitations of transformers @xcite . the prta system was used to conduct a study of covid-19 misinformation and associated propaganda techniques in bulgaria @xcite and qatar @xcite . this paper presents our work on designing propaganda detection guidelines for the participation in fignews 2024 @xcite , a shared task on news media narratives and part of the second arabic natural language processing conference (arabicnlp 2024). this shared task is more akin to a research-focused datathon with a strong emphasis on the development of improved annotation guidelines for complex opinion data tasks. the rest of the paper is organized as follows: section 2 explains our methodology and annotation process. section 3 provides some examples of the annotation process and in section 4 we present the results as reported by the organizers of the shared task. in section 5 we discuss how the guidelines were used in the annotation process. section 6 gives some concluding remarks and future work. ## examples of annotated news items here we provide some examples of annotated news posts to showcase our annotation process. these examples highlight the identification of various propaganda techniques and provide an interpretation of the annotation results. ## results in the propaganda subtask, there were 51 annotators across 6 teams, and they annotated together a total of 41,300 news posts. for each subtask, there are four evaluation tracks: quantity, quality, consistency, and guidelines. as shown on ## conclusion this research paper presented our work on designing a set of annotation guidelines for propaganda detection in news items. these guidelines were designed through an iterative process involving thorough discussions, and empirical testing. the proposed guidelines detail how to identify and label propaganda techniques in text and define clear, unambiguous categories. furthermore, to address propaganda interpretation issues, illustrative examples were included into the guidelines. this method aids in guaranteeing that all annotators possess an identical comprehension of the criteria for annotation. the guidelines have shown that they significantly improve inter-annotator agreement, indicating annotation consistency. this improvement is essential for datasets for propaganda detection, machine learning model training, and evaluation. future work will refine these guidelines based on feedback from the research community. additionally, the team plans to incorporate these guidelines into automated annotation tools to optimize the process. in conclusion, we believe that our data annotation guidelines will help tp advance the systematic study of news media propaganda.