DRAGON at FIGNEWS 2024 Shared Task: a Dedicated RAG for O ctober 7th conflict News (2024)

Abstract: In this study, we present a novel approach to annotating bias and propaganda in social media data by leveraging topic modeling techniques. Utilizing the BERTopic tool, we performed topic modeling on the FIGNEWS Shared-task dataset, which initially comprised 13,500 samples. From this dataset, we identified 35 distinct topics and selected approximately 50 representative samples from each topic, resulting in a subset of 1,812 samples. These selected samples were meticulously annotated for bias and propaganda labels. Subsequently, we employed multiple methods like KNN, SVC, XGBoost, and RAG to develop a classifier capable of detecting bias and propaganda within social media content. Our approach demonstrates the efficacy of using topic modeling for efficient data subset selection and provides a robust foundation for improving the accuracy of bias and propaganda detection in large-scale social media datasets.

## introduction in response to the evolving landscape of media representation and discourse surrounding the gaza-israel 2023-2024 war, the fignews shared task @xcite has been initiated to delve into the intricate nuances of bias and double standards prevalent in news articles. this task aims to explore diverse perspectives, cultures, and languages, fostering a comprehensive understanding of these events through the lens of major news outlets across the globe. the overarching objective is to establish a shared corpus for comprehensive annotation across various layers, crafting annotation guidelines shaped by the diverse range of conflicting discourses around this sensitive topic. this endeavor seeks to highlight both challenges and commendable aspects within the data, fostering a collaborative community and nurturing the growth of the next generation of nlp researchers. our approach aligns with the goals of the fignews shared-task by utilizing topic modeling, specifically the bertopic @xcite tool, to select a well-generalized and suitable subset of social media data for annotation. from an initial dataset of 13,500 samples, we identified 35 distinct topics and selected approximately 50 samples from each topic, resulting in a final subset of 1,812 samples. these samples were annotated for bias and propaganda labels. we then used multiple methods, including rag, knn 2 @xcite , svc 3 @xcite , and xgboost @xcite , to create a classifier for these labels. our study demonstrates the efficacy of using topic modeling for efficient data subset selection and provides a robust foundation for improving the accuracy of bias and propaganda detection in large-scale social media datasets. ## annotation methodology and examples in this section, we describe the pipeline for selecting suitable samples for human annotation. after annotating this subset of data, we use it to train multiple models. these models, along with the rag technique, are then employed to annotate the rest of the dataset using a voting mechanism. the implementation details can be found on the github page 4 . ## team composition and training first, we define the bias and propaganda tags and share these definitions with the annotators. our team includes two human annotators and several artificial annotators trained on a subset of data chosen using bertopic, as explained in section 2.2. one of the artificial annotators employs the rag technique, integrating label definitions into the llm 5 model's prompts, which our two human annotators also use for their annotations. however, other classic classification models do not utilize these label definitions during their training or inference processes. ## task participation and results the team's performance was bolstered by employing a sophisticated classification approach combining embedding and classifier models, experimenting with knn, svc, and xgboost, with knn emerging as the superior classifier. the innovative use of retrieval rag further enhanced this approach, involving dynamic prompts tailored for each news article. by retrieving and including examples and definitions based on the most similar annotated documents, focusing on unique labels among these documents, the team refined their prompts, leading to improved classification outcomes. experiments showed that using 5 similar documents and including one example per unique label in the prompts yielded satisfactory results. this methodology was facilitated by the gpt-3.5turbo model, which efficiently generated text and classified news articles, underscoring the advantage of integrating examples and definitions in promptbased classification compared to related works relying on static or less dynamic prompting techniques. the results of propaganda and bias are shown in the tables 5 and 6. in ## discussion the team's findings suggest that increasing the size of the manually annotated subset used to train artificial annotators could enhance performance, indicating that a larger, more comprehensive training dataset can improve the accuracy and reliability of automated classification systems. however, the team also observed that annotator bias and conflicts arising from differing cultural and personal beliefs influenced the labeling process(our annotators are from iran and the usa), reflecting a significant challenge in creating unbiased datasets. this highlights the importance of developing methods to mitigate annotator bias and ensure consistency across diverse teams. these insights contribute to the field by emphasizing the need for larger and more diverse training datasets and robust strategies for managing annotator bias, thereby advancing the development of more accurate and fair automated annotation systems in the context of detecting propaganda and bias. ## conclusion the key insights from this paper underscore the critical need to address the issue of unbalanced subdata, as the low frequency of some labels adversely affects the performance of artificial annotators. ad-ditionally, the reliance on only 2 human annotators from different countries has introduced biases and inconsistencies in the annotations, highlighting the necessity of expanding the annotator pool to improve the diversity and reliability of the curation process. moreover, the current lack of thorough curation, with only 2 people annotating the sub-data, points to the need for a more systematic and collaborative approach. these findings emphasize that incorporating thorough examples and a comprehensive understanding of related work is essential for enhancing the accuracy and efficacy of annotation tasks, ultimately contributing to the development of more robust artificial annotators.