Narrative Navigators at FIGNEWS 2024 Shared Task: New Frontiers in Bias and Propaganda Annotation Techniques (2024)

Abstract: This paper presents our team’s contribution to the FIGNEWS 2024 Shared Task, which involved annotating bias and propaganda in news coverage of the Israel-Palestine conflict. We developed comprehensive guidelines and employed a rigorous methodology to analyze 2,200 news posts from several official Facebook accounts of news websites in multiple languages. Our team, Narrative Navigators, achieved third place in both the Bias Guidelines and Bias Consistency tracks, demonstrating the effectiveness of our approach. We achieved an IAA Kappa score of 39.4 for bias annotation and 12.8 for propaganda detection. These findings and our performance underscore the need for enhanced media literacy and further research to counter the impact of biased and misleading information on public understanding of the conflict.

## introduction in the context of the 2023-2024 war on gaza, the concepts of "propaganda" and "bias" are crucial in media studies. bias refers to a disproportionate leaning towards or against certain ideas, entities, or nations, often leading to skewed representations @xcite . these narratives frequently integrate propagandistic elements to sway attitudes and opinions @xcite . media outlets employ various techniques to mold contexts and foster ideologies that support their agendas @xcite . propaganda is strategically used to manipulate public opinion through biased information to promote political objectives, significantly shaping domestic and international discourse (jones, 2019). the need for arabic-specific propaganda detection tools is emphasized by @xcite , who reveal the challenges posed by diverse linguistic and cultural contexts. @xcite address emotion manipulation and logical fallacies in arabic digital content, highlighting the importance of balanced training models. deep learning approaches for analyzing propaganda techniques in news posts have been introduced by @xcite , demonstrating effectiveness in recognizing propaganda. combining machine learning with human judgments to improve bias detection accuracy is explored by @xcite , showcasing the value of integrating diverse insights. @xcite detailed an annotation and modeling process for bias detection, while @xcite introduced the babe methodology, using distant supervision by experts to train neural models in detecting biased news sentences. the fignews 2024 shared task proposed by @xcite aims to investigate media representation of the israel-palestine conflict by focusing on bias and propaganda in news posts across multiple languages. our team, 'narrative navigators,' focused on the critical tasks of identifying biased content and detecting propaganda techniques in media articles. we developed and adhered to tailored annotation guidelines that allowed us to navigate through narratives with precision. this systematic application of our guidelines ensured consistent and reliable annotations. despite advancements in detecting biases and propaganda, challenges persist, especially in politically charged content. potential biases in annotators' judgments due to demographic backgrounds and personal perceptions remain a concern. zaghouani and colleagues have developed extensive annotation guidelines for various arabic nlp tasks including punctuation @xcite , author profiling @xcite ), text analytics in social media @xcite , diacritization @xcite and nonnative arabic texts @xcite . their work on large-scale arabic error annotation guidelines and framework @xcite is particularly relevant to our study, as it demonstrates the importance of well-defined annotation guidelines for ensuring data quality and consistency. while these studies focus on arabic, we adapt their methodologies and insights to develop guidelines for annotating bias and propaganda in multilingual news posts related to the israel-palestine conflict in the fignews 2024 shared task. this paper presents an overview of our participation, methodology, annotation process, and key findings. as narrative navigators, we encountered several challenges which we discuss in detail, proposing strategies to enhance the identification of bias and propaganda. our efforts underscore the importance of advancing media literacy and fostering informed public discourse on the conflict, true to our team's guiding principle of navigating narratives. ## methodology the fignews 2024 shared task focuses on creating a comprehensive corpus to annotate media representations of the 2023-2024 israel-palestine conflict, aimed at uncovering biases and double standards in global and local news coverage. our team, comprising seven members with diverse backgrounds, developed guidelines for annotating bias and propaganda. we analyzed news posts across different languages and cultural contexts. ## task participation and results our team actively participated in the fignews 2024 shared task, which focused on the meticulous annotation of news posts related to the israel-palestine conflict across various languages. the task required a comprehensive, multilingual approach to identify instances of bias and propaganda in the news posts. our annotation strategies were informed by insights gleaned from academic literature and comparable tasks from previous years. by integrating these insights into our methodology, we were able to refine our annotation schemes and enhance our analytical techniques, ensuring that our methods were current, effective, and aligned with established best practices. this alignment with related work allowed us to robustly validate and compare our results, confirming the reliability and relevance of our findings. ## conclusion our participation in the fignews 2024 shared task @xcite , reveals significant findings: 42% of texts showed bias against palestine, 15% against israel, and 68% contained propaganda. these results align with studies like jones (2019), who highlighted propaganda's role in manipulating public opinion, and @xcite , who demonstrated effective deep learning approaches for detecting propaganda. the high prevalence of propaganda supports @xcite and @xcite , who emphasize the need for nuanced tools to address complex linguistic and cultural media content. this study explored bias and propaganda in media coverage of the israel-palestine conflict using a comprehensive annotation approach. results indicate a complex landscape: 29.2% of 129,800 data points showed bias against palestine, 10.9% against israel, and 46.1% contained propaganda. such imbalanced coverage underscores the media's influence on public perception of the conflict. identifying bias and propaganda posed challenges, with some texts unclear or not applicable, highlighting the need for advanced detection methods. these findings stress the importance of media literacy and critical thinking among news consumers. these findings underscore the necessity for balanced reporting and advanced nlp tools to enhance media literacy and public understanding of the israel-palestine conflict. ongoing scrutiny of media practices, especially in conflict reporting, is crucial for ensuring fair, accurate, and unbiased coverage. developing advanced nlp tools can enhance media literacy and public understanding of global issues, promoting responsible journalism and informed public discourse. will stand at "givaot olam", the hill where he grew up and loved so much, at 3:00 p.m. and we will pay our last respects to him." annotator1: not propaganda annotator 2: propaganda. notes: annotators disagreed on whether the text segment is propaganda, reflecting differing interpretations of its intent and emotional tone.