NLPC olab at F ig N ews 2024 Shared Task: Challenges in Bias and Propaganda Annotation for News Media (2024)

Abstract: In this paper, we present our methodology and findings from participating in the FIGNEWS 2024 shared task on annotating news fragments on the Gaza-Israel war for bias and propaganda detection. The task aimed to refine the FIGNEWS 2024 annotation guidelines and to contribute to the creation of a comprehensive dataset to advance research in this field. Our team employed a multi-faceted approach to ensure high accuracy in data annotations. Our results highlight key challenges in detecting bias and propaganda, such as the need for more comprehensive guidelines. Our team ranked first in all tracks for propaganda annotation. For Bias, the team stood in first place for the Guidelines and IAA tracks, and in second place for the Quantity and Consistency tracks.

## introduction this paper details the fignews 2024 shared task participation by the nlpcolab team, which is a collaborative effort of fatima jinnah women university, rawalpindi, pakistan, beaconhouse national university, lahore, pakistan and national university of science and technology, islamabad, pakistan. the fignews 2024 shared task aimed to look into the subtle complexities of bias and propaganda prevalent in news articles, covering multiple languages, in the context of the gaza-israel 2023-2024 war. the primary goal for fignews 2024 is to create a shared corpus for detailed annotation and to develop annotation guidelines. the initiative also aims to foster collaboration between natural language processing (nlp) researchers. fignews 2024 describes the task as "a researchoriented datathon with a strong focus on the devel-opment of improved annotation guidelines for complex opinion data tasks" @xcite . fignews curated a corpus comprising news article headlines and advertising posts from facebook, all of which are collectively termed posts, in english, arabic, hebrew, french, and hindi. the posts were collected from verified accounts of news agencies from around the world, from october 1, 2023 to january 31, 2024. the focus query was "gaza", and there is a focus on significant moments during this conflict, e.g., the official war declaration, bombings at al-shifa hospital, ceasefire and hostage release. fignews released a subset of the full corpus to the shared task teams. machine translations into english and arabic were provided. the subset was shared in the form of 15 batches, with 1000 posts per batch. the first subtask was on bias annotation and teams were given 7 labels: unbiased, biased against palestine, biased against israel, biased against both palestine and israel, biased against others, unclear, not applicable. one example was provided for each label @xcite . the second subtask was on propaganda annotation and teams were given 4 labels: propaganda, not propaganda, unclear, not applicable. one example was provided for each label @xcite . shared tasks and datathons have become increasingly popular during the last two decades to advance research and development in various fields including nlp. some examples that are significant in the context of the fignews shared task include semeval-2018 task 1: affect in tweets, which annotates the intensity of emotion in tweets using best-worst scaling. this shared task implemented a defined process to identify malicious an-notations. it involved a set of internally annotated tweets, referred to as gold tweets, that were shared for additional annotation, allowing the tracking of annotator performance on these specific tweets @xcite . semeval-2019 task 4: hyperpartisan news detection annotated entire articles using 5 classes @xcite . nlp4if-2019 shared task on fine-grained propaganda detection focused on text fragments instead of entire articles, and is closer in approach to the fignews task @xcite . semeval-2020 task 11 on detection of persuasion techniques in news articles was similar to the fignews task in that it was focusing on multiple types of annotations including misleading information and propaganda @xcite . semeval-2021 task 6 on detection of persuasion techniques in texts and images focused on annotating memes and was unique in that it took combinations of image and textual information into consideration @xcite . in wanlp-2022 shared task on propaganda detection in arabic the task involved identified the type of propaganda technique being used and identified the fragment of the text where it was used @xcite . semeval-2023 task 3 look at categories of persuasion in news articles in 9 languages @xcite , and they used machine translations, similar to how fignews did, among other techniques. such tasks help create standardized datasets and annotation guidelines. the collaborative annotations lead to enhanced quality and reliability of datasets. they allow for the creation of large, high-quality datasets that might be too resourceintensive for single researchers or small teams to develop independently, and usually, these datasets end up becoming valuable publicly available resources. ## annotation methodology and examples for both subtasks, bias and propanda, the team started off with the examples provided by fignews for each label and proceeded to develop annotation methodologies. ## team composition and training the nlpcolab team operated in multiple groups to complete the annotations. the annotation team each annotation team was composed of four members and completed two batches, including the iaa sets as per the fignews guideline. a comprehensive training was provided to annotators to ensure a thorough understanding of the guidelines and annotation process. open discussions and consensus-building were encouraged among annotators to resolve disagreements and clarify ambiguous cases. regular feedback was incorporated iteratively to improve the guidelines and streamline the annotation workflow. ## task participation and results evaluation for each task was conducted on four tracks: guidelines, inter-annotator agreement (iaa) quality, quantity, consistency @xcite a total of 17 teams, including around 140 overall members from across the globe participated in the fignews shared task. the nlpcolab team participated in both bias annotation and propaganda subtasks and successfully annotated all 15 batches for both. iaa analysis was conducted on 200 posts as per the fignews guideline. the nlpcolab team stood in first place for all tracks of the propaganda subtask. for bias, the team stood in first place for the guidelines and iaa tracks, and in second place for the quantity and consistency tracks. ## discussion it was apparent during early discussions that annotators may have varying perceptions of bias and propaganda. to mitigate this, the team implemented detailed guidelines and regular training sessions. annotators engaged in regular feedback and review meetings to ensure a common understanding of and consistent application of the guidelines. it also became clear as the task unfolded that detecting subtle forms of bias and propaganda was particularly challenging, as it requires linguistic proficiency, a deep awareness of socio-political contexts and the ability to discern underlying agendas. during reviews, the team detected several instances of subtle cues that were overlooked. additionally, subtle bias and propaganda can vary significantly across different cultures, further complicating the annotation process. in the context of these issues, a significant strength of our approach was the multidisciplinary composition of our team, which included members from media and communication sciences as well as computer science (specifically nlp students). this diversity brought a broad range of perspectives and expertise to the annotation pro-cess, encouraging richer discussions and a more comprehensive understanding of what constitutes bias and propaganda. the collaboration between these disciplines allowed us to leverage the theoretical and practical insights from media studies alongside the technical rigor of nlp, enhancing the overall quality and depth of our annotations. one interesting question raised during review meetings was whether the translation of the posts amplified or de-emphasized the presence of bias and propaganda. however, it is important to note that fignews considers machine translations to be valid stand-ins for the annotation tasks, reflecting the reality that many people rely on such technologies @xcite . given the sensitive and polarizing nature of the palestine-israel conflict, annotators engaged in continuous self-reflection and employed strategies to remain as objective as possible. this conscious effort to recognize and mitigate personal biases was essential to ensuring the integrity and reliability of the annotated data, fostering a more balanced and nuanced interpretation of the news items. in this context, it is to be noted that the fignews guideline referred to the area of focus as both "war" and "conflict", and this paper consequently uses both terms as the result of a conscious decision. our experience highlights the importance of continuous education and awareness in annotation tasks, particularly in complex and sensitive domains. regular training sessions on recognizing and addressing personal biases, along with discussions on ethical considerations, were vital in maintaining high standards of annotation quality and integrity. ## conclusion our participation in the fignews 2024 shared task employed a multi-faceted approach including team selection, task division, and data analysis. the use of examples and comparisons with related work significantly influenced our methodology and performance. by reviewing existing guidelines, we identified gaps and formulated new strategies to improve the annotation process. despite the inherent subjectivity in annotating bias and propaganda, our iaa scores demonstrate that our detailed guidelines and multidisciplinary approach helped maintain consistency. one final recommendation for future work is to leverage technology to automate parts of the annotation process. steps such as initial tagging or flagging potentially biased content, can increase efficiency and allow annotators to focus on more subjective and complex decisions.