The FIGNEWS Shared Task on News Media Narratives (2024)

Abstract: We present an overview of the FIGNEWSshared task, organized as part of the Arabic-NLP 2024 conference co-located with ACL2024. The shared task addresses bias and pro-paganda annotation in multilingual news posts.We focus on the early days of the Israel War onGaza as a case study. The task aims to fostercollaboration in developing annotation guide-lines for subjective tasks by creating frame-works for analyzing diverse narratives high-lighting potential bias and propaganda. In aspirit of fostering and encouraging diversity,we address the problem from a multilingualperspective, namely within five languages: En-glish, French, Arabic, Hebrew, and Hindi. Atotal of 17 teams participated in two annota-tion subtasks: bias (16 teams) and propaganda(6 teams). The teams competed in four evalua-tion tracks: guidelines development, annotationquality, annotation quantity, and consistency.Collectively, the teams produced 129,800 datapoints. Key findings and implications for thefield are discussed.

## introduction the fignews 2024 shared task, 2,3 henceforth fignews, addresses the critical need for analyzing bias and propaganda in multilingual news discourse surrounding the israel war on gaza. this task aligns with the nlp community's growing efforts to create datasets and guidelines for complex opinion analysis through collaborative shared tasks and datathons. such a meta-task allows for the exploration of various annotation frameworks in particular for complex and challenging subjective tasks. fignews focuses on a diverse multilingual corpus, emphasizing the development of guidelines 1 fignews: framing the israel war on gaza news. 2 website: https://sites.google.com/view/fignews 3 code and data: https://github.com/camel-lab/ fignews-2024 & https://huggingface.co/datasets/ camel-lab/fignews-2024 with rich examples, while fostering a researchoriented collaborative environment. by simultaneously examining multiple languages, comparing and contrasting various narratives, fignews aims to unravel the layers of possible bias and propaganda within news articles with alternative media narratives. this is especially critical at times of any war or conflict. the media's portrayal of events during such events has significant implications on public perception, policy-making, and international relations. by addressing bias and propaganda, we hope to illuminate the varied ways in which news can shape, and sometimes distort, public understanding of complex geopolitical events. this initiative seeks to explore diverse perspectives, cultures, and languages, thereby fostering a comprehensive understanding of events through the lens of major news outlets across the globe. developing guidelines for complex data is a challenging task. the problem is exacerbated when the data is contemporaneous hence the annotators and the task organizers might have a stance on the subject matter. fignews is our attempt at addressing the creation of annotation guidelines, addressing what relevant best practices should be. we use the israel war on gaza as a use case to highlight some of these aspects. to that end, we curate a shared corpus for comprehensive annotation across various layers, crafting annotation guidelines shaped by the diverse range of conflicting discourses around this sensitive topic. this endeavor facilitates the development of robust methodologies and metrics for detecting and analyzing bias and propaganda, which are crucial for ensuring fair and accurate media reporting. the curated corpus, along with meticulously developed guidelines and annotations, will serve as a valuable resource for future research in nlp and related fields. this initiative also seeks to bring to light both challenges and commendable aspects within the data, fostering a collaborative community that can learn from each other's approaches and findings. we believe that a collaborative, research-oriented environment is essential for tackling the intricate task of bias and propaganda detection. the fignews shared task thus represents a significant step forward in the field of data annotation, media analysis and nlp. a step towards an annotation science. it not only provides a platform for examining critical issues of bias and propaganda but also promotes the development of best practices in data annotation and analysis. through this shared task, we aim to contribute to the broader goal of improving media literacy and fostering a more informed and critically engaged public. ## related work propaganda and bias can have far-reaching implications depending on the context and medium in which they are propagated. polarization, conflict and injustice are but some of the side effects they can create in any context. in the context of politics, they can alter the outcomes of elections and change the face of nations @xcite @xcite . news media framing and narratives around wars and socio-political events have been extensively studied, with a focus on identifying bias, propaganda, offensive language detection, and diverse perspectives @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite . several works have proposed computational approaches to detect media bias through analysis of word choice, labeling, and factual reporting @xcite @xcite . @xcite defines framing as the process of selecting certain aspects of perceived reality and constructing a narrative that emphasizes their connections to promote a specific interpretation. this foundational work highlights how news framing can influence public perception by emphasizing particular elements over others. @xcite further elaborates on this concept, providing a comprehensive framework for understanding how media frames can shape political and social realities. several studies have developed methods and datasets to detect news bias. @xcite used crowdsourced content analysis to quantify media bias, while @xcite compared computational approaches for detecting framing in political news. tools like @xcite help users gauge news bias, quantifying liberal or conservative leanings. in the context of multilingual and multi-label news framing analysis, @xcite explored the complexities of news framing across different languages. their work is crucial for understanding how bias and framing manifest in multilingual contexts, aligning closely with the objectives of the fignews shared task. similarly, the study by @xcite offers a valuable dataset and linguistic insights from two multilingual disinformation websites, providing a foundation for further exploration of language-specific disinformation techniques and their impact on news framing. recent advances in bias detection have also emphasized the importance of detailed and wellannotated datasets. @xcite introduced the mbic dataset, which includes detailed information about annotator characteristics and provides labels for bias identification at both the word and sentence levels. this dataset represents a significant step forward in creating reliable groundtruth data for bias detection. additionally, @xcite developed tassy, a text annotation survey system that enhances the quality control of annotation processes in nlp tasks. annotation of biased language and framing in news articles has been explored using techniques like expert annotation @xcite and crowdsourcing @xcite . quality control and guidelines for annotation processes in nlp tasks have also been investigated @xcite . @xcite developed eras, a system designed to enhance quality control in nlp tasks, which is particularly relevant for ensuring the reliability of annotations in bias and propaganda detection. further contributions to the detection of bias and propaganda include the work of @xcite , who analyzed language in fake news and political fact-checking, and @xcite , who organized news based on their propaganda-prone content. these studies provide foundational techniques for the identification and analysis of biased and propaganda language in news media. the shared task on propaganda detection at semeval-2020, organized by @xcite , is directly relevant to the fignews task. this task focused on detecting propaganda techniques in news articles, providing valuable benchmarks and methodologies that can be applied to the detection of bias and propaganda in social media posts. in the context of the israel-related wars and conflicts, @xcite conducted a sentiment analysis study to detect bias in western media coverage. this work highlights the specific challenges of detecting bias in a highly polarized and sensitive geopolitical context. additionally, @xcite introduced a new dataset designed for detecting and classifying various forms of offensive language in hebrew on social media. furthermore, the wojoodner shared task 2024 offered a new ner dataset related to the israeli war on gaza called wojood gaza (jarrar et al., 2024). research on detecting political bias in social media includes @xcite , who identified bias in user-generated content, and @xcite , who surveyed sentiment analysis techniques. the propitter corpus @xcite was developed using distant supervision with refined annotations. @xcite advanced media bias detection by focusing on automated identification through word choice and labeling. @xcite analyzed factual reporting to reduce bias, stressing the need for objectivity. among the studies forming a strong basis for understanding bias and propaganda detection in media are the following. @xcite examined news media profiling using text and social media analysis, showing how news context affects bias perception. @xcite , @xcite , and @xcite studied fake news spread during the 2016 us election, highlighting the need for reliable information. @xcite detailed fake news detection methods, while horne and adali (2017) emphasized high-quality data for combating misinformation. @xcite surveyed fake news identification techniques. zhou and zafarani (2020) reviewed fake news detection theories and methods, pointing to future research opportunities. while prior work has made notable contributions, our shared task focuses on comprehensive annotation of media narratives and framing around a specific war to uncover new insights. we build upon existing annotation frameworks @xcite to de-velop improved guidelines and foster a collaborative exploration of media discourses through a multilingual, multicultural lens. ## data collection and selection we used the crowdtangle foot_0 platform to collect facebook posts related to the israel war on gaza in five languages: english, french, arabic, hebrew, and hindi. specifically, we retrieved public posts containing the keyword "gaza" foot_1 from verified bluecheck accounts, publicly posted between october 7, 2023, and january 31, 2024. for each language, we collected approximately 300, 000 posts. to narrow this down to 3, 000 posts per language and have a balanced distribution, we focused on ten key moments in the first few months of the war: starting with the events of october 7, 2023 and the declaration of war, and including the bombings in jabalia refugee camp, al-shifa hospital, and st. porphyrius church, as well as the ceasefire and hostage release, gaza mass arrests, and the coverage of the international court of justice case till the end of january 2024. for each of the ten moments, we selected the top 200 posts per language, ranked by total interaction count (likes, shares, comments, etc.). as a result, we selected 15, 000 posts with the highest interactions, covering different key moments of the war, in five languages. we divided the 15, 000 posts into 15 batches ({b01, b02, ..., b015}) with 1, 000 posts in each batch. each batch contains 200 posts per language. additionally, each batch included 20 posts for each of the 10 key moments during the war. since the annotators may not speak all five languages, we provided machine translations (google translate) into english and arabic to facilitate annotation across the multilingual corpus. while machine translation inevitably introduces some noise, we considered this to be reflective of the real-world reliance on such technologies. to compare the inter-annotator agreement (iaa) among all participating teams in a fair manner, we randomly selected 20 posts from each language (i.e., 100 posts, which is 10%) from each batch, totaling 1, 500 posts. all participants received the dataset in a google sheet file with two sheets: "main" and "iaa". the "main" sheet con-tains 13, 500 posts, while the "iaa" sheet contains 1, 500 posts. see appendix b for a screenshot of the interface. it is important to note that participants were provided only with the original posts and their translations. information such as account owner, date, and total interactions were not given to any participating team (see the ethical consideration section). ## subtasks and evaluation tracks this section presents the shared task subtasks, evaluation tracks and minimal requirement for teams to qualify. further details are in appendix a. ## discussion we now consider the label distribution patterns for bias and propaganda, independently and together. while this seems consistent with what would be expected of the news media of a country at war, it is possible that there is an additional priming bias from knowing the source language of the text. perhaps in future editions, we could compare with a setting where all source language information is hidden and only translations are provided. labels bias labels propaganda not propaganda not applicable unclear biased against palestine 74.3% biased against israel 15.2% biased against both palestine and israel 13.7% biased against others 23.1% unbiased 78.7% not applicable 85.9% unclear 57.6% -61.2% -20.1% -21.7% -7.2% -12.5% -6.4% -9.0% -7.2% -3.2% -20.0% -7.0% -4.0% -69.8% -10.4% -2.3% -27.9% -14.7% 25.5% -25.9% -9.7% 31.2%