Sina at F ig N ews 2024: Multilingual Datasets Annotated with Bias and Propaganda. (2024)

Abstract: The proliferation of bias and propaganda onsocial media is an increasingly significant concern,leading to the development of techniquesfor automatic detection. This article presents amultilingual corpus of 12, 000 Facebook postsfully annotated for bias and propaganda. Thecorpus was created as part of the FigNews2024 Shared Task on News Media Narrativesfor framing the Israeli War on Gaza. It coversvarious events during the War from October7, 2023 to January 31, 2024. The corpuscomprises 12, 000 posts in five languages (Arabic,Hebrew, English, French, and Hindi), with2, 400 posts for each language. The annotationprocess involved 10 graduate students specializingin Law. The Inter-Annotator Agreement(IAA) was used to evaluate the annotationsof the corpus, with an average IAA of 80.8%for bias and 70.15% for propaganda annotations.Our team was ranked among the bestperformingteams in both Bias and Propagandasubtasks. The corpus is open-source and availableat https://sina.birzeit.edu/fada

## introduction since october 7, social media has been flooded with posts, articles, images, and videos related to the israeli war on gaza. such posts are often divided by hate, bias, and fake news either in favor of or against one of the parties or by remaining neutral, see e.g., "framing the israeli war on gaza" is a shared task on news media narratives @xcite , which is part of the 2 nd arabicnlp conference. the task aims to create a multilingual corpus that unravels the layers of bias and propaganda within news articles in various languages. such shared tasks and datathons are crucial in the nlp community to foster collaboration and advance research in specific areas. previous efforts, such as semeval-2020 task 11 @xcite and @xcite have provided valuable resources for propaganda detection in news articles. the dual focus of fignews on bias and propaganda is a novel approach that addresses the evolving nature of misinformation on social media platforms. the detection of propaganda on social media is crucial @xcite , as it can polarize public sentiment, foster violent extremism and hate speech, and eventually erode democracies and diminish trust in democratic procedures @xcite . notably, only a few corpora have been recently built to address these issues. recent work by @xcite involved establishing a hebrew dataset comprising 15, 881 tweets for detecting offensive language. this dataset was manually annotated with four labels: hate, abusive, violence, and pornographic. their work focused on detecting hate speech in hebrew tweets and implemented in sinatools @xcite . additionally, the wojoodner shared task 2024 offered a new ner dataset related to the israeli war on gaza called wojood gaza @xcite . other notable works include @xcite , qprop @xcite , and ptc @xcite . tshp-17 and qprop are documentlevel corpora, while ptc is a sentence-level corpus. while semeval-2020 @xcite task 11 is similar to fignews @xcite in its objective, they differ in their data sources and focus areas. this paper describes our participation in the fignews. our contributions are: remark: the corpus presented in this article does not cover the genocide, ethnic cleansing, or starvation events as they mostly happened after collecting the corpus. the article is organized as follows: section 2 describes the methodology, 3 presents our team composition and training; section 3 presents our participation and results; section 5 analyzes some errors, and section 6 concludes the paper. ## annotation methodology the objective of the task is to address the complex landscape of social media discourse related to the israeli war on gaza 2023-2024. the task organizers provided participants with 15k posts from verified facebook accounts, selected between october 6, 2023, and january 31, 2024, using "gaza" as a query keyword across 5 languages: arabic, hebrew, english, french, and hindi. the dataset consists of 15 batches, each containing 1000 posts. ## error analysis and discussion despite training and supervision, errors may arise from subjective interpretation, ambiguous guidelines, or complex content. we explored the errors ## conclusion this article presents our contribution to the fignews 2024, where we annotated a multilingual corpus of 12, 000 facebook posts for bias and propaganda across five languages. we extended the annotation guidelines for better consistency and accuracy, providing a foundation for future work in detecting bias in social media. our plans include expanding the corpus to cover more critical events of the war and leveraging neural and large language models to automatically detect bias and propaganda on social media posts.