The C yber E quity Lab at FIGNEWS 2024 Shared Task: Annotating a Corpus of F acebook Posts to Label Bias and Propaganda in G aza- I srael War Coverage in Five Languages (2024)

Abstract: This paper presents The_CyberEquity_Lab team’s participation in the FIGNEWS 2024 Shared Task (Zaghouani, et al., 2024). The task is to annotate a corpus of Facebook posts into bias and propaganda in covering the Gaza-Israel war. The posts represent news articles written in five different languages. The paper presents the guidelines of annotation that the team has adhered in identifying both bias and propaganda in coverage of this continuous conflict.

## introduction the ongoing conflict between palestinians and israelis has been a subject to traditional media bias and propaganda for years, layers over layers of misinformation accumulated over decades did not help bridge the gap between palestinian and israeli public points of views. then comes the event of october 7 th , coinciding with the era of social media, where people can see and hear the suffering of innocent war victims through numerous social platforms, alternative to traditional media outlets. one would think that in the age of information, controlling a narrative would be out of the reach, however since the events of 7 th of october and followed by israel's military response, social media has been plagued with bias and propaganda, and the same narratives and talking points from media, globally, are echoing all over social networks. detecting bias and propaganda from news or social media posts has been deeply explored in the literature and a variety of methodologies and techniques have been presented. for instance, detecting bias through sentiment analysis on news reports using natural language processing (nlp) and machine learning (ml) techniques @xcite . @xcite presented sentiment analysis, subjectivity analysis, modality and other analysis to detect bias in short sentences @xcite . @xcite has presented a semi-automatic annotation framework to detect misinformation on covid-19 pandemic in social media posts @xcite . @xcite clustering on arabic tweets in order to detect sentiment (d. @xcite . generating ml models, through nlp techniques, rely on large corpora for training to achieve satisfying level of accuracy. ml models learn from such corpora and use its knowledge to perform numerous operations on new data. in order to facilitate the use of such models to detect bias and propaganda in the israel-gaza social media coverage, trained annotation teams are required to label a large portion of the corpora while following well-defined labeling guidelines. this research work has been done as participation in fignews 2024 shared task at the arabicnlp 2024 conference. the task aims at constructing a corpus of facebook posts and crowdsourcing an annotation outline to label bias and propaganda in these posts @xcite . the task is split into two subtasks one for labeling bias and the other is for labeling propaganda. the corpus consists of 15000 posts collected from october 1 st , 2023 till january 31 st , 2024 and written in five different languages: arabic, hebrew, french, hindi, and english. the posts are translated into arabic and english using translation tools. the corpus was collected by searching for the keyword "gaza" in the five aforementioned languages. in this paper, the cyberequity lab proposes two sets of guidelines to define and identify both bias and propaganda in the coverage of israel-gaza war. the rest of the paper is organized as follows: section 2 will discuss annotation process and methodology, section 3 will present team ## annotation methodology labeling bias and propaganda in full news articles is has been always a challenge. however, what is more challenging is when labeling in short social media posts, some might be as short as a single sentence. in order to develop a comprehensive set of guidelines for such challenging annotation task, our team had to go through several revisions and endless debates, scratching our head: how to define bias and propaganda, what is the reference, what is the yardstick. williams @xcite defined bias as willful, influential, and threatening to widely held conventions. to apply williams's definition on our task, one must ask an important question: which conventions are we talking about, palestinian or israeli? the next section will discuss the development process that led our team to the concluded annotation guidelines. ## team composition and training our team is composed of five staff members from the department of computer science, birzeit university. all members of the team are palestinians from west bank. four of the five members are male and one female. since the team members are from the same department, it was convenient for us to hold regular meetings and discussions, feedback, and trial and error sessions. the team leader has individually met with each team member and worked together for several hours discussing as many cases as possible and observing the adherence to mutual standards. the team would leave ambiguous posts to be discussed as a group. it is worth to mention that some of the team members are specialists in nlp and ml, and since all members are palestinians, they are deeply immersed in political dialogues concerning the palestinian/israeli conflict and they have been observing related news throughout their whole lives. ## task participdsxaation, results and discussion in this section, a comparison of corpora sizes is done between our teams' annotation output and related nlp and manual annotated corpora. the volume of the corpora plays a significant role in deciding the accuracy of the nlp models developed after training on the corpora. therefore, in order to evaluate the output of our team, we have taken a look at the size of corpora used to train existing models, in order to develop a perception of how large should the corpus be in order to produce an impactful corpus. the cyberequity team has managed to annotate ~3000 posts out of 15000, 10% of annotated posts are part of iaa set. comparing our performance with related participations in such shared tasks, for example, sora et. al. (sora lim, 2020) has covered 966 news articles, çağrı (çöltekin, 2020) user a large 36,000 records to detect offensive language in turkish, but the team has manually annotated a small portion of the corpus. marta et. al. @xcite presents a similar shared task, in which they crowd sourced a best-practice guideline. however, the paper does not explicitly mention the size of the corpus. hutto et. al. @xcite has used an average of 91 human annotated labels as test set. having looked at related work and having compared the volume of the corpora, it comes clear that 3000 records of manually annotated labels is an acceptable amount of data to train highly accurate nlp models. moreover, our team has achieved competitive results compared with other participating teams in fignews 2024 shared task, especially in the propaganda subtask where our team has secured third rank out of sixteen teams in: ## conclusion the paper presents the process of manual annotation of facebook posts written in five languages: arabic, hebrew, english, french, and hindi. the labels adopted highlight the bias and propaganda in the coverage of gaza-israel war. the paper has proposed a comprehensive guideline to identify and label posts into bias and propaganda. the research work has been dedicated as a response to fignews 2024 call for participation in two subtasks. our team has participated in annotating 3000 different posts, including 10% for iaa agreements analysis. comparing the volume of our annotated data with related earlier work on both manual and computational annotation, it can be concluded that the volume produced can be used to train ml models and can be used to evaluate clustering models.