G roningen A nnotates G aza at the FIGNEWS 2024 Shared Task: Analyzing Bias in Conflict Narratives (2024)

Abstract: In this paper we report the development of our annotation methodology for the shared task FIGNEWS 2024. The objective of the shared task is to look into the layers of bias in how the war on Gaza is represented in media narrative. Our methodology follows the prescriptive paradigm, in which guidelines are detailed and refined through an iterative process in which edge cases are discussed and converged. Our IAA score (Krippendorff’s \alpha ) is 0.420, highlighting the challenging and subjective nature of the task. Our results show that 52% of posts were unbiased, 42% biased against Palestine, 5% biased against Israel, and 3% biased against both. 16% were unclear or not applicable.

## introduction news articles usually claim to be objective and neutral reports. however, in practice, they often deviate wildly from these standards. the deliberate stress on certain content and the nuanced shifts in word choice or writing style can profoundly influence a reader's perception, particularly in political discourse @xcite @xcite . this phenomenon is referred to as news bias @xcite . identifying such bias is a critical yet difficult task in media studies, as bias can manifest subtly, such as through the minimal differences in meaning between words or in specific lexical choices, exemplified by terms like "illegal immigrants" versus "undocumented immigrants" @xcite . ## team composition and training the annotation process was carried out by a team seven annotators with expertise in computational linguistics, linguistics, and middle eastern studies, and with diverse linguistic backgrounds (native speakers of italian, dutch, arabic, and bengali); five annotators are aged between 25 and 34, one between 18 and 24 years old, and one between 35 and 44 years old. ## task participation and results as can be seen in figure 1 , hebrew has the highest percentage of texts labelled as biased against palestine. the hebrew texts also contain the lowest percentage of unbiased texts, around 25%, in comparison to 50%-70% of unbiased texts in the other languages. the french texts contain the highest percentage of texts biased against israel and the hindi texts as the highest percentage of unclear texts. figure 1 : frequencies per label per language in the batches b01 and b02 of the main sheet. ## discussion the high percentage of bias against palestine hebrew texts could be due to the sources of these texts, which are most likely to be israeli. as we have not annotated propaganda, we cannot conclude whether the hebrew texts contain israeli propaganda. we can however conclude that the hebrew texts contain the strongest bias against palestine in this dataset and that the sources most likely portray the israeli narrative. on the contrary, it is harder to interpret the high percentage of bias against israel in french texts and unclear labels in hindi texts. as the sources of the texts are not included in the dataset, it is unclear what influence the sources have on the bias presented in the texts, as the french sources could be from france, but also could be from news items aimed at areas from the former french colonial empire. notably, texts from hindi platforms were often incomprehensible and sensational. besides, some texts were merely lists of hashtags, offering little substantive content. moreover, certain texts indicated contradictory biases, simultaneously supporting perspectives on palestine and israel. ## conclusion our paper presented the guideline methodology developed for the participation in the shared task fignews 2024, aimed at annotating media narrative bias in representing israel's war on gaza. we followed a prescriptive paradigm to craft our guidelines, including two rounds of discussion phase for guidelines refinements based on identified edge cases. we added an extra level of annotation, which is the type of frame that supports either the israel or the palestine narrative. our annotation shows that hebrew texts have the highest percentage of biases against palestine, while the french texts are the most biased against israel and english texts are mostly unbiased. as for future work, we plan to refine our annotation by expanding our set of frames and annotating additional texts on the topic. we also aim to adapt our annotation scheme and apply it to other existing datasets relating to the palestinian-israeli conflict, for example the bitterlemons initiative. 4 thereafter, we intend to develop a computational model for detecting frames and examine the relationship between frames and different types of bias.