Eagles at FIGNEWS 2024 Shared Task: A Context-informed Prescriptive Approach to Bias Detection in Contentious News Narratives (2024)

Abstract: This research paper presents an in-depth examination of bias identification in media content related to the Israel-Palestine war. Focusing on the annotation guidelines and process developed by our team of researchers, the document outlines a systematic approach to discerning bias in articles. Through meticulous analysis, key indicators of bias such as emotive language, weasel words, and loaded comparisons are identified and discussed. The paper also explores the delineation between facts and opinions, emphasizing the importance of maintaining objectivity in annotation. Ethical considerations, including the handling of sensitive data and the promotion of multipartiality among annotators, are carefully addressed. The annotation guidelines also include other ethical considerations such as identifying rumors, false information, exercising prudence and selective quotations. The research paper offers insights into the annotation experience, highlighting common mistakes and providing valuable guidelines for future research in bias identification. By providing a comprehensive framework for evaluating bias in media coverage of the Israel-Palestine war, this study contributes to a deeper understanding of the complexities inherent in media discourse surrounding contentious geopolitical issues.

## introduction the world recently witnessed an unprecedented escalation of the war between palestine and israel which resulted in contentious international media coverage of which warrants close scrutiny. the element of subjectivity depending on various factors, such as proximity and exposure to certain narratives, linguistic barriers, etc. also confers an additional challenge. the fignews 2024 shared task @xcite addresses this area of research by adopting a collaborative approach, leveraging on a diverse pool of researchers and machine translation technologies to create a corpus of global news media narratives surrounding the israel-palestine war that would serve as a foundation for further research in natural language processing. there are two separate subtasks, one tailored to bias annotation and the other to propaganda annotation. as a relatively new team, we decided to focus on the bias annotation alone in our contribution. the shared task emphasized creation of annotation guidelines, a pivotal step of the nlp pipeline that influences the viability and efficacy of any model. thus, we sought to develop comprehensive prescriptive annotation guidelines that accounted for the contextual sensitivities of diverse and conflicting discourse. this paper details the rationale behind the creation of the annotation guidelines in addition to team composition and training, data annotation process, task participation and result. existing research that leveraged shared tasks have predominantly focused on data tasks such as named entity recognition @xcite , language identification @xcite , and information extraction @xcite , where prescriptive standards could be established and applied. there have been exceptions such as literary annotations @xcite , which indicate the potential for shared tasks with respect to efforts that involve higher degrees of variability and subjectivity. given the inherent diversity in any representative dataset for a global, polarized discourse such as the israel war on gaza, it is thus prudent to approach corpus creation in a diverse, collaborative fashion. ## team composition and training our team was composed of researchers with interests in linguistics, computational linguistics, and media studies. each member comes from a different academic background and demographic (egypt, lebanon, tunisia, and malaysia), although three members of the team were not residing in their home country at the time of the shared task, with the annotator from lebanon based in egypt, the annotator from tunisia based in italy and the annotator from malysia based in qatar. the team of researchers were relatively new to annotation, with only one researcher having had practical exposure to data annotation. the first few meetings prioritized knowledge transfer, with team members sharing their relevant expertise, i.e., on the annotation process and rationale of quantifying consistency via the separation of datasets into the main and iaa portions, the linguistic considerations of assessing bias, etc. the team also shared online resources on annotation, such as the annotation instructions tutorial (qcri, 2024), in preparation for the annotation task. the team worked together in a collaborative virtual environment through weekly synchronous meetings and asynchronous group chat. unfortunately, due to external constraints one researcher was highly limited in their contributions, hence the remaining three team members coordinated on an ad hoc basis to ensure completion of the task. the team coordination included delegating tasks, group discussions, taking notes of the team's discussions and important points, and having indepth discussions of confusing items. for instance, the delineation between "unclear" and "not applicable." some items had historical references that needed to be researched to understand their context and their relation to current events and therefore be able to decide the appropriate annotation label. ## task participation and results of the 1,800 texts annotated, our team found that 1,064 (59.1%) texts were classified as unbiased, 473 (26.3%) as biased against palestine, 87 (4.8%) as not applicable, 121 (6.7%) as unclear, 38 (2.1%) as biased against others, 10 (0.6%) as biased against israel, and 7 (0.4%) as biased against both palestine and israel (figure 1 ), which was similar to the overall findings foot_0 (figure 2 ). ## discussion this shared task establishes a promising alternative to generating crowdsourced annotations for natural language processing that addresses some key concerns in annotation guidelines and practice, particularly in highly sensitive and complex subject areas. for instance, @xcite expounds upon instruction bias, where natural language understanding is driven by benchmarks where annotators generate examples in a top-down approach, based on annotation instructions. this can be equated to the "ground truth" mentioned in previous sections. however, in refining the annotation guidelines based on a large dataset derived from real world sources, the shared task ensures that the collected data is robust and representative of real-world content. in view of the widespread conflation of palestinians with hamas or other political figures, we equated bias against hamas and political leaders as bias against palestinians. this was done to meaningfully assess the sentiments of the text in accordance with existing views and does not reflect the team's opinions on the plurality of palestinian political views. similarly, the assessment of fact versus opinion was based on up-to-date knowledge of current events. this introduces some variability to the annotations, since some content may be proven or disproven at different points in time, for example, headlines claiming genocide released on october 8 th , 2023, could reasonably be construed as biased against israel, but an identical headline published in january 2024 should certainly be viewed differently. this is impactful given the extent of disinformation surrounding the israel-palestine war. notably, this may explain the slight divergence in our team's findings compared to the overall findings regarding the presence of bias against israel. in discussion, we concluded that many texts that were initially labelled as biased against israel were in fact unbiased due the factual content, based on external research and factchecking. consequently, the resulting annotations are bound to recency, and efforts to update them based on current information are paramount to preserve the accuracy and integrity of the data. since the shared task focused on news media content, the team also considered ethical reporting and journalism standards in ascribing bias (appendix e). ## conclusion this paper proposes a comprehensive framework for bias detection in news media narratives surrounding the israel-palestine war. given the objective of the annotation guidelines to provide a robust prescriptive framework for bias detection, the development thereof took on an iterative approach, that via a systematic annotation process, mitigated the risk of inter-annotator subjectivity. the annotation process revealed the subtle complexities of detecting bias, highlighting the role of diverse human perspectives, robust discussion, and up-to-date information to mitigate annotator bias in such data tasks. we believe that these resources and the insights gained from the production thereof will inform the understanding of the unique challenges of analyzing conflicting discourse in highly contentious topics.