The Guidelines Specialists at FIGNEWS 2024 Shared Task: An annotation guideline to Unravel Bias in News Media Narratives Using a Linguistic Approach (2024)

Abstract: This article presents the participation of “The Guideline Specialists” in the FIGNEWS 2024 Shared Task, which aims to unravel bias and propaganda in news media narratives surrounding the Gaza-Israel 2023-2024 war. Leveraging innovative annotation methodologies and drawing on a diverse team of annotators, our approach focuses on meticulously annotating news articles using a linguistic approach to uncover the intricate nuances of bias. By incorporating detailed examples and drawing on related work that show how language structure represented in the use of passive voice or the use of nominalization and the choice of vocabulary carry bias, our findings provide valuable insights into the representation of the Gaza-Israel conflict across various languages and cultures. The guideline we developed detected the bias against Gaza, against Israel and others by setting keywords that are based on linguistic background tested by the AntConc concordance tool. The result was an annotation guideline that have a solid base. Through this collaborative effort, we developed a guideline that contributes to fostering a deeper understanding of media narratives during one of the most critical moments in recent history.

## introduction the gaza-israel conflict of 2023-2024 has not only been a pivotal moment in recent history but has also underscored the critical role of media narratives in shaping public perception and understanding of complex geopolitical events. in response to the multifaceted portrayal of this conflict across various news outlets, the fignews 2024 shared task, entitled "framing the israel war on gaza: a shared task on news media narratives," emerges as a timely and imperative initiative. this shared task, as outlined in the fignews overview @xcite , seeks to study and examine the biases and double standards prevalent in news media narratives surrounding the gaza-israel conflict. adhering to the format prescribed by the shared task's title, the fignews overview paper meticulously outlines the objectives and methodology of the initiative, emphasizing the need for a comprehensive exploration of media narratives to foster a nuanced understanding of the conflict. this alignment underscores the shared task's commitment to addressing the overarching goal of unravelling bias and propaganda within news articles, thereby contributing to a more informed and balanced discourse on the gaza-israel conflict. in recent years, shared tasks and similar collaborative efforts have proliferated within the natural language processing (nlp) community, reflecting a growing recognition of the importance of large-scale datasets and standardized annotation guidelines in advancing research in this field. works such as the semeval series @xcite and the conll shared tasks (conll, 2019) have played pivotal roles in facilitating benchmarking, fostering innovation, and driving progress in nlp tasks, including sentiment analysis, named entity recognition, and discourse parsing. however, while these initiatives have significantly contributed to the advancement of nlp research, they often focus on generic linguistic tasks and lack the specificity required to address domain-specific challenges such as media bias analysis. in contrast, the fignews 2024 shared task represents a novel and targeted effort to tackle the complexities of bias and double standards in news media narratives, particularly in the context of a high-stakes geopolitical conflict. by honing in on this specific domain, the shared task not only offers a unique opportunity to deepen our understanding of media representations but also paves the way for developing specialised methodologies and techniques tailored to address these challenges. in this introduction, we situate the participation of our team, "the guideline specialists," within the broader context of shared tasks in the nlp community, highlighting the distinctiveness of the fignews 2024 shared task and the novel contributions it offers to the field. through our engagement in this initiative, we aim to advance the state-of-the-art in media bias analysis and contribute to a more nuanced and informed discourse on the israeli war on gaza by introducing a linguistically and statistically informed guideline to guide the annotation of the news under investigation. the next section presents the methodology that is used to develop our guideline to detect bias in the media news on israel's war on gaza. ## annotation methodology and examples the guideline followed two approaches. the first approach is a linguistic one that is partly based on the work of @xcite that represents how bias is detected through language structure, especially in the use of the passive voice. verbs in the passive voice of violent and destructive meaning that are not followed by any reference to the doer are biased against the recipient/victim of these violent actions. "the passive voice can also be used to purposely leave the performer of the action unspecified, a strategy that the reporter can use to avoid ascribing direct responsibility for an action to anybody in particular" @xcite . the second part of the linguistic approach according to @xcite is the use of nominalization to avoid mentioning any doer of the actions making the action with no doer that receives no accountability which is biased against the receiver of this action. nominalization is the "transformation which reduces a whole clause to its nucleus, the verb, and turns that into a noun" @xcite . the third linguistic criterion is lexicon-based as the choice of certain words that label certain groups to direct the reader not to sympathize with them. on the other hand, using words referring to relatives, friends and different relationships in the narrative draws on moving the feelings of reader to sympathise with the group with these more words that refer to family members and to feel angry with the other party which represents bias against this party in media narratives. examples of the different methods will be presented in the following subsections. ## team composition and training the team composition and training for the fignews shared task on news media narratives were carefully structured to leverage diverse expertise and ensure effective collaboration. annotator demographics were a key consideration, reflecting a commitment to inclusivity and cultural sensitivity. the team consisted of two female members: samar amer, a linguist from egypt, and ghizlane bourahouat, a data scientist from morocco. this diverse composition enriched the annotation process by providing insights into linguistic nuances and cultural contexts specific to egypt and morocco. training sessions were conducted with precision to equip team members with the requisite skills and knowledge for accurate annotation. these sessions covered a spectrum of topics, including annotation guidelines, data interpretation techniques, and quality assurance protocols. samar and ghizlane, ## task participation and results the guidelines specialists' participation in the fignews shared task on news media narratives was marked by a meticulous and systematic approach, which yielded highly reliable results. central to their methodology was the strategic use of detailed examples and comparative analysis with related work, both of which significantly informed their annotation process and outcomes. the use of examples played a pivotal role in guiding the annotation process. detailed examples were included in the annotation guidelines to serve as clear benchmarks, helping to minimize ambiguities and ensure consistent application of the guidelines across various pieces of news. these examples were particularly beneficial during training sessions, where they provided practical references that enabled annotators to internalize the guidelines effectively and apply them uniformly. in addition to leveraging examples, the team conducted a comprehensive review of related work in the field of news media narrative analysis. by comparing their methodology with established practices and findings from previous studies, the team was able to adopt proven techniques and avoid common pitfalls. this comparative approach informed their strategy, leading to more accurate and consistent annotations. we annotated 1509 sentences. descriptive analysis of our annotations shows that the 'biased against palestine'label is the most common type in the annotated corpora that was equally distributed as 25% (arabic), 25%, (english), 25% (french), 25% (hindi) and 25% (hebrew). the effectiveness of our team's approach was validated through the calculation of cohen's kappa, a measure of inter-annotator agreement. the team achieved a cohen's kappa score of 85%, as depicted in figure 1 , indicating a high level of agreement among annotators. this high kappa score reflects the annotation guidelines' clarity and the training sessions' success in preparing the annotators. the annotation scores for "the guideline specialists" on project b01 using the squad dataset, as evaluated by the organization team, are as follows: a precision of 0.003, a recall of 0.098, and an f1 score of 0.068. the overall performance score for the team is 0.740. ## discussion the team's findings from the fignews shared task on news media narratives offer significant insights into the detection and analysis of bias in news media. by employing a well-defined methodology that incorporated detailed examples and comparative analysis with related work, the team not only achieved high reliability in their annotations but also contributed valuable knowledge to the field of media studies and computational linguistics. the high cohen's kappa score of 0,85 underscores the effectiveness of the annotation guidelines and the training process. this level of agreement among annotators highlights the clarity and precision of the guidelines, which were informed by previous work, particularly the studies of @xcite and @xcite on linguistic indicators of bias such as passive voice and nominalization. the use of these linguistic features as part of the annotation criteria proved effective in identifying bias, aligning with the findings of related research. our team's approach and findings also have broader implications for the study of media narratives. by systematically applying linguistic criteria to detect bias, the team demonstrated how structured and replicable methods can enhance the objectivity and consistency of media analysis. this methodological rigour contributes to the ongoing efforts to develop automated tools for bias detection, which are crucial in an era of increasing media consumption and misinformation. furthermore, the team's work enriches the discourse on how specific linguistic structures and word choices can influence readers' perceptions. the findings support the notion that passive structures and nominalizations can obscure responsibility and accountability, thereby introducing bias. this aligns with @xcite observations on the manipulative potential of language in media. in comparison with related work, such as the work of @xcite that presented comprehensive guidelines for consistent and effective annotation of text corpora with emotion labels that led to substantially (30%) higher agreement scores among human annotators, the team's focus on combining linguistic criteria with practical annotation tools like antconc and lancsbox showcases an innovative approach to bias detection. this integration of computational tools with linguistic theory not only enhances the accuracy of bias identification but also provides a replicable framework that can be applied to other datasets and languages. the annotation scores for the team "the guideline specialists" on project b01 using the squad dataset, as evaluated by the organization team, demonstrate performance with notable variance. the key metrics reveal a precision of 0.003, a recall of 0.098, and an f1 score of 0.068. these results indicate that while the recall is relatively higher, suggesting the team captures a fair number of relevant instances, the precision and f1 score is quite low, pointing to challenges in achieving accuracy and overall balanced performance. with an overall performance score of 0.740, there's room for improvement, particularly in enhancing precision to improve the f1 score. overall, the guidelines specialists team's findings contribute to the field by providing a validated and replicable methodology for bias detection in news media narratives. the high interannotator agreement achieved demonstrates the potential for these methods to be used in largerscale studies, potentially informing the development of automated systems for real-time bias detection in media. this work advances academic understanding and practical applications in promoting media literacy and accountability. ## conclusion the fignews shared task on news media narratives demonstrated the critical role of thorough examples and a deep understanding of related work in achieving high-quality annotations. the use of detailed examples ensured consistent application of guidelines, while insights from related studies informed a robust methodological framework. the team's success, reflected in the high inter-annotator agreement score, highlights the effectiveness of combining linguistic theory with practical annotation tools. these contributions not only advanced the task but also provided valuable methodologies for future research in media bias detection.