Uot1 at FIGNEWS 2024 Shared Task: Labeling News Bias (2024)

Abstract: This paper outlines the University of Tripoli’s initiative in creating annotation guidelines to detect bias in news articles concerning the Palestinian-Israeli conflict. Our team participated in the Framing of Israeli Gaza News Media Narrative (FIGNEWS 2024) shared task. We developed annotation guidelines to label bias in news articles. Using those guidelines we managed to annotate 3,900 articles with the aid of our custom-developed annotation tool. Among 16 participating teams, we scored 48.7 on the macro F1 measure in the quality track in which we ranked 4th. In the centrality track we were ranked at the 6th position using the macro F1 avg measure, however, we achieved the 4th best kappa coefficient. Our bias annotation guidelines was ranked in the 9th position.

## introduction framing can be defined as the process of selecting certain elements of perceived reality and constructing a narrative that emphasizes their connections to promote a specific explanation @xcite . news framing involves emphasizing certain aspects of specific issues in news reports to encourage a particular interpretation. while recent nlp studies have examined framing in english news, there has been limited exploration of how this analysis can be applied to other languages and in a multi-label context @xcite . this paper is a contribution to the creative shared task on news media narratives (framing the israel war on gaza) @xcite to explore into the complex intricacies of bias and double standards evident in news coverage. the main goal is to create a common database for thorough annotation across different levels, shaping annotation rules based on the various and sometimes conflicting discussions on this delicate subject. this effort aims to highlight both difficulties and positive aspects within the data and to encour-age a cooperative environment. by dissecting news articles in various languages, this joint endeavor aims to uncover the underlying biases and propaganda, encouraging a collective investigation into the media narratives surrounding this significant historical period @xcite . text annotation is a crucial activity across various fields that involves several essential steps includes: formatting the text, defining annotation categories, organizing large volumes of documents for annotation, assessing inter-annotator agreement on the same documents, and preparing annotated documents in formats compatible with nlp software @xcite . in natural language processing (nlp) research, text annotation plays a fundamental role in generating datasets used to train and assess automated techniques @xcite . in the realm of news reporting, it's crucial for both writers and audiences to uphold principles of fairness, accuracy, and impartiality by presenting a balanced array of viewpoints. nevertheless, bias in news articles has emerged as a significant concern. despite numerous news outlets asserting their commitment to objectivity, individual sources often harbor distinct perspectives on societal, political, and other subjects. additionally, the imperative to engage readers to ensure profitability can inadvertently foster biased reporting practices, potentially yielding harmful outcomes @xcite . this task focuses on examining how journalists' reporting of important events can shape consumers' views on political matters @xcite . the inherent bias of news media is a fundamental flaw in the entire news production process, encompassing stages from gathering to writing and editing. at each step, the subjective judgments of producers and external pressures from owners and advertisers likely influence the news content. these influences often lead to diverse framings of reality, as producers emphasize different aspects, employ varied tones, and present stories in distinct styles. consequently, this bias exacerbates political polarization and contributes to misunderstandings of crucial issues. it becomes challenging for readers to attain a balanced perspective on realities, especially when they primarily access news through limited channels, often just one @xcite . thus, recognizing and addressing news bias is crucial, not only for informing readers but also for guiding authors in maintaining a neutral writing style @xcite . the importance of detecting bias in media news has been recognized and studied by several researchers @xcite @xcite . advanced bias detection techniques rely on artificial intelligence and machine learning @xcite , thus requiring a well-annotated datasets to train ai models. existing datasets often suffer from limitations such as lack of representativeness, insufficient diversity, and inadequate information about annotator characteristics. addressing these challenges is crucial for contextualizing and interpreting bias annotations accurately. previous research efforts have contributed to the development of datasets for bias detection, such as those by @xcite , @xcite , @xcite , and @xcite . however, these datasets often have limitations such as small topic coverage, focus on specific framing effects, one-language news source, or lack of annotations at the word level. @xcite addressed the data gap by presenting the mbic (media bias including characteristics) dataset, which contains 1,700 statements annotated by ten annotators each. this dataset represents various instances of media bias, including framing and epistemological bias, and provides labels for bias identification at both the word and sentence levels. notably, mbic is the first dataset to include detailed information about annotator characteristics and their individual backgrounds, enhancing the reliability and interpretability of bias annotations. in the context of israeli-gaza war, there exist few reliable dataset that can be used to train ai models effectively. most existing datasets are sourced from one language and use two to three annotation labels. the co-authors of this paper focus their effort in establishing comprehensive guidelines for annotating biased news reports concerning the current conflict in gaza as pointed out in fignews 2024 shared task @xcite . section 2 describes the process of annotation including the annotators training and team coordination, section 3 discusses the team contributions and section 4 provides some concluding remarks and points to future work. ## results the shared task included 23 participating teams. only 17 teams qualified by finishing the required batches. results were published by the organizers and made public foot_1 . different measures were calculated to assess the quantity, quality, and centrality (consistency). our rank in the three measures ranges from 4 th to 8 th position across all measures. in the quantity track, we were placed at the 7 th position with 3900 points. we believe that teams would have been ranked better in this measure if effort per an annotator is considered as team size ranges between 2 to 21 annotator. in the quality track, we were ranked 4 th using the macro f1 average measure and in the centrality track we were ranked 4 th using the kappa measure. our average rank is 6 cross all ranks. if applied to all teams, we could have been ranked in the second position. ## discussion annotating bias in news articles is a complex and meticulous task that requires careful execution. despite the design of labels and provision of examples, annotators encountered several challenges that led to mistakes in labeling articles correctly. firstly, the annotation process itself is tedious, demanding both patience and concentration. the progress of annotators was notably faster during the initial rounds compared to the subsequent rounds, indicating the need for sustained effort and focus over time. secondly, the reliance on translation introduced additional difficulties. for example, the term "hamas," which refers to a leading palestinian organization in the conflict, also means "enthusiasm" in arabic. this dual meaning led to misleading annotations in some cases. for instance, in an article translated from english titled "cheering hamas because a released hostage smiled is a pretty low bar for humanity | haaretz today," the term "hamas" was misinterpreted as "enthusiasm" in the arabic version, resulting in annotation discrepancies. to address some of these challenges, we initially added two more labels to better capture the nuances in the text. however, we were later instructed by the organizers to remove these additional labels and re-annotate the articles using the original labels specified in the shared task. this re-annotation process underscored the importance of adhering to standardized guidelines to maintain consistency and accuracy in the annotations ## conclusions the university of tripoli's initiative to develop guidelines for annotating bias in news articles about the palestinian-israeli conflict has yielded significant progress in understanding and detecting media bias. by participating in the fignews 2024 shared task, we successfully annotated over 3,000 articles and created an annotation tool that nonexperts can use to identify bias effectively. our findings underscore the complexities involved in annotating bias, especially given the challenges posed by multilingual datasets and translation nuances. this effort highlights the necessity for clear, detailed guidelines to ensure accuracy and consistency in annotations. the collaborative aspect of fignews 2024 has been instrumental in achieving reliable inter-annotator agreement, which is crucial for the validity of the annotated data. this project makes a valuable contribution to natural language processing by offering new resources and insights into media bias detection. it lays the groundwork for future research and innovation, aimed at promoting balanced and fair news reporting. moving forward, we plan to further refine and expand our guidelines and tools, enhancing the capacity of researchers and practitioners to identify and mitigate bias in media narratives.