Bias Bluff Busters at FIGNEWS 2024 Shared Task: Developing Guidelines to Make Bias Conscious (2024)

Abstract: This paper details our participation in the FIGNEWS-2024 shared task on bias and propaganda annotation in Gaza conflict news. Our objectives were to develop robust guidelines and annotate a substantial dataset to enhance bias detection. We iteratively refined our guidelines and used examples for clarity. Key findings include the challenges in achieving high inter-annotator agreement and the importance of annotator awareness of their own biases. We also explored the integration of ChatGPT as an annotator to support consistency. This paper contributes to the field by providing detailed annotation guidelines, and offering insights into the subjectivity of bias annotation.

## introduction media bias detection is a rapidly evolving field dedicated to developing tools that help news producers, analysts, and readers identify and mitigate biased content. such biases can profoundly shape public perception and understanding of events. by improving transparency and promoting objectivity, these tools aim to foster a more informed and balanced media landscape, ensuring that diverse perspectives are fairly represented. however, the field faces significant challenges. data scarcity, lack of standard definitions for bias and annotation @xcite , and generally low annotator agreement hinder the development of effective bias detection models. existing large-scale datasets typically contain one to two thousand sentences annotated by experts or micro-jobbers @xcite @xcite . ## team composition and training the team (table . 1) comprises four female members residing in three different countries in western europe. two annotators are native italian speakers, while the others are native german and russian speakers, respectively. all members are fluent in english and hold ma, ba, or phd degrees in linguistics and computational linguistics. three annotators have no stake in the conflict or direct links to the involved parties or religions, whereas one annotator is originally from eastern europe and of jewish ethnicity. the team met weekly online to plan the work, discuss annotation cases, make guideline decisions, and coordinate. to familiarize themselves with the task, the annotators independently tagged 20 texts, followed by thorough discussions of the findings and uncertainties, which informed the initial drafting of the guidelines. during the annotation of the main section of the dataset, annotators reported ambiguous or edge case texts in a discussion document to ensure continuous alignment. communication was maintained through a group chat platform, enabling prompt feedback exchanges and fostering collaboration among team members throughout the project duration. ## task participation and results our team focused on developing robust guidelines, prioritizing the discussion of annotation challenges, documenting decisions on edge cases, and deriving general principles to help annotators recognize their own biases and make more grounded decisions. due to time constraints, we acknowledged the limitation in providing extensive training and alignment opportunities for better annotator convergence. one key takeaway from participating in the task was the realization that bias is inherent in human nature and in human-created products, such as chat-gpt. consequently, some level of bias in the annotators is unavoidable and will inevitably affect annotation decisions. a good practice, therefore, would be to have a large and diverse group of annotators to represent multiple perspectives. nonetheless, we found that explicitly identifying bias and propaganda indicators fostered a deeper sense of awareness and helped annotators make more informed decisions. the team annotated one-way the 1800 text from the main section of the dataset and 4-way 200 texts from the iaa section. looking at the main section, the labels assigned by the annotators show a skewed distribution. fig. 1 and 2 in c show that roughly one third of texts were considered 'unbiased' and another third as expressing 'bias against others', largely referring to hamas. texts biased against israel and palestine represented around 10% and 2.5% respectively. we also identified over 21% 'unclear' cases, partly covering texts expressing native gender age positive bias. ## conclusion in this paper, we have detailed our approach and findings from participating in the fignews-2024 shared task, focusing on the annotation of bias and propaganda in news texts. our contributions include the development of comprehensive guidelines that emphasize the need for annotator awareness of their own biases and the integration of linguistic clues to detect bias and propaganda. despite achieving moderate inter-annotator agreement, our work highlights the inherent subjectivity in bias annotation and the necessity for extensive training and alignment among annotators. our findings suggest that bias annotation guidelines 4 should be tested with a diverse pool of annotator to ensure high iaa is the product of quality guidelines rather then of annotators' homogeneity. future work should consider introducing labels for positive bias, refining annotation processes to account for different voices within texts, and addressing the challenges posed by translated texts and lack of context. this study underscores the complexity of media bias detection and the ongoing need for refined methodologies and diverse perspectives in the annotation process.