J ustice L eague at FIGNEWS 2024 Shared Task: Innovations in Bias Annotation (2024)

Abstract: In response to the evolving media representation of the Gaza-Israel conflict, this study aims to categorize news articles based on their bias towards specific entities. Our primary objective is to annotate news articles with labels that indicate their bias: “Unbiased”, “Biased against Palestine”, “Biased against Israel”, “Biased against both Palestine and Israel”, “Biased against others”, “Unclear”, or “Not Applicable”.The methodology involves a detailed annotation process where each article is carefully reviewed and labeled according to predefined guidelines. For instance, an article reporting factual events without derogatory language is labeled as “Unbiased”, while one using inflammatory language against Palestinians is marked as “Biased against Palestine”.Key findings include the identification of various degrees of bias in news articles, highlighting the importance of critical analysis in media consumption. This research contributes to the broader effort of understanding media bias and promoting unbiased journalism. Tools such as Google Drive and Google Sheets facilitated the annotation process, enabling efficient collaboration and data management among the annotators.Our work also includes comprehensive guidelines and examples to ensure consistent annotation, enhancing the reliability of the data.

## introduction in the realm of natural language processing (nlp), the annotation of datasets is a critical step in developing models that can understand and interpret human language accurately. several key studies have laid the groundwork for our approach. for instance, (2) @xcite explored the detection of hate speech against immigrants and women in tweets, providing insights into the nuances of hate speech and its implications. similarly, (3) mohammad et al. @xcite focused on detecting stance in tweets, which involves understanding whether a tweet is in favor, against, or neutral towards a given target, adding another layer of complexity to bias detection. the justiceleague team at fignews 2024 shared task has taken on the challenge of annotating news articles to categorize their bias towards specific entities involved in the gaza-israel conflict. this shared task is of paramount importance as it aims to shed light on the often subtle and nuanced biases present in media reporting, which can significantly influence public opinion and policy decisions. the title of our paper, "justiceleague at fignews 2024 shared task: innovations in bias annotation," reflects our commitment to adhering to the rigorous standards set forth by the shared task organizers. our objectives align closely with the goals of the task, which seek to enhance the understanding of media bias and contribute to the development of unbiased journalism. in the broader context of the nlp community, shared tasks and datathons have become instrumental in advancing research by providing standardized datasets and guidelines. for instance, previous efforts such as the semeval tasks have focused on various aspects of semantic analysis, while other initiatives have tackled sentiment analysis and hate speech detection. these endeavors have laid the groundwork for our current task, which specifically targets the identification of bias in news articles. our approach to the shared task diverges from previous efforts in several ways. (7) francisco-javier rodrigo-ginés et al. (2024) provide a systematic review of media bias detection methods, which has informed our categorization and analysis techniques. firstly, we have developed a comprehensive set of annotation guidelines that not only identify bias but also categorize it into specific entities, such as "biased against palestine" or "biased against israel." this level of granularity allows for a deeper analysis of the nature and extent of bias in media reporting. secondly, our use of collaborative tools like google drive and google sheets has streamlined the annotation process, ensuring efficient data management and facilitating real-time communication among annotators. to illustrate the effectiveness of our methodology, consider the following examples: these examples underscore the importance of our task and the need for a systematic approach to identifying and categorizing bias in news articles. the psychological aspects of language use, as detailed by ( 8 ) @xcite , also play a significant role in our annotation process. by contributing to the body of research on media bias, our work aims to promote transparency and accountability in journalism, ultimately fostering a more informed and discerning public. ## team composition and training our annotation team consisted of three computer science students from helwan university in cairo, egypt, each bringing a unique perspective to the task despite our shared academic background. leveraging our technical expertise, we approached the annotation of news articles with a methodical and analytical mindset, focusing on the systematic application of the bias categorization guidelines. before the annotation process, we conducted a focused training session to ensure a cohesive understanding of the guidelines and the nuances of each bias label. we emphasized the use of examples to clarify the criteria for each category, which was crucial for maintaining consistency in our annotations. the training also included discussions on the ethical considerations of bias annotation, reinforcing the importance of impartiality and accuracy in our work. to facilitate effective team coordination, we utilized digital tools such as google drive and google sheets for efficient data management and annotation tracking. a dedicated chat group was established for immediate communication and to resolve any annotation discrepancies. regular team meetings were held to discuss progress, address challenges, and refine our annotation approach, ensuring the reliability and precision of our bias categorization efforts. ## task participation and results the justiceleague team's participation in the fignews 2024 shared task was characterized by a meticulous and informed approach to the annotation of news articles for bias. as computer science students from helwan university, we leveraged our technical acumen to systematically apply the bias categorization guidelines, with a particular emphasis on the inclusion of examples in our annotation process. these examples served as critical benchmarks, guiding our team in maintaining a high degree of consistency and accuracy in our labeling decisions. our comparative analysis with related work within the nlp community further refined our strategy, allowing us to anticipate challenges and adapt our methodology to ensure reliable and consistent annotations. despite being our first annotation task, the outcomes of our participation demonstrated a strong alignment with the task's objectives, underscoring the effectiveness of our approach and the importance of well-defined examples in achieving high-quality bias categorizations. ## discussion our team's participation in the fignews 2024 shared task, despite being our first annotation endeavor, has yielded significant insights and contributions to the field of media bias annotation. the nuanced landscape of bias identified in the news articles we annotated, ranging from overt to subtle forms, aligns with the complexities previously documented in related work. for example, (7) francisco-javier rodrigo-ginés et al. (2024) provide a detailed review of media bias detection techniques, which has informed our approach. the nuanced landscape of bias identified in the news articles we annotated, ranging from overt to subtle forms, aligns with the complexities previously documented in related work. the pivotal role of including detailed examples in our annotation guidelines, which facilitated standardized and accurate bias categorization, underscores a best practice that can inform future tasks. our findings have profound implications for media literacy, critical news consumption, and the ethical development of journalism practices, as they provide a framework for discerning bias and promoting objective reporting. moreover, our comparative analysis with related work has refined our methodology, highlighting the value of learning from past efforts. the dataset, annotation guidelines, and examples we have generated are substantial contributions to the nlp community, offering empirical evidence and resources for advancing research in bias detection algorithms. in essence, our work has not only enriched the understanding of media bias but has also set a precedent for rigorous and informed annotation practices in this vital area of study. ## conclusion our participation in the fignews 2024 shared task has yielded significant insights and results that underscore the effectiveness of our approach to bias annotation in news articles. as a team of three computer science students from helwan university, we brought a unique perspective to this challenging task, combining our technical expertise with a nuanced understanding of media bias. our contribution of 2,400 annotated data points across two main batches and six inter-annotator agreement (iaa) batches represents a substantial input to the task dataset. this volume of work allowed us to gain deep insights into the nuances of bias in news reporting across multiple languages and contexts. the cornerstone of our success was our meticulous approach to developing annotation guidelines. by including detailed examples for each bias category, we created a robust framework that served as a clear and consistent reference for our team. this approach was instrumental in achieving our high interannotator agreement scores, culminating in our second-place achievement in the iaa-quality track. our impressive iaa metrics (kappa: 64.4, accuracy: 83.7%, macro f1 average: 63.8, f1 bias*: 73.6) reflect the consistency and reliability of our annotation process. our methodology was further strengthened by our thorough integration of related work within the nlp community. by critically examining previous efforts in bias detection and annotation, we were able to anticipate challenges and refine our approach accordingly. this comparative analysis not only enhanced the quality of our annotations but also contributed to the broader discourse on best practices in bias annotation. the official results provided fascinating insights into the distribution of bias across different languages and contexts. the prevalence of unbiased content (42.8% of all annotations) is an encouraging finding, suggesting that a significant portion of news coverage maintains neutrality. however, the high proportion of content biased against palestine (29.2%), particularly in hebrew-language sources (10.4%), highlights ongoing challenges in balanced reporting in certain contexts. the varying distributions of bias across different languages underscore the importance of multilingual approaches in bias detection tasks. our centrality metrics (kappa: 19.9, accuracy: 43.3%, macro f1 average: 19.6, f1 bias*: 46.5), while lower than our internal iaa scores, reflect the inherent challenges in achieving consensus across different teams in bias annotation. these results highlight the subjective nature of bias perception and the need for continued refinement of annotation guidelines and methodologies. as we reflect on our contributions, we recognize the broader implications of our work. by develop-ing robust annotation guidelines and demonstrating their effectiveness, we have contributed to the ongoing effort to improve media literacy and promote unbiased journalism. our dataset and methodologies offer valuable resources for future research in automated bias detection and analysis. moreover, our experience underscores the importance of interdisciplinary approaches in tackling complex challenges like media bias. as computer science students, we brought technical rigor to the task, but our success was equally dependent on our ability to understand and analyze the nuances of language and media representation. looking forward, we see several avenues for future research. there is potential for developing more sophisticated machine learning models for automated bias detection, building on the insights gained from our manual annotation process. additionally, further investigation into the relationship between bias and propaganda, as hinted at by our correlation analysis, could yield valuable insights for combating misinformation. in conclusion, our participation in the fignews 2024 shared task has not only produced valuable data and insights but has also demonstrated the effectiveness of a well-structured, example-driven annotation process. by combining technical expertise with a nuanced understanding of media bias, we have made a meaningful contribution to this critical field of study. as we move forward, we hope that our work will serve as a foundation for continued research and development in the crucial task of identifying and mitigating media bias.