BSC - LANGTECH at FIGNEWS 2024 Shared Task: Exploring Semi-Automatic Bias Annotation using Frame Analysis (2024)

Abstract: This paper introduces the methodology of BSC-LANGTECH team for the FIGNEWS 2024 Shared Task on News Media Narratives. Following the bias annotation subtask, we apply the theory and methods of framing analysis to develop guidelines to annotate bias in the corpus provided by the task organizators. The manual annotation of a subset, with which a moderate IAA agreement has been achieved, is further used in Deep Learning techniques to explore automatic annotation and test the reliability of our framework.

## introduction being the first and primary source of information, online news articles play a key role in how people shape their opinions and engage with topics of societal relevance @xcite . however, media coverage is often far from being unbiased. considering the impact of media bias on public opinion, @xcite propose framing the israel war on gaza: a shared task on news media narratives @xcite . this shared task focuses on exploring the media narratives and digging deep into the biases and propaganda in news articles about the gaza-israel 2023-2024 war from multiple perspectives, cultures and languages @xcite . the main objective is to develop guidelines to annotate a corpus of news posts. such corpus was compiled from international news article headlines and advertising posts from facebook in the languages above mentioned. the posts selected date from october 1, 2023 to january 31, 2024, and they all include the word query "gaza" in any of the different languages. in addition, the corpus also contains machine translations into english and arabic for all posts. with this aim, this shared task proposed two possible subtasks to focus: annotating either bias or propaganda. as the title suggests, our work uses practical conceptualisations from framing theory to define each label and constrain corner cases within * equal contribution the bias annotation task; and leverages encoderdecoder classification models to aid the annotation process, while also being useful to evaluate the robustness of the proposed annotation framework within the automatic modelling of media framing. efforts to annotate and detect bias have been already made @xcite @xcite , some of them focusing on the palestinian-israeli conflict @xcite @xcite . in addition, some shared tasks have also been devoted to delve into bias in news articles, like the clef-2023 checkthat! shared task @xcite . however, most of these works do not include detailed information about the annotation, and the guidelines used are not publicly available neither. in this work, we apply the theory and methods of framing analysis to develop guidelines for bias annotation in a specific topic. specifically, we combine linguistic frame indicators from @xcite and narrative framing from @xcite to solve the bias detection task as a singlelabel classification problem. computational approaches in framing device annotation are numerous, but neural network models have been the most popular approach for the most recent years, including bert @xcite and roberta @xcite classification models @xcite . for this reason, we decide to explore automatic multi-label prediction of our frames using the base version of roberta model @xcite , which allows us to test the reliability of our framework. the paper follows with section 2, which explains the guideline creation and data annotation process, and discusses the application of the proposed framework to the dataset. section 3 introduces the team of annotators and the quality check procedures used to ensure consistency and expectations between annotators. section 4 discusses the inter-annotator agreement (iaa) results and the experiments with the supervised prediction of our labels. ## team composition and training annotator demographics. both annotators in charge of the whole annotation task, a man and a woman, are also the two main authors of this paper. we fall within the age range of 18 to 24 years old, and we share a cultural, linguistic and educational background. coming from spain, our native language is spanish, and we both have a translation and linguistics background, now working in the area of nlp. ## task participation and results our team processed a total of 2,200 instances across 2 main batches, where the iaa subset of each batch, corresponding to 200 out of 2000 instances, was annotated individually by each annotator. centrality measures show agreement across different teams in both the main and iaa annotations, where our kappa ranked 5 th place with a 26.5%. our accuracy ranked 1 st place with a 46.6%, and a macro f1 average of 29.2% in 5 th place, and an f1 bias of 60.2% in the 9 th place. these centrality measures highlight a high overall accuracy, while also shows the challenges in creating annotations that are consistent across other guidelines. regarding the automatic annotation, however, the overall agreement between the automatic and human annotations is lower (39.27% for model_1 and 62.83% for model_2) due to the complete lack of agreement in the remaining classes, showing that the models were not able to learn them. this could be due to the small amount of data on which the model was trained and its unbalanced label distribution, where for model_1 the label not applicable was significantly less present than biased and unbiased. similarly, in the training data of model_2, biased against both, biased against others and unclear are less prominent than the others. ## discussion our goal was to define annotation guidelines for identifying media frames that can be used to detect bias as a single-label classification problem. to this end, we adapt a set of linguistic and narrative frame indicators from the discussed literature into a deductive framework. we then apply the framework to annotate the fignews 2024 corpus, which contains news articles about the gaza-israel war of 2023-2024. our proposed framework was used to annotate and explore supervised deep learning techniques using roberta models, which showed a higher correlation with human annotation in the most prominent labels of the training dataset. however, our contribution faces some limitations: the annotation task was fully carried out by the same people developing the guidelines. introducing some external annotators not familiar in the task would be helpful to get a deeper insight of the explainability and consistency of the guidelines. regarding the iaa results, our annotations achieved only a moderate agreement in the iaa analysis. a high level of agreement can be challenging in such complex annotation tasks, but our result suggests a meaningful alignment in the interpretations of the annotation guidelines. moving forward, we can further refine our guidelines and potentially enhance agreement in future annotations. in addition, among the 15 batches available, only the first 2 of them have been annotated and used in this work. having limited data has also constrained the results of our fine-tunings. having a larger amount of training data would improve the performance of the models. in relation with the fine-tuning experiments carried out, we could explore further the impact of training with a different set of hyperparameters, models or even training data scenarios. ## conclusion our work contributes to the creation of detailed, consistent guidelines considering both theory and methods of framing analysis for a complex annotation task such as the presented in the fignews 2024 shared task on news media narratives. more specifically, we have developed guidelines to annotate bias on news posts about the gaza-israel 2023-2024 war. in addition, we have explored stateof-the-art deep learning techniques to explore the automatization of the annotation task to easen it. of press and television news. journal of communication, 50(2):93-109. thomas wolf, lysandre debut, victor sanh, julien chaumond, clement delangue, anthony moi, pierric cistac, tim rault, rémi louf, morgan funtowicz, et al. 2020. transformers: state-of-the-art natural language processing. in proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pages 38-45. wajdi zaghouani, mustafa jarrar, nizar habash, houda bouamor, imed zitouni, mona diab, samhaa r. el-beltagy, and muhammed abuodeh, editors. 2024. the fignews shared task on news media narratives. association for computational linguistics, bangkok, thailand.