SQU ad at FIGNEWS 2024 Shared Task: Unmasking Bias in Social Media Through Data Analysis and Annotation (2024)

Abstract: This paper is a part of the FIGNEWS 2024 Datathon Shared Task and it aims to investigate bias and double standards in media coverage of the Gaza-Israel 2023-2024 conflict through a comprehensive analysis of news articles. The methodology integrated both manual labeling as well as the application of a natural language processing (NLP) tool, which is the Facebook/BART-large-MNLI model. The annotation process involved categorizing the dataset based on identified biases, following a set of guidelines in which categories of bias were defined by the team. The findings revealed that most of the media texts provided for analysis included bias against Palestine, whether it was through the use of biased vocabulary or even tone. It was also found that texts written in Hebrew contained the most bias against Palestine. In addition, when comparing annotations done by AAI-1 and AAI-2, the results turned out to be very similar, which might be mainly due to the clear annotation guidelines set by the annotators themselves. Thus, we recommend the use of clear guidelines to facilitate the process of annotation by future researchers.

## introduction the fignews 2024 datathon shared task mainly aims to shed light on the bias and double standards found in the media coverage of the gaza-israel 2023-2024 war. the objective of this collaborative task is to build a shared corpus for comprehensive annotation by setting guidelines tailored to address conflicting views on such a sensitive topic. the task also helps build a generation of nlp researchers who will be able to handle raw data and set their own guidelines to analyze complex datasets. the paper adheres to the required title format and aligns with the shared task's goals by addressing the challenges of bias in media coverage of the gaza-israel war. it also considers the influence of language and culture on the type of bias found in the articles. through our participation, our team contributes to the development of a comprehensive corpus and guidelines to deal with media bias. the fignews 2024 datathon shared task requires the use of nlp on social media under two categories: bias and propaganda. the squad team focused on one subtask, which is detecting bias. the annotation process involved identifying bias against palestine, israel, both, others, or if the text is unbiased, unclear, or not applicable. furthermore, manual and automated techniques were used to detect bias in a dataset in different languages (english, arabic, french, hebrew, and hindi). scholars such as @xcite have previously attempted to explain the manual process of content analysis for identifying media bias. he mentioned that the analysis questions and hypotheses should be defined first. for our task, this step was already done by the fignews task organizers. the researchers divided the actual annotation process into two steps: inductive content analysis in which annotators annotate text without any previous knowledge, other than the analysis question. in the same sense, the squad team started to analyze texts manually by doing inductive content analysis for a part of the given dataset. deductive analysis in which annotators then set some rules to guide the annotation process, based on the findings of the inductive content analysis. @xcite addressed the squad at fignews 2024 shared task: unmasking bias in social media through data analysis and annotation asmahan al-mamari fatma al-farsi sultan qaboos university, oman sultan qaboos university, oman s133847@student.squ.edu.om s133941@student.squ.edu.om najma al zidjaly sultan qaboos university, oman najmaz@squ.edu.om in addition to manual content analysis approaches, there are also automated analysis methods. regardless, they have taken a limited view, simply equating media bias with "diverse opinions" @xcite rather than analyzing it comprehensively. an example of advanced automated approaches is those that aim to detect bias through the process of word choice and labeling (wcl). @xcite developed a method to detect possible biased words in news articles. their model inspected factors such as word sentiment, named entities, and parts of speech to flag biased language. the scholars managed to demonstrate that the method they developed was effective in identifying biased words. targetspecific sentiment classification (tsc) is another example of automated techniques used to analyze media bias. yu and jiang (2019) developed a tsc model using both textual and visual information from news articles. their model adapts a bertbased text encoder and a cnn-based image encoder to classify sentiment toward specific targets. the scholars found that their multimodal model outperformed text-only and image-only baselines. however, they note that tsc approaches still face challenges when applied to complex news texts. for this task, the squad team used the facebook/bart-large-mnli model through zeroshot classification to label the posts under the "main" category. the process will be further explained in the following section. ## annotation methodology and examples to ensure an accurate annotation process, clear guidelines were developed for this task. ## team composition and training squad team has two omani annotators specialized in english education at sultan qaboos university. both annotators are arabic native speakers and learners of english as a second language. they share an academic foundation stemming from english education, literature, and linguistics. in addition, they specialize in english which aids their linguistic analysis abilities. annotators underwent some consultation meetings to discuss the task, and they were held with some professors at squ specifically from the english department of college of arts and social sciences, including dr. najma al zidjaly who later became the supervisor of the team. some meetings were held to increase the annotator's awareness of the complexities of the israel-palestine conflict to recognize the different forms of bias that could exist in media. according to some researchers, increasing the knowledge about news events can be a major factor that helps in bias annotation @xcite ). in addition, nlp experts, such as professors and the task organizers, were contacted to provide guidance for the team members when needed. also, relevant work related to the nlp filed was used for reference when needed. the team members used online platforms such as whatsapp, virtual and face-to-face meetings, and email updates which facilitated the communication process for them. the coordinated efforts of the team were overseen by a supervisor to address any issues or concerns raised by any of the team members. furthermore, each member's role and responsibility were outlined to streamline decision-making processes. also, regular check-ins were scheduled to identify any issues and monitor the team's performance. ## task participation and results as our team contained two members only, we focused on the subtask related to detecting bias only. setting clear guidelines helped the team members carry out a smooth annotation process. in addition, the annotators believed that including examples was important to facilitate the understanding of different forms of bias. for example, it was agreed on by both annotators that the use of words such as "terrorists" or "monsters" to describe palestinian resistance groups is considered biased towards them. in addition, whenever the team faced any ambiguity, they maintained constant communication and referred to the guidelines they set earlier. based on the final results decided by the fignews organizers, it was found that the nlp tool we used to annotate the data in the "main" sheet scored a low centrality score on the kappa scale @xcite . regarding the iaa sheets, which were annotated manually, it was found the annotators had similar results under the different bias categories as shown in the following two tables. this is similar to @xcite case, where the annotation guidelines they set helped them boost their inter-annotator agreement up to 92%. ## discussion the team relied on a manual approach to annotate the iaa section of the data provided, however, as hamborg (2020) has pointed out, manual analysis is time-consuming. also, prior knowledge about the conflict was helpful in the bias annotation process as lim, jatowt, farber, & yoshikawa (2020) have mentioned. based on tables 1 & 2, the percentages for each bias category under aai-1 and aai-2 turned out to be similar, which proves the importance of having guidelines. both annotators found that the hebrew data contained the most bias against palestine, with over 50% in both cases. regarding bias against israel, there was a noticeable difference between the two annotators. for example, in the text "these are the idf's merkava tanks that caused devastation in gaza," one annotator found the text to be biased against israel while the other categorized it as unbiased. this might be due to misinterpretation of the guidelines at times. lastly, it was noticed that the data in french contained the lowest percentage of bias against palestine. overall, all languages showed a level of bias. it was noticed that the nlp model that was used in the "main" section of the data provided could not pick on context cues the way humans do. as advanced as the current models are, they don't have human-level comprehension. this supports @xcite beliefs that automated approaches take a limited view rather than analyzing the dataset comprehensively. moreover, algorithms don't always capture the intended meanings as their interpretations depend on the context, which aligns with yu and jiang (2019) findings. furthermore, the model we used does not detect sarcasm, irony, and other forms of figurative language well. also, the data we used was divided into chunks which made the process easier. analyzing unstructured text at a large scale might be more challenging. lastly, nlp models, including the one we used, provided us with output but without explanations of their reasoning. this can sometimes make it difficult to understand the findings. the team's guidelines can provide a framework for categorizing different forms of bias to facilitate further research on bias detection in nlp. in addition, this kind of task can provide the next generation with the skills needed to detect bias. also, identifying the limitations of nlp models can help scholars focus on developing them. ## conclusion in conclusion, the bias annotation task was done based on the guidelines set by the annotators, with reference to some related work. in this paper, the annotators discussed the methodology employed and the process followed to carry out this study. in addition, the findings demonstrate bias against palestine was found to be the category with the highest frequency regardless of the language of the text. overall, the use of examples is crucial to ensure consistency in the annotation process. based on the results concluded by the fignews organizers, the guidelines set by our team scored the highest rate. as for the quantity of the annotation, the team ranked 5th place out of the 16 teams, and our total data points were 4,400. however, our team did not score the highest overall ranking (11th place) @xcite , which may be due to scoring low on the kappa scale under some categories. lastly, future research could examine the inclusion of bias annotation techniques in nlp models to develop more bias-aware natural language processing systems.