Ceasefire at FIGNEWS 2024 Shared Task: Automated Detection and Annotation of Media Bias Using Large Language Models (2024)

Abstract: In this paper, we present our approach for FIGNEWS Subtask 1, which focuses on detecting bias in news media narratives about the Israel war on Gaza. We used a Large Language Model (LLM) and prompt engineering, using GPT-3.5 Turbo API, to create a model that automatically flags biased news media content with 99% accuracy. This approach provides Natural Language Processing (NLP) researchers with a robust and effective solution for automating bias detection in news media narratives using supervised learning algorithms. Additionally, this paper provides a detailed analysis of the labeled content, offering valuable insights into media bias in conflict reporting. Our work advances automated content analysis and enhances understanding of media bias.

## introduction social media platforms and digital newspapers/articles are seen as a crucial source of information that nearly substituted conventional news media platforms on tv, radio, etc. despite widely acknowledged bias in news media sources, many individuals perceive information from such online platforms as reliable sources for major or current events/conflicts @xcite . people trust social media news as they present updates almost instantaneously, having a big impact on society and public opinion. in this paper, we introduce the fignews 2024 shared task (framing the israel war on gaza) @xcite , which aims at exploring news media narratives and biases on the israel war on @xcite . the goal of this task is to annotate and label a corpus that contains content from news headlines, articles, and social media posts into different bias categories. the labeling is based on a set of determined bias categories. this task intends to equip future natural language processing (nlp) researchers with the skills to be able to automate the annotation process by using supervised learning algorithms on media narratives. the israel war on gaza has brought the world's attention to media posts and news due to its nature of providing instantaneous, global updates. recent studies have emphasized the significance of addressing biases in news and social media through automation. for instance, lim (2020) talks about the difficulties in identifying bias in news stories by referring to published datasets. another study focuses on the development of a comprehensive methodology that automatically annotates news articles based on their bias @xcite . the study uses human-written guidelines for generating prompts, with algorithms and llm apis for processing and annotating. the purpose of this was to annotate a corpus and train the model to enhance the accuracy of bias detection. furthermore, spinde (2021) introduced a feature-based method for identifying terms in news articles that cause bias. in his paper, @xcite proposes a set of guidelines to enhance the use of llms for text annotation using chatgpt as the main prompt construction basis. moreover, in another study, @xcite explored the use of a generative pre-trained transformer in automating the social media annotation process by fine-tuning a model with expert-labeled guidelines to train the model. this model was developed to create a zerogap potential between human and automated annotation. in this paper, we we developed a robust framework and utilize openai's gpt-3.5 turbo api to automate the process of detecting and analyzing media bias concerning the israel war on gaza. by annotating sentences based on our written prompt with the llm model, we aim to streamline and enhance the accuracy of media bias detection. additionally, we present detailed guidelines developed to ensure consistency during both manual and automated annotation processes. ## team composition and training our team consists of three female members, all currently pursuing their master's degree at hamad bin khalifa university qatar from the college of science and engineering. while two of the members hail from south asia and speak hindi/urdu, the third member is from a gcc country and speaks arabic as her native language. all the members fall within the age range of 18-24. we all used the english translations of the statements to label them, except for one member who used arabic to analyze the arabic statements. in terms of the contribution, all the members contributed equally to the labeling (100 main and 200 iaa). while some of the manually labeled statements we labeled together as a team, others were equally split among us and checked by the other members for consistency and accuracy. our study involves two types of training. the first involves training the team members, and the second involves training the model. both were done as a team of three from the beginning of this project. all team members trained for the labeling together by establishing guidelines together based on the literature and further developing the guidelines based on the application of the guidelines on the statements and revising. the model was trained with the statements we labeled as a team and manually checked for accuracy. any errors were then reflected in the prompt with additional explanations. ## task participation and results the performance of the team was improved in various stages by having a well-organised plan and strategy, which was initialized by previous studies and by using the methodologies that was initiated in the studies. as stated previously, we developed the prompt using llms in conjunction with gpt-3.5. this was done in order to ensure that the annotation process is automated, which led to a better accuracy. the prompt consisted of specific examples for each label (biases categories), which helped enhanced the output. the results of our model are depicted in ## discussion analyzing the labels gives us insights into how different media represent the israel war on gaza. based on the 1800 statements labeled from the first two batches, we deduce the following: the summary graph in figure 1 indicates that around 40% of the analyzed statements were biased against palestine, while 19% were biased against israel, and only 29% were unbiased. this indicates a disproportionate representation of the war in media, which is problematic as it not only distorts the public perception of the war but is also unethical to provide unbiased reporting. this underscores the critical need for unbiased journalism. as seen from figure 1 , the topmost labels were biased against palestine, biased against israel, and unbiased. the word cloud of the top 20 words were analyzed in these three labels to gain insights into the prominent themes and biases. in the word cloud representing bias against palestine as shown in figure 2 , the term "hamas" is the most prominent. this prevalence indicates that most of the biases against palestine were linked to hamas, suggesting that the portrayal of hamas significantly contributes to the negative bias towards palestine. words such as "terrorist" and "attack" frequently co-occur with "hamas," reinforcing this negative perception. in contrast, the word cloud for bias against israel features terms such as "israel," "gaza," and "palestinian." figure 3 depicts the prominence of "gaza" and "palestinian" alongside "israel" suggests that the negative bias against israel often revolves around its actions and policies towards gaza and the palestinian people. words like "war," "attack," and "killed" also appear frequently, indicating that the discourse often highlights violent interactions and casualties associated with israel. for the unbiased category, figure 4 shows a balanced representation of terms related to both sides of the conflict, such as "israel," "hamas," "gaza," "war," and "people." the terms "live," "world," and "us" suggest a broader, more neutral discussion that encompasses various perspectives without showing explicit favor or prejudice towards either party.the portrayal of hamas and israel's actions in gaza significantly drives bias against palestine, while the unbiased category provides a comprehensive view that considers multiple aspects of the conflict. ## conclusion in this paper, we presented our methodology for automating the annotation/labelling of the bias categories to the dataset provided by the shared task. the team automated the annotation process using the prompt construction -llms in conjunction with chatgpt-3.5 by providing clear guidelines and examples to ensure consistency in the labelling of bias categories across different sets of sentences from several news headlines and media platforms. the methodology for automating the annotation process had limitations, including the model's time-consuming nature (1 hour for 500 statements), which limited the number of examples we could add. we reduced the number of examples to 10 and also incurred the cost of running the model due to its closed-source nature. in future work, we aim to improve the iaa and improve the prompt to get a better accuracy.