L exicon L adies at FIGNEWS 2024 Shared Task: Identifying Keywords for Bias Annotation Guidelines of F acebook News Headlines on the I srael- P alestine 2023 War (2024)

Abstract: News bias is difficult for humans to identify, but even more so for machines. This is largely due to the lack of linguistically appropriate annotated datasets suitable for use by classifier algorithms. The FIGNEWS Subtask 1: Bias Annotation involved classifying bias through manually annotated 1800 headlines from social media. Our proposed guidelines investigated which combinations of keywords available for classification, across sentence and token levels, may be used to detect possible bias in a conflict where neutrality is highly undesirable. Much of the headlines’ percentage required contextual knowledge of events to identify criteria that matched biased or targeted language. The final annotation guidelines paved the way for a theoretical system which uses keyword and hashtag significance to classify major instances of bias. Minor instances with bias undertones or clickbait may require advanced machine learning methods which learn context through scraping user engagements on social media.

## introduction headlines are the first and often the only element of news which readers interact with @xcite , especially on social media. what this meant for online and social media journalism was that readers could now filter their 'feeds' or 'timelines'. most news is not viewed by a general audience anymore and instead are pushed by algorithms further or closer to the viewer @xcite , depending on their preferences and ideologies. from there we could explore how journalists and public figures used targeted language to cater to specific groups or parties (in the case of political news coverage) or to attempt to reach and influence the mindset of other groups. such endeavor is referred to as strategic communication and is a contributor to media bias @xcite . to track news bias, researchers often devise their own datasets. many of the available datasets focus their efforts on bias detection on the document level, by news outlet source, or by using automated methods which lack proper inference of implicit bias. examples of such methods, and perhaps the closest and most elaborative studies on keyword classification of news sentiment (leading to bias) are the works including hamborg, spinde, and gipp 2021. in their often combined research efforts on news bias in conventional news @xcite on social media @xcite , they bridged the context gap between news outlets and readers. one of their methods involved building a keyword lexicon to utilize in the training of word embeddings. attempts to identify bias triggers discovered the possibility of relying on linguistic features for shallow or simple bias detection @xcite , providing some characteristics of bias-inducing words. @xcite crowdsourced again a dataset of news, this time consisting of 966 sentences for 4 different events, concluding that prior knowledge about the context surrounding news is a major factor in bias detection. context is a difficult subject. it cannot be easily inferred from specific keyword presence only. ultimately, it depends on the readers' ideologies, stances, and knowledge @xcite . research on the biased language of reporting on the palestinian-israeli conflict has been studied in several attempts, however, the resulting datasets are relatively small and may not be sourced from a variety of news sources @xcite @xcite . better bias detection classification models require valid training and evaluation of diverse corpora of news on the conflict. this paper tries to explore the development of bias recognition specifics for this conflict, especially on social media. the fignews subtask 1: bias annotation was done through a linguistic viewpoint, in an attempt to compile a sound semantic and contextually backed understanding of the discourse differences surrounding the israeli war on the gaza strip @xcite . one important aspect of the dataset's headlines is that they are extracted from social media, namely facebook. social media news has the additional feature of permitting user interaction (when applicable) under news to contextualize reporting. depending on the audience reaction, biased news may be balanced to objectivity through user input or they may be further amplified towards the biased direction they intended @xcite . facebook as a platform has long been complicit in giving emphasis to israeli and zionist news and voices @xcite . the biased content moderation of palestinian content has been studied during several israeli campaigns on palestinian territories, such as the sheikh garrah crisis @xcite and the guardian of the walls operation @xcite . the extent to which social media sites seek to censor or "reduce the visibility" @xcite of certain ideologies, including pro-palestinian posts and headlines, has caused several commentators to similarly censor their intentions while maintaining bias towards the palestinian cause. several headlines required external knowledge of the surrounding events and context to correctly label them as biased @xcite . therefore the task is pragmatic by nature, with semantic considerations to the underlying meanings of words and expressions, as well as media literacy and considerations of culture and audience @xcite . most of the previous attempts at annotating datasets regarding news headlines focused on credibility and misinformation identification @xcite . while the aspect of veracity is a part of bias detection, our focus is on the required knowledge of the language used to identify biased headlines. as language models do not fully acknowledge cultural context, and rather measure the statistics surrounding entities and events, we attempt to illustrate a theoretical system based on overall sentence sentiment and token-level signs that may be of use to build models of better bias classification. we ask the following: if a system were to analyze the resulting short (headline) and long-form (posts) news, what combination of keywords would be required to infer the implicit bias or lack thereof? furthermore, how much does world context, user engagement with social media news, and media literacy affect the overall detection of possible bias? ## team composition and training the annotation team responsible for the task consisted of 3 women (ages range from 22 to 24, "generation "z"") and recent graduates in applied linguistics with a native arabic background of different dialects. all members were proficient in english and two annotators had working knowledge of french. personal preference was granted for the choice of language to annotate headlines in. most of the annotations were done in the mt english version, therefore the final guidelines used english keywords to recognize and filter possible bias. two annotators had very little annotation experience prior to the task. further training included offline discussions on labeling specifics and deciding on where to scan headlines for potential bias inducing words. ## task participation and results we tried to produce results of higher quality than quantity, in which we implemented our contextual and linguistic knowledge to the labeling efforts. this knowledge included: 1. known social media engagement tactics, 2. known political and humanitarian stances of news outlets and countries, and 3. one's own bias regarding the war. the highest recorded label was "biased against palestine", with a percentage of 34% out of the total 1800 annotations (figure 1 ). it was followed by "unbiased" (27%), "biased against israel" (16%), "unclear" (8%), "biased against both palestine and israel" (6%), "biased against others" (6%), and "not applicable" (3%). our guidelines set out to explore if there was a method to detect possible bias amidst a conflict of high sensitivity and relatively unfair coverage in non-arabic news. the keywords we decided on within the guidelines included words of reference to two main actors in the conflict: the israeli forces and the palestinian movement. these words and the entities or events they refer to can be found in table @xcite . we analyze the collocation frequencies of each of the words provided and assign them bias labels, which is further explored in the next section. the guidelines helped reached an f1 centrality score of 33.1 across all other participating teams' main batches 1 and 2. this score was the highest in terms of bias annotation consistency. ## conclusion we tackled the task with a linguistic perspective in hopes of offering a better understanding of the pragmatics that surround bias detection. due to the task's underlying complexity even to people, having machines perform statistical classifications on such sensitive matters may carry turbulence and misguidance. language inference is closely connected to adherence to cultural and world context. while the original notion of journalism is objectivity, that becomes increasingly difficult as the news covered becomes more sensitive to every party involved. we found that the polarity of the conflict has generated keywords which have a high possibility of better bias detection, yet a headline may be further flawed by contradictory hit word injection, or in the case of social media, by inclusion of irrelevant hashtags.