Hate Personified: Investigating the role of LLM s in content moderation (2024)

Abstract: For subjective tasks such as hate detection, where people perceive hate differently, the Large Language Model’s (LLM) ability to represent diverse groups is unclear. By including additional context in prompts, we comprehensively analyze LLM’s sensitivity to geographical priming, persona attributes, and numerical information to assess how well the needs of various groups are reflected. Our findings on two LLMs, five languages, and six datasets reveal that mimicking persona-based attributes leads to annotation variability. Meanwhile, incorporating geographical signals leads to better regional alignment. We also find that the LLMs are sensitive to numerical anchors, indicating the ability to leverage community-based flagging efforts and exposure to adversaries. Our work provides preliminary guidelines and highlights the nuances of applying LLMs in culturally sensitive cases.

## introduction human evaluators from diverse backgrounds are necessary to provide coverage against hate speech, which the un defines as "any kind of communication that attacks or uses pejorative or discriminatory language with reference to a person or a group on the basis of who they are." variation in annotations with respect to demographics matters as it is reflective of their lived experiences. however, the background of the evaluators also contributes to annotation biases @xcite @xcite . on the other hand, the role of large language models (llms) @xcite to help content moderation is now being explored @xcite . motivation. in a recently proposed hate speech dataset crehate @xcite , annotators * equal contribution 1 disclaimer: the paper contains examples of strong and hateful language. this is how they work. they are domestic terrorists. they are taking over corporations world wide and nothing good will come of it. figure 1 : annotations of hate/non-hate (red/green) for usa, australia, uk, south africa, and singapore, by a) annotators from respective countries (circle) and b) prompting gpt-3.5 with 'the following statement was made @xmath0: <post>. is the given statement hateful?' (square). the posts and human labels are verbatim from the crehate dataset @xcite . we, thus, investigate variations in llm's output when primed with context, which, under a similar setting for humans, causes variability. in a twoparty setup consisting of a human and a zero-shot llm as annotators, we prompt the llm with contextual information and observe the inter-annotator variation (w.r.t human label) in the llm's output. borrowing from the demographic identities of crehate, we assess if llms qualify as a proxy for annotators of a given identity. the direct implication is obtaining cheaper and faster soft labels via llms for a demographic that is underrepresented among human annotators. we also investigate the role of geographical cues, similar to the use of geolocation metadata by online platforms, in annotating hate. apart from the post's content, numerical ## experimental setup this section outlines the datasets, models, and evaluation metrics employed in the study foot_0 . datasets. an overview of the datasets employed in this study is provided in evaluation metric. for p base , we have the ground labels (majority voted gold labels). so, we employ a weighted f1 score to compare the p base outputs. among the prompt variants in rqs, where we may have a direct ground truth, we analyze the performance disparities via the cohen-κ inter-annotator agreement (iaa) @xcite . it has been observed that iaa and f1 are positively correlated when the dataset is not skewed @xcite . in our case, the skewness is controlled by having samples that are almost equal in both classes (as noted in rectified scores for hallucination. as the aim is to know if the post is hateful, for our use-case, any output not in the form of 'yes/no', 'hate/nonhate', or 'hateful/non-hateful' can be considered as a 'hallucinated' label falling outside the range of the expected answers. we specify the prompt suffix 'answer in one word only.' we also perform a manual evaluation to access the unique outputs per setup, and where the output could be salvaged, they were updated. in line with the existing literature @xcite , after all filters, the outputs that still did not qualify as acceptable are discarded. recording the number of hallucinations (discarded outputs), we introduce 'rectified f1/iaa scores', score rectif @xmath1, where h and t are the hallucinated and total samples, respectively. any mention of f1 and iaa in our study means 'rectified weighted f1' and 'rectified iaa.' models. we begin with flant5 performance metrics on all shortlisted llms (with the total number of parameters in that model) are captured in we also examine language-specific llms to corroborate the role of zero-shot prompting setup of general-purpose llm (gpt-3.5) with cultural context vs language-specific models. the results ( prompts. tables 12 and 13 provide the list of verbalized prompts under english and multilingual settings, respectively. for the latter, after employing google translate, the sample prompts were verified by two linguistic experts. note on hate speech definition. existing literature has observed that providing explicit definitions to llms may not lead to performance gain @xcite . we, thus, conduct our experiments building on the latent notions of hate speech in llms. for a sanity check, we prompt flant5-xxl and gpt-3.5 with 'define: hate speech' and obtain following responses: background. humans from different countries are prone to variability when flagging the same post as 'hateful' or 'non-hateful' @xcite . given that ai-assisted content moderation systems often have access to geolocation and language markers along with a post, we investigate whether 'akin to humans, do geographical cues influence llm's predictions for an incoming hateful post.' setup. inspired by @xcite , we reproduce and extend their analysis on the language tag and multilingual datasets. as described in section 3, we modify their prompt to suit our format. we compare the change in iaa when considering the human annotator of the respective country vs. p base against the human annotator vs. p con + p base or p lang + p base . here, p @xmath3.', where @xmath4, australia (aus), united kingdom (uk), south africa (sa), singapore (sg)}. meanwhile, p @xmath5.', where @xmath6, french (fr), german (de), hindi (hi)}. for analyzing p con , we employ crehate for both flant5-xxl and gpt-3.5. for p lang , we employ the hasoc-2020 (german & hindi) and mlma (arabic & french) on gpt-3.5's english and language-specific prompting. insights. we discuss the results in two broad settings: for country and language tags. country. from figure 3 (b) with p con + p base , gpt-3.5 exhibits disparity in lack of context about social constructs of the global south (sa, sg) compared to higher alignment with so-called western nations (usa, uk, aus) @xcite @xcite . surprisingly, for flant5-xxl (figure 3 (a) ), we observe an improvement in all countries. it is further corroborated by significance testing on flant5-xxl, where we observe (appendix d) that adding the country cue leads to a notable change in output. one hypothesis for the difference in p con results can be the nature of training the llm. due to only instruction tuning of flant5-xxl, it develops country-specific bias implicitly from the training data; however, for gpt-3.5 the implicit bias is augmented with explicit human feedback. it surely calls into question how llm pretraining mechanisms impact the subjective (non-gleu) downstream tasks such as hate speech detection @xcite . language. from figure 3 (c), we conjecture that p lang nudges gpt-3.5 to language-specific subspaces leading to a visible higher iaa with ground truth labels when we add p lang . as we infer in appendix a, the delta increase in performance in p lang + p base vs. p base follows the same order of magnitude irrespective of whether the prompts are in english or the respective language. takeaways. firstly, our observations emphasize the fact that both corpus and paradigm in training play a role in the geographical sensitivity of the llms. judging why some cues and llms garner more improvement than others is challenging to access from prompting alone. this calls for more transparency in the llm training to decode the geographical bias during training epochs. secondly, as an actional insight, the findings encourage incorporating geographical cues in the zero-shot prompt to ensure higher human-llm alignment. interestingly, even if the prompt is in the respective language, explicitly nudging helps. background. socio-cultural experiences of humans color their outlook about hate and cause variation in annotations @xcite . however, without direct signals about mental state, we rely on demographic attributes as markers of human conditioning @xcite a.k.a personas. meanwhile, llms only possess statistical socio-cultural experience. when employing llms for crowdsourcing, we need to assess 'do llm's emulation of demographics at a fine-grained level leads to variations in annotating hate?'. we further hypothesize that differences in a vulnerable group's projection) can lead to variability in the hate perceptiveness of llms. we consider a vulnerable group as those who have historically been mistreated based on identity @xcite . setup. borrowing the annotator demographic list from crehate, we define our persona attributes in terms of @xmath7, ethnicity (d e ), political orientation (d p ), religion (d r ), education level (d q )}. each demographic in d * ∈ d has further sub-classes as outlined in intrigued by the success of the language tag in rq1 (section 4), we deep dive into persona traits that closely represent each language's demographic. here, ∀@xmath12, french, german, hindi}, we introduce prompts p l * @xmath13, and @xmath14.' in the prompt, i enlists a base/majoritarian persona vs. a vulnerable/minority persona of the respective geography examined on gpt-3.5. steps for obtaining the following personas are provided in appendix b. we examine the hateful (h) and non-hateful (n ) queries here as well, both with english and multilingual prompts. as none of the datasets provide demographic-specific labels, we rely on iaa among predicted and the majority-voted gold labels for our assessment. for p h/n trait , we provide in the prompt if the persona identifies a statement as hate/non-hate speech. our objective here is to assess the role of these traits in persuading llms to increase/decrease the number of hate labels in their responses. we thus utilize the predicted hate label ratio (phlr) metric in addition to iaa. insights. we discuss the results broadly for english and multilingual datasets. english: under gender demographic for gpt-3.5, the percentage of hate labels is higher for 'non-binary' than 'males' (for p h trait ). it aligns with the former being a more susceptible subclass of gender in the real world. meanwhile, for flant5-xxl, the presence of the non-hate tag p n trait dominates the demographic information in the context. however, the opposite is not valid for p h trait . despite the ground labels associated with samples being balanced across classes, p n trait for flant5-xxl still predicts the majority of the labels as 'non-hate.' multilingual. from figure 4 (a-b), we hypothesize that explicitly known vulnerabilities like islamophobia (arabic + muslim) and casteism (hindi + lower caste) are better captured by the llm than ethnicity, leading to higher sensitivity of these pairs towards hate when prompted in english. while the patterns persist under multilingual prompting for the rest of the languages, its lack of variation in p h trait hindi is puzzling ( as the highest gap in p * trait english is observed for german, we deep dive into this prompt (native speaker vs non-native speaker) and repeat this for all languages under english and multilingual prompting. from figure 4 (c-d ) and takeaways. given the sensitivity of llms toward the combination of persona + label, we urge the practitioners not mindlessly to rely on llm as a substitute for demographic attributes. the persistence of results in english and multilingualism is an advantage for the researchers working on multilingual datasets. it can allow faster assessment of the llms without requiring separate prompt engineering for both setups. background. anchoring bias occurs when humans rely too heavily on the anchor information (relevant or not) to influence their decision-making. while some anchors like language and geolocation, as established in rq1 (section 4), are helpful for llmbased content moderation, the influence of numerical cues under the zero-shot setting is unknown. numerical features/cues can be defined as aggre-gated real-world or simulated values representing countable metadata associated with the posts. it can range from the total/average views, likes, and comments a post/user receives to the number of people who have in the past reported/flagged the post/user as hateful. the former set of metadata is publicly available and has been successfully employed in hate speech detection to improve finetuned accuracy @xcite . meanwhile, the influence of previously flagged counts is not known, as such metrics are not available to the public. here, using voting statistics as a proxy for the crowd's opinion about the post, we are motivated to examine whether 'made-up voting percentages in the prompt lead to manipulation in llm's output?' setup. the base prompt p base , is prefixed with p h/n @xmath15.', where @xmath16, 25%, 50%, 75%, 100%}. the two variants p h/n vote capture the hateful (h) or non-hateful voting (n) label. we represent z in the percentage to give a relative sense of majority voting. at @xmath17, 50%, annotators consider the post is hateful, and 50% say it is non-hateful is not the same as p base . we conduct this experiment with hatexplain on flant5-xxl and gpt-3.5 with p vote + p base . the z% alludes to annotators in general and not a specific persona. further, it should be noted that these percentages are not available as a part of hatexplain, and added by us to introduce the "anchoring" information. insights. in line with existing evidence @xcite of majority labeling bias in few-shot learning, we also establish that llms are prone to labeling bias even under zero-shot settings if the context mentions voting percentages. from figures 5 (a-b ) regarding p h vote , it is evident that alignment in hate labeling is more consistent when the percentages lie closer to each other and decrease as one moves away. succinctly, 5 (c-d )). takeways. corroborated by significance testing (appendix d) and controlling for decoding temperature (appendix c), we discover that while llms understand relative percentages, they are prone to emphasize this information over the post's content. as in our rq, voting serves as a proxy of numerical metadata in the content moderation pipeline; it implies that corruption (accidental or intentional) can lead to instance misclassification. despite numerical/statistical features being used earlier to enhance the finetuning in hate speech detection @xcite , they cannot be directly extrapolated as features in prompting for hate speech annotations. ## discussion this section summarizes the implications of the research. we also provide recommendations for llm-assisted content moderation. except for the flant5-xxl non-hate persona, the results across models, datasets, languages, and prompts are aligned. cognizant of adversarial attacks on prompting, the study balances both feature importance (corroborated by significance testing) and cautions against adversary features. despite how the perception of hatefulness manifests via p base , we observe that the inclusion of cues nudges the llms to explore a contextual definition of hate. geographical sensitivity. how the latent spaces are triggered at the mention of geographical cues @xcite is intractable from prompting alone. this becomes especially tricky for overlapping signals like 'arabic,' 'muslims,' and 'islamophobia' (figure 3 (c) ). it once again highlights the need for more transparency in the training and cultural alignment of llms. as a quick fix, we suggest augmenting the language/country tags for an incoming post being prompted to improve alignment with people representative of a given region/geography. demographic sensitivity. our work establishes that the manner of personification of demographic attributes can lead to variation in hate labeling, with assumed persona (p a trait ) being closer to the inherent knowledge and biases an llm possesses. here, we caution practitioners looking to adopt llms for crowdsourcing to experiment with different framing of the personas and identity traits. numerical sensitivity. as evident from our experiments, llms do not have a clear way of discovering noise from the informativeness in the prompt. our advice to practitioners is to refrain from adding any numerical statistics in the prompt unless they are quality-checked and to mask such adversarial expressions written by the authors of the input samples. ## conclusion in the current llm space, both alignment and robustness do not have a singular non-overlapping definition, which makes our study sit at the intersection of robustness in hate speech detection as well as conceptualization of the human-llm alignment. over multiple rqs and prompt setups, we explore how well-suited llms are for assisting humans at different stages of the content moderation pipeline. our results on demographic sensitivity, cultural priming, and anchoring bias are evident over multiple annotations on 6 datasets, 5 languages, and 2 llms. our analysis reiterates using llms as an assistive system rather than substituting human moderators. we would like to work on more datasets and explore intersectional demographic attributes in the future. 15855 first and foremost, the list of research questions and prompts analyzed, while significant in number, is not exhaustive. given that the adoption of llms in the hate speech moderation pipeline is nascent, the scope for research is more vast than what we can account for in current research. we hope our findings and primary analysis motivate future research. besides, there is a shortage of opensource instruction-tuned multilingual llms, which restricts our multilingual and multicultural analysis to gpt-3.5. owing to the lack of datasets with ground truth labels where multiple annotators are considered and demographic-specific annotators are released, performing one-to-one mapped interannotator agreement analysis between llms and demographic personas is challenging. the current study only utilizes textual content. in the future, we need to conduct assessments for content moderation on other modalities like memes, short videos, etc. it should be noted that geography or language only acts as a proxy for cultural and linguistic recalls and does not represent an absolute assessment for multilingual llm evaluations. consequently, when employing geographical or demographic attributes, we need to be cognizant of the fact that the sensitivity of the llms can also be a source of their implicit bias.