Probing LLM s for hate speech detection: strengths and vulnerabilities (2023)

Abstract: Recently efforts have been made by social media platforms as well as researchers to detect hateful or toxic language using large language models. However, none of these works aim to use explanation, additional context and victim community information in the detection process. We utilise different prompt variation, input information and evaluate large language models in zero shot setting (without adding any in-context examples). We select two large language models (GPT-3.5 and text-davinci) and three datasets - HateXplain, implicit hate and ToxicSpans. We find that on average including the target information in the pipeline improves the model performance substantially ( \sim20-30\% ) over the baseline across the datasets. There is also a considerable effect of adding the rationales/explanations into the pipeline ( \sim10-20\% ) over the baseline across the datasets. In addition, we further provide a typology of the error cases where these large language models fail to (i) classify and (ii) explain the reason for the decisions they take. Such vulnerable points automatically constitute ‘jailbreak’ prompts for these models and industry scale safeguard techniques need to be developed to make the models robust against such prompts.

## introduction abusive language has become a perpetual problem in today's online social media. an ever-increasing number of individuals are falling prey to online harassment, abuse and cyberbullying as established in a recent study by pew research @xcite . in the online setting, such abusive behaviours can lead to traumatization of the victims @xcite , affecting them psychologically. furthermore, widespread usage of such content may lead to increased bias against the target community, making violence normative @xcite . many gruesome incidents like the mass shooting at pitts-burgh synagogue foot_0 , the charlottesville car attack foot_1 , etc. have all been caused by perpetrators consuming/producing such abusive content. in response to this, social media platforms have implemented moderation policies to reduce the spread of hate/abusive content. one important step in content moderation is filtering of abusive content. a common way to do this is to train language models on human annotated contents for classification. however there are challenges in this approach in the forms of heavy resources required in terms of labour and expertise to annotate these hateful contents. this exercise also exposes the annotators to a wide array of hateful contents that is almost always psychologically very taxing. therefore, recently, many works have tried to understand if large language models (llms) can be used for detecting such abusive language 3 @xcite , but none of these study the role of additional context as input (output) to (from) such llms. we, for the first time, introduce several prompt variations and input instructions to probe two of the llms (gpt 3.5 and text-davinci) across three datasets -hatexplain @xcite , implicit hate @xcite and toxic-spans @xcite . note that all these three datasets contain ground truth explanations in the form of either rationales @xcite or implied statements @xcite that tells why an annotator took a particular labelling decision. in addition, two of the datasets also contain the information about the target/victim community against whom the hate speech was hurled. in particular, we design prompts that contain (a) only the hate post as input and a query for the output label (vanilla) (b) post as well as the definition of hate speech as input and a query for the output label (c) post (-/+ definition) and the target community information as input and a query for the output label, (d) post (-/+ definition) and the explanation as input and a query for the output label, (e) post (-/+ definition) as input and a query for the output label and the target community, and (f) post (-/+ definition) as input and a query for the output label and the explanation. we record the performance of all these approaches and also identify the most confusing cases. in order to facilitate future research, we further provide a typology of the error cases where the llms fail to classify and usually provide poor explanations for the classification decisions taken. this typology would naturally constitute the 'jailbreak' prompts 4 to which such llms are vulnerable thus pointing to the exact directions in which industry scale safeguards need to be built. we make the following observations. ## related works large language models: based on the training setup, the model architecture and the use cases llms can be broadly classified into encoder-only, encoder-decoder based and decoder-only types. in recent years decoder-only llms foot_5 have seen a huge surge with industry scale releases like chatgpt, gpt-4, bard, llama, etc. decoder-only llms have been used for benchmarks like glue @xcite where there is no downstream application involved. in the classification setting llms have been extensively used for sentiment analysis @xcite . they have also been heavily used in nli and qa tasks on multiple datasets @xcite . in the generation setting, llms have found applications in summarization @xcite , machine translation @xcite and open-ended generation @xcite . llms for hate speech detection: in @xcite , the authors use chatgpt to relabel various datasets one of which is on hate speech detection and found that the agreement with human annotations is still quite poor. the authors in @xcite use chatgpt to classify a comment as harmful (i.e., hateful, offensive, or toxic -hot) and found that the model is better in identifying non-hot comments than hot comments. finally in @xcite the authors attempted to classify implicit hate speech using chatgpt. however, their prompt was framed as a 'yes/no' question (rather than based on the exact classes i.e., implicit hate, explicit hate, non-hate as in the original study @xcite ) which makes the problem lose its original fervour. ## datasets and metrics for all these datasets, we utilise (create) a test dataset for our experiments. implicit hate dataset: the implicit hate @xcite corpus is a specialized collection of data aimed at detecting hate speech. it provides detailed labels (implicit_hate, explicit_hate or non_hate) for each message, including information about the implied meaning behind the content. the dataset comprises 22,056 tweets sourced from major extremist groups in the united states. we test on a subset of 2147 samples (108 entries labelled explicit_hate, 710 entries labelled implicit_hate, and 1329 entries labelled not_hate) which are sampled from the entire dataset in a stratified fashion. note that we do not have explanations and targets for all the posts. the implied statements and targets were available only for the samples with label implicit_hate. hence, when we experiment with explanation as input (see section 5), we pass the input post as the implied statement. this is for the explicit_hate and not_hate data points. both these categories do not required any additional implied statement as there is nothing implied in them. in case of targets as inputs (see section 5), we remove the explicit_hate datapoints since they are targetting some victim community but the targets are not present in the annotated dataset. the targets for not_hate are set as 'none'. hatexplain dataset: hatexplain @xcite is a benchmark dataset specifically designed to address bias and explainability in the domain of hate speech. it provides comprehensive annotations for each post, encompassing three key perspectives: classification (hate speech, offensive, or normal), the targeted community, and the rationales -which denote the specific sections of a post that influenced the labelling decision (hate, offensive, or normal). we test on the already released test dataset containing 1924 samples (594 entries labelled as hate speech, 782 entries labelled as normal, and 548 entries labelled as offensive). note that we do not have rationales for the normal posts. in the explanations as input experiments (see section 5), the complete post (tokenized) is taken as their rationale for the normal posts. toxicspans dataset: the toxicspans (pavlopoulos et al., 2022) dataset is a subset (containing 11,006 samples labelled toxic) of the civil comments dataset (1.2m posts). the dataset also contains the toxic spans, i.e., the region of the texts found toxic. not all the posts had toxic spans annotated. we create a test set of 2000 samples by picking 1000 samples labelled toxic from this dataset (where the toxic spans were available), and 1000 samples labelled non-toxic from the civil comments dataset @xcite . note that we have the spans/rationales marked only for the toxic data points. for the non-toxic posts for explanations as input experiments (see section 5), the complete post (tokenized) is taken as the rationale. metrics: for primary evaluation, we rely on classification performance. we use precision, recall, accuracy and macro f1-score to measure classification performance, which are all standard metrics. the data points which have rationales are additionally evaluated using other generation metrics. for the natural language explanations (implicit hate dataset) we use bertscore averaged over all data points. bertscore @xcite computes a similarity score for each token in the candidate sentence with each token in the reference sentence. however, instead of exact matches, they compute token similarity using contextual embeddings. for extractive explanation (hatexplain/toxicspans dataset), we use average sentence-bleu @xcite score which is the standard among the generation metrics. ## models for our experiments we utilise three llms. two of these are from the proprietary gpt-3.5 model series foot_6 while the third is an open source one from the t5 series. gpt-3.5 models are better than their predecessors gpt 3 @xcite . both of these models are highly advanced language models capable of generating human-like text based on the provided prompts but they differ in some key ways. as per their documentation foot_7 , gpt-3 was optimized on code completion tasks to create code-davinci-002. this was further improved using instruction finetuning @xcite to create text-davinci-002. this was later upgraded to text-davinci-003 which was trained on a larger dataset (ope) making it better at higher quality text generation, following instructions and generating longer context. gpt-3.5-turbo-0301 is an improvement over text-davinci-003. we choose these two models in our study namely the gpt-3.5-turbo and the text-davinci-003. the third model we use is the open source flan-t5-large which is an instruction finetuned variant of the popular t5 model @xcite . as per their documentation the model was instruction finetuned with an emphahsis on scaling the number of tasks, scaling the model size and introducing chain of thought data in the finetuning pipeline. the authors have claimed that this particular sort of finetuning has benefited the t5 models greatly by outperforming models of higher size like gpt-3. ## prompts in this section, we list the prompt variations used in this work. a concise summary of the different variants is noted in the appendix a ( ## results in this section, we note the results of the different prompt variations on the models for the three datasets. as a baseline, we consider bert-hatexplain @xcite ) and run the model on the implicit hate and toxicspan datasets. we have taken the results for the hatexplain dataset from the original paper. our key results are noted in tables 3, 4 and 5 for the hatexplain, implicit hate and toxicspans dataset respectively. ## error analysis and attack points in this section we shall first discuss the cases where the models encounter a large number of misclassifications and then outline a technique to induce a typology of the different attack points to which the models are vulnerable. significance test: we compare the results of the vanilla model for each dataset with the best performing prompt plus model combination using the mann-whitney u test. we find that the results are significant. we next present the p-values for each -(a) gpt-3.5-turbo-0301 + hatexplain: @xmath0.0042 (vanilla vs targetat-input), (b) gpt-3.5-turbo-0301 + implicit hate: @xmath1.0 (vanilla vs target-at-input), (c) gpt-3.5-turbo-0301 + toxicspans: @xmath2.00088 (vanilla vs explanation-at-input), (d) text-davinci-003 + hatexplain: @xmath3.946e-1 (vanilla vs target-at-input), (e) text-davinci-003 + implicit hate: @xmath4.441e -16 (vanilla vs targetat-input), (f) text-davinci-003 + toxicspans: @xmath5.01524 (vanilla vs definition-at-input), (g) flan-t5-large + hatexplain: @xmath6.0 (vanilla vs definition+target-at-input), (h) flan-t5-large + implicit hate: @xmath7.000358 (vanilla vs definitionat-input), (i) flan-t5-large + toxicspans: @xmath8.0 (vanilla vs explanation -at-input). cases of misclassification: for the implicit hate dataset we observe that in case of the gpt-3.5-turbo-0301 model, for all the different prompt variants the largest number of misclassifications is from the non-hate to the implicit hate class. for the text-davinci-003 model the major observations are as follows. in the vanilla setup (with or without definition), the implicit hate class is most often confused with the explicit hate class. however, if the prompt has an explanation component (either at input or at output), once again, there is a large number of misclassifications from the nonhate to the implicit hate class. for flan-t5-large we observe that the model fails to classify the implicit hate speech class. the implicit hate is classified as non-hate or explicit hate following the trend of the other two models. for the hatexplain dataset we observe that in case of the gpt-3.5-turbo-0301 model, for all the different prompt variants the largest number of misclassifications is from the normal to the offensive class. another confusion that the model often faces is between the hate and the offensive class; if the prompts do not contain the definition (of hate/offensive speech), then offensive speech is largely mislabelled as hate speech while the results are exactly reversed if the prompts contain the definition. for the text-davinci-003 model once again we observe for all the different prompt variants the largest number of misclassifications is from the normal to the offensive class. further, irrespective of whether the prompts contain the definition or not, hate speech is heavily mislabelled as offensive speech. for flan-t5-large both offensive and hatespeech classes are missclassified as normal speech for all prompt variations. for the toxicspans dataset in case of the gpt-3.5-turbo-0301 model, the non-toxic data points are heavily misclassified as toxic for all prompt variants. curiously, adding definition to the prompts increases this misclassification. these observations remain similar for the text-davinci-003 model. for flan-t5-large non-toxic data points are misclassified as toxic for all the prompt variations except for when target is asked to be generated in which case toxic datapoints are misclassified as non-toxic. in order to better elicit the above observations, we present the confusion matrices for the best prompt combinations for each model and each dataset in appendix c. typology induction: in order to analyse the data points where the models are most vulnerable we employ the following heuristics for each dataset. in these heuristics we always consider the vanilla + defn + exp (output) setup. for the implicit hate dataset we sort the data points in non-decreasing order based on the bertscore between the ground truth implied statement and the generated explanation at the output. starting from the top of this list, we consider the data points that are either not_hate in the ground truth and misclassified as implicit_hate or vice versa by all the three models. note that these data points constitute the cases where the models misclassify and provide poor explanation for their classification decision. these data points are then passed through lda @xcite (number of topics, @xmath9. for the hatexplain and toxicspans datasets we sort the data points in non-decreasing order based on the sentence-bleu scores between the ground truth rationales and the generated rationales at the output. we select 80 low-scoring data points according to the bleu/bertscore measure. for the hatexplain (toxicspans) dataset, starting from the top of this list, we consider the data points that are either normal (non_toxic) in the ground truth and misclassified as hate speech/offensive (toxic) or vice versa by all the three models. these points are then passed through lda (number of topics, @xmath10. for each topic, we choose four words which have the highest probability of association with the topic. these collected set of words are then manually coded with topic names by two researchers with long experience in hate speech research. in order to obtain a semantic clarity upon the derived topic names, a meticulous manual annotation process is undertaken. this annotation task is done by three domain experts, and notably, unanimous consensus is reached among all annotators regarding the semantic interpretation and nomenclature of each delineated topic name. we note the emerging typologies in the observations from the induced typology: for each dataset, we note some of the interesting typologies that emerged. (1) for the implicit hate dataset, one curious case of misclassification (@xmath11, while in another category we find the model marked something as implicit hate because it has pro-white sentiments. even when post representing news articles or opinion pieces contain sensitive words like 'blacks', 'antifa' etc., the models label them as implicit hate. (2) for the hatexplain dataset, we find that hateful or offensive posts get misclassified to normal due to multiple reasons -(a) presence of negations like 'neither', 'nor', 'no' etc. in the post, (b) vocabulary gap where the models do not know the usage of words like 'muzrat' used in the post, and (c) presence of polysemous words in the post with one of the meanings usually being a slang (usually not found in standard english dictionaries), e.g., 'dyke', 'furries', 'furfaggotry' etc. (3) for toxicspans dataset, we note that ideological posts and posts with fact-checking related statements about political news often get misclassified as toxic. on the other hand, posts that are implicitly toxic often get misclassified as non-toxic dataset topic cluster examples gt pr type implicit hate antifa, self, make, right, time, rt, memorial, requires, government, say bring back huac rt : you can be racist all you want but socialism is another beast n_h imp_h racist these are the same people who yell about how they're not racist . this thread is so sad n_h imp_h racist white, people, good, anti, nationalist, right, chief, staff, south, cruz not enough power to make a white lol n_h imp_h pro-white how is appreciating your heritage white privilege ? we're the only ones not allowed to do it n_h imp_h pro-white racist, white, america, make, people, americans, know, really, trump, new yeah but who is defining intolerance . antifa are intolerant . n_h imp_h stmt_with_sensitive_words black #unemployment rate : 9.2 % more than double white unemployment of 4.4 % via #hireamerican n_h imp_h stmt_with_sensitive_words hatexplain toxic, talk, men, masculinity, freakin, attack, nigger, slut, people, women tbtc take back the country toxic masculinity patriarchy harm men noxious femaleness and public slut posturing empower women neither feminism nor now could have pulled this off this is the doing of taxpayer supported state university system maga o n negation note no nigger was harmed during the making of this film. h n negation white, like, line, did, large, muzzies, women, tactics, scarf, counter why he can not get a hoe that work minus the crazy o n vocab_gap massive muzrat espionage h n vocab_gap common, kike, christian, better, stick, prefer, pissing, discussion, sure, provide additionally furries can also transmit the infectious disease of furfaggotry to others h n polysemy a dyke puts the mannish moves on a femme at a lesbian bar in @xmath12, right, good, man, state, like, stupid, don, money, did right charlie the media fabricated all those live broadcast during the primaries. then they altered everything trump said during the campaign. do you have twitter running 24/7 to keep up with donald's latest rants? n_t t ideology the liberals leasing the trudeau name for their leadership is turning out to be a big, fat, failed experiment!! n_t t ideology people, trump, just, know, tax, like, think, don, vote, need the headline for this article has changed at least twice since it was originally posted yesterday. here's the latest update: unhinged trump re-emerges, defending first month in white house n_t t fact_check_pol_news this article is entirely wrong! an ongoing deficit will disintegrate the financial system and the country in less than 30 years. . . . computer projections by more than one analyst suggest a "kinetic" outcome within 15 years. . . n_t t fact_check_pol_news just, like, make, stupid, sure, don, person, people, trump, does oh, gay and black, you just caused all our white christian friends here to start salivating at the same time, for what i'm not sure. t n_t implicit_semantics this individual does need therapy, i hope he gets the help he needs as well as one of those ridiculously light sex offender sentence. this system is broken when it comes to crimes of this nature. the list was designed to be a warning for parents, and yet t n_t implicit_semantics ## conclusion in this work, we extensively study three llms across three datasets on a variety of prompt setups. overall these models, though advanced, still face lot of challenges in hate speech detection in their vanilla setting. our prompt strategies, when applied individually, improve the performance of such models. however, merging of the prompt strategies do not help much. last, we do a detailed error analysis and typology coding to find 'jailbreak' points where these models are vulnerable.