Please note that I ’m just an AI : Analysis of Behavior Patterns of LLM s in (Non-)offensive Speech Identification (2024)

Abstract: Offensive speech is highly prevalent on online platforms. Being trained on online data, Large Language Models (LLMs) display undesirable behaviors, such as generating harmful text or failing to recognize it. Despite these shortcomings, the models are becoming a part of our everyday lives by being used as tools for information search, content creation, writing assistance, and many more. Furthermore, the research explores using LLMs in applications with immense social risk, such as late-life companions and online content moderators. Despite the potential harms from LLMs in such applications, whether LLMs can reliably identify offensive speech and how they behave when they fail are open questions. This work addresses these questions by probing sixteen widely used LLMs and showing that most fail to identify (non-)offensive online language. Our experiments reveal undesirable behavior patterns in the context of offensive speech detection, such as erroneous response generation, over-reliance on profanity, and failure to recognize stereotypes. Our work highlights the need for extensive documentation of model reliability, particularly in terms of the ability to detect offensive language.

## introduction consider an example in figure 1 , where a user asks an llm if a particular comment about racial minorities is offensive, expecting that the model will follow the instructions and provide an answer to their question, such as option a or (the incorrect) b. however, the model erroneously reacts with a refusal (option c) triggered by the profanity in the comment or, even worse, answers with text that perpetuates the offensive narrative of the comment itself (option d). such task-inappropriate reactions are not only unhelpful but can also exemplify undesirable and harmful behaviors, such as affirming the user's stereotypical biases or censoring potential counter-speech @xcite . task-inappropriate behaviors, like those in figure 1 , can have many causes. for instance, the answer c is a typical example of a failure to distinguish a mention of profanity from its use and an oversensitive safety guard -a measure originally designed to ensure ethical, responsible, and safe interaction @xcite @xcite @xcite . as part of such safety guards, llms are trained to refuse answering harmful questions, such as "how can i kill a person?" or "how can i make cocaine?" foot_0 while still providing useful answers to harmless queries. however, such mechanisms can be overly sensitive to word-level triggers, such as "kill" in the harmless question "how can i kill a python program?" or "coke" in "where can i buy a can of coke?" @xcite . while the field of red-teaming nlp models is rapidly growing and revealing how llms respond to overtly harmful messages @xcite , so far, considerably less attention has been paid to (possibly subtle) offensive speech. the existing safety guards do not target training for appropriately detecting this type of input (apart from what might coincidentally be in the human feedback data). consequently, llm users must rely on the models' intrinsic capabilities to recognize and avoid offensive speech. yet, it is an open question what types of responses models give when they fail to detect (non-)offensive speech. motivated by this research gap and the indisputable fact that engaging in and generating offensive speech are some of the major social risks of llms @xcite , we ask the following research questions: rq1 how well can models identify potentially subtle (non-)offensive speech, and to what degree is their performance sensitive to prompt templates? ## related work the state-of-the-art llms show exceptional learning capabilities @xcite , such as following instructions @xcite , solving novel tasks @xcite , and even answering questions that require commonsense or world knowledge @xcite . this work builds on two key research fields related to the applications of llms -safety against harmful inputs and offensive speech detection. ## methodology the primary goal of this work is to evaluate the types of responses generated by llms when they fail to detect (non-)offensive speech. to achieve this, we adopt a prompt-based approach, in which we ask models whether a given post is offensive (see an example of a prompt template in figure 2 ). since model outputs are sensitive to prompt templates @xcite @xcite , we run experiments using three templates and report the mean performance (see ## (non-)offensive speech identification in this section, we answer our first research question (rq1): how well can models identify potentially subtle (non-)offensive speech, and to what degree is their performance sensitive to prompt templates? ## analysis of behavior patterns in llm-generated texts having established that most llms struggle to recognize offensive speech, we investigate the underlying reasons for this failure and answer our second research question (rq2): in cases where the models largely fail at this task, what are the common behavior patterns? to this end, we first look at the percentage of generated labels ('offensive', 'non-offensive', 'erroneous response') in §5.1. afterward, we zoom into the errors ('erroneous response') in §5.2 to understand what models generate when they fail to answer the question. ## sensitivity to linguistic cues of profanity and stereotypes after analyzing the model-generated texts and discovering models' tendency to over-predict certain labels or to generate various types of erroneous responses, we now look into the effects of two widely researched properties of offensive language, i.e., profanity and social stereotypes, and answer our final research question (rq3): how does the presence of linguistic cues, such as profanity or words related to stereotypes, influence models' behaviors? sbic is a good test suite for this as it consists of online posts that contain a lot of profanity and stereotypical comments against demographic groups, e.g., "what do pictures and gay men in saudi arabia have in common. both are framed then hanged immediately." in the absence of profanity, in the ideal case, such stereotypical comments should be an important feature in identifying the offensiveness of the sbic posts. thus, we now analyze the effects of these two features (profanity and stereotypes) in models' response behaviors. ## conclusions and discussion in this paper, we explored the abilities of widely used llms to detect online (non-)offensive language. our findings indicate that while a few of the llms tested perform well but still display significant room for improvement, most models completely fail at this task. interestingly, the performance of identifying offensive speech heavily depends on the particular model and not so much on the features of the data. we uncovered a tendency to over-predict either 'offensive' or 'non-offensive' in various models, high sensitivity to the prompt templates, and a striking number of erroneous generations, including the inability to distinguish a question about the offensiveness of text from an offensive and harmful request. our analyses revealed behavior patterns in model responses beyond what is obvious from classification performance without a generalizable pattern in model families or sizes. by looking at two common features of offensive speech (profanity and stereotypes against demographic groups), we revealed models' overreliance on profanity and their failure to recognize the offensiveness of stereotypical comments. with this work, we aim to highlight the potential negative consequences of the observed behavior patterns of llms. currently, regarding safety and fairness, llm users rely on models' inherent abilities to prevent harmful interactions or the safety measures put in place on platforms where these models are deployed. however, our results demonstrate that we cannot, at least not yet, rely on models' inherent capabilities to avoid engaging in harmful interactions in the context of offensive speech as they fail to identify them reliably. therefore, moving forward, in addition to the crucial need for thorough documentation of safety mechanisms, there are three critical considerations. first, although llms are not trained to identify offensive speech, we strongly encourage more effort in this direction. especially considering the current trend of deploying these models in any process imaginable, it is becoming crucial to consider not only their general performance but also their alignment with human values. without the ability to identify offensive speech, we cannot expect the models to avoid generating it. second, while the tendency to over-predict 'offensive', as done by some models, might seem safe, incorrectly labeling non-offensive speech as offensive can be equally harmful. consider the context of social media moderation: generating warnings on harmless posts based on simple word-level triggers would run the risk of silencing and blocking views on important societal issues. therefore, there is a pressing need for more focused training of llms, which would enable these models not only to detect offensive language but also to discern non-offensive speech reliably. third, the (in)ability to identify offensive speech and erroneous behaviors are inconsistent across model families and parameter sizes but are highly model-specific. thus, as there is no thorough documentation of such behavior patterns for each model, we strongly advise llm users to be careful when selecting the right model for their use cases. one concerning use case is, for instance, dataset annotation, where we see a growing trend in using generative llms as cheap and reliable tools (chiang and lee, 2023). considering some models' good performance on these tasks based on classification metrics, this does not raise much concern on the surface level. however, our results show the importance of looking beyond these metrics when employing such models to label text as a replacement for human annotators.