Oddballs and Misfits: Detecting Implicit Abuse in Which Identity Groups are Depicted as Deviating from the Norm (2024)

Abstract: We address the task of detecting abusive sentences in which identity groups are depicted as deviating from the norm (e.g. Gays sprinkle flour over their gardens for good luck). These abusive utterances need not be stereotypes or negative in sentiment. We introduce the first dataset for this task. It is created via crowdsourcing and includes 7 identity groups. We also report on classification experiments.

## introduction abusive language is commonly defined as hurtful, derogatory or obscene utterances made by one person to another person. 1 examples are (1)-( 2 ). due to the rise of abusive language online, a common application is content moderation on social media, where nlp methods guide human reviews to the most relevant microposts. the detection of implicitly abusive language @xcite , i.e. abusive language not conveyed by unambiguously abusive words (e.g. scum, tosser), remains challenging @xcite @xcite : implicitly abusive language often targets identity groups, i.e. groups of people united by a common characteristic, fostering a sense of community (e.g. jews, gay people, women etc.). abusive language towards such groups can often be regarded as a form of othering, i.e. a means of stigmatizing the target as not fitting in within the norms of a social group @xcite . a common form of stigmatizing identity groups is by imposing negative stereotypes on them: recent datasets on implicitly abusive language cover instantiations of such stereotypes fairly well @xcite @xcite @xcite . in this paper, we focus on the detection of a subset of abusive instances of othering that do not coincide with negative sentiment and may not even necessarily represent stereotypes: (7)-(11) depict the identity groups as deviating from general norms of modern-day society. such norm-contraventions are frequently perceived as abusive by the targeted identity groups. a classifier specializing in detecting this form of abuse could supplement existing classifiers for abusive language detection since, as we will show, the latter are unable to detect this type of abuse reliably. in order to quantify the phenomenon, we manually annotated the abuse directed at identity groups on the dataset for implicit abuse by @xcite , which is a union of 7 previous datasets. 65% of the (declarative) sentences can be considered instances of othering. while only 5% of all abusive instances lack negative sentiment, in 80% of these cases the identity groups are depicted as deviating from the norm. thus, deviating from the norm represents a prominent subset of the difficult non-negative abusive sentences. compared to the figures reported by @xcite on other subtypes of implicit abuse, this phenomenon is similarly frequent as dehumanization, euphemisms or comparisons, all of which have previously been examined @xcite @xcite . we also had crowdworkers, all native english speakers without specific backgrounds, compare the severity of different examples of implicit abuse (20 examples for each type). the examples were presented in pairs without revealing their types. crowdworkers had to decide which example they considered more severe. overall, our novel type of abuse was judged even more severe than euphemisms or comparisons. 2 since our above sample is too small for a proper study we created 2 new english datasets: the first represents true-to-life examples extracted from twitter. the second comprises sentences constructed by crowdworkers. due to ethical concerns, the crowdworkers are not asked to form sentences targeting a specific identity group but an unspecific group of people represented by a 3rd person pronoun (e.g. they). we then instantiate these pronouns with identity groups and have the resulting sentences validated as abusive language by other crowdworkers. thus, we establish that depicting a group of people as deviating from the norm is a general property of implicitly abusive language that can be observed across different identity groups. we focus on sentences that can be interpreted without any additional context. the task is a binary (sentence-level) classification problem in which norm-compliant behaviour is to be distinguished from norm-contravening behaviour. we also report on classification experiments and show that this task benefits from recent language models. our contributions are the following: all data created as part of this research are available upon request. there has already been previous research related to othering beyond stereotypes: @xcite and @xcite analyze the juxtaposition of 1st and 3rd person pronouns to contrast the norm (us) with identity groups (them). @xcite examine non-conformist views, sentences expressing negative sentiment towards targeted groups. our focus on othering differs as it is not limited to specific lexical items, like pronouns, nor solely to negative sentiment. our task is also related to framing @xcite since the presentation of identity groups selects aspects of a perceived reality and makes them more salient in a communicating text @xcite . these aspects do not have to apply to the identity groups in reality. thus, they can also be considered misinformation @xcite . our work is also anchored in social psychology: @xcite observed that what is common (=norm-compliance) is often regarded as moral and that rare positive behaviour, e.g. altruism, is judged less moral than common positive behaviour. our work echoes this sentiment, suggesting that norm-compliance is perceived positively, while deviation from norms is viewed negative. @xcite and @xcite find that guilt and shame are social emotions typically experienced when individuals transgress a (social) norm. the audience of sentences depicting norm-contravention may likewise associate similar negative emotions with these utterances, potentially leading to strong disapproval. people are often motivated to punish those who violate societal norms, even if they are not personally affected by these violations @xcite . authors who engage in abusive language targeting norm violations may aim to trigger this punitive reflex in their audience. after all, call for action is a typical characteristic of implicitly abusive language. finally, the fact that identity groups are the target of abusive sentences displaying norm-contravention might be explained by the fact that these groups are also sanctioned more frequently for norm violations than others @xcite . ## data our new data represents a subtype of abuse that depicts identity groups as deviating from the norm. this is a form of othering. the utterances may be stereotypes but they do not have to be. they may also coincide with examples of framing or misinformation. we do not consider abusive utterances that are explicitly negative in sentiment. such sentiment, which is conveyed by words unambiguously negative in meaning, 3 e.g. poor or sad, has been dealt with in previous work @xcite . we create 2 datasets: due to the rareness of this phenomenon, we created a constructed dataset, i.e. a dataset in which sentences are invented ( §3.1). in this way, we can cover various areas of life. however, since that dataset does not necessarily reflect texts in social media, we also produce a twitter dataset comprising attested sentences ( §3.2). 4 the annotation of all datasets was produced via crowdsourcing. as a platform we used prolific academic. 5 we did not specifically target any particular profession, age group, gender or ethnicity for our sample. we randomly sampled crowdwork-3 more information on sentiment is given in appendix b.3. 4 we consciously avoided sampling sentences from so-cial-chem-101 @xcite as that work is strongly related to moral foundations @xcite and the predominant norms in the dataset demonstrate explicit sentiment. considering the large number of participants, the platform's extensive base of several thousand native english speakers in western countries, and the absence of further demographic restrictions, we are confident that our ratings offer a representative cross-section of western society. ## set-up for transformers two transformers are used as learning methods: bert (bert-base-uncased) @xcite and deberta (deberta-large) @xcite . we compare bert, a foundational model, with deberta, which introduces advanced architecture and benefits from more extensive training data. we fine-tune the pretrained models on the given training data using the flair-framework @xcite with the hyperparameter settings from @xcite , a study closely related to ours. we always report the average over 5 training runs (+ standard deviation). appendix a contains details on the settings of all classifiers. we also use large language models, such as gpt-4 (openai et al., 2024), for getting state-of-the-art text completions as outlined in the following. ## norm-compliance in this section, we focus on our constructed normcompliance dataset. the task is to distinguish between sentences in which a generic group of people (i.e. they) is presented as deviating from the norm short prompt is this common? long prompt is this common in our western society? ## conclusion we addressed the task of detecting abusive sentences in which identity groups are depicted as deviating from the norm. we created novel datasets with sentences that do not express (explicitly) negative sentiment for this type of abuse via crowdsourcing. previous classifiers are unable to detect this form of abuse sufficiently. this is a phenomenon not tied to specific lexical units. therefore, only large language models produce good results, in our case, deberta fine-tuned on data augmented by gpt-4. our approach also handles negation and addresses non-abusive instances that are inherent to an identity group but not common for the western society.