Explainable Abuse Detection as Intent Classification and Slot Filling (2022)

Abstract: To proactively offer social media users a safe online experience, there is a need for systems that can detect harmful posts and promptly alert platform moderators. In order to guarantee the enforcement of a consistent policy, moderators are provided with detailed guidelines. In contrast, most state-of-the-art models learn what abuse is from labeled examples and as a result base their predictions on spurious cues, such as the presence of group identifiers, which can be unreliable. In this work we introduce the concept of policy-aware abuse detection, abandoning the unrealistic expectation that systems can reliably learn which phenomena constitute abuse from inspecting the data alone. We propose a machine-friendly representation of the policy that moderators wish to enforce, by breaking it down into a collection of intents and slots. We collect and annotate a dataset of 3,535 English posts with such slots, and show how architectures for intent classification and slot filling can be used for abuse detection, while providing a rationale for model decisions.1

## introduction the central goal of online content moderation is to offer users a safer experience by taking actions against abusive behaviors, such as hate speech. researchers have been developing supervised classifiers to detect hateful content, starting from a collection of posts known to be abusive and non-abusive. to successfully accomplish this task, models are expected to learn complex concepts from previously flagged examples. for example, hate speech has been defined as ''abusive speech targeting specific group characteristics, such as ethnic origin, religion, gender or sexual orienta-tion'' @xcite , but there is no clear definition of what constitutes abusive speech. recent research @xcite has shown that supervised models fail to grasp these complexities; instead, they exploit spurious correlations in the data, they become overly reliant on low-level lexical features and flag posts because of, for instance, the presence of group identifiers alone (e.g., women or gay). efforts to mitigate these problems focus on regularization, for example, preventing the model from paying attention to group identifiers during training @xcite , however, they do not seem effective at producing better classifiers @xcite . social media companies, on the other hand, give moderators detailed guidelines to help them decide whether a post should be deleted, and these guidelines also help ensure consistency in their decisions (see let us illustrate this with the following example. assume we are shown two posts, the abusive ''immigrants are parasites'', and the non-abusive ''i love artists'', and are asked to judge whether a new post ''artists are parasites'' is abusive. although the post is insulting, it does not contain hate speech, as professions are not usually protected, but we cannot know that without access to moderation guidelines. based on these two posts alone, we might struggle to decide which label to assign. we are then given more examples, specifically the non-abusive ''i hate artists'' and the abusive ''i hate immigrants''. in the absence of any other information, we would probably label the post ''artists are parasites'' as non-abusive. the example highlights that 1) the current problem formulation (i.e., given post p and a collection of labeled examples c, decide whether p post: artists are parasites policy: posts containing dehumanizing comparisons targeted to a group based on their protected characteristics violate the policy. protected characteristics include race, ethnicity, national origin, disability, religious affiliation, caste, sexual orientation, sex, gender identity, serious disease, and immigration status. old formulation: is the post abusive? our formulation: does the post violate the policy? in this work we depart from the common approach that aims to mitigate undesired model behavior by adding artificial constraints (e.g., ignoring group identifiers when judging hate speech) and instead re-define the task through the concept of policy-awareness: given post p and policy p , decide whether p violates p . this entails that models are given policy-related information in order to classify posts like ''artists are parasites''; for example, they know that posts containing dehumanizing comparisons targeted to a group based on their protected characteristics violate the policy, and that profession is not listed among the protected characteristics (see we create and make publicly available the policy-aware explainable abuse detection (plead) dataset, which contains (intent and slot) annotations for 3,535 abusive and non-abusive posts. to decide whether a post violates the policy and explain the decision, we design a sequenceto-sequence model that generates a structured representation of the input by first detecting and then filling slots. intent is assigned deterministically based on the filled slots, leading to the final abusive/non-abusive classification. experiments show our model is more reliable than classification-only approaches, as it delivers transparent predictions. ## related work we use abuse as an umbrella term covering any kind of harmful content on the web, as this is accepted practice in the field @xcite . abuse is hard to recognize, due to ambiguity in its definition and differences in annotator sensitivity @xcite . recent research suggests embracing disagreements by developing multi-annotator architectures that capture differences in annotator perspective @xcite @xcite . while this approach better models how abuse is perceived, it is not suitable for content moderation where one has to decide whether to remove a post and a prescriptive paradigm is preferable @xcite . @xcite adopt a more objective approach, as they aim to detect content that is illegal according to eu legislation. however, as they explain, illegal content constitutes only a tiny portion of abusive content, and no explicit knowledge about the legal framework is provided to their model. the problem is framed as the combination of two binary tasks: whether a post contains a protected characteristic, and whether it incites violence. the authors also create a dataset which, however, is not publicly available. most existing work ignores these annotation difficulties and models abuse detection with transformer-based models @xcite ; in our work, we formalize the problem of policy-aware abuse detection as an instance of intent classification and slot filling (icsf), where slots are properties like ''target'' and ''protected characteristic'' and intents are policy rules or guidelines (e.g., ''dehumanization''). while ahmad et al. (2021) use icsf to parse and explain the content of a privacy policy, we are not aware of any work that infers policy violations in text with icsf. state-of-the-art models developed for icsf are sequence-to-sequence transformers built on top of pretrained architectures like bart @xcite , and also represent the starting point for our modeling approach. ## problem formulation given a policy for the moderation of abusive content, and a post p, our task is to decide whether p is abusive. we further note that policies are often expressed as a set of guidelines @xmath0, r 2 , . . . r n } as shown in ## the plead dataset post selection to validate our problem formulation and for model training we created a dataset consisting of posts with slot annotations (e.g., target, threateningspeech). we built our annotation effort on an existing dataset associated with the policy guidelines introduced in section 3 and extended it with additional spanlevel labels. this dataset @xcite was created by providing annotators with a classification model trained on 11 other datasets, and asking them to write hateful and non-hateful sentences such that they fooled the model in predicting the opposite class (hateful for non-hateful and vice versa). the process was iterative, we used sentences from the second round onwards, which were annotated with policy violations. the dataset is not balanced, that is, some policies are violated more frequently than others. to mitigate this and reduce annotation costs, we selected all posts from the less popular policies and a random sample of posts from the most popular ones. we further merged posts annotated with derogation and animosity classes as they are similar, the main difference being the extent to which the negative opinion is implied. the number of selected posts per intent is shown in annotation task we performed two annotation tasks, one for hateful posts and one for nonhateful ones. for hateful posts, annotators were presented with the post, information about the target(s), its characteristics, and the slots. they were then asked to specify the spans of text corresponding to each slot. the dataset already con-tains annotations about which policy is being violated. for instance, for posts labeled as pro hate crime, annotators look for spans corresponding to hateentity and support. information about the target and its characteristics is also present in metadata distributed with the dataset, and we used it to steer annotators towards a correct reading of the posts. in general the original posts, metadata, and labels are of high-quality; @xcite report extremely high agreement for instances created during round 2, moderate for the following rounds and disagreements were resolved by an expert annotator. each post can contain multiple targets, and each target can be associated with multiple protected characteristics (e.g., black woman indicates both the race and gender of a target). our annotation scheme assumes that only one opinion is annotated for each post. for instance, the post ''i love black people but hate women'' contains both a non-hateful and hateful opinion, but we only elicit annotations for the hateful one. likewise, when a post contains more than one hateful opinion, 4 annotators select the one that better fits the associated policy and target description. equally, for non-hateful posts, we asked annotators to focus on a single opinion, with a preference for opinions that resemble hateful messages (e.g., the second opinion in ''i love cats, but i wish all wasps dead''). annotators could specify as many spans (and associated slots) as they thought appropriate, including none. if enough elements were selected for a post to violate a rule (e.g., both hateentity and support were specified), annotators were asked whether the post contained counter speech (and if so, to specify a span of text for negativestance) or derogatory terms used as reclaimed identity terms (e.g., the n-word used by a member of the black community). annotator selection we recruited annotators resident in english-speaking countries through the amazon mechanical turk crowdsourcing platform. to ensure high-quality annotations we designed a quiz for each policy rule and assessed the fairness of the quiz through a two-phase pilot study: in the first phase annotators were shown the instructions and asked to annotate eight sentences. these annotations were then used as possible correct answers for the quiz or to clarify the instructions. during the second phase, new annotators were shown the updated instructions and asked to pass a quiz consisting of three questions. the pilot showed that most crowdworkers who understood the task were able to pass the quiz, but no one was able to pass the quiz without understanding the task. only successful annotators were granted a guideline-specific qualification that allowed them to annotate real instances. to enforce consistency, annotators were prompted to pass a quiz after every 30-post batch, and each batch contained posts associated with one rule. to ensure the data was annotated correctly, we included two control questions in each batch. these were not simple attention checks, but regular posts for which the correct answers were known (from the pilot study). for the annotation of non-hateful posts, we only admitted annotators who had submitted at least 300 annotations for hateful posts, and used the first batch of annotations as a further qualification test. overall, 75% of annotations were produced by women, 91% by people who identify as straight, and 75% by people with ethnicity other than white. we release a full breakdown of demographic information with our dataset. annotation quality we collected three annotations per post. we measured inter-annotator agreement, as the length of the longest common subsequence (lcs) normalized by the length of the longest span. lcs better reflects our annotation task compared to more standard measures like cohen's kappa @xcite which could be calculated on the token level. however, our annotators do not consider words indvidually, they primarily make decisions for sequences of tokens. aside from presuming a different annotation task, token-based kappa leads to extremely unbalanced data with the un-annotated tokens vastly outnumbering the labels of interest, artificially inflating agreement. as shown in ## abuse detection model icsf is traditionally modeled as a sequenceto-sequence problem where the input utterance represents the source sequence, and the target sequence is a linearized version of the corresponding tree. for instance, the linearized version of the tree in figure 1 posts can contain multiple sentences, all of which might have to be considered to discover policy violations (e.g., because of coreference), we adopt the conversational approach to icsf introduced in @xcite . in this setting, all sentences are parsed in a single session (rather than utterance-by-utterance) which is pertinent to our task, as we infer intent after filling the slots, and would otherwise have no information on which slots to carry over (e.g., detecting a target in the first utterance does not constrain the set of slots that could occur in the following ones). our sequence-to-sequence model is built on top of bart @xcite . however, in canonical icsf, bart generates the intent first, and then uses it to look for the slots associated with it. in our case, intent is inferred post-hoc, based on the identified slots, not vice versa. our model adopts a two-step approach where bart first generates a coarse representation of the input, namely, a meaning sketch with coarse-grained slots, and then refines it @xcite . the meaning sketch is a tree where non-terminal nodes are slots, and leaves @xmath2. the sketch for the example in figure 1 and its refined version are shown in figure 2 . specifically, we first encode source tokens w i (figure 2a ): and refine it (see figure 2b ) by first re-encoding the source tokens jointly with the meaning sketch (which is gold at training time, predicted otherwise): a second decoder generates then a new probability distribution over the vocabulary: at inference time, we use beam search to generate the final representation starting from p f . the training objective is to jointly learn to generate the correct sketch z for post x, and the correct tree t from x and z. we define our loss function for tuple (x, z, t) as: although this loss penalizes the model for hallucinating or missing slots, it does not discriminate between errors that cause the prediction of a wrong intent, and those that are less relevant (e.g., hallucinating a threat when no target has been detected). in fact, intent is not part of our sequence-to-sequence task since it is only predicted post-hoc. to help the model learn how combinations of slots relate to intents, we include intent classification as an additional training task. we essentially predict intent starting from the probability of each slot to appear in the sketch (figure 2a ). in other words, we restrict p c t to slot tokens (e.g., sl:target) and normalize it, to obtain a new probability distribution q t over the set of slots. we aggregate these probabilities by taking the maximum value over the sequence length, thus obtaining a single score for each slot. since each intent can be modeled as a disjunction of slot combinations (e.g., the nothateful intent could result from a tree containing only a target, or only a target and a protected characteristic), we pass the slot scores through two linear layers with activation functions: the new loss aims to assign higher penalty to meaning sketches that lead to intent misclassification. the two linear layers are trained on gold intents and sketches. the layers are then added to the bart-base architecture while kept frozen, so that the model cannot modify its weights to ''cover up'' wrong sketches by still mapping them to the right intents. note that this additional classification task is only meant to improve the quality of the generated sketches: intent is added post-hoc in the output tree depending on the slots that have been detected (figure 2a ). ## experimental results we performed experiments on the plead dataset (section 4). rather than learning complex structures with nested slots, we post-process an instance with t targets into t equivalent instances, one per target. furthermore, we discarded instances with reclaimed identity terms as these are not taken into account by our current modeling of the policy, and are too infrequent (< 0.01%). we split the dataset into training, validation, and test set (80%/10%/10%), keeping the same intent distribution over the splits. ## discussion the overwhelming majority of approaches to detecting abusive language online are based on training supervised classifiers with labeled examples. classifiers are expected to learn what abuse is based on these examples alone. we depart from this approach, reformulate the problem as policyaware abuse detection, and model the policy explicitly as an intent classification and slot filling task. our experiments show that conventional black-box classifiers learn to model one of the phenomena represented in the dataset, but small changes such as different random initialization can lead the very same model to learn different ones. our icsf-based approach guides the model towards learning policy-relevant phenomena, and this can be demonstrated by the explainable predictions it produces. we acknowledge that policies for hate speech, as most human developed guidelines, leave some room for subjective interpretation. for instance, moderators might disagree on whether a certain expression represents a dehumanizing comparison. however, the more detailed the policy is (e.g., by listing all possible types of comparisons), the less freedom moderators will have to make subjective judgments. the purpose of policies is to make decisions as objective as possible, and our new problem formulation shares the same goal. while our model still makes errors, the proposed formulation allows us to precisely pinpoint where these errors occur and design appropriate mitigation strategies. this is in stark contrast with existing approaches, where instability is the consequence of spurious correlations in the data, it is hard to isolate errors and, consequently, mitigation strategies are often not grounded in human knowledge about abuse. for example, our analysis showed that our model can sometimes fail to generate the correct tree by mixing the targets and sentiments of multiple opinions. this suggests that it would be useful to have nested slots, for example, a derogatory opinion as the child of its corresponding target. this could also help the model learn the difference between derogatory opinions (nested within a target node) and negative stance (nested within an opinion node), facilitating the detection of counter speech examples. introducing a slot for the proponent of an opinion could also help, as the model would then recognise when a hateful opinion is expresed by someone other than the author. finally, we would like to emphasize that our modeling approach is not policy-specific and could be adapted to other policies used in industry or academia. our formulation of abuse detec-tion and the resulting annotation are compatible with more than one dataset (e.g., @xcite and could be easily modified-for example, by removing or adding intents and slots. extending our approach to other policies would require additional annotation effort, however, this would also be the case in the vanilla classification setting if one were to use a different inventory of labels. ## conclusions in this work we introduced the concept of policy-aware abuse detection which we argue allows to develop more interpretable models and yields high-quality annotations to learn from. humans who agree on the interpretation of a post also agree on its classification label. our new task requires models to produce human-readable explanations that are specific to the input post. to enable models to reason over the policy, we formalize the problem of abuse detection as an instance of icsf where each policy guideline corresponds to an intent, and is associated with a specific set of slots. we collect and release an english dataset where posts are annotated with such slots, and design a new neural model by adapting and enhancing icsf architectures to our domain. the result is a model which is more reliable than existing approaches, and more ''rational'' in its predictions and mistakes. in the future, we would like to investigate whether and how the explanations our model produces influence moderator decisions. nikita moghe, and tom sherborne for their valuable feedback. finally, we thank the action editor and the anonymous reviewers for their helpful comments. this work was supported in part by huawei and the ukri centre for doctoral training in natural language processing, funded by the ukri (grant ep/s022481/1) and the university of edinburgh, school of informatics. lapata gratefully acknowledges the support of the uk engineering and physical sciences research council (grant ep/w002876/1) and the european research council (award 681760).