Rethinking Offensive Text Detection as a Multi-Hop Reasoning Problem (2022)

Abstract: We introduce the task of implicit offensive text detection in dialogues, where a statement may have either an offensive or non-offensive interpretation, depending on the listener and context. We argue that reasoning is crucial for understanding this broader class of offensive utterances, and release SLIGHT, a dataset to support research on this task. Experiments using the data show that state-of-the-art methods of offense detection perform poorly when asked to detect implicitly offensive statements, achieving only {\sim} 11\% accuracy. In contrast to existing offensive text detection datasets, SLIGHT features human-annotated chains of reasoning which describe the mental process by which an offensive interpretation can be reached from each ambiguous statement. We explore the potential for a multi-hop reasoning approach by utilizing existing entailment models to score the probability of these chains, and show that even naive reasoning models can yield improved performance in most situations. Analysis of the chains provides insight into the human interpretation process and emphasizes the importance of incorporating additional commonsense knowledge.

## introduction with the development and popularity of online forums and social media platforms, the world is becoming an increasingly connected place to share information and opinions. however, the benefit that these platforms provide to society is often marred by the creation of an unprecedented amount of bullying, hate, and other abusive speech foot_0 . such toxic speech has detrimental effects on online communities and can cause significant personal harm. efforts by the nlp community to address this problem has led to the development of models capable of identifying toxic speech in specific domains (sexism @xcite , racism @xcite , or otherwise hateful text @xcite @xcite ), but the problem of identifying harmful text can also involve more complex pragmatic reasoning. consider a scenario where a young girl runs into her elderly neighbor who remarks, "your piano playing has really improved lately!" most people (and classifiers) would likely take this comment as a compliment. however, in some circumstances, the intent may be the opposite. the neighbor can only have knowledge of the girl's piano progress if she is able to hear it, and being able to hear it may indicate that it is too loud, implying that the girl is inconsiderate of her neighbors. 2 through this reasoning process, we may reach the less complimentary interpretation, namely that the neighbor is annoyed by the playing and the comment is a subtle attempt to convey it. this work considers how current models of offensive text detection (otd) perform when faced with such ambiguous examples of offensive text. following the classification proposed in @xcite , we consider two categories of otd: (1) explicit offensive text, which is unambiguous in its potential to be offensive and often includes overtly offensive terms, such as slurs, and (2) implicit offensive text, which is more ambiguous and may use sarcasm, innuendo, or other rhetorical devices to hide the intended nature of the statement. we hypothesize that there exists a direct relationship between these tasks and that each implicitly offensive statement corresponds to an explicitly offensive statement which is realized through the interpretation process. this explicitly offensive statement is closer to the sentiment the listener feels when interpreting the statement as offensive. consider the example in figure 1 , a dialogue between two speakers, s1 and s2: s1: "i love bookclubs, i go every week" s2: "some places with free food, right?" by itself, the statement by s2 is innocuous and could be interpreted as a simple prompt for more information about the bookclub. however, other interpretations of this statement could lead s1 to arrive at a number of explicitly offensive statements, such as (1) "you are poor." (2) "you are fat." (3) "you are not smart/sophisticated." thus we consider the chain of reasoning which constitutes the interpretation a crucial part of recognizing implicitly offensive statements. to study this phenomenon, we use human annotators to construct a dataset consisting of (1) an implicitly offensive statement, (2) a corresponding explicitly offensive statement, and (3) a chain of reasoning mapping (1) to ( 2 ). when evaluated on the explicitly offensive examples, state-of-the-art models perform well, @xmath1. however, when applied to the implicit otd examples, the accuracy of the models drops to an average of @xmath2. we then explore using a multi-hop reasoning-based approach by utilizing a pre-trained entailment model to score the transitions along each "hop" of the reasoning chain. when incorporating additional knowledge (from human annotations) into the premises of each entailment, we achieve higher accuracy than comparable methods which do not utilize the reasoning chain. we present this as the evidence that a multi-hop reasoning-based approach is a promising solution to this problem and release our data to support further research on the topic. our contributions in this work are threefold: you go to bookclubs because of free food. you love free food and eating. you love eating a lot which makes you fat. you are fat. is that how you became fat? otd model offensive ! non-offensive ! multi-hop reasoning model offensive ! chain of reasoning context explicit implicit explicit otd task implicit otd task otd model cases. in addition, we provide an analysis of which types of reasoning are most challenging and which types of external knowledge are required. otd in text classification early approaches to otd relied primarily upon dictionaries like hatebase 3 to lookup offensive words and phrases. the creation of otd datasets enabled the development of ml-based approaches utilizing simple features, such as bag-of-word representations @xcite . with the advent of social media platforms, many resources have been developed for identifying toxic comments in web text @xcite , including many deep learning-based methods @xcite @xcite @xcite . notably, all of these methods can be described as building a contextual representation of a sentence (whether trained end-to-end or on top of existing pre-trained language models) and making a classification based on this representation. ## data we propose slight foot_2 , a dataset for the study of implicit otd as a multi-hop reasoning problem or as a diagnostic to test models' ability to identify implicitly offensive statements. each example in the dataset consists of three parts: annotations are crowdsourced using amazon mechanical turk (amt). we performed four rounds of pilot experiments in which high-quality annotators were identified, and the annotation instructions were refined to address any observed confusion in the annotation process. the final instructions can be found in appendix c. due to the nature of the data, all participants were briefed that the task would involve offensive content and were knowledge only the best can win contests. classic things are usually old. grown-ups don't play with dolls. parents want children to be independent. overworking makes people exhausted. all workers were paid an average hourly wage of @xmath02 @xcite , we pay relatively more to encourage high-quality annotations of a challenging task. we did not limit the location of annotators, requiring only english proficiency. this allows for a diverse range of viewpoints to help understand how statements may be interpreted in different ways by different cultures @xcite . ## experiments we evaluate the difficulty of the implicit otd task using existing state-of-the-art models before exploring a multi-hop approach to implicit otd using existing entailment models to score transitions in the reasoning chains. ## discussion we introduced this work based on a hypothesis that a reasoning-based approach has a conceptual advantage over existing approaches to offensive text detection, in that humans must each be performing some reasoning process in order to find statements either offensive or non-offensive in different situations. we then showed that this conceptual advantage could translate to an empirical one and showed performance gains over current approaches. however, we do so under strong assumptions and access to additional information. how realistic is our experimental setup? ## conclusion in this work, we aim to broaden the scope of offensive text detection research to include the nuanced utterances. improvements in these models have applications ranging from distant futures where humans frequently interact with dialogue systems in situated ways which require such pragmatic reasoning to avoid unintended offense to today's online forums, where often a cat-and-mouse game of increasingly more creative offensive text creation and moderation occurs. in addition to providing a dataset of implicitly offensive text, which can itself be used purely as a diagnostic of systems' ability to identify more subtle instances of offensive text, we also provide chains of reasoning annotations which we hope can provide insight to how statements lead to offensive interpretations in certain situations. our experiments provide a proof of concept of how multi-hop reasoning models have the potential to outperform directly classifying offensive text using current state-of-the-art approaches and identify areas for improvement via future research in commonsense knowledge base construction and inference.