An Interactive Exploratory Tool for the Task of Hate Speech Detection (2022)

Abstract: With the growth of Automatic Content Moderation (ACM) on widely used social media platforms, transparency into the design of moderation technology and policy is necessary for online communities to advocate for themselves when harms occur. In this work, we describe a suite of interactive modules to support the exploration of various aspects of this technology, and particularly of those components that rely on English models and datasets for hate speech detection, a subtask within ACM. We intend for this demo to support the various stakeholders of ACM in investigating the definitions and decisions that underpin current technologies such that those with technical knowledge and those with contextual knowledge may both better understand existing systems.

## introduction the field of natural language processing (nlp) is organized into tasks, definitions of which minimally include the combination of a modeling paradigm and benchmark datasets @xcite ; @xcite ; @xcite ; see also big-bench 1 ). this organization, however, is not necessarily apparent to those outside of nlp research. making these established tasks outwardly visible is one step towards the recent push for accessible documentation of nlp @xcite @xcite @xcite and promoting the importance of careful data treatment @xcite . one task that has attracted sustained interest in nlp is the problem of content moderation. while many manual and hybrid paradigms for content moderation exist @xcite , several major platforms have invested heavily in automated methods that they see as necessary to support scaling 1 https://github.com/google/big-bench/ up moderation to address their colossal content loads @xcite . automatic content moderation (acm) includes strategies that range from keyword-or regular expression-based approaches, to hash-based content recognition, to data-driven machine learning models. these approaches employ different families of algorithms, resulting in various downstream effects and necessitating documentation and algorithmic accountability processes that address the needs of a variety of stakeholders. synchronizing research around consistent modeling paradigms and benchmark datasets is an ongoing problem for acm @xcite , with experts calling for more grounding in related areas in the social sciences, communication studies and psychology @xcite . without this grounding and without consideration for the contexts into which acm is integrated, the technology intended to prevent harms ends up magnifying them, especially for vulnerable communities @xcite . the present paper proposes an interactive tool aimed at allowing a diverse audience to explore examples of nlp data and models used in data-driven acm, focusing on the subtask of hate speech detection. our tool outlines various aspects of the social and technical considerations for acm, provides an overview of the data and modeling landscape for hate speech detection, and enables comparison of different resources and approaches to the task. our goal is to understand the role of multidisciplinary education and documentation in promoting algorithmic transparency and contestability @xcite . we provide a brief overview of acm as well as the interactions between its many stakeholders ( §2) and describe related work in dataset and model exploration ( §3). we then present our demo ( §4), highlighting its constituent sections and describing our rationale for each. we conclude with a summary of limitations and future work ( §5). 2 background: content moderation content moderation is the process by which online platforms manage which kinds of content, in the form of images, video, or text, that users are allowed to share. policies for content moderation, which vary across platforms, are often guided by a combination of legal, commercial, and social pressures. broadly, these policies tend to prohibit explicit sexual content, graphic depictions of violence, hate speech foot_0 , and harassment or trolling between platform users @xcite . platforms take a variety of actions to moderate content, including removal of the offending content, reducing the visibility of the content, adding a flag or warning, and/or suspending accounts that violate content guidelines. moderation decisions can, however, lead to undesired reactions. for example, removing conspiracy theory content tends to reinforce conspiracy theory claims, and ousting hateful groups from larger platforms can result in these groups flocking to smaller platforms with fewer resources for moderation @xcite . conflicts in moderation decisions often arise due to the size and diversity of a platform's community members and a divergence in priorities between community members and platform managers. a report from the brennan center for justice found that 'double standards' pervade in content moderation actions, and that inconsistently applied content policies overwhelmingly silence marginalized voices @xcite . for example, facebook erroneously labeled hashtags referencing al-aqsa, a mosque in a predominately palestinian neighborhood of jerusalem, as pertaining to a terrorist organization, and was also found to censor deroga-tory speech against white people more frequently than slurs against black, jewish, and transgender people @xcite . to address the often stark gap between model performance on intrinsic metrics and performance in real-world, user-facing scenarios for toxic content classifiers, @xcite propose an evaluation paradigm that takes into account inter-annotator disagreements on training data. even when moderation rules are applied consistently, they may result in over-moderating communities that use terms that are deemed 'explicit' outside the community but are acceptable to the community members themselves, as often happens for lgbtq communities online @xcite . these kinds of harms show that content moderation algorithms must be developed with transparency, care for the context in which the algorithms will be integrated, and mechanisms for the community to contest moderation decisions. one approach to consulting diverse perspectives on 'toxic' content relies on jury learning, as in a model proposed by @xcite . in addition to calling for more inclusion by various stakeholders in decision-making processes for each platform, pershan (2020) advocates for the development of regional policies that consider the moderation styles of smaller platforms as well as larger ones. regional policies are especially important as the large platforms, primarily located in the us, are ported outside the us with moderation policies that are ill-equipped to support local communities appropriately, for example in india where hate speech may also occur on the basis of caste @xcite . approaches to content moderation commonly involve a hybrid strategy that uses reports from users and algorithmic systems to identify content that may violate platform guidelines, and then relies on human review to determine a course of action (i.e., retain, obscure, or remove the content). this process exposes human moderators to high volumes of violent and hateful content (roberts, 2014), motivating a push for enhanced automatic methods to alleviate the burden on human moderators. automated content moderation can rely on analyses of the content itself using nlp or computer vision (cv), features of user dynamics, and hashing to match instances of pre-identified forbidden content. within the realm of text-based acm, approaches vary from wordlist-based approaches to data-driven models. when platforms opt not to build their own systems, perspective api foot_1 is commonly used to flag various kinds of content for moderation. forbidding hate speech on online platforms is seen as a way to prevent the proliferation of hateful discourse from leading to hate-driven violence offline foot_2 . common datasets used for training and evaluating hate speech detectors can be found at https://hatespeechdata.com/ . we refer readers to @xcite for a comprehensive overview of definitions, resources, and ethical challenges incurred in the development of hate speech detection technologies. ## related work: interactive dataset and model exploration a variety of methods and tools that enable dataset users to explore and familiarize themselves with the contents of the datasets have been proposed. for example, know your data foot_3 , provided by google's pair research group, aims to provide users with views of datasets that surface errors or issues with particular instances, systematic gaps in representation, or problematic content that requires human judgment to assess. this tool thus far has focused on image datasets. the dataset cartography method, proposed by swayamdipta et al. (2020), uses model training dynamics to create maps of dataset instances organized by difficulty or ambiguity, which can surface problematic instances. recently, xiao et al. (2022) released a tool for comparing datasets aimed at enabling dataset users to understand potential sources of bias in the data. while much previous work has focused on ex-ploratory tools for dataset users, our tool is meant to cater to an audience who will not necessarily be training machine learning models, but constitute a variety of impacted or interested stakeholders. wright et al. ( 2021 ) tackle the problem of interrogating a toxicity detection model using a tool they call recast. they fine-tune a bert-based transformer model on the jigsaw kaggle dataset of toxic comments from wikipedia and provide an online text-editing application that visually highlights words that the models detects as toxic, suggesting alternate phrases that may be less toxic using both word embeddings and language modeling predictions. they evaluate the tool using a text-editing task, presenting user study participants with comments drawn from both the kaggle dataset and twitter threads, and show that the users in their study are learning about the model behavior by editing toxic comments to be less toxic according to the model prediction scores. ## demo development and structure we aim to make the exploration tool as accessible and useful as possible to the many stakeholders involved in acm. particularly in light of the closed nature of many contemporary content moderation pipelines that impact people who use social media, our demo familiarizes these stakeholders with the general framework of how such systems might work behind the scenes. in order to conceptualize the breadth of uses that acm stakeholders may have for such an exploratory tool, we considered the stakeholders and their goals detailed in pershan (2020) using the framework developed by @xcite . rather than identifying stakeholders based on their roles, they propose mapping stakeholders based on the type of knowledge they hold and the context of that knowledge, such as technical, domain, and contextual knowledge. in mapping out our envisioned stakeholders, we tried to consider how they might use the tool towards their goals. policymakers, journalists and impacted communities may use the demo to understand where and how things go wrong in hate speech detection in order to advocate for changes to platform policies. domain experts may use the tool to understand where their work is used in a pipeline, such as in label definitions, and envision potential locations in the pipeline where additional domain information could be useful. students and current developers may use the tool to reflect upon their own design decisions in light of the historical and sociotechnical framing we provide for acm and consider new possibilities for research development. finally, we imagine that our demo may generally provide common ground for these and other stakeholders in order to facilitate more productive discussions on how to develop acm technologies. additionally, in order to more fully understand the perspectives of stakeholders outside of the academic context, we discussed our demo and the state of the field of hate speech detection with several experts in the field, particularly those with experience deploying models in the industry context and working with non-technical stakeholders. following these discussions, we built the interactive, openly available demo using streamlit 6 , the first page of which is shown in fig. 1 . we provide screenshots of the other modules in appendix a. ## limitations and future work while our tool is aimed at promoting a shared vocabulary and common ground between (1) those who build and design hate speech detection datasets and models, (2) those who are on the receiving end of moderation decisions on social media platforms, and (3) researchers and journalists who are interested in understanding some of the mechanics of automated content moderation, the tool is not designed to be a platform for facilitating connection and engagement between these groups. however, the tool can serve as a foundation for such discussions and could be integrated into a larger system designed for engagement. we plan to update the demo based on feedback from the questionnaire. once the demo has been finalized, user studies aimed at gathering perspectives from a broader set of stakeholders, including those we did not consider in our initial design process such as content moderation workers, would help to outline how different stakeholders actually use the tool and evaluate the effectiveness of the tool with respect to the participants' use cases and contexts. following these studies, future versions of the tool could expand to consider more issues within content moderation beyond hate speech detection or be designed to provide context for other kinds of nlp tasks. while this current demo is focused on english resources, future versions could also include resources and contexts for other languages as well as more complex configurations of datasets and models beyond binary labeling schemas. we began this work with the intention to help provide clarity into the organization of the field of nlp into various tasks. while this demo has focused on the task of acm, we would expect that similar demos could be developed to contextualize other well-known tasks in nlp such as machine translation, information retrieval, and automatic speech recognition. very thoughtful and helpful suggestions for improving this publication. a.p. is supported by the national institutes of health, national library of medicine (nlm) biomedical and health informatics training program at the university of washington (grant nr. t15lm007442). the content is solely the responsibility of the authors and does not necessarily represent the official views of the national institutes of health. tu vu, tong wang, tsendsuren munkhdalai, alessandro sordoni, adam trischler, andrew mattarella-micke, subhransu maji, and mohit iyyer. 2020. exploring and predicting transferability across nlp tasks. in proceedings of the 2020 conference on empirical methods in natural language processing (emnlp), pages 7882-7926, online. association for computational linguistics. austin p wright, omar shaikh, haekyu park, will epperson, muhammed ahmed, stephane pinel, duen horng chau, and diyi yang. 2021. recast: enabling user recourse and interpretability of toxicity detection models with interactive visualization. proceedings of the acm on human-computer interaction, 5(cscw1):1-26. yang xiao, jinlan fu, weizhe yuan, vijay viswanathan, zhoumianze liu, yixin liu, graham neubig, and pengfei liu. 2022. datalab: a platform for data analysis and intervention. arxiv preprint arxiv:2202.12875. figure 2 shows the context of automatic content moderation module (section 4). by introducing the demo users to some of the relevant context outlined in section 2 and to selected writings both by content platforms and independent writers on their approach to (automatic) content moderation, we aim to help them better understand the information presented in the following sections. figures 4, 5, and 6 correspond to the model exploration section (4).