Critical Perspectives: A Benchmark Revealing Pitfalls in P erspective API (2022)

Abstract: Detecting “toxic” language in internet content is a pressing social and technical challenge. In this work, we focus on Perspective API from Jigsaw, a state-of-the-art tool that promises to score the “toxicity” of text, with a recent model update that claims impressive results (Lees et al., 2022). We seek to challenge certain normative claims about toxic language by proposing a new benchmark, Selected Adversarial SemanticS, or SASS. We evaluate Perspective on SASS, and compare to low-effort alternatives, like zero-shot and few-shot GPT-3 prompt models, in binary classification settings. We find that Perspective exhibits troubling shortcomings across a number of our toxicity categories. SASS provides a new tool for evaluating performance on previously undetected toxic language that avoids common normative pitfalls. Our work leads us to emphasize the importance of questioning assumptions made by tools already in deployment for toxicity detection in order to anticipate and prevent disparate harms.

## introduction social media and online discussion forums are plagued with toxic content. 41% of american adults have personally experienced online harassment @xcite . this problem can be further exacerbated by learning algorithms with unchecked incentives. on a platform like facebook, incendiary comments are highly visible due in large part to the design of its ranking algorithms @xcite . some experiments by @xcite ; @xcite have shown that users are more likely to write toxic or "troll" posts when the initial comments in a conversation exhibited toxic or "trolling" behavior, which can be interpreted as evidence that "trolling is contagious." this creates the potential for a negative feedback loop where companies are incentivized to promote toxic content and toxic content in turn provokes users to post more malicious content. in response, researchers and corporations alike have invested in systems that can detect toxicity at massive scales to help moderate content on social media and internet forums @xcite @xcite . some of these models have, however, been shown to be susceptible to simple adversarial attacks @xcite ). ## benchmarking with sass the sass benchmark contains 250 manually created natural language examples across 10 nuanced "toxicity" categories (e.g. stereotyping, classism, blackmail). these categories were selected via a process of literature review and vulnerability testing on perspective and other toxicity tools, to determine their weaknesses/strengths. as we sought to challenge perspective and other toxicity tools, we believe this to be a sufficient process for determining our categories, although acknowledge that it introduces some unavoidable author bias. the examples are each 1-2 sentences long and are designed to exploit vulnerabilities in toxicity detection systems like perspective. samples from sass in each category are shown in eight of sass's categories are aimed at generating "false negative" (fn) scores (a score that significantly underestimates the toxicity of some text), one category is aimed at "false positive" (fp) scores (a score that overestimates toxicity), and one category is "neutral," a control, demonstrating the model's performance on "normal," non-toxic sentences. sass is heavily biased towards examples that generate a fn score, which we argue may be more harmful than a fp score, as a fn means toxic content has gone undetected. for each category, the benchmark contains 15 "filtered" and 10 "unfiltered" examples, drawing inspiration from @xcite . we generate filtered examples by brainstorming toxic comments and evaluating the comments with perspective to ensure a toxicity score @xmath0.5. then, we generate an additional set of 10 examples per category using the knowledge gained from creating the filtered examples without first testing them on perspective. human ground truth the benchmark also contains a "human" toxicity @xmath1, 1] for each comment, which can be used as a baseline for evaluating toxicity detection tools using sass. the human toxicity scores are an average of the toxicity scores of the authors per comment (scored blindly). here, we scored examples on a scale of 0-10, using jigsaw's definition of toxicity, i.e. "how likely [the example is to] make [a user] leave the discussion" (0=highly unlikely, 10=highly likely). significantly, we aligned these ratings with assumptions laid out in a.2.2 (in appendix) for consistency and to combat benchmarking pitfalls @xcite . we further performed z-normalization, as per @xcite . each author may have treated the "0-10 toxicity scale" differently, so this normalization process ensures that the final aggregate scores are not overly biased by any single author's interpretation of the scale. in ## experiments and discussion binary toxicity classification we showcase the utility of sass by evaluating perspective and gpt-3 against the human baseline in a binary classification setting. it's important to note that perspective and gpt-3 are very different systems, trained with distinct objectives, amounts and the binarized ground truth human labels on sass contain 72.4% toxic labels versus 27.6% non-toxic labels. we use these thresholded human labels as ground truth and evaluate perspective and gpt-3's performance on sass in model description perspective uses a transformer model with a state-of-the-art charformer encoder. the model is pretrained on a proprietary corpus including data collected from the past version of perspective and related online forums. this dataset is mixed in equal parts with the mc4 corpus, which contains multilingual documents @xcite . gpt-3, created by openai in 2020, is a state-of-the-art autoregressive transformer-based language model @xcite . gpt-3 is trained on a massive amount of internet text data, predominately common crawl and webtext2 @xcite , and generates human-like language in an open prompt setting. ## ethical considerations sass, the new benchmark proposed in this paper, seeks to address normative claims made by toxicity detection tools that rely on majority opinion to determine malicious content. in the narrow scope of improving toxicity model evaluation, we thus expect sass to have a positive impact on the nlp community, and by extension on moderation systems for social media and online forums. however, thousands of content moderators, whose job descriptions include toxic content de-tection, are currently employed by companies such as meta. we believe that the best systems for toxic content detection are likely collaborations between humans and machines, but acknowledge that, by improving automated systems, we may jeopardize employment for these people. still, it is unclear that content moderation is a task that people should take part in, and automating toxicity detection may reduce the exposure of people to harmful content that could have severe mental health consequences @xcite . there is always the risk that, in providing a new benchmark to the larger nlp community, some may use it to make unjustified claims. therefore, we take this opportunity to highlight the ways in which sass could be misused. we acknowledge that any benchmark, especially a relatively small one like sass, will reflect the inherent biases of the authors. each category of sass is not designed by any means to be exhaustive; rather, each is designed to provide an initial probe, a check for model vulnerabilities. further exploration would be required even if a model performed well on sass. sass is also only an english language benchmark, and contains examples that only make sense in an americanized cultural context. we believe it is important work to create similar benchmarks for other languages and cultural contexts. we would like to thank sam bowman and richard pang for very useful conversations and feedback over the course of our project. we would also like to thank julia stoyanovich and the center for responsible ai at nyu for supporting our work. ellery wulczyn, nithum thain, and lucas dixon. 2017. ex machina: personal attacks seen at scale. in proceedings of the 26th international conference on world wide web, pages 1391-1399.