On the Challenges of Using Black-Box API s for Toxicity Evaluation in Research (2023)

Abstract: Perception of toxicity evolves over time and often differs between geographies and cultural backgrounds. Similarly, black-box commercially available APIs for detecting toxicity, such as the Perspective API, are not static, but frequently retrained to address any unattended weaknesses and biases. We evaluate the implications of these changes on the reproducibility of findings that compare the relative merits of models and methods that aim to curb toxicity. Our findings suggest that research that relied on inherited automatic toxicity scores to compare models and techniques may have resulted in inaccurate findings. Rescoring all models from HELM, a widely respected living benchmark, for toxicity with the recent version of the API led to a different ranking of widely used foundation models. We suggest caution in applying apples-to-apples comparisons between studies and call for a more structured approach to evaluating toxicity over time.

## introduction detecting and measuring toxicity in language is a complex task that requires expertise in language subtleties and contextual awareness that can vary by geography and cultural norms. moreover, with the ever-expanding size of datasets, auditing for toxicity has become infeasible for human annotators @xcite @xcite . human annotation is not only increasingly expensive but also poses a serious mental health risk to evaluators exposed to highly toxic content, leaving them vulnerable to lasting psychological harm @xcite . automatic toxicity detection tools, which often use machine learning algorithms to quickly analyze large amounts of data and identify patterns of toxic language, are a popular and cost-effective method of measurement @xcite . for example, black-box commercial apis are a widely used tool for evaluating toxicity for online content moderation. these commercial apis, such as perspective api foot_1 , have also been widely adopted for academic benchmarking of toxicity-related work. for example, the realtoxicityprompts (rtp) @xcite dataset leveraged the perspective api to generate toxicity scores in order to investigate the tendency of language models (lms) to generate toxic text. this dataset is frequently used to benchmark the toxicity of widely used open-source and closed-source models, and also for academic benchmarking to assess the relative merits of new proposed toxicity mitigation methods. despite the usefulness of automatic toxicity detection tools such as the perspective api, relying on commercial apis for academic benchmarking poses a challenge to the reproducibility of scientific results. this is because black-box apis are not static but frequently retrained to improve on unattended weaknesses and biases @xcite . updates to the api are often poorly communicated and we observe that updates appear to have occurred in the absence of any formal communication to users. as a result, this can impact static datasets with outdated toxicity definitions and scores, such as the rtp dataset, or the reuse of previously released results that had generated continuations scored with an older version of the api. more broadly, reproducibility difficulties are true for any black-box api that does not inform of model updates or provides model versioning for users. nowadays, only a handful of enterprises and groups have access to the amount of computing nec-essary to train the most powerful language models, for example, and users have access to those exclusively through an api. similar to the difficulties we found when using perspective, previous work has shown the lack of reproducibility in general use text generation apis @xcite . we believe these work, in conjunction with ours, to be of extreme importance for setting clear limitations (and room for improvement) for the usage of machine learning algorithms through apis. in this work, we ask how have changes to the api over time impacted the reproducibility of research results? our results are surprising and suggest that the use of black-box apis can have a significant adverse effect on research reproducibility and rigorous assessment of model risk. we observe significant changes in the distributions of toxicity scores and show that benchmarking the same models at different points in time leads to different findings, conclusions, and decisions. our findings suggest caution in applying like-for-like comparisons between studies and call for a more structured approach to evaluating toxicity over time. our contributions are four-way: ## methodology cohen et al. ( 2018 ) define reproducibility as "a property of the outcomes of an experiment: arriving -or not -at the same conclusions, findings or values". the authors propose three dimensions of reproducibility: 1) of conclusions, or validity of inductions made based on results from research; 2) of findings, a repeatable discovery based on the relationship between values; and 3) of values measured or calculated. our goal in this work is to understand how reliance on black-box apis for toxicity evaluation in research impact all three axes. we rescore previously generated text using the perspective api and evaluate the implications of this rescoring on research reproducibility. we measure the impact of changes in the perspective api in three settings: 1) in toxicity scores and statistics of the widelyused realtoxicityprompts (rtp), 2) on the comparison of models' toxicity over time in a living benchmark and 3) on the reproducibility of proposed findings of toxicity mitigation research techniques. ## recommendations in this section, we lay recommendations to improve reproducibility and confidence in results for applications that rely on black-box apis for large-scale evaluations, such as toxicity-related research. in order for these recommendations to be effective, community collaboration and awareness of an evaluation's limitations are required elements. assume unreleased generations have outdated scores and are not safely comparable. perspective api on every model addition. if the toxicity metrics for that control set change, all previous models should be rescored. if a model cannot be rescored due to access restrictions, add a note regarding outdated results or remove the results from that benchmark version. toxicity detection and evaluation are some of the first steps towards safe use and deployment of language models @xcite . these are challenging first steps, though, because the perception of toxicity and hate-speech is known to vary among different identity groups @xcite and genders @xcite . the quality of human-based toxicity detection is correlated to the expertise of the annotator @xcite or to being part of the group which was targeted by the toxic comment @xcite . however, even experts are prone to generating biased annotations in this context @xcite . on the hazards of the task, human-based toxicity evaluation is known for negatively impacting moderators' psychological well-being @xcite . on top of that, the ever-larger amounts of data for either content moderation or dataset curation are often infeasible to be manually annotated. automatic toxicity evaluation not only stabilizes processes but also adds consistency in decisions @xcite . those tools have their own drawbacks, such as outputting higher toxicity scores for non-normative and minority communities @xcite , and exhibiting variations in scores for paraphrases @xcite , but act as a low-cost first measure of toxicity @xcite . ## conclusion in this work, we present some of the challenges of using black-box apis in research, specifically in the toxicity evaluation of language models. the joint usage of outdated and fresh scores prevents a fair comparison of different techniques over time and leads authors to biased conclusions. that was showcased with changes in the just-published results from uddia @xcite and the living benchmark helm @xcite , which has been adding new models and benchmarking at different times since its release in november 2022. while perspective api does not announce all model updates nor allows for api calls with previous model versions, we urge authors to be cautious when directly comparing to other work.