Watching the AI Watchdogs: A Fairness and Robustness Analysis of AI Safety Moderation Classifiers (2025)

Abstract: AI Safety Moderation (ASM) classifiers are designed to moderate content on social media platforms and to serve as guardrails that prevent Large Language Models (LLMs) from being fine-tuned on unsafe inputs. Owing to their potential for disparate impact, it is crucial to ensure that these classifiers: (1) do not unfairly classify content belonging to users from minority groups as unsafe compared to those from majority groups and (2) that their behavior remains robust and consistent across similar inputs. In this work, we thus examine the fairness and robustness of four widely-used, closed-source ASM classifiers: OpenAI Moderation API, Perspective API, Google Cloud Natural Language (GCNL) API, and Clarifai API. We assess fairness using metrics such as demographic parity and conditional statistical parity, comparing their performance against ASM models and a fair-only baseline. Additionally, we analyze robustness by testing the classifiers’ sensitivity to small and natural input perturbations. Our findings reveal potential fairness and robustness gaps, highlighting the need to mitigate these issues in future versions of these models.

## introduction ai safety moderation (asm) classifiers are designed to mitigate hateful, unsafe, toxic, and problematic content for two primary applications: (1) content moderation online on social media platforms (e.g. facebook), and (2) as safety guardrails to ensure that large language models (llms) are not fine-tuned on harmful data. the access to these asm models is often provided in a closed-source black-box manner (openai). asm models play a major and consequential role in the aforementioned applications. for instance, given the exponential growth in content generation across social media platforms @xcite , asm classifiers are essential in automating moderation tasks that would otherwise be impractical to manage only manually @xcite . similarly, as asm models moderate what user content llms can be fine-tuned on by filtering training data, @xcite @xcite , they directly impact the behaviors the models learn. for instance, openai's moderation api (openai) needs to be used prior to fine-tuning their gpt models @xcite . with this growing dual use of asm classifiers for social media content moderation and llm finetuning, it's vital to ensure they are unbiased, robust and safe to use. due to their closed-source nature, asm models may unfairly target or overlook marginalized groups, leading to biased outcomes in content moderation and llms trained on filtered data. bias in moderation can damage trust in online social media platforms, potentially suppress essential voices, and perpetuate inequalities in ai systems trained on the moderated data. similarly, a lack of robustness can allow exploitative behaviors to bypass moderation efforts, compromising both user safety and data integrity for any subsequent ai training both these case scenarios are visualized in figures 1 and 2 . to our best knowledge, large scale end-user audits have only been conducted on one asm model (perspective api), particularly highlighting issues that affect marginalized communities @xcite . however, these evaluations required users to highlight the issues manually and did not utilize a fairness analysis framework relying on analytical fairness metrics. to our knowledge, no formal fairness analysis has been conducted on close-sourced asm models to date. through this paper, we seek to bridge this gap and study fairness and robustness for four commonly used closed-source asm classifiers, namely, openai moderation api, perspective api, google cloud natural language (gcnl) api (palm2based moderation) and clarifai api, across multiple predictive tasks. in summary, we make the following contributions: ## related works progress has been made in evaluating fairness in social media content moderation @xcite and measuring bias in open-source text classification asm models @xcite . in @xcite , the authors show that german content is moderated more than other languages by the perspective api. however, recent research emphasizes the need for fairness evaluation and improved asm models for closed-source llm services @xcite . in @xcite , methods to jailbreak asm models and fine-tune llms to induce bias and make them unsafe are discussed. research in @xcite shows that llms can produce unsafe content through prompt-based techniques. in @xcite , the authors utilize llms as toxicity classifiers and show performance improvement over perspective api. overall, while the broader problem of bias in llms has been explored @xcite ; the analysis of fairness and robustness in closed-source asm models remains unaddressed. ## discussion more fine-grained fairness analysis. through our experiments, we observe that there are clear fairness issues in openai, perspective, and clarifai asm models, especially when considering sexual orientation as a sensitive attribute. while the analysis does not flag any significant fairness issues for the gcnl asm model, an additional experiment specific to the domain could be performed by downweighting the labels provided by this model. this is because the model provides 16 labels which might not be related to safety in all the practical scenarios (see additional details in appendix f where we show that the ratio of unsafe to safe comments is higher for the gcnl api as compared to the other asm models for all the regard labels). minimal perturbations lead to significant asm robustness issues. we show that minimal llmbased perturbations using gpt-3.5 turbo can cause all asm models to change their initial predictions (see figure 4b ) and this error in robustness is the highest for openai moderation api asm across all the datasets (see understanding impact of perturbations on harmful inputs. our llm based perturbation paraphrases the input text into a similar text while preserving its semantic meaning. to understand the effect of this llm-based perturbation on harmfulness of originally harmful inputs, we manually evaluate the perturbed inputs. specifically, we select 50 inputs each from the jigsaw datasets (gender, ethnicity, disability and sexual orientation) and, select 100 harmful examples from the reddit-ideology dataset to label as harmful/harmless post perturbation. we find that for the jigsaw datasets, 19 out of 200 harmful inputs become harmless and for reddit-ideology, 16 out of 100 harmful inputs become harmless, indicating that perturbed inputs retain semantically relevant harm information. intersectional fairness studies. in our work, we mainly focus on cases where only one protected attribute is present, as motivated by prior work on fairness @xcite . in appendix i, we highlight the need for an intersectional analysis of fairness and perform experiments to study the same using the openai asm model. future research in this direction can focus on larger scale intersectional studies on asm fairness. choosing asm model thresholds. the asm models provide an output score upon which a threshold is applied to obtain the binary safe and unsafe labels. in our study, we use a threshold of 0.5 to conduct a fair comparison study. however, in appendix j, we show the impact of applying a threshold of 0.7 on the asm model fairness. we observe that the choice of theshold may improve or worsen the fairness of asm models and thus, future work can provide more insights on threshold selection and its impact of fairness of asm models. ## conclusion we perform a fairness and robustness analysis foot_0 on the ai safety moderation classifiers (openai, perspective, gcnl and clarifai) that are used for social media content moderation and as guardrails for fine-tuning closed-source llms. we highlight the issues in fairness and robustness based on the predictions made by asm models on two datasets with several sensitive attributes (gender, ethnicity, disability, sexual orientation and ideology). notably we observe that there are significant issues with asm models in terms of robustness. our work highlights the potential risks associated with the use of current asm models and the dire need to mitigate these in future work.