  Biased associations have been a challenge in the development of classifiers
for detecting toxic language, hindering both fairness and accuracy. As
potential solutions, we investigate recently introduced debiasing methods for
text classification datasets and models, as applied to toxic language
detection. Our focus is on lexical (e.g., swear words, slurs, identity
mentions) and dialectal markers (specifically African American English). Our
comprehensive experiments establish that existing methods are limited in their
ability to prevent biased behavior in current toxicity detectors. We then
propose an automatic, dialect-aware data correction method, as a
proof-of-concept. Despite the use of synthetic labels, this method reduces
dialectal associations with toxicity. Overall, our findings show that debiasing
a model trained on biased toxic language data is not as effective as simply
relabeling the data to remove existing biases.
