  Online social media platforms increasingly rely on Natural Language
Processing (NLP) techniques to detect abusive content at scale in order to
mitigate the harms it causes to their users. However, these techniques suffer
from various sampling and association biases present in training data, often
resulting in sub-par performance on content relevant to marginalized groups,
potentially furthering disproportionate harms towards them. Studies on such
biases so far have focused on only a handful of axes of disparities and
subgroups that have annotations/lexicons available. Consequently, biases
concerning non-Western contexts are largely ignored in the literature. In this
paper, we introduce a weakly supervised method to robustly detect lexical
biases in broader geocultural contexts. Through a case study on
cross-geographic toxicity detection, we demonstrate that our method identifies
salient groups of errors, and, in a follow up, demonstrate that these groupings
reflect human judgments of offensive and inoffensive language in those
geographic contexts.
