Representing and Clustering Errors in Offensive Language Detection (2025)

Abstract: Content moderation is essential in preventing the spread of harmful content on the Internet. However, there are instances where moderation fails and it is important to understand when and why that happens. Workflows that aim to uncover a system’s weakness typically use clustering of the data points’ embeddings to group errors together. In this paper, we evaluate the K-Means clustering of four text representations for the task of offensive language detection in English and Levantine Arabic. We find Sentence-BERT (SBERT) embeddings give the most human-interpretable clustering for English errors and the grouping is mainly based on the targeted group in the text. Meanwhile, SBERT embeddings of Large Language Model (LLM)-generated linguistic features give the most interpretable clustering for Arabic errors.

## introduction content moderation systems are used to mitigate the spread of offensive content online. these systems are usually successful at flagging offensive language, but may also incorrectly remove nonoffensive content, and this incorrectly flagged nonoffensive content is disproportionately shared by people who identify with a marginalized group. previous works have shown bias in hate speech detection systems when it comes to text written in african american english @xcite @xcite . moreover, moderation systems struggle to classify implicit offensive language. @xcite tested dog whistle detection on the perspective api foot_1 and found that it assigned lower ratings to examples that used dog whistles (subtle, potentially harmful messages intended to only be understood by certain groups) instead of slurs. in order to work toward correcting these types of issues, offensive language detection models must be examined more closely to understand how and why they are making mistakes. evaluation metrics like f1-score and accuracy provide a compact and high-level means of scoring models, but are not enough to fully understand a model's behavior. to uncover where a model underperforms, researchers have recently shifted to automating aspects of the error analysis process and providing a systematic approach to analyzing a model's performance. these approaches are presented as error analysis tools @xcite @xcite @xcite @xcite or slice detection models (sdms) @xcite @xcite . error analysis tools provide a user-interface that allows practitioners to closely examine their systems and sdms partition the data to "slices", aiming to identify those partitions on which the model underperforms, without the need for explicitly labeled subgroups. these tools and models typically involve grouping the data points according to some humanunderstandable concept (e.g., gender, race). clustering textual data requires them to be converted to a vector representation, like contextual embeddings, which gained popularity with the rise of pre-trained language models. sdms and error analysis tools frequently use contextual embeddings when developing their frameworks @xcite @xcite @xcite @xcite . embeddings of text from neural networks encode information that can go beyond the label and these interpretable features or subclasses are not prompt linguistic features embedding sentence-bert (sbert) llm generated linguistic features sentence transfomer concatenated embeddings llm dataset last hidden state language model embeddings generated a. finetune model b. generate embeddings for misclassified examples c. cluster finetuned model sbert embedding of example sbert embedding of linguistic features generated k-means forward pass on finetuned models to retrieve the [cls] token figure 1 : overview of the methodology followed in the paper. a. two pretrained language models are finetuned on offensive language datasets (one on english and one on arabic). b. we take the misclassified examples and generate embeddings to then cluster. we experiment with four types of embeddings: (1) last hidden state (lhs) are generated by extracting the [cls] token from the last layer of the finetuned models. (2) sentence-bert (sbert) are generated by running a sentence transformer model trained to generate semantically meaningful sentence embeddings. (3) linguistic features are generated by prompting an llm to generate linguistic features for the example, then the generated features are encoded using the same models as in embedding type 2. (4) concatenated embeddings are generated by concatenating the embeddings from 2 and 3. c. the final step is running k-means clustering on the generated embeddings. ## background ad-hoc approaches to understand model performance for nlp classification tasks involve manu-ally grouping the errors and giving each group/cluster a label. the process of having humans provide the label is laborious and subjective, leading to results that are often not reproducible @xcite . recent works that propose systematic error analysis frameworks for nlp classification tasks use clustering algorithms like k-means and hierarchical clustering to group misclassified instances in an attempt to understand where the model underperforms @xcite @xcite . similarly, popular slice detection models (sdm)s are based on gaussian mixture models (a generalized version of k-means clustering) @xcite @xcite . a popular vector representation used to cluster the textual data points is the last hidden layer of deep learning models, because it contains the learned representation of the entire sequence of tokens. when using pre-trained language models based on the transformer architecture like bert @xcite and roberta @xcite the final representation of their [cls] token is commonly used. these vector representations are used for understanding model performance in error analysis tools and sdms for nlp classification tasks @xcite @xcite . other works such as @xcite use sentence-bert (sbert) embeddings as the representation of the data points. sbert embeddings @xcite use siamese network structures @xcite to build a sequence-level text representation, which shows improvements over previous state-ofthe-art sentence embedding methods on semantic textual similarity tasks. prior works focused on quantitative evaluation of groups of embeddings with limited evaluation of how the choice of embedding approach might impact the final result @xcite @xcite . in this work we leverage two embedding types that have been commonly used to perform error analysis, last hidden state embeddings and sbert embeddings, to build representations of the misclassified examples. moreover, we propose a new method of representing errors which uses llms to generate linguistic features present in the errors. we evaluate the interpretability of the clusterings and provide insights into the type information the embeddings hold. ## conclusion contextual embeddings are frequently used as a vector representation of textual data when performing error analysis. in this work, we evaluate four types of text representations of erroneously classified text in the context of offensive language in english and arabic. we find that sbert clustering provides the most human-interpretable clustering of english text, with each cluster focusing mainly on one target group. for arabic we find that the sbert embeddings of llm generated features give the most interpretable clustering and the only approach to have above baseline performance. we notice the clustering of lhs in both english and arabic is based on the predicted label. this paper builds on a growing area of research in error analysis for offensive language detection and provides insights into what information about the errors is encoded in their representation. future work should explore other clustering algorithms and the effects of them on the interpretability and usefulness for error analysis, as well as automatic methods to generate informative labels about the clusters.