Sensitive Content Classification in Social Media: A Holistic Resource and Evaluation (2025)

Abstract: The detection of sensitive content in large datasets is crucial for ensuring that shared and analysed data is free from harmful material. However, current moderation tools, such as external APIs, suffer from limitations in customisation, accuracy across diverse sensitive categories, and privacy concerns. Additionally, existing datasets and open-source models focus predominantly on toxic language, leaving gaps in detecting other sensitive categories such as substance abuse or self-harm. In this paper, we put forward a unified dataset tailored for social media content moderation across six sensitive categories: conflictual language, profanity,sexually explicit material, drug-related content, self-harm, and spam. By collecting and annotating data with consistent retrieval strategies and guidelines, we address the shortcomings of previous focalised research. Our analysis demonstrates that fine-tuning large language models (LLMs) on this novel dataset yields significant improvements in detection performance compared to open off-the-shelf models such as LLaMA, and even proprietary OpenAI models, which underperform by 10-15% overall. This limitation is even more pronounced on popular moderation APIs, which cannot be easily tailored to specific sensitive content categories, among others.

## introduction consider the case of a researcher or a data analyst who needs to filter sensitive content from a large dataset. such task is crucial to ensure that data shared or analysed does not include harmful or inappropriate material. one might initially consider using external tools like perspective foot_0 or openai moderation apis foot_1 to assess and filter sensitive content. however, this approach often falls short, presenting important limitations for an effective identification of inappropriate content online @xcite . for instance, they usually offer limited customisation capabilities (e.g., how can the model be improved if it fails on specific domains or keywords?), and limited sensitive categories coverage (lacking in detecting self-harm @xcite , for example). finally, these tools rely on external servers, which raises concerns about data privacy and security @xcite . alternatively, one might consider using existing datasets and open-source models for sensitive content detection. this could be a viable option if the primary focus was on detecting toxic language, given the abundance of resources available in this area. however, if the goal extends to identifying additional sensitive categories such as sexually explicit content, drugs, self-harm or spam, the situation becomes more challenging. data on these less-explored categories is limited and sometimes outdated. for instance, those categories could be covered on datasets that are biased @xcite , old or inaccessible even in an anonymized manner @xcite , too small-scaled or rely on an handful of keywords to extract the data @xcite . this limited approach can result in incomplete or less accurate detection of sensitive content. existing solutions either require sending data to external servers or fail to address the full spectrum of sensitive content categories. in response to these challenges, this paper proposes a new holistic ap-proach: a unified dataset for detecting sensitive text across a broad range of categories, including (1) conflictual language, (2) profanity, (3) sexually explicit material, (4) drug-related content, (5) selfharm, and (6) spam foot_2 . this dataset can then be used for both evaluation and fine-tuned models to all these categories under a single framework. our approach involves collecting and reannotating data to ensure consistent quality across sensitive classes. the alternative of putting together a collection of existing datasets to create one single dataset would include several limitations, as (1) there would be bias towards the data distribution (different retrieval strategies, topics, source platforms), (2) annotation guidelines and quality would differ, and (3) each text would include only one sensitive dimension (even if the text includes multiple sensitive categories). in short, we propose a holistic approach when it comes to sensitive content moderation in social media, overcoming common shortcomings of previous works and providing the following contributions: the x-sensitive dataset, as well as the best performing models built upon it, are made openly available. x-sensitive is available at https://huggingface. co/datasets/cardiffnlp/x_sensitive . best multi-label and binary models are available at https://huggingface.co/ cardiffnlp/twitter-roberta-largesensitive-multilabel and https: //huggingface.co/cardiffnlp/ twitter-roberta-large-sensitive-binary, respectively. ## related work our current work aims to bridge the gap between current academic research in content moderation and the needs of content moderators in realistic scenarios. while hate speech and toxic language are widely studied in nlp, there is little research on other types of sensitive content that platforms seek to detect and moderate, such as sexually explicit content or content about illicit substances @xcite . to that end, our work is situated at the intersection of nlp research on harmful language detection and research on platform governance and content moderation. ## x-sensitive dataset in order to study sensitive content in x, we construct a new dataset, x-sensitive. as a first step, we conceptualise a topic taxonomy based on community guidelines from several social media platforms. ## results the scores for both binary and multi-label scenarios, across all models tested in the fine-tuning and zero-/few-shot settings, are presented in in general, fine-tuning leads to clear improvement for all models, which reinforces the importance of our dataset not only to evaluate models, but to build specialised models based based on it. llama3-8b performs best overall, with macro-f1 scores of 85.6 in the binary setting, and 69.8 in the more challenging and fine-grained multi-label setting. ## error analysis aiming to understand better the dataset and the challenges that the models face on identifying sensitive content we consider the best performing model, the fine-tuned llama3-8b, and try to understand better its performance. in the binary setting the model displays a strong performance and achieves high precision and recall values, 85.5%, 85.7% respectively, signifying its ability to effectively identify true positive cases with a relatively low number of false positives (figure 4 , appendix c.4). in contrast, for the multi-label setting, the model seems to struggle with several categories as seen in figure 2 . despite the strong performance of llama-8b in classifying profanity and sex labels, the model struggles with the conflictual, drugs, self-harm, and spam categories. as recall increases, precision for these categories drops significantly. this poses a particular challenge for health-related categories like drugs and self-harm, where high recall is critical, as missing cases could have serious consequences. ## conclusions in this paper, we presented a complete research approach into sensitive content moderation in social media. going beyond hate speech, we focus on categories that need to consistently be monitored in social media, let it be to filter to adult users or to remove from the platform, among others. we construct a multi-label dataset using six categories. the results show that lms fine-tuned on our datasets are generally robust, although there are some categories where they are less precise, and hence these models are probably to be used as a support for human moderators. nonetheless, the fact that these models perform at a high accuracy represents a useful tool to filter the most relevant messages for each category.