Fine-Grained Transfer Learning for Harmful Content Detection through Label-Specific Soft Prompt Tuning (2025)

Abstract: The spread of harmful content online is a dynamic issue evolving over time. Existing detection models, reliant on static data, are becoming less effective and generalizable. Developing new models requires sufficient up-to-date data, which is challenging. A potential solution is to combine existing datasets with minimal new data. However, detection tasks vary—some focus on hate speech, offensive, or abusive content, which differ in the intent to harm, while others focus on identifying targets of harmful speech such as racism, sexism, etc.—raising the challenge of handling nuanced class differences. To address these issues, we introduce a novel transfer learning method that leverages class-specific knowledge to enhance harmful content detection. In our approach, we first present label-specific soft prompt tuning, which captures and represents class-level information. Secondly, we propose two approaches to transfer this fine-grained knowledge from source (existing tasks) to target (unseen and new tasks): initializing the target task prompts from source prompts and using an attention mechanism that learns and adjusts attention scores to utilize the most relevant information from source prompts. Experiments demonstrate significant improvements in harmful content detection across English and German datasets, highlighting the effectiveness of label-specific representations and knowledge transfer.

## introduction the increasing proliferation of harmful content and its evolving nature require effective and generalizable detection methods. one solution for detecting hateful content in new domains involves preparing new adequate data, which presents challenges in data collection and annotation. another approach is to utilize existing datasets and transfer their knowledge. however, directly applying these datasets for transfer learning is not straightforward. various datasets have been developed to identify offensive language, hate speech, sexism, or racism, but nuanced differences among classes, variations in annotation styles, and differences in scope make it challenging to apply these datasets to new, unseen tasks @xcite @xcite . in contrast to previous work that addresses these challenges through data augmentation @xcite , domain-adapted models @xcite , or specific model instructions @xcite , we argue that effectively differentiating between classes of harmful language requires modeling fine-grained class representations. with these representations, we can analyze label information across multiple tasks @xcite and transfer knowledge between tasks in a fine-grained manner @xcite . large language models (llms) @xcite @xcite have demonstrated strong performance across various tasks, but fine-tuning them is challenging and costly. parameter-efficient fine-tuning (peft) addresses this by training only a small number of parameters @xcite @xcite . our method builds on soft prompt tuning (sp), a peft approach introduced by @xcite , using fewer parameters to achieve competitive results. sp scales well with larger models and effectively captures tasklevel information @xcite . however, it learns a single-task representation, which we argue is insufficient for harmful content detection, where more fine-grained classlevel representations are necessary. we present an edition of the sp approach for harmful text detection by introducing label-specific for example, for a task with two classes, our method employs two label-specific prompts, prompting the llm to predict if the given label-specific prompt and the instance's label match. recent work leveraged transfer learning to boost performance in low resource settings @xcite . however, finding the right existing tasks (source) for a new unseen task (target) is not straightforward. firstly, harmful content detection datasets often have partly overlapping label sets, e.g. the @xcite dataset has labels hateful, offensive and normal, while the dataset of @xcite has hateful, abusive, spam and normal. secondly, labels of the same name often have different definitions, e.g. hate speech and offensive are defined as two separate classes in some datasets @xcite ; however, some others include hate speech in the offensive class @xcite . label-specific soft prompts enable fine-grained knowledge transfer between tasks. instead of transferring all knowledge, including irrelevant or misaligned labels, we propose transferring only essential information by selecting the appropriate label soft prompts. this can be achieved simply by initializing the label-specific soft prompts with the same type of source prompts directly or by aver-aging them as depicted in the left side of figure 2 . alternatively, as shown on the right side of figure 2 , the model can be trained to select the required information from source label prompts autonomously. to achieve this, we use an attentional mixture of soft prompts across all label prompts, allowing the model to learn how to utilize them effectively. our proposed attentional transfer learning method, designed for label-specific learning and inspired by @xcite , automatically identifies which source label prompts are most useful for each target label. additionally, this method measures the relation between source and target labels by providing attention scores. evaluation of our fine-grained transfer learning method across five tasks demonstrates the effectiveness of label-specific knowledge transfer compared to baseline methods. based on label types (names), the initialization method performs well, showing that label names can guide the transfer and selection of source label prompts. the attention method, by contrast, removes the dependency on specific label types and uses attention scores to identify the most effective source label prompts for the target labels. more importantly, it outperforms other methods, demonstrating that target labels can benefit from a broader range of source prompts, rather than being limited to those of the same type. additionally, our first stage-label-specific source prompt tuning-outperforms task-specific prompt tuning, even when the tunable parameters for taskspecific prompts are increased. overall, our method exhibits these characteristics: transfer learning for harmful text detection. fine-tuning pre-trained language models (plm) for harmful content detection is a well-established research approach, focusing on adapting llms to specific tasks such as detecting harmful text. in contrast, hard prompting methods, which involve manually designing textual templates and querying the model @xcite @xcite , can be less robust. these methods rely on altering prompts to change responses, which often lacks the stability and effectiveness achieved through fine-tuning. hatebert @xcite retrained the bert @xcite model for detecting abusive language. sarkar et al. (2021) and okpala et al. (2022) applied transfer learning techniques to offensive language identification. sabry et al. (2022) explore the effectiveness of t5 for hate speech detection by investigating data augmentation techniques and ensemble methods, and adewumi et al. (2023) introduces hat5, a text-to-text transformer based on t5. however, these papers did not consider various types of harmful content and the generalizability of their transfer learning methods. while a prior study he et al. (2024) explored soft prompts for general toxic content detection without modification, our work focuses on harmful contents with label-specific prompts tuned using peft methods. this allows our model to capture the subtleties of different harmful categories. incorporating label-specific information in training. work such as wang et al. (2018); li et al. (2022); liu et al. (2022b) utilizes label embeddings to create more informative text representations and enhance text classification. in the paper by xiao et al. ( 2019 ), a multi-view attention mechanism is proposed to learn label-specific representations for text classification. @xcite introduce a text classification approach using only label types for self-training llms. none of these works measured the knowledge transfer of their label-specific representations. müller et al. (2022) introduced label tuning, which adapts models to new tasks by fine-tuning label embeddings. unlike our method, they use label text as hard prompts and transfer encoder weights instead of soft prompts, which is less efficient for transfer learning, especially when dealing with multiple source tasks. generalization capability to diverse harmful detection tasks. recent research highlights concerns about the generalizability of current hate speech detection models @xcite . although they perform well on their own test sets, their accuracy drops significantly when tested on different datasets @xcite . this indicates that existing test data do not accurately represent future cases, leading to overestimating these models' generalization capabilities @xcite . @xcite analyze hate speech detection models, finding topic-diverse training data improves their ability to generalize across different hate speech types. @xcite addresses abusive content detection by training a model on multiple datasets and adapting it to specific tasks with minimal data, achieving strong multilingual performance. however, retraining a model is less efficient compared to using soft prompts. our experiments demonstrate that our approach has strong generalization capabilities and efficiently transfers fine-grained knowledge to target tasks. ## methodology the methodology is organized into two stages: source prompt tuning (label-specific soft prompt tuning) and target prompt tuning (transfer learning for label soft prompts). the second stage has two approaches: initialization and attention-based methods. each subsection starts with a problem statement, followed by its formulations. ## evaluation although our main contribution is the transfer learning of label-specific soft prompts, to maintain coherence with the methodology section, we first present the evaluation and visualizations of the first stage. then, we discuss the two transfer learning methods: initializing and attentional, in that order. ## conclusion we introduce transfer learning of label-specific soft prompt tuning for harmful content detection, where prompts are tailored to individual labels to capture fine-grained distinctions between content types. moreover, we propose attentional knowledge transfer at the class level, enabling the model to learn label nuances from source prompts. experimental results show that label-specific prompt tuning improves performance on english and german datasets, achieving higher f1-macro scores than the baseline. few-shot experiments further validate the method, demonstrating enhanced performance in target tasks with limited data.