Implicit Hate Target Span Detection in Zero- and Few-Shot Settings with Selective Sub-Billion Parameter Models (2025)

Abstract: This work investigates the effectiveness of masked language models (MLMs) and autoregressive language models (LLMs) with fewer than one billion parameters in the detection of implicit hate speech through fine-grained span identification. The evaluation spans zero-shot, few-shot, and full supervision settings across two core benchmarks—SBIC and IHC—and an auxiliary testbed, OffensiveLang.RoBERTa-Large-355M emerges as the strongest zero-shot model, achieving the highest F1 scores of 75.8 (SBIC) and 72.5 (IHC), outperforming larger models like LLaMA 3.2-1B. ModernBERT-125M closely matches this performance with scores of 75.1 and 72.2, demonstrating the advantage of architectural efficiency. Among instruction-tuned models, SmolLM2-135M Instruct and LLaMA 3.2 1B Instruct consistently outperform their non-instructed counterparts, with up to +2.3 F1 gain on SBIC and +1.7 on IHC. Interestingly, the larger SmolLM2-360M Instruct does not outperform the 135M variant, highlighting that model scale does not always correlate with performance in implicit hate detection tasks.Few-shot fine-tuning with SmolLM2-135M Instruct achieves F1 scores of 68.2 (SBIC) and 64.0 (IHC), trailing full-data fine-tuning by only 1.6 and 2.0 points, respectively, with accuracy drops under 0.5 points. This illustrates the promise of compact, instruction-aligned models in data-scarce settings, particularly when optimized with Low-Rank Adaptation (LoRA).Topic-guided error analysis using Latent Dirichlet Allocation (LDA) reveals recurring model failures in ideologically charged or euphemistic discourse. Misclassifications often involve neutral references to identity, politics, or advocacy language, underscoring current limitations in discourse-level inference and sociopragmatic understanding.

## introduction warning: this paper contains offensive content and may be distressing. ## introduction warning: this paper contains offensive content and may be distressing. implicit hate speech is a covert and insidious form of prejudice that avoids overtly offensive language while still conveying harmful social attitudes or exclusionary ideologies. unlike explicit hate-typically marked by recognizable slurs or hostile phrasing-implicit hate is expressed through subtle lexical patterns, contextualized inferences, and culturally situated cues that require deeper semantic reasoning @xcite . this makes implicit hate particularly challenging to detect and annotate. crucially, the harmful implication often resides in localized linguistic expressions-such as group references, ideologically charged phrases, or euphemistic constructions-that serve as the semantic anchors of bias (see figure 1 ). identifying these implicit target spans is essential for token-level modeling, supporting more fine-grained supervision, enhancing interpretability, and enabling targeted interventions in applied settings such as moderation or legal auditing. sentence-level classification alone fails to capture the internal structure of implicitly hateful utterances, treating all tokens uniformly and offering limited interpretability and granularity @xcite . span-level identification addresses this shortcoming by isolating the linguistic elements responsible for the hateful implication. this capability is especially critical in high-stakes applications such as platform moderation, forensic audits, and responsible nlp pipelines, where traceability and accountability are essential. despite growing interest in implicit hate detection, most prior work has concentrated on binary classification @xcite . only a limited number of studies tackle the dual challenge of both detecting and localizing implicit bias within text @xcite . furthermore, while recent advances llms and mlms have demonstrated impressive zero-and few-shot capabilities, their ability to identify subtle, context-sensitive expressions of hate remains underexplored, particularly in low-resource training regimes @xcite . although larger llms have shown strong performance in explicit hate detection tasks @xcite , their deployment for implicit content must also consider efficiency and operational scalability. especially for deployment in real-world moderation systems or edge computing environments, lightweight models under 1 billion parameters present an attractive balance of interpretability, performance, and resource efficiency. in this study, we benchmark a diverse set of mlms and instruction-tuned llms-focusing exclusively on sub-billion parameter architectures-to evaluate their capacity to detect and ground implicit hate speech spans. our approach integrates instruction prompting with span-level supervision to test whether these models can infer indirect hostility across sbic, ihc, and offensive-lang datasets. to better understand the limitations of these systems, we perform a detailed error analysis using lda, a topic modeling technique that enables us to surface the latent themes behind systematic model failures. these include conflations of political discourse with hateful intent and misinterpretations of socio-cultural insinuations, revealing persistent challenges in context-aware language understanding. we organize our investigation around the following research questions: our main contributions are as follows: ## related work early approaches to hate speech detection predominantly relied on traditional machine learning methods, such as support vector machines (svms) and logistic regression, which leveraged handengineered linguistic features like n-grams, syntactic dependencies, and sentiment lexicons @xcite . while interpretable, these models lacked the capacity to capture nuanced or implicit hate speech, often leading to high false-negative rates and limited generalization across domains @xcite . the introduction of deep learning architectures, including recurrent neural networks (rnns) and bi-grus, improved sequence modeling by capturing contextual dependencies in text @xcite . however, the most substantial performance improvements came with transformer-based models such as bert @xcite , roberta @xcite , and hatebert @xcite , which enabled richer semantic representations. despite their effectiveness, these models were often trained on datasets dominated by explicit hate content, limiting their ability to recognize more subtle or indirect forms of toxicity. to address this, techniques like implicit target span detection (itsd) were proposed to identify the latent linguistic triggers of hate within a sentence @xcite . the emergence of llms such as llama @xcite and mistral @xcite has further pushed the boundaries of hate speech detection. these models demonstrate strong zeroshot and few-shot capabilities, particularly when instruction-tuned to align with task-specific goals @xcite . efficient fine-tuning strategies such as lora) @xcite offer scalable solutions for adapting large models to domain-specific tasks involving subtle and context-dependent hate expressions @xcite . beyond performance gains, recent research has emphasized interpretability and model behavior analysis. studies by @xcite investigate how llms represent and generalize hate-related knowledge. in parallel, rationaleguided methods @xcite and explanatory frameworks like hatexplain @xcite promote transparent decision-making by aligning model predictions with human-understandable justifications. data curation and augmentation also play a central role in enhancing detection systems. advances in dataset quality include the incorporation of hard negatives for robustness @xcite , gpt-driven paraphrastic augmentation for annotation diversity @xcite , and normalization techniques that reformulate hate speech into less toxic equivalents @xcite . additionally, the expansion of annotated resources-such as offen-sivelang, ihc, and vihos for vietnamese-has contributed to better cross-lingual generalization and cultural relevance in detection efforts @xcite . a complementary line of work explores the nar-rative framing of hate speech and its dissemination dynamics. for instance, antoniak et al. @xcite examine how storytelling structures influence the perception and spread of harmful content on social platforms, underscoring the need for models that account for discourse-level context. ## implicit target span identification implicit target span identification is a key subtask in detecting covert hate speech that lacks overtly toxic markers. the goal is to localize specific lexical spans using the standard bio (begin-inside-outside) tagging scheme. figure 2 presents the end-to-end pipeline for itsi. the process begins with standard input preprocessing, including tokenization, normalization, and subword segmentation. the processed input is then passed through a span detection model that outputs token-level bio labels. these predicted labels are subsequently used to extract span candidates, which are further refined through postprocessing. this includes merging overlapping spans and filtering out vague predictions. the final output is a set of contextually grounded target spans. we compare two modeling paradigms: (i) mlms, fine-tuned with supervised learning on annotated span data, and (ii) lightweight llms, evaluated under zero-and few-shot conditions using prompt-based inference. we conduct evaluations across sbic, ihc, and offensivelang datasets with consistent optimiza-tion protocols. mlms are trained using full supervision as well as limited-shot settings (5 and 10 examples per dataset). llms are prompted directly for span outputs. given the limited availability of humanannotated span-level data in the mentioned corpora, we employed an openai gpt model @xcite as an automated annotator to generate the span-level labels used in our experiments. our experimental design follows the research questions outlined in section 1, examining the influence of model scale (rq1), the benefits of instruction tuning (rq2), the viability of few-shot learning (rq3), and error analysis via topic-guided modeling using lda on false negatives to identify recurring blind spots in model behavior (rq4). an example of the span extraction prompt used for llm inference is provided in appendix a.9. ## error analysis our error analysis highlights several persistent challenges faced by contemporary hate speech detection systems, particularly in identifying subtle, context-dependent, or inferential forms of toxicity. a significant source of error arises from false positives, where models incorrectly classify nonhateful or mildly opinionated statements as offensive. for instance, the sentence "non-binary is a concept for progressive people" was frequently misclassified as hate speech. this suggests that current models disproportionately rely on surfacelevel lexical cues, reacting to the presence of politically charged or socially contentious terms such as "non-binary" and "progressive" without sufficient regard for the speaker's intent or the broader discourse context. such misclassifications indicate a fundamental limitation: an oversensitivity to identity-related language that is not inherently toxic, resulting in unwarranted censorship of benign or affirming discourse. in contrast, false negatives were prevalent in examples where toxicity was conveyed implicitly through tone, insinuation, or ideological framing. statements like "f-ing liberals and their policies" or "women are too emotional for politics" were frequently overlooked by the model, despite containing overt ideological hostility or gender-based stereotypes. these errors reveal that models struggle to capture offensive content when it is expressed indirectly, particularly in instances where dismissive tone, sarcasm, or culturally encoded bias replace explicit slurs. this suggests a systemic gap in the model's ability to detect the pragmatics of hate speech-namely, the subtle communicative acts through which social exclusion or denigration is performed. to better understand the structure of model misclassifications, we applied lda to the subset of false negatives from the sbic dataset. the model was implemented using scikit-learn, trained on tf-idf-weighted unigram representations of the misclassified examples. we initially extracted 10 latent topics, using default symmetric dirichlet priors (α = 1.0, β = 1.0) and trained for 1000 iterations. from these, we manually selected three themati-cally coherent topics for in-depth analysis, based on relevance to sociolinguistic bias. the resulting topics were visualized with matplotlib. figure 3 illustrates these three dominant clusters. the first is centered on feminist and gender-rights discourse, with salient terms such as "women," "rights," and "movements." models often misinterpret advocacy-focused or feminist language as neutral, missing subtle implications of group targeting. the second topic involves political ideology, including terms like "liberals," "progressive," and "values," indicating that politically charged but nontoxic language is frequently overlooked due to its subjective tone. the third topic relates to social identity and gender constructs, with terms such as "non," "binary," and "concept," where models struggle to identify implicit bias embedded in discussions of gender diversity. ## discussion and conclusion in addressing the impact of model size (rq1), our findings indicate that scaling parameter counts-particularly within the sub-1b range-does not guarantee better performance for implicit hate detection. roberta-large-355m consistently outperforms the much larger llama 3.2-1b, achieving top f1 scores on both sbic (75.8) and ihc (72.5). similarly, modernbert-125m matches llama's performance with a significantly smaller footprint, illustrating that architecture refinement and task-aligned pretraining objectives can outweigh sheer scale. this trend also holds among instruction-tuned models: smollm2-135m delivers competitive f1 scores that surpass its larger non-instructed sibling smollm2-360m, indicating diminishing returns from scaling when instruction alignment or domain adaptation is absent.