Can Prompting LLM s Unlock Hate Speech Detection across Languages? A Zero-shot and Few-shot Study (2025)

Abstract: Despite growing interest in automated hate speech detection, most existing approaches overlook the linguistic diversity of online content. Multilingual instruction-tuned large language models such as LLaMA, Aya, Qwen, and BloomZ offer promising capabilities across languages, but their effectiveness in identifying hate speech through zero-shot and few-shot prompting remains underexplored. This work evaluates LLM prompting-based detection across eight non-English languages, utilizing several prompting techniques and comparing them to fine-tuned encoder models. We show that while zero-shot and few-shot prompting lag behind fine-tuned encoder models on most of the real-world evaluation sets, they achieve better generalization on functional tests for hate speech detection. Our study also reveals that prompt design plays a critical role, with each language often requiring customized prompting techniques to maximize performance.

## introduction hate speech is a worldwide issue that undermines the safety of social media platforms, no matter the language @xcite . it can violate platform rules, damage user trust, influence opinions, and reinforce harmful biases against individuals or groups targeted @xcite @xcite . however, most recent advancements in hate speech detection have focused primarily on english, as the majority of datasets and language models are centered on english content. this has led to limited-but not negligible-attention to other languages @xcite . since users on social media write and engage with content in many languages-not just english-it is crucial to find tools that can detect hate speech across various languages. instruct-tuned large language models (llms) have demonstrated exceptional performance across a wide range of text-related tasks @xcite . many of these models possess multilingual capabilities, enabling them to process and understand text in various languages @xcite . this makes them suitable for tasks like hate speech detection, even without additional fine-tuning. although finetuning is possible, it requires computational and resource costs, which leads many users to rely on prompt-based use instead @xcite . while their effectiveness in detecting hate speech in english has been studied extensively @xcite @xcite , their performance on non-english datasets remains underexplored. to evaluate the capabilities of multilingual instruction-tuned llms in detecting hate speech in various languages, we conduct a study using several prompting techniques, including zero-shot prompting (e.g., vanilla, chain-of-thought, role-play), fewshot prompting, and combinations of these prompts. we evaluate performance across eight non-english hate speech detection tasks, covering spanish, portuguese, german, french, italian, turkish, hindi, and arabic, using real-world foot_1 and hate speech functional test sets. this study seeks to address the following research questions: (1) how well do llms perform on hate speech detection across various non-english languages? (2) does few-shot prompting improve performance compared to zero-shot prompting? (3) how does llm performance compare to that of traditional fine-tuned models? our findings highlight the importance of prompt design in multilingual hate speech detection. while performance varies by the prompting strategy, ex-perimenting with different techniques leads to reasonably strong results. in most languages, few-shot prompting combined with other techniques outperforms zero-shot prompting, suggesting that providing a few task-specific examples is beneficial. compared to fine-tuned encoder models, prompting llms shows lower performance on real-world test sets. however, in functional test cases, prompting often performs better. further analysis of languages where prompting underperforms on realworld data suggests that prompting can still be a practical option when only limited training data is available. nonetheless, with access to larger training sets, fine-tuning encoder models remains the more effective approach. overall, instruction-tuned llms demonstrate stronger generalization in controlled functional benchmarks, without the need for additional training. ## related work the ability of instruction-tuned llms to perform a wide range of nlp tasks without the need for finetuning or training data has drawn growing interest, particularly in applications like hate speech detection. recent studies have explored llm-based hate speech detection, primarily in english. @xcite reports low agreement between llm predictions and human annotations, while @xcite finds that llms are more effective at identifying non-hateful content. @xcite examines the use of llms for generating explanations of implicit hate, and @xcite shows that including target-specific information in prompts improves performance. another study examines how in-context learning, combined with few-shot examples and task descriptions, boosts the performance of hate speech detection by llms @xcite . @xcite investigates using llms for real-world hate speech detection using four diverse prompting strategies and finds that few-shot and chain-ofthought prompts help. while these works have explored prompting techniques, they primarily assess the capabilities of llms for hate speech detection in english and do not examine a broad range of prompting strategies across languages. there have been efforts to investigate the capabilities of llms for non-english hate speech. @xcite tested prompt strategies only in chinese and bangla, respectively. ahmad et al. (2025 ) used an llm for hate speech detection in urdu, outperforming bert in detecting both explicit and implicit hate. moving beyond isolated languages, @xcite assesses llms' sensitivity to geographical priming and persona attributes in five languages, showing that geographical cues can improve regional alignment in hate speech detection. similarly, zahid et al. (2025) uses geographical contextualization into prompts for five languages. these motivate our use of culture-aware prompts; however, these studies do not explore a wide range of prompting strategies, such as few-shot, chain-of-thought, etc. he et al. ( 2024 ) introduced a multilingual benchmark for offensive language detection in eight languages, focusing on offensive language and model alignment over prompt design. @xcite evaluate hate speech detection in eight languages using real-world and functional test sets, but rely solely on vanilla prompting. similarly, @xcite applied prompting llms to three low-resource south asian languages, finding that translating inputs to english outperformed prompting in the original language. this motivated us to prompt the llm to translate before classifying. in contrast to these efforts, our work covers eight languages and evaluates a broader range of prompt designs on real-world and functional test sets. ## datasets we selected datasets with explicit hate speech labels that adhere to definitions commonly used in social science and by social media platforms: abusive language that targets a protected group or individuals for being part of that group. the datasets, along with their overall sizes and the percentage of hateful instances, are summarized as follows: ous19_ar (ousidhoum et al., 2019): contains 3,353 arabic tweets, with 22.5% labeled as hateful. ous19_fr (ousidhoum et al., 2019): consist of 4,014 french tweets, with 11.0% labeled as hateful. bas19_es (basile et al., 2019): compiled for semeval 2019, it includes 4,950 spanish tweets, 41.5% of which are labeled as hateful. has21_hi @xcite : collected for hasoc 2021, it contains 4,594 hindi tweets, with 12.3% labeled hateful. san20_it @xcite : created for evalita 2020, it includes 8,100 italian tweets, 41.8% of which are hateful. for19_pt @xcite : consists of 5,670 portuguese tweets, with 31.5% labeled as hateful. gahd24_de @xcite : a german adversarial dataset consisting of 10,996 tweets, 42.4% of which are labeled as hateful. xdo-main_tr @xcite : a large-scale, multi-domain turkish dataset consisting of 38k samples, with a class imbalance rate of 74.4%. for functional hate speech evaluation, we used the hatecheck benchmark @xcite , a benchmark for evaluating the robustness of hate speech detection systems across languages. it includes functional test cases-controlled examples designed to test specific capabilities such as handling implicit hate, negation, and non-hateful slurs. originally developed for english, it has been extended by @xcite to multiple languages to support cross-lingual evaluation and reveal systematic model weaknesses not captured by standard datasets. ## models we evaluate four instruction-tuned multilingual llms for hate speech detection across eight languages: llama-3.1-8b-instruct @xcite ): meta's instruction-tuned decoder model, optimized for reasoning tasks and primarily designed for english, with multilingual support. qwen2.5-7b-instruct @xcite : a multilingual decoder model by alibaba cloud, supporting 30+ languages with strong instruction-following capabilities. aya-101 @xcite ): cohere's multilingual model trained on 100+ languages, tuned for equitable cross-lingual nlp, including hate speech detection. bloomz-7b1 @xcite ): a decoder model by bigscience, fine-tuned via multitask instruction tuning on 46 languages for cross-lingual instruction following. for the encoder-based baseline, we fine-tuned two multilingual models with strong performance on classification tasks: xlm-t @xcite : an xlm-r extension pre-trained on 198m twitter posts in 30+ languages. mdeberta @xcite : a multilingual encoder covering 100+ languages, effective in zero-shot and low-resource settings. see appendix a for model versions and additional details. ## experimental setup for each dataset, we randomly sampled 2,000 samples to serve as the test set for evaluating both prompting-based and fine-tuned models. due to limited dataset sizes, the test sets for arabic and french were restricted to 1,000 and 1,500 samples, respectively. instruction-tuned multilingual llms were evaluated in inference-only mode, without additional fine-tuning, on both real-world and functional test sets. the models were prompted such that they responded with yes if the input text was hateful and no otherwise. each experiment was repeated with three random seeds, and we alternated the order of yes and no in the prompt to reduce positional bias. for the encoder-based models, after setting aside the test set, we held out 500 samples for validation and used the rest for training. after training, we evaluated the models on both their respective test sets, representing real-world evaluation, and on their language-specific subsets of the hatecheck benchmark, representing functional test evaluation. model outputs and labels were mapped to binary values: 0 for non-hateful and 1 for hateful. each experiment was run with five different random seeds, and the final results were averaged across these runs. moreover, since several of the datasets are imbalanced, we report f1-macro as the primary evaluation metric to ensure fair assessment across classes. further implementation details and hyperparameters are provided in appendix a. ## prompts we assess instruction-tuned multilingual llms using a range of prompting strategies for hate speech detection, such as: directly asking whether a comment is hateful (vanilla); prompting the model to act as a classifier (classification); chainof-thought prompting for step-by-step reasoning (cot); natural language inference-inspired prompts (nli); language-aware prompts that consider linguistic and cultural context (cultural); assigning the llm the role of a community moderator (roleplay); translate then classify prompts (translation); definition-based prompts that explain hate speech (definition); and defining related forms of abusive content to help the model differentiate them from hate speech (distinction), etc. we also include fewshot prompting, where we retrieve and insert example instances from the training set into the prompt. we also explore combinations of these strategies. for full prompt texts and implementation details, see appendix b. ## results we evaluate instruction-tuned llms with various prompt types over three runs in the inference mode and report the average f1-macro scores. in zero-shot settings, qwen and llama3 generally outperform the other models, with similar overall performance. however, qwen performs better in most real-world test cases, whereas llama3 leads on functional benchmarks. few-shot prompting (typically five-shot) improves performance, especially on functional tests, as examples help the model apply contextual distinctions more effectively. on real-world tests, improvement is less consistent-even with examples from the same training data. this suggests that few-shot effectiveness depends not only on data quality but also on prompt clarity and structure. overall, instructiontuned llms perform notably well on functional tests and reasonably well on real-world tests in different languages. however, their effectiveness depends heavily on prompt design and the inclusion of few-shot examples. appendix d contains detailed performance results. to understand when prompting is preferable, we conducted additional experiments comparing encoder model performance at varying training set sizes to that of prompting. figure 1 presents results for three languages where prompting underperforms compared to fine-tuned models. depending on the language, prompting becomes competitive when training data is limited-for example, with 100-200 examples in spanish, 300-400 in hindi, or 600-700 in german. beyond that, finetuning generally yields better performance. see appendix c for more results across other languages. ## conclusion in this study, we explore the capabilities of multilingual instruction-tuned llms in detecting hate speech across eight non-english languages. the findings suggest that different prompting techniques work better for different languages, indicating that it is beneficial to experiment with various prompt designs when addressing a new language. in real-world scenarios, where the data is more culturally dependent, prompting llms is less effective than training encoder models with taskspecific data. however, in functional hate speech tests, llms tend to perform better and offer more flexibility. incorporating few-shot examples into prompts in such cases may further enhance the llms' performance.