Evaluating the Effectiveness of Natural Language Inference for Hate Speech Detection in Languages with Limited Labeled Data (2023)

Abstract: Most research on hate speech detection has focused on English where a sizeable amount of labeled training data is available. However, to expand hate speech detection into more languages, approaches that require minimal training data are needed. In this paper, we test whether natural language inference (NLI) models which perform well in zero- and few-shot settings can benefit hate speech detection performance in scenarios where only a limited amount of labeled data is available in the target language. Our evaluation on five languages demonstrates large performance improvements of NLI fine-tuning over direct fine-tuning in the target language. However, the effectiveness of previous work that proposed intermediate fine-tuning on English data is hard to match. Only in settings where the English training data does not match the test domain, can our customised NLI-formulation outperform intermediate fine-tuning on English. Based on our extensive experiments, we propose a set of recommendations for hate speech detection in languages where minimal labeled training data is available.

## introduction hate speech is a global issue that transcends linguistic boundaries, but the majority of available datasets for hate speech detection are in english @xcite . this limits the capabilities of automatic content moderation and leaves most language communities around the world underserved. creating labeled datasets is not only slow and expensive but also risks psychological impacts on the annotators @xcite . although the number of non-english datasets is increasing, most languages still have limited or no datasets available @xcite . 1 we make our code publically available at https:// github.com/jagol/xnli4xhsd . @xcite . via c) and d), we explore natural language inference as an additional intermediate fine-tuning step. the natural language inference formulation further allows hypothesis engineering @xcite . consequently, there is a pressing need to develop methods that can efficiently expand hate speech detection into languages with less labeled data. repurposing natural language inference models for text classification leads to well-performing zero-shot and few-shot classifiers @xcite . recently, @xcite showed that zeroshot nli-based setups can outperform standard few-shot finetuning for english hate speech detection. this raises the question of whether natural language inference can be used to expand hate speech detection into more languages in a data-efficient manner. in this paper, we aim to systematically evaluate the effectiveness of nli fine-tuning for languages beyond english where we do not have an abun-dance of labeled training data. we give an overview of the experiment setup and compared approaches in figure 1 . unlike @xcite , and inspired by @xcite , we do not restrict ourselves to a zero-shot setup but further analyse the usefulness of an nli-formulation when we have access to a limited amount of labeled examples in the target language as well as additional english data for intermediate fine-tuning. we believe that this mirrors a more realistic setup and allows us to offer clear recommendations for best practices for hate speech detection in languages with limited labeled data. our experiments with 0 up to 2000 labeled examples across five target languages (arabic, hindi, italian, portuguese, and spanish) demonstrate clear benefits of an nli-based formulation in zero-shot and few-shot settings compared to standard fewshot fine-tuning in the target language. while @xcite approach of fine-tuning on english data before standard few-shot learning on the target language proves to be a strong baseline, we reach similar performance when fine-tuning nli-based models on intermediate english data. building on the results by @xcite , who showed that targeted hypothesis engineering can help avoid common classification errors with nli-based models, we find that such strategies offer an advantage in scenarios where we have expert knowledge about the domain but no indomain english data for intermediate fine-tuning. overall, our contributions are the following: ## related work hate speech is commonly defined as attacking, abusive or discriminatory language that targets protected groups or an individual for being a member of a protected group. a protected group is defined by characteristics such as gender, sexual orientation, disability, race, religion, national origin or similar @xcite @xcite . the automatic detection of hate speech is typically formulated as a binary text classification task with short texts, usually social media posts and comments, as input @xcite . despite most work being focused on english @xcite , in the last years there has been a growing trend to expand into more languages @xcite @xcite . in what follows, we first review the relevant literature for multi-and cross-lingual hate speech detection, and then move on to the previous work in zero-shot and few-shot text classification with a specific focus on nli-based methods, and finally focus on hypothesis engineering @xcite . ## experiment setup our experiment setup is largely based on the one created by @xcite and can be seen in figure 2 . in what follows, we first describe their setup, which we replicated as a baseline for our results. we then describe how we expanded their setup for the nli-based experiments. ## evaluation following @xcite , we evaluate each setting displayed in figure 2 on two test-sets: in what follows, we group the results according to research questions. the full results are given in appendix d. overall, we view our results as a confirmation of their findings. in order to simplify our evaluation, and since x + fen and x + ken have very similar results, we will only include x + fen as a repre- ## conclusion in this work, we systematically evaluated the effectiveness of an nli task formulation for hate speech detection in scenarios where only few labeled data in the target language are available. we were able to reproduce results by @xcite , who showed that intermediate fine-tuning on english hate speech is beneficial in such scenarios. following their setup with our nli-based experiments, we answered the following questions: (1) how does nli fine-tuning compare to english hate speech fine-tuning? our results showed that while nli fine-tuning leads to strong improvements over only fine-tuning in the target language it is outperformed by english hate speech fine-tuning. (2) are there benefits to combining nli fine-tuning with english hate speech fine-tuning? we observed minor improvements when only 0 or 20 target language examples are available, but no benefits when more examples are available. (3) can hypothesis engineering further improve performance on the previously best nli-based setting? our experiments demonstrated that hypothesis engineering can outperform other approaches only when the domain of the input data at inference time is known, but no matching training data is available. based on our results, we offer the following recommendations for hate speech detection in languages where little labeled data is available in the target language: finally, we want to highlight two areas for future work that arise from our experiments: