Data-Efficient Strategies for Expanding Hate Speech Detection into Under-Resourced Languages (2022)

Abstract: Hate speech is a global phenomenon, but most hate speech datasets so far focus on English-language content. This hinders the development of more effective hate speech detection models in hundreds of languages spoken by billions across the world. More data is needed, but annotating hateful content is expensive, time-consuming and potentially harmful to annotators. To mitigate these issues, we explore data-efficient strategies for expanding hate speech detection into under-resourced languages. In a series of experiments with mono- and multilingual models across five non-English languages, we find that 1) a small amount of target-language fine-tuning data is needed to achieve strong performance, 2) the benefits of using more such data decrease exponentially, and 3) initial fine-tuning on readily-available English data can partially substitute target-language data and improve model generalisability. Based on these findings, we formulate actionable recommendations for hate speech detection in low-resource language settings.

## introduction hate speech is a global phenomenon, but most hate speech datasets so far focus on english-language content @xcite . this hinders the development of effective models for detecting hate speech in other languages. as a consequence, billions of non-english speakers across the world are less protected against online hate, and even giant social media platforms have clear language gaps in their content moderation systems @xcite . zero-shot cross-lingual transfer, where large multilingual language models are fine-tuned on one source language and then applied to another target language, may appear like a potential solution to the issue of language-specific resource mhc is multilingual hatecheck @xcite . so, how do we build hate speech detection models for hundreds more languages? we need at least some labelled data in the target language to make models effective, but data annotation is difficult, time-consuming and expensive. it requires resources that are often very limited for non-english languages. annotating hateful content in particular also risks exposing annotators to harm in the process @xcite . in this article, we explore strategies for hate speech detection in under-resourced languages that make efficient use of labelled data, to build effective models while also minimising annotation cost and risk of harm to annotators. for this purpose, we conduct a series of experiments using mono-and multilingual models fine-tuned on differently-sized random samples of labelled hate speech data in english as well as arabic, spanish, hindi, portuguese and italian. our key findings are: based on these findings, we formulate and discuss five recommendations for expanding hate speech detection into under-resourced languages: ## discussion and recommendations based on the sum of our experimental results, we formulate five recommendations for the dataefficient development of effective hate speech detection models for under-resourced languages. ## related work cross-lingual transfer learning only a very small number of the over 7,000 languages in the world are well-represented in natural language processing resources @xcite . this has motivated much research into cross-lingual transfer, where large multilingual language models are finetuned on one (usually high-resource) source language and then applied to another (usually underresourced) target language. cross-lingual transfer has been systematically evaluated for a wide range of nlp tasks and for varying amounts of shots, i.e. target-language training data. evidence on the efficacy of zero-shot transfer is mixed, across tasks and for transfer between different language pairs @xcite @xcite @xcite @xcite . in comparison, few-shot approaches are generally found to be more effective @xcite @xcite . in many-shot settings, multilingual models are often outperformed by their languagespecific monolingual counterparts @xcite @xcite . in this paper, we confirmed and expanded on key results from this body of prior work, such as the effectiveness of few-shot learning, specifically for the task of hate speech detection. we also introduced ols linear regression as an effective method for quantifying data efficiency, i.e. dynamic costbenefit trade-offs in data annotation. ## conclusion in this paper, we explored strategies for hate speech detection in under-resourced language settings that are effective while also minimising annotation cost and risk of harm to annotators. we conducted a series of experiments using mono-and multilingual language models finetuned on differently-sized random samples of labelled hate speech data in english as well as arabic, spanish, hindi, portuguese and italian. we evaluated all models on held-out test sets as well as out-of-domain target-language test suites from multilingual hatecheck, and used ols linear regression to quantify the marginal benefits of targetlanguage fine-tuning data. in our experiments, we found that 1) a small amount of target-language fine-tuning data is needed to achieve strong performance on held-out test sets, 2) the benefits of using more such data decrease exponentially, and 3) initial fine-tuning on readily-available english data can partially substitute target-language data and improve model generalisability. based on our findings, we formulated five actionable recommendations for expanding hate speech detection into under-resourced languages. most hate speech research and resources so far have focused on english-language content. however, hate speech is a global phenomenon, and hate speech detection models are needed in hundreds more languages. with our findings and recommendations, we hope that we can facilitate the development of models for yet-unserved languages and thus help to close language gaps that right now leave billions of non-english speakers world less protected against online hate. this article was completed during paul röttger's 2022 research visit to dirk hovy's research group at bocconi university in milan. paul röttger was funded by the german academic scholarship foundation. debora nozza received funding from fondazione cariplo (grant no. 2020-4288, monica). federico bianchi was a member of the bocconi institute for data science and analysis (bidsa) when parts of this work were completed. dirk hovy received funding from the european research council (erc) under the european union's horizon 2020 research and innovation program (grant agreement no. 949944). debora nozza and dirk hovy are members of the data and marketing insights (dmi) unit of the bocconi institute for data science and analysis ( @xcite . we thank the milanlp lab and the pierrehumbert language modelling group for their helpful comments, and all reviewers for their constructive feedback.