Multilingual Auxiliary Tasks Training: Bridging the Gap between Languages for Zero-Shot Transfer of Hate Speech Detection Models (2022)

Abstract: Zero-shot cross-lingual transfer learning has been shown to be highly challenging for tasks involving a lot of linguistic specificities or when a cultural gap is present between lan- guages, such as in hate speech detection. In this paper, we highlight this limitation for hate speech detection in several domains and languages using strict experimental settings. Then, we propose to train on multilingual auxiliary tasks – sentiment analysis, named entity recognition, and tasks relying on syntactic information – to improve zero-shot transfer of hate speech detection models across languages. We show how hate speech detection models benefit from a cross-lingual knowledge proxy brought by auxiliary tasks fine-tuning and highlight these tasks’ positive impact on bridging the hate speech linguistic and cultural gap between languages.

## introduction given the impact social media hate speech can have on our society as a whole -leading to many small-scale overton window effects -the nlp community has devoted considerable efforts to automatic hate speech detection using machine learning-based approaches, and proposed different benchmarks and datasets to evaluate their techniques @xcite @xcite @xcite . however, these systems are designed to be efficient at a given point in time for a specific type of online content they were trained on. as hate speech varies significantly diachronically @xcite and synchronically @xcite , hate speech detection models need to be constantly adapted to new contexts. for example, as noted by @xcite , the occurrence of new hate speech domains and their associated * these authors contributed equally. in this low-resource scenario for a given target language and domain, if annotated data is available in another language, the main option for most nlp tasks is to perform zero-shot transfer using a multilingual language model @xcite . however, in our case, hate speech perception is highly variable across languages and cultures; for example, some slur expressions can be considered not offensive in one language, denoting an informal register nonetheless, but will be considered offensive, if not hateful, in another @xcite . despite the cross-lingual transfer paradigm being extensively used in hate speech detection to cope with the data scarcity issue @xcite @xcite or even the use of models trained on a translation of the initial training data @xcite , this strong hate speech cultural and linguistic variation can lower the transferability of hate speech detection models across languages in a zero-shot setting. to overcome this limitation, in the absence of training data or efficient translation models for a target language, the cultural and linguistic information specific to this language needs to be found elsewhere. in this paper, we propose to capture this information by fine-tuning the language model on resource-rich tasks in both the transfer's source and target language. indeed, even though hateannotated datasets are not available in both lan-guages, it is likely that similarly annotated data in the source and target language exist for other tasks. a language model jointly fine-tuned for this other task in the two languages can learn some patterns and knowledge, bridging the gap between the languages, and helping the hate speech detection model to be transferred between them. in summary, our work focuses on zero-shots cross-language multitask architectures where annotated hate speech data is available only for one source language, but some annotated data for other tasks can be accessed in both the source and target languages. using a multitask architecture (van der goot et al., 2021b) on top of a multilingual model, we investigate the impact of auxiliary tasks operating at different sentence linguistics levels (pos tagging, named entity recognition (ner), dependency parsing and sentiment analysis) on the transfer effectiveness. using nozza (2021)'s original set of languages and datasets (hate speech against women and immigrants, from twitter datasets in english, italian and spanish), our main contributions are as follows. training on intermediate labeled-data tasks (stilt) by phang et al. (2018), who perform sequential task-to-task pre-training. more recently, pruksachatkun et al. (2020) perform a survey of intermediate and target task pairs to analyze the usefulness of this intermediary fine-tuning, but only in a monolingual setting. phang et al. ( 2020 ) turn towards cross-lingual stilt. they fine-tune a language model on nine intermediate language-understanding tasks in english and apply it to a set of non-english target tasks. they show that machine-translating intermediate task data for training or using a multilingual language model does not improve the transfer compared to english training data. however, to the best of our knowledge, using intermediate task training data on both the source and the target language for transfer has not been tested in the literature. auxiliary tasks for hate speech detection. auxiliary task training for hate speech detection has been done almost exclusively with the sentiment analysis task @xcite , and only in monolingual scenarios. but additional information is sometimes added to the hate speech classifier differently. @xcite , among the best systems on the haspeede task of evalita 2020, use pos-tagged text as input of the classification systems, which is highly beneficial for spanish and a bit less for german and english. furthermore, the effect of syntactic information is also investigated by narang and brew (2020), using classifiers based on the syntactic structure of the text for abusive language detection. @xcite evaluate the impact of manually extracted pos, stylometric and emotion-based features on hate speech detection, showing that the latter two are robust features for hate speech detection across languages. zero-shot cross-lingual transfer for hate speech detection due to the lack of annotated data on many languages and domains for hate speech detection, zero-shot cross-lingual transfer has been tackled a lot in the literature. among the most recent work, @xcite investigates the impact of a preliminary training of a classification model on hate speech data languages different from the target language; they show that language models pre-trained on a small number of languages benefit more of this intermediate training, and often out-performs massively multilingual language models. to perform cross-lingual experiment, glavaš et al. ## auxiliary tasks experiments we define several training tasks whose effects on cross-lingual transfer of hate speech detection mod-els are to be evaluated: a sequence-level task, sentiment analysis, and several token-level tasks: named entity recognition (ner) and a set syntactic tasks that we group -by misnomer -under the term "universal dependency" (ud). we hypothesize that sentiment analysis and ner tasks allow the model to learn high-level, semantic information, while the ud tasks convey syntactic skills to the model. ## results on auxiliary tasks training we analyze the training effect of adding different auxiliary tasks on top of xlm-r, jointly with monolingual hate speech detection. results can be found in to help with the interpretation, we aggregate the results according to the monolingual (mono), and zero-shot cross-lingual (cross) settings. in the zero-shot cross-lingual transfer scenario, we hypothesized that the additional information on the source and target languages could bridge the gap between the languages and improve the transfer for hate speech detection. looking at the scores for cross-lingual transfer, sentiment analysis and ner lead to an average improvement of respectively of 1.42 and 1.48 points for the immigrants domains; combined (last row of ## diagnosis: effect of auxiliary task training there is an extensive literature on how performance metrics aggregated over the full test set are far from conveying enough information to fully evaluate and compare the strengths and weaknesses of models @xcite , including for the task of hate speech detection @xcite . here, we use the hatecheck test suite in english @xcite and its recent multilingual version mhc @xcite , which includes our two other target languages, spanish and italian. these are test sets covering a wide range of hate speech detection aspects that the authors call functionalities, testing detection models with hateful and non-hateful sentences of various styles, vocabulary, syntax and hate speech targets. all 29 functionalities are grouped into 11 classes and 7 protected groups as targets foot_7 , and the various test cases of each functionality lead to a total of 3,901 sentences classified as hateful or not hateful. the protected groups vary across languages in the mhc test set; the authors selected them to better adapt to the cultural context of each language. the target group "women" is covered for our three languages, but the target group "immigrants" is not covered in spanish; instead, we match it with the group "indigenous people". 10 moreover, to ease the interpretation, we perform the analysis on the aggregated 11 classes of functionalities. we do not evaluate the performance of our various models on the test suite intrinsically: what we want to measure is the effect of zero-shot crosslingual transfer and auxiliary tasks training on the hate speech functionalities. first, we measure the difference between monolingual and zero-shot cross-lingual training on the various functionalities: what the model "loses" by not being trained on the same language as the test set. we rank the functionalities by average difference across the two domains ( the impact of multilingual auxiliary task training compared to training on hate speech detection only (baseline model), on the various functionalities. for the two domains and for each source-target language pair, we measure the hatecheck functionality score of the baseline model, and jointly on every auxiliary task. for each auxiliary task, we compute the relative difference in score with the baseline model; this difference represents the effect of the joint training. however, we focus here on the joint training impact for zero-shot crosslingual transfer; thus, we separate the impact of auxiliary task training in a monolingual setting and in a cross-lingual setting. in ## discussion on the impact of each auxiliary task training, we experimented with jointly training hate speech detection and different auxiliary tasks: sentiment analysis, ner and ud tasks. in the immigrants domain, the ner and sentiment auxiliary tasks led to the best improvement on hate speech detection. the cross-lingual transferability of ner was facilitated by the fact that many named entities are the same across languages (e.g. person and organisation names); indeed, many successful unsupervised cross-lingual transfer systems for this task can be found in the literature @xcite . compared with the first two tasks, adding syntactic information had the lowest positive impact on hate speech detection, often decreasing the performance for zero-shot cross-lingual settings. this is in line with results from the literature that agree on the positive effect on sentiment analysis @xcite , but face varying conclusions when it comes to ud tasks. narang and brew (2020) showed the positive impact of syntactic features on top of noncontextualized embeddings for hate speech detection; @xcite , among the best systems on the evalita2020 hate speech detection task, used pos-tagged text as input for classification. on the contrary, in a monolingual setting, @xcite showed that morphological features added to lstm and bert-based hate speech detection models did not help with comment filtering. similarly, using sequential auxiliary training of tasks such as pos tagging, @xcite showed that the resulting additional low-level skills often led to negative transfer for many downstream tasks. in our cross-lingual setting, our goal was to use these tasks as a proxy to fill the mismatch between languages and facilitate the transfer. we hypothesize that when working on tweets, their constrained style -short sentences, generally with low syntactic complexity -makes additional syntactic knowledge unhelpful (especially in a more difficult to parse user-generated content context) for a downstream task such as hate speech detection, which benefits more from semantic information. regarding the non-usage of pos taggers that could have been optimized for our user-generated content-based datasets, we investigated this possibility and conducted preliminary experiments for english -using the tweebank @xcite as data source-, that showed that using a tagger trained on it did not bring much in terms of performance compared to "classic" ud pos taggers. part of the reasons might come from the fact that our pre-processing step removes hashtags and normalized other twitter's idiosyncrasies and hence make the data somewhat simpler to tag. another reason to not investigate this further lies in the lack of availability of a ugc treebank for spanish, breaking thus the symmetry of our experimental protocol. last but not least, another reason we hypothesized for this lack of much improvement we noticed comes from the fact that the multilingual language model we used (xmlr and xmlr-t) were already providing strong results on ugc. this was corroborated by @xcite , who experimentally verified the robustness of language models when facing noisy ugc. moreover @xcite showed that subword-based language models were able to capture a significant amount of character-level alteration typical of ugc @xcite , explaining their surprising level of robustness when facing noisy content. however, we agree that better handling ugc content would be an interesting step, if not the next step, especially if we can demonstrate that many idiosyncrasies align across languages in our target domains and hence are alleviated by the use of optimized tagging and parsing, eventually multilingual, models. this, in our minds, warrants another full-scale study with a thorough error analysis of cross-lingual syntactic transfer in noisy scenarios. we leave this for future work. cross-lingual zero-shot transfer on a domain with a gap between languages. in section 3, we observed that using larger pre-trained multilingual language models, and if possible, multilingual models trained on corpora from the same source as the downstream task, improves cross-lingual zeroshot transfer. this adaptation has a significant and consistent positive impact. this is in line with the findings of @xcite , who demonstrated the superiority of mlm over other tasks in a crosscorpora transfer setting. similarly, van der goot et al. (2021a) jointly trained auxiliary tasks with a downstream task (in their case, spoken language understanding) in a cross-lingual setting to find that mlm fine-tuning consistently improves the downstream task. beyond the obvious improvement due to the mlm training on more adapted data, we would have expected xlm-t to increase the impact of auxiliary tasks fine-tuning; a more adapted language model helping to bridge the gap between hate speech in the source and target languages. here, the twitter data used for the xlm-t training may not be optimal for the observed linguistic specificities and cultural gap. it was trained on tweets published between 05/2018 and 03/2020, while the hate speech corpora range from 2017 to 2018, depending on the language; moreover, some events were specifically targeted when scraping twitter for hate speech detection (e.g., gamergate victims for the italian datasets on hate speech towards women @xcite ). furthermore, contrarily to wikipedia where corpora are highly similar from one high-resource language to another in term of domains, twitter data can significantly differ between languages due to cultural differences and events in the respective countries. overall, when we used xlm-t, the model is only adapted to the form and style of twitter data (small sentences, with mentions and urls. . . ). the tweets' content, topic, and vocabulary might differ a lot between the hate speech corpora, the xlm-t training data, and the sentiment analysis corpora. we can only hypothesize on these variations. however, they should be quantified to understand better the impact of fine-tuning on these data and to distinguish between corpus variations and the actual cultural and linguistic gap. discussions on computational costs and ethical considerations for this work can be found in appendix 9. ## conclusion in this work, we highlighted situations where zeroshot cross-lingual transfer of hate speech models fails because of the linguistic and cultural gap. we quantified the effect of the choice of multilingual language model and of auxiliary task training on these "failed" cases, showing the positive effect of ner and sentiment analysis multilingual training, but their limited improvement in the domain of hate speech against women. we performed a preliminary analysis on the effect of auxiliary tasks by hate speech functionality using the hatecheck test suite, hinting at which kind of hate speech benefits from transferring knowledge in both the source and the target languages for the three auxiliary tasks. finally, we discussed limitations related to training data for language model pre-training, auxiliary tasks, and hate speech detection. all of our datasets with their new splits and models are freely available. 11 , hoping that the sound experimental framework we designed will help strengthen future studies on cross-lingual hate-speech detection.