Transferring Knowledge via Neighborhood-Aware Optimal Transport for Low-Resource Hate Speech Detection (2022)

Abstract: The concerning rise of hateful content on online platforms has increased the attention towards automatic hate speech detection, commonly formulated as a supervised classification task. State-of-the-art deep learning-based approaches usually require a substantial amount of labeled resources for training. However, annotating hate speech resources is expensive, time-consuming, and often harmful to the annotators. This creates a pressing need to transfer knowledge from the existing labeled resources to low-resource hate speech corpora with the goal of improving system performance. For this, neighborhood-based frameworks have been shown to be effective. However, they have limited flexibility. In our paper, we propose a novel training strategy that allows flexible modeling of the relative proximity of neighbors retrieved from a resource-rich corpus to learn the amount of transfer. In particular, we incorporate neighborhood information with Optimal Transport, which permits exploiting the geometry of the data embedding space. By aligning the joint embedding and label distributions of neighbors, we demonstrate substantial improvements over strong baselines, in low-resource scenarios, on different publicly available hate speech corpora.

## introduction with the alarming spread of hate speech (hs) in social media, natural language processing techniques have been used to develop automatic hs detection systems, typically to aid manual content moderation. although deep learning-based approaches @xcite have become state-of-the-art in this task, their performance depends on the size of the labeled resources available for training @xcite . annotating a large corpus for hs is considerably time-consuming, expensive, and harmful to human annotators @xcite @xcite . moreover, models trained on existing labeled hs corpora have shown poor generalization when evaluated on new hs content @xcite @xcite . this is due to the differences across these corpora, such as sampling strategies @xcite , varied topics of discussion @xcite , varied vocabularies, and different victims of hate. thus, to address these challenges, here we aim to devise a strategy that can effectively transfer knowledge from a resource-rich source corpus with a higher amount of annotated content to a lowresource target corpus with fewer labeled instances. one popular way to address this is transfer learning. for instance, @xcite fine-tune a large-scale pre-trained language model, bert @xcite , on the limited training examples in hs corpora. further, a sequential transfer, following @xcite , can be performed where a pre-trained model is first finetuned on a resource-rich source corpus and subsequently fine-tuned on the low-resource target corpus. since this may risk forgetting knowledge from the source, the source and target corpora can be mixed for training @xcite . besides, to learn target-specific patterns without forgetting the source knowledge, @xcite augment pre-trained neurons from the source model with randomly initialized units for transferring knowledge to low-resource domains. recently, @xcite argue that traditional transfer learning strategies are not systematic. therefore, they model the relationship between a source and a target corpus with a neighborhood framework and show its effectiveness in transfer learning for content flagging. they model the in-teraction between a query instance from the target and its neighbors retrieved from the source. this interaction is modeled based on their label agreement -whether the query and its neighbors have the same labels -while using a fixed neighborhood size. however, different neighbors may have varying levels of proximity to the queried instance based on their pair-wise cosine similarities in a sentence embedding space. therefore, intuitively, the neighbors should also be weighted according to these similarity scores. we hypothesize that simultaneously modeling the pair-wise distances between instances from the low-resource target and their respective neighbors from the resource-rich source, along with their label distributions should result in a more flexible and effective transfer. with this aim, we propose a novel training strategy where the model learns to assign varying importance to the neighbors corresponding to different target instances by optimizing the amount of pair-wise transfer. this transfer is learned without changing the underlying model architecture. such optimization can be efficiently performed using optimal transport (ot) @xcite @xcite due to its ability to find correspondences between instances while exploiting the underlying geometry of the embedding space. our contributions are summarised as follows: ## proposed approach our problem setting involves a low-resource target corpus x t with a limited amount of labeled training data (x t train , y t train ) = {x t i , y t i } nt @xmath0 given such a setting, we hypothesize that transferring knowledge from the nearest neighbors in the source should improve the performance on the insufficiently labeled target. furthermore, to provide additional control to the model, we propose a systematic transfer. with this transfer mechanism, a model can learn different weights assigned to the neighbors in x s train based on their proximity to the instances in x t train simultaneously in a sentence embedding space and the label space. for this, we incorporate neighborhood information with optimal transport (ot), as ot can learn correspondences between instances from x s train and x t train by exploiting the underlying embedding space geometry. ## conclusion and future work since our framework uses neighborhood information for transferring knowledge, it relies on the degree of proximity of the neighbors. however, if all of the source and target instances are very distant semantically, all the nearest neighbors from the source may have very low cosine similarity to the corresponding target instances. in such scenarios, the framework may yield limited improvements over the vanilla fine-tuning as the available neighborhood information would be much weaker. in such cases, the performance would mainly depend on the label consistency of the neighbors. for future work, our framework can be explored for transferring knowledge from resource-rich languages, such as english, to low-resource languages. this can be done by extracting the cross-lingual neighbors using multilingual sentence embedding models like labse @xcite . besides, the framework can be applied for transferring knowledge in other text classification tasks, such as sentiment classification, bragging detection @xcite , etc., as the methodology is not restricted to only hate speech detection.