Generalizable Multilingual Hate Speech Detection on Low Resource I ndian Languages using Fair Selection in Federated Learning (2024)

Abstract: Social media, originally meant for peaceful communication, now faces issues with hate speech. Detecting hate speech from social media in Indian languages with linguistic diversity and cultural nuances presents a complex and challenging task. Furthermore, traditional methods involve sharing of users’ sensitive data with a server for model training making it undesirable and involving potential risk to their privacy remained under-studied. In this paper, we combined various low-resource language datasets and propose MultiFED, a federated approach that performs effectively to detect hate speech. MultiFED utilizes continuous adaptation and fine-tuning to aid generalization using subsets of multilingual data overcoming the limitations of data scarcity. Extensive experiments are conducted on 13 Indic datasets across five different pre-trained models. The results show that MultiFED outperforms the state-of-the-art baselines by 8% (approx.) in terms of Accuracy and by 12% (approx.) in terms of F-Score.

## introduction in the age of social computing utilizing social media as a communication tool motivated by anonymity and ease has grown significantly in size and significance @xcite . microblogs provide instantaneous worldwide idea sharing, information acquisition, company promotion, etc. this encourages free speech and makes the world more connected. unfortunately, it has also led to an increase in abusive interactions, such as cyberbullying, profanity, hate speech, etc., directed at specific people and groups @xcite . in addition to detracting from important conversations, the increasing amount of inappropriate information on the internet has the potential to instigate aggressiveness, which can lead to violent real-world situations @xcite . social media platforms are forced to carefully monitor user postings and actions in order to halt the spread of hate speech @xcite . therefore, there exists a pressing necessity to establish effective automated frameworks for identifying hate speech, ensuring a proactive approach to its mitigation. the majority of existing work in hate speech detection is based on european languages or languages with a wider international user base @xcite @xcite . whereas only a limited amount of work is done on other languages due to limited resources and linguistic nuances. specifically, in india, we have 22 official languages and people like to communicate in their regional languages @xcite . this makes social media interactions lack formal structure and frequently involve grammatical errors, emoticons, spelling issues, etc. moreover, the mixing of one or more languages that can even be code-mixed makes hate speech detection extremely challenging @xcite . as a result, we have a limited digital presence of training data. hence, a very limited amount of work focuses on hate speech detection in indian languages. this motivated us to work on hate speech detection for indian languages. to address data scarcity for hate speech detection in indian languages, various works sourced datasets from social media platforms @xcite . also, competitions such as hasoc foot_0 and indoml foot_1 have been launched to promote research in indian languages by providing high-quality datasets. however, the scale and linguistic coverage of these datasets is sparse @xcite . therefore, most of the works in hate speech are either based on monolingual datasets or utilize only a few languages. consequently, the model trained in fewer languages might not yield accurate results in other languages @xcite . to solve this, a few recent studies attempted to widen the coverage of training data by combining samples from multiple languages @xcite @xcite . however combining various smaller datasets into one large-scale dataset may introduce inconsistency due to differences in annotation guidelines, data source platform, class imbalance, etc @xcite . in such scenarios, the performance of the centrally trained model may be highly impacted and biased. therefore, adapting a single model to give a generalized performance in diverse domains while maintaining its effectiveness in each domain is challenging and complex. further, the success of a centralized computational framework relies on public datasets. but sensitive topics demand caution even with publicly available social media texts. the texts become private if a user makes their account private, impacting prior posts. such privacy can be preserved if the model training utilizing these sensitive data happens at the user devices without the need to send the data to a common server. this is the motivation to use federated learning, suitable for sensitive data tasks emphasizing user privacy @xcite @xcite . apart from privacy risks, fl addresses user volume, info loss, and label imbalance @xcite . the objective of this paper is to design a generalized model using federated learning capable of detecting hate speech in multiple indian languages. for this, we consider datasets from 13 indian languages collected from various social media platforms. the publicly available different stateof-the-art bert models such as xlm-roberta @xcite , multilingual-bert @xcite , muril, indic-bert @xcite similar to @xcite , and xlnet constitutes our centralized baselines for multilingual hate speech detection task. in summary, we make the following contributions: recently, more studies have attempted hate speech in indian languages like hindi @xcite , marathi @xcite , bengali @xcite , etc. however, users often mix english with their native language (code-mixed data) for communication ease, complicating hate speech detection. despite progress, research in bilingual, codemixed, or multilingual tasks is nascent. for example, @xcite proposed standardization classes across publicly available datasets and studied the generalization capabilities of bert, fasttext, and svm models. @xcite uses datasets for 3 different languages (english, italian, and german) and trains different models such as lstms, grus, bidirectional lstms, etc. the work claims to have a robust neural architecture for hate speech detection across different languages. our work develops models that are far more generalizable and trained on a much larger dataset of languages. @xcite constructed a multilingual twitter hate speech corpus from 5 languages that they augmented with demographic information to study the demographic bias in hate speech classification. (aluru et al., 2020) use datasets from 8 different languages and obtain their embeddings using laser 3 and muse 4 . though performance is decent across languages, fine-tuned models work best only for the 8 languages. the study focuses on low-resource performance, ignoring external impact. to address poor generalization of models due to biases in hate speech datasets, a suite of functional tests (hat-echeck) is introduced. hatecheck offers insights into hate speech detection models but only tests english and modalities in text @xcite . the extension of the english hatecheck functional test suite is multilingual hatecheck (mhc), identifying model weaknesses for monolingual and cross-lingual applications, providing insights for the development of better multilingual hate speech detection models @xcite . federated learning prior work on hate speech detection has primarily focused on privacy-agnostic machine learning paradigms, using centralized models for classification. in this work, we use a privacy-centric paradigm of machine learning, i.e., federated learning (fl). fl is a decentralized training strategy of machine learning models, a strategy reminiscent of parameter servers across a group of clients @xcite . clients train the model locally keeping data private from the server and peers. the parallel nature of training reduces the overall training time and enables training of models on a large corpus. this prevents the system from computational bottlenecks and minimizes the effect of heterogeneity in data on the global model performance @xcite . recently several studies applied fl in many nlp tasks @xcite @xcite . the authors of @xcite discussed training of nlp models and acknowledged the potential of methods like differential privacy, federated learning, and homomorphic encryption to handle sensitive data, e.g., medical records. however, the paper does not provide insights into the scalability or efficiency of the proposed framework for training nlp models on privacy-protected data. it is experimentally proved that fl ensures faster convergence when the number of clients is increased @xcite . but the study lacks a discussion on framework limits in handling bigger datasets. in a similar study, the fl method is utilized as a privacy-preserving training paradigm for hate speech detection on eight datasets, surpassing centralized models aiming to address the lack of privacy in current approaches @xcite . another study applied fl for multilingual emoji prediction in clean and attack situations, emphasizing its privacy and distributed advantages. however, it overlooks discussing how data distributions and biases could affect fl model performance @xcite . similar work is done in emoji detection for hindi texts collected from twitter considering fedprox @xcite and a modified version of causalfedgsd @xcite using federated approach @xcite . ## results and discussion we assess the performance of our proposed mul-tifed approach on five pre-trained bert-based models under iid and non-iid data distribution settings. firstly, we compare the generalized performance over the low-resource indic languages (see please note that we reported the best results obtained in various experiments carried out for multi-fed. considering the performance for iid and non-iid settings from after the preliminary exploration, it is clearly visible that our proposed multifed framework outperformed the baseline by a margin of 7.78% & 11.58% in terms of accuracy and by 9.59% & 11.91% in terms of f1-score under iid and non-iid settings, respectively. we see that xlm-roberta achieves the best performance in all experimental settings while xlnet gives suboptimal performance. it is interesting to observe the performance of baseline models that are pre-trained on various indic languages (such as muril @xcite and indicbert @xcite ). although these models are pre-trained on similar indic languages, xlmr and mbert beat them in most of the cases. this suggests the importance of continued training for domain adaptation. as evident from we also tested multifed on the centralized test data from each language separately to quantify the personalized performance of the proposed model (see to understand this better, it is worth discussing the behavior of these utilized pre-trained models in our fl setup with different linguistic variability. all these models are pre-trained on large-scale diverse corpus of multilingual texts, including various indian languages. their capability to capture semantic similarity and differences across the languages tend to give significant performance across wide-range of languages without any extensive finetuning. however, the performance is dependent on the lexical similarity to the languages it was pretrained. in this work, we fine-tuned these models using fl after creating groups of similar clients. thus, these models leverage knowledge gained from one language to improve the performance in other facilitating the transferable characteristics across languages. ## conclusion in this work, we propose a federated approach to hate speech detection, coined multifed, which aims to localize users' data and prevent it from being exposed while training models for hate speech detection. it also tackles the challenge of diverse linguistic data by personalizing the performance of clients. through empirical experimentations, we find that federated learning along with preserving privacy achieves a higher performance level than centralized baselines in various scenarios. in future work, we intend to personalize the performance, improve fairness in the selection of clients, and reduce the resource consumption of each client.