Privacy-Preserving Federated Learning for Hate Speech Detection (2025)

Abstract: This paper presents a federated learning system with differential privacy for hate speech detection, tailored to low-resource languages. By fine-tuning pre-trained language models, ALBERT emerged as the most effective option for balancing performance and privacy. Experiments demonstrated that federated learning with differential privacy performs adequately in low-resource settings, though datasets with fewer than 20 sentences per client struggled due to excessive noise. Balanced datasets and augmenting hateful data with non-hateful examples proved critical for improving model utility. These findings offer a scalable and privacy-conscious framework for integrating hate speech detection into social media platforms and browsers, safeguarding user privacy while addressing online harm.

## introduction protecting personal data while enabling effective machine learning is a critical challenge, especially in low-resource languages where data scarcity compounds the difficulty of detecting hate speech. traditional models primarily focus on high-resource languages, leaving underrepresented languages unsupported. federated learning (fl) with differential privacy (dp) offers a solution by enabling collaborative model training without sharing sensitive data. however, the trade-off between privacy and performance in low-resource settings remains a significant concern. this paper investigates the use of privacy-preserving fl for hate speech detection in low-resource languages, specifically afrikaans and russian, which are considered low-resource with regard to labeled hate speech resources, addressing three research questions: the main contribution of this work is the adaptation of differential privacy within a federated learning framework for hate speech detection in a lowresource environment, and the understanding of the challenges imposed by such systems. ## related work hate speech detection has primarily focused on high-resource languages like english. efforts to address low-resource languages include ranasinghe and zampieri (2021), who applied transfer learning to fine-tune transformer models for arabic, bengali, and hindi, showing that pre-trained bertbased models, like albert, work well in these contexts. fine-tuning pre-trained models remains a dominant approach, with studies like geet d'sa et al. (2020) and @xcite demonstrating its effectiveness. however, bert fine-tuning can be unstable, particularly with small datasets, as noted by @xcite . privacy concerns, driven by regulations like the eu's gdpr (of the european union, 2016), have led to federated learning adoption for decentralized data processing. while early work like zampieri et al. (2024) showed fl's promise, vulnerabilities in shared model weights have been identified, as seen in @xcite . differential privacy, introduced by @xcite , mitigates such risks by adding noise to gradients, ensuring privacy while enabling collaborative learning. both global @xcite and local @xcite ) dp methods in federated learning have shown effectiveness and limitations, as reviewed by ouadrhiri and @xcite . while recent approaches, such as @xcite , leverage fl for few-shot hate speech detection in low-resource languages, this paper adapts dp to further enhance model security and evaluate its impact on performance. ## discussion this paper investigates federated learning with differential privacy for hate speech detection in lowresource environments. results show that this approach is feasible for fine-tuning models, even with limited data, but models react differently to added noise. albert models (base and xxlarge) performed the best due to parameter sharing, which might have mitigated the noise. deeper and multilingual models experienced greater utility loss, though further research is needed to confirm these findings. achieving strong privacy guarantees remains challenging. at ϵ ď 1, performance dropped below the non-fine-tuned baseline, highlighting the difficulty of selecting optimal ϵ values, which depend on the model, dataset, and parameter interactions. more local data per client improved results, with 50 sentences per client showing consistent gains. however, limited data hampers effective learning under differential privacy. balanced datasets are critical, but a slight imbalance towards hateful sentences helped overcome the noise added by differential privacy. sampling non-hateful examples is crucial for effective training. despite challenges, federated learning with differential privacy remains advantageous where privacy is paramount. addressing the research questions: ## conclusion this paper explored federated learning with differential privacy for hate speech detection in lowresource settings. fine-tuning a pre-trained albert model showed improved performance at moderate privacy levels. key findings included the importance of nearly-balanced datasets and the impact of differential privacy parameters (ϵ, δ, and c), with albert outperforming other bert-based models. the results addressed the research questions, highlighting both strengths and areas for improvement. in conclusion, despite challenges in lowresource environments, federated learning with differential privacy can effectively detect hate speech while ensuring user privacy.