V i H ate T 5: Enhancing Hate Speech Detection in V ietnamese With a Unified Text-to-Text Transformer Model (2024)

Abstract: Recent advancements in hate speech detection (HSD) in Vietnamese have made significant progress, primarily attributed to the emergence of transformer-based pre-trained language models, particularly those built on the BERT architecture. However, the necessity for specialized fine-tuned models has resulted in the complexity and fragmentation of developing a multitasking HSD system. Moreover, most current methodologies focus on fine-tuning general pre-trained models, primarily trained on formal textual datasets like Wikipedia, which may not accurately capture human behavior on online platforms. In this research, we introduce ViHateT5, a T5-based model pre-trained on our proposed large-scale domain-specific dataset named VOZ-HSD. By harnessing the power of a text-to-text architecture, ViHateT5 can tackle multiple tasks using a unified model and achieve state-of-the-art performance across all standard HSD benchmarks in Vietnamese. Our experiments also underscore the significance of label distribution in pre-training data on model efficacy. We provide our experimental materials for research purposes, including the VOZ-HSD dataset, pre-trained checkpoint, the unified HSD-multitask ViHateT5 model, and related source code on GitHub publicly.

## introduction hate speech refers to harmful expression targeting individuals or groups based on their inherent characteristics, potentially inciting violence or discrimination @xcite . its detrimental impact on mental well-being includes different levels of anxiety, depression, or stress among affected individuals @xcite . due to the rise of social media on the internet, where people can easily leave their toxic content that may negatively affect anyone who reads it, the consequences that hate speech brings to use become worse and worse. to address these issues, automatic systems have been explored to detect harmful content online and mitigate its dissemination @xcite @xcite . different languages have unique forms of harmful expressions, necessitating specific textprocessing methodologies for developing hsd systems. in the context of english, one of the most prevalent languages, there exist several effective strategies for addressing hsd-related tasks, such as employing machine learning models @xcite or deep learning models for identifying harmful content @xcite . furthermore, transfer learning approaches have garnered considerable research interest, showcasing the effectiveness in solving hate speech detection tasks @xcite . in the case of low-resource languages, numerous studies have been conducted, yielding promising results in tackling this issue @xcite @xcite . vietnamese, considered a low-resource language, has seen limited research in the field of nlp concerning high-quality datasets and pretrained models. recent efforts in hate speech detection tasks based on vietnamese language characteristics have yielded significant achievements @xcite @xcite . however, current state-of-the-art models only fine-tune general transformer-based models, which may have been pre-trained on formal textual data sources @xcite . moreover, even pre-trained on social media text, models like visobert @xcite still necessitate separate fine-tuning for specific tasks, resulting in system fragmentation. this paper presents a novel approach to address the existing limitations of hsd systems in vietnamese. the contributions of this research are outlined as follows: this paper is structured into distinct sections. section 2 examines relevant prior research on vietnamese hsd-related tasks. subsequently, section 3 introduces vihatet5, our text-to-text model, covering its automatically generated pre-training dataset, pre-training methodologies, and finetuning for downstream tasks. section 4 presents experimental results obtained by comparing various baseline methods with our proposed vihatet5 model across a range of hate speech detectionrelated tasks and address discussions. section 5 concludes the paper with a summary of our findings. section 6 addresses the current limitations of our proposed method, while section 7 provides ethical statements related to our research. ## related work since the emergence of the transformer architecture @xcite , numerous challenges in nlp have been successfully addressed, including tasks related to hate speech detection. additionally, domain-specific models like hatebert @xcite , fbert @xcite , or toxicbert foot_4 have been introduced. however, there remains a deficiency in hate-speech-focused pre-trained models for low-resource languages like vietnamese, hindering the effective resolution of hsd tasks. besides, there exist diverse endeavors on hsd tasks, which involve the contribution of large-scale, high-quality datasets, thus facilitating precise research in this domain @xcite @xcite . furthermore, competitions such as the vlsp-2019 shared-task @xcite , dubbed hate speech detection on social networks, aiming at identifying harmful content on internet-based social media, yielding remarkable outcomes. additionally, transformerbased models have demonstrated remarkable efficacy across various nlp tasks. the advent of monolingual pre-trained models in vietnamese, which have been observed to surpass their multilingual counterparts @xcite , has paved the way for the creation of precise systems for hate speech detection. on the contrary, despite the remarkable performance achieved by current bert-based approaches, developing separate systems tailored to individual tasks is necessary. addressing this issue of fragmentation, a unified text-to-text-based model on t5 architecture @xcite such as ft5 for english or mft5 @xcite has demonstrated its effectiveness in amalgamating multiple tasks ranging from syllable-level to sentence-level-based hsd challenges. hence, in this study, we introduce vi-hatet5, a pre-trained text-to-text model designed explicitly for the hate speech domain, aiming to streamline complex separate systems while ensuring optimal performance in addressing hsd issues in vietnamese. ## vihatet5 this section reveals the methodologies for the creation of pre-training data, the pre-training techniques utilized, and the fine-tuning procedures undertaken to assemble the unified vihatet5 model. ## conclusions advancements in hate speech detection tasks in vietnamese have recently gained notable progress thanks to the use of transformer models. however, these efforts remain fragmented due to the reliance on separate fine-tuned models for distinct tasks. hence, our research aims to introduce a unified text-to-text transformer model, vihatet5, with the potential to address prevailing issues in hate speech detection in vietnamese and attain stateof-the-art performance. moreover, vihatet5's pre-training on domain-specific datasets enables it to grasp the nuances of social media content in vietnamese deeply. the open-source nature of both the dataset and the model facilitates researchers and developers in leveraging our work, fostering further advancements in vietnamese nlp and online safety.