Text Detoxification as Style Transfer in E nglish and H indi (2023)

Abstract: This paper focuses on text detoxification, i.e., automatically converting toxic text into nontoxic text. This task contributes to safer and more respectful online communication and can be considered a Text Style Transfer (TST) task, where the text’s style changes while its content is preserved. We present three approaches: (i) knowledge transfer from a similar task (ii) multi-task learning approach, combining sequence-to-sequence modeling with various toxicity classification tasks, and (iii) delete and reconstruct approach. To support our research, we utilize a dataset provided by Dementieva et al. (2021), which contains multiple versions of detoxified texts corresponding to toxic texts. In our experiments, we selected the best variants through expert human annotators, creating a dataset where each toxic sentence is paired with a single, appropriate detoxified version. Additionally, we introduced a small Hindi parallel dataset, aligning with a part of the English dataset, suitable for evaluation purposes. Our results demonstrate that our approach effectively balances text detoxification while preserving the actual content and maintaining fluency.

## introduction text detoxification is a process that involves the automatic conversion of toxic text into non-toxic or detoxified text @xcite . text detoxification can be viewed as a subtask within the broader context of text style transfer (tst) @xcite . tst aims to alter the style of text while preserving its core content. in the case of text detoxification, the source style is toxic language, and the target style is non-toxic, with the primary objective being the transformation of text from a harmful and offensive nature to a neutral or positive one, all while retaining the rest of the original text's meaning. existing detoxification methods often rely on rule-based removal of toxic words or phrases @xcite , but this approach is not very efficient and can make sentences sound unnatural. it also does not consider whether the meaning of the sentence is affected by these removals. additionally, because we have limited resources in terms of datasets @xcite for transforming toxic text to non-toxic text, simple sequence-to-sequence training may not be enough for better results @xcite . to improve this, we propose three approaches: (i) knowledge transfer from a similar task (ii) multi-task learning approach, utilizing sequenceto-sequence modeling coupled with various toxic and civil text classification tasks, and (iii) delete and reconstruct approach. these approaches enhance the sequence-to-sequence transformation and lead to better outcomes compared to basic sequence-to-sequence training. to conduct our research and experiments, we leverage a dataset provided by @xcite . this dataset contains various versions of text that have undergone detoxification, offering valuable resources for our investigations. we have taken a meticulous approach to ensure dataset quality. this includes soliciting the expertise of human annotators to select the most suitable detoxified versions for each toxic text, resulting in a carefully curated dataset where each source toxic sentence is paired with its corresponding non-toxic text. furthermore, to contribute to the development of multilingual parallel datasets for text detoxification, we present a novel dataset containing 500 parallel toxic and non-toxic sentences in hindi, aligned with their english counterparts (see section 3). hindi is one of the 22 scheduled and official indian languages and the largest speech community in india @xcite . this dataset serves as a validation resource for research in hindi and will open the possibility of extending it to other indian languages as well as in multilingual domains. our contributions are summarized as follows: ## related work text style transfer (tst) text style transfer (tst) is a task where texts with different styles but similar content are transformed @xcite . for instance, @xcite used a model to turn modern english into shakespearean english, and @xcite explored tst with limited parallel data resources. however, tst is often tricky due to the shortage of matching data, with examples such as yelp (sentiment) @xcite and the detoxification datasets of @xcite having the order of a few thousand examples at most. to tackle this, two main strategies have emerged: (i) simple text replacement, which involves explicitly finding and swapping specific style-related phrases @xcite , (ii) implicit style separation, techniques like back-translation and autoencoding help separate style from content through hidden representations @xcite @xcite @xcite . toxicity several datasets have been built for the detection of toxic/hateful/offensive content from various social media platforms such as twitter (now called x), reddit, facebook @xcite @xcite @xcite @xcite @xcite and wikipedia talk pages @xcite @xcite . these datasets often contain various types of toxicity, including between offensive language and hateful/aggressive speech, whereas other datasets focus only on specific types of toxicity @xcite , e.g., sexism, religious discrimination, or racism. these datasets are explored to build automatic hate speech/offensive text detection models using various machine and deep learning approaches @xcite @xcite @xcite . however, given that the focus is on toxic content detection and not detoxification, these datasets are not parallel (with aligned toxic and non-toxic sentence pairs), except for small data in english @xcite and russian @xcite , which leads to the same problems in detoxification as with tst in general @xcite . to our best knowledge, the following models exist for the detoxification task: duplicate, a rulebased delete, rut5, and ruprompts baseline models @xcite . duplicate keeps input text intact, establishing a trivial baseline. delete is an unsupervised rule-based method that removes toxic words using a predefined vocabulary, akin to censoring on tv. rut5 involves supervised fine-tuning of the t5 model. ruprompts uses the ruprompts library for continuous prompt tuning of language models @xcite . ## dataset development the original dataset @xcite is a collection of user-generated comments which are toxic in nature. the features of the dataset can be summarised as follows: ## detoxification models we first introduce the seq2seq baseline (seq2seq, section 4.1), then present three distinct extension approaches: knowledge transfer from a similar task (kt, section 4.2), multitask learning (section 4.3), and delete and reconstruct (del_recon, section 4.4) for the text detoxification task. ## experimental settings to ensure consistency in our experiments, we partitioned the english datasets into 508 examples for training, 100 for development, and 500 for testing. for the hindi dataset, we created training and development sets of the same size as the english dataset through machine translation. we utilized the facebook nllb-200-3.3b model @xcite available from huggingface. for evaluation in hindi, we employed our newly provided dataset consisting of 500 samples (see section 3). we employed the mbart-large-50 model @xcite ) from the huggingface library @xcite for both english and hindi. to optimize model performance, we conducted hyperparameter tuning, leading to the selection of a learning rate of 1e-5 and a batch size of 3. throughout the network, dropout was applied with a rate of 0.1. additionally, we introduced l2 regularization with a strength of 0.01. the training was executed over 5 epochs. ## evaluation metrics the evaluation process involves three primary aspects: accuracy of the toxic to civil text transfer (detoxification accuracy), content preservation, and fluency. detoxification accuracy is assessed using our fine-tuned mbert classifier, which used the same training set for finetuning as our primary tst task (see section 3). classifier accuracies of toxic and non-toxic text in english and hindi languages are shown in as automated metrics for language generation may not correlate well with human judgments @xcite , we also run a small-scale human evaluation with language expert annotators on a random sample of 50 sentences from the test set for each language. outputs are rated on a 5point likert scale for detoxification accuracy, content preservation, and fluency. ## conclusion we explored the task of text detoxification, aiming to automatically transform toxic text into nontoxic text while preserving content and fluency. our findings suggest that our approaches successfully address the task of text detoxification, achieving notable detoxification accuracy while preserving content and maintaining fluency. in future research, we will focus on exploring text detoxification in various linguistic and cultural contexts. we anticipate that our work will pave the way for more effective and ethical online communication by mitigating the harmful impact of toxic text.