M ulti P ara D etox: Extending Text Detoxification with Parallel Data to New Languages (2024)

Abstract: Text detoxification is a textual style transfer (TST) task where a text is paraphrased from a toxic surface form, e.g. featuring rude words, to the neutral register. Recently, text detoxification methods found their applications in various task such as detoxification of Large Language Models (LLMs) (Leong et al., 2023; He et al., 2024; Tang et al., 2023) and toxic speech combating in social networks (Deng et al., 2023; Mun et al., 2023; Agarwal et al., 2023). All these applications are extremely important to ensure safe communication in modern digital worlds. However, the previous approaches for parallel text detoxification corpora collection—ParaDetox (Logacheva et al., 2022) and APPADIA (Atwell et al., 2022)—were explored only in monolingual setup. In this work, we aim to extend ParaDetox pipeline to multiple languages presenting MultiParaDetox to automate parallel detoxification corpus collection for potentially any language. Then, we experiment with different text detoxification models—from unsupervised baselines to LLMs and fine-tuned models on the presented parallel corpora—showing the great benefit of parallel corpus presence to obtain state-of-the-art text detoxification models for any language.

## introduction we formulate text detoxification task as stated in @xcite so the objective is to paraphrase a toxic text to a text that: (i) has neutral style (register); (ii) saves the meaningful content as much as possible; (iii) is fluent at least at the same level as the input text. before, many unsupervised approaches for text detoxification were presented @xcite @xcite addressing the task based only on available toxic or hate speech classification corpora which are most commonly non-parallel. however, in @xcite and appa-dia @xcite the benefit of parallel corpus for text detoxification was illustrated-the seq2seq models like bart @xcite and t5 @xcite fine-tuned on the presented corpora outperformed previous unsupervised baselines in both manual and automated evaluations. while the parallel detoxification corpora are already available together with their collection pipelines, they were only presented for english language. however, we strongly support the idea of such corpus availability for any language would lead to fair and safe lms development equally for all languages @xcite . in this work, we aim to extend paradetox collection pipeline to a multilingual format confirming the hypothesis that it can be used to collect parallel text detoxification data for any language foot_0 . thus, the contributions of this work are as following: ## multiparadetox pipeline we adapt paradetox @xcite collection pipeline as it was designed to automate the data collection as well as verification with crowdsourcing. the pipeline consists of three tasks: task 1: rewrite text in a polite way annotators need to provide the detoxified paraphrase of the text so it becomes non-toxic and the main content is saved or to skip paraphrasing if the text is not possible to rewrite in non-toxic way; task 2: do these sentences mean the same? check if the content is indeed the same between the original toxic text and its potential non-toxic paraphrase; task 3: is this text offensive? verification of the provided paraphrase if it is indeed non-toxic. we extend this pipeline to multiparadetox supporting any new language (see figure 1 ): step 1: toxic corpus preparation firstly, we need to prepare toxic samples that will serve as input to the paradetox pipeline. in the annotation, we focus only on explicit toxicity types @xcite . (i) if there already exists binary toxicity classification dataset, then it is enough to select from it toxic or hate part preferably with labels like "toxic", "offensive", "obscene"; (ii) if there is not such dataset, then samples with explicit toxicity can be selected by finding toxic keywords substrings in the texts. as we want sentences to have meaningful content, only sentences with less then 1/2 of toxic keywords fraction should be chosen. step 2: tasks language adaptation then, the paradetox tasks needed to be adapted for the target language. this can be done with combination of automated translation followed by language native speakers texts proofreading. step 3: tasks settings adjustment finally, for the crowdsourcing tasks, the language and country settings should be chosen accordingly to the target language. for the quality control, we follow the procedure described in paradetox utilising training, exam, and control tasks. we claim that these tasks can be also translated from the original ones with slight edits by native speakers according to special cases of toxicity for the language. ## collection of new parallel datasets with multiparadetox we applied the described above pipeline to obtain new parallel datasets for three languages-russian, ukrainian, and spanish. the choice of language was done based on the availability of native speakers of these languages. the data collection was done via toloka platform. 3 for translations, we used deepl foot_2 api (tasks texts are presented in appendix a). we accepted to the annotation only workers who proved the corresponding language fluency with a test. the general information about the datasets is presented in as russian toxicity classification datasets were available for russian language, we selected toxic sentences from russian language toxic comments competitions @xcite . in case of the ukrainian language, there was no binary toxicity classification corpus available. we filtered from ukrainian tweets corpus @xcite the explicitly toxic samples that contain obscene lexicon from the predefined list @xcite . for spanish language, we selected samples for annotation from two datasets: hate speech detection one @xcite as well as filtered by keywords spanish tweets corpus @xcite . we collected the data for each language depending on the available input data and resources. the nature of the original data influenced the process. thus, the lowest ratio of non-detoxified sample filtering was observed for ukrainian language. for spanish, this ratio is higher as the input data labels were more from the hate speech domain. nevertheless, for each language it was possible to collect at least several hundreds of pairs with 1-3 paraphrases per each toxic input. ## text detoxification experiments to enhance the validation of the data collected using multiparadetox, we conduct text detoxification experiments, comparing baselines with fine-tuned models with the newly obtained data. ## results the results of the systems evaluation are presented in delete methods reaches the highest content similarity as it was designed to modify the original sentence slightly. however, it does not filter all toxic language and gains the lowest sta scores. the condbert method fails to make substitutions with correct words and obtains not good enough fluency scores. llama achieves very high sta scores concurrently with the lowest sim scores. the model can hallucinate and even generate text not in a target language as can be observed from the examples. the models fine-tuned on the translated datasets fail for each language in sta scores. the rationale lies in the diversity of toxic phrases across languages. for instance, russian and ukrainian, being morphologically rich languages, encompass a multitude of toxic expressions that cannot be directly translated from english. moreover, there exists a strong correlation between language and culture, manifesting in specific discussion topics and expressions unique to each language's online informational space. finally, the models fine-tuned on the proposed data never fail in any of the evaluation parameters and outperform unsupervised baselines based on j score with a high gap. this attests to the reliability of our data and necessity of parallel text detoxification corpora in acquiring state-of-the-art text detoxification models. for spanish, a slight drop of the results can be cased by significantly lower amount of the training data. even in this case, the model shows promising results while other models still did not produce qualitative results (llama got high sta scores but the content output text was just random). in the end, we also presented the results for multilingual text detoxification model fine-tuned for all three languages. the obtained results on par with monolingual models confirm the possibility to obtain single multilingual model for the multilingual text detoxification task. ## conclusion we presented multiparadetox-the extension of paradetox pipeline for parallel data collection for the text detoxification task to new languages. the target language corpus collection can be prepared only with three steps: provision of input toxic corpus, crowdsourcing tasks language adaptation, and corresponding settings adjustments. we tested our proposed pipeline extension on three new languages-russian, ukrainian, and spanishcollecting corresponding new corpora. by native speakers. finally, the data efficacy was confirmed with text detoxification systems comparison where the models fine-tuned on our data outperformed unsupervised baselines and zero-shotprompted llms.