Exploring Cross-lingual Text Detoxification with Large Multilingual Language Models. (2022)

Abstract: Detoxification is a task of generating text in polite style while preserving meaning and fluency of the original toxic text. Existing detoxification methods are monolingual i.e. designed to work in one exact language. This work investigates multilingual and cross-lingual detoxification and the behavior of large multilingual models in this setting. Unlike previous works we aim to make large language models able to perform detoxification without direct fine-tuning in a given language. Experiments show that multilingual models are capable of performing multilingual style transfer. However, tested state-of-the-art models are not able to perform cross-lingual detoxification and direct fine-tuning on exact language is currently inevitable and motivating the need of further research in this direction.

## introduction the task of textual style transfer (textual style transfer) can be viewed as a task where certain properties of text are being modified while rest retain the same foot_0 . in this work we focus on detoxification textual style transfer (dos @xcite . it can be formulated as follows: given two text corpora d @xmath0, x 2 , . . . x n } and d @xmath1, y 2 , . . . , y n }, where x, y -are two sets of all possible text in styles s x , s y respectively, we want to build a model f θ : @xmath2, such that the probability p(y gen |x, s x , s y ) of transferring the style s x of given text x (by generation y gen ) to the style s y is maximized (where s x and s y are toxic and non-toxic styles respectively). some examples of detoxification presented in textual style transfer gained a lot of attention with a rise of deep learning-based nlp methods. given that, textual style transfer has now a lot of specific subtasks ranging from formality style transfer @xcite and simplification of domain-specific texts @xcite to emotion modification @xcite and detoxification (debiasing) @xcite . there exist a variety of textual style transfer methods: from totally supervised methods @xcite @xcite which require a parallel text corpus for training to unsupervised @xcite @xcite that are designed to work without any parallel data. the latter sub-field of research is more popular nowadays due to the scarcity of parallel text data for textual style transfer. on the other hand, if we address textual style transfer task as a machine translation task we get a significant performance boost @xcite . the task of detoxification, in which we focus in this work, is relatively new. first work on detoxification was a sequence-to-sequence collaborative classifier, attention and the cycle consistency loss @xcite . a recent work by @xcite introduces self-supervised model based on t5 model @xcite with a denoising and cyclic auto-encoder loss. both these methods are unsupervised which is an advantage but it comes from the major current problem of the textual style transfer. there is a lack of parallel data for textual style transfer since there exist only few parallel datasets for english @xcite and some other languages @xcite . when it comes to detoxification there are only two parallel detoxification corpora available now and they both appeared only last year @xcite . most state-of-the-art methods rely on large amounts of text data which is often available for some well-researched languages like english but lacking for other languages almost entirely. therefore, it is important to study whether cross-lingual (or at least multilingual) detoxifica- ## results & discussion as for cross-lingual style transfer, results are negative. none of the models have coped with the task of cross-lingual textual style transfer. that means that models produce the same or almost the same sentences for the language on which they were not fine-tuned so that toxicity is not eliminated. we provide only some scores here in the despite the fact that our hypothesis about the possibility of cross-language detoxification was not confirmed, the presence of multilingual models pretrained in many languages gives every reason to believe that even with a small amount of parallel data, training models for detoxification is possible. a recent work by @xcite shows that cross-lingual formality textual style transfer is possible. @xcite achieve this on xfor-mal dataset @xcite by adding language-specific adapters in the vanilla mbart architecture @xcite ) -two feed-forward layers with residual connection and layer normalization @xcite . we follow the original training procedure described by @xcite by training adapters for english and russian separately on 5 million sentences from news crawl dataset foot_9 . we use batch size of 16 and 200 thousand training iterations. we also then train cross-attentions on our parallel detoxifcation data in the same way. however, models tend to duplicate input text without any detoxification. thus, while the exact same original setup did not work for detoxification, more parameter search and optimization could lead to more acceptable results and we consider the approach by @xcite as a promising direction of a future work on multilingual and cross-lingual detoxification. ## conclusion in this work we have tested the hypothesis that multilingual language models are capable of performing cross-lingual and multilingual detoxification. in the multilingual setup we experimentally show that reformulating detoxification (textual style transfer) as a nmt task boosts performance of the models given enough parallel data for training. we beat simple (delete method) and more strong (condbert) baselines in a number of metrics. based on our experiments, we can assume that it is possible to fine-tune multilingual models in any of the 100 languages in which they were originally trained. this opens up great opportunities for detoxification in unpopular languages. however, our hypothesis that multilingual language models are capable of cross-lingual detoxification was proven to be false. we suggest that the reason for this is not a lack of data, but the model's inability to capture the pattern between toxic and non-toxic text and transfer it to another language by itself. this means that the problem of cross-lingual textual style transfer is still open and needs more investigation.