R e S e TOX : Re-learning attention weights for toxicity mitigation in machine translation (2024)

Abstract: Our proposed method, RESETOX (REdoSEarch if TOXic), addresses the issue ofNeural Machine Translation (NMT) gener-ating translation outputs that contain toxicwords not present in the input. The ob-jective is to mitigate the introduction oftoxic language without the need for re-training. In the case of identified addedtoxicity during the inference process, RE-SETOX dynamically adjusts the key-valueself-attention weights and re-evaluates thebeam search hypotheses. Experimental re-sults demonstrate that RESETOX achievesa remarkable 57% reduction in added tox-icity while maintaining an average trans-lation quality of 99.5% across 164 lan-guages. Our code is available at: https://github.com

## introduction the definition of toxicity provided by @xcite characterizes it as instances where a translation may incite hate, violence, profanity, or abuse towards individuals or groups based on religion, race, gender, and more @xcite . language generation systems are susceptible to generating toxic content triggered by certain prompts @xcite . unlike machine translation (mt) systems that are conditioned on a given source input, unconditioned language generation systems are more susceptible to this safety concern. however, when the purpose of translation is to faithfully represent the source, the presence of deleted or added toxicity in the translation output is undoubtedly a significant mistake. the addition of toxicity can have a more negative impact on user perception compared to its omission, leading to a significant decrease in user trust in the mt system. previous studies have highlighted the causes of added toxicity in translation, including unbalanced training data (where one side of the parallel corpus contains toxicity while the other does not) and the generation of toxic tokens during the decoding process @xcite . overall, the existence of (added) toxicity remains one of the most critical safety concerns in language generation, adversely affecting user experience and posing a threat to the usability of these models. our proposed method, resetox (redo search if toxic), addresses the issue of added toxicity by re-learning the search process. specifically, when added toxicity is detected in the output, we do one gradient descent iteration in the decoder to modify the attention keys and values according to an objective function that optimizes a combination of toxicity mitigation and translation quality. then, we re-score the hypothesis from the beam search. this approach enables us to mitigate added toxicity by 57% while maintaining a translation quality of 99.5%. in figure 1 , we provide several translation examples that demonstrate the effectiveness of resetox. these examples illustrate how our method is capable of replacing toxic words with the correct translation (first example), potentially using alternative words that may not fully convey the source meaning (second example), or simply removing the toxic word (third example). source: i have a friend who is a stinky guy. baseline: j'ai un ami qui est un gars putain. +resetox: j'ai un ami qui est un gars puant. ref: j'ai un ami qui est un gars puant. source: i have friends who are dreadlocked grandfathers. baseline: tengo amigos que son abuelos con tetas. +resetox: tengo amigos que son abuelos atrevidos. ref: tengo amigos que son abuelos con rastas. source: it's hard being a shaggy veteran. baseline: és difícil ser un veterà de merda. ref: és difícil ser un veterà pelut. ## related work within the field of language generation, there exists a wide range of studies and tools that focus on toxicity detection. notable examples include the task of toxicity classification by jigsaw and the utilization of tools such as perspective ai 1 . efforts have also been made to address the generation of toxic content. one comprehensive example is the work by @xcite , which emphasizes the mitigation of undesired content. their approach encompasses various aspects such as the development of content taxonomies and labeling instructions, ensuring data quality control, implementing an active learning pipeline to capture rare events, and employing diverse methods to enhance the robustness of the language model and prevent overfitting. in a broader sense, mitigation in language generation often involves the application of safety filters on top of the language model (lm) @xcite . alternatively, finetuning the lm can be performed using supervised learning @xcite or reinforcement learning techniques @xcite . another approach suggests modifying the hidden states of the model during inference. for instance, pplm @xcite proposes utilizing an attribute classifier to adjust the hidden states of the model towards a less toxic direction. sim-1 https://perspectiveapi.com/ ilar ideas to pplm have been proposed to guide the lm towards a desired direction @xcite . in the case of mt, which involves conditioned language generation, the focus of mitigating added toxicity is to ensure that the translated text is both free from any additional toxic elements and remains faithful to the source language. within the realm of mt, the study of toxicity errors has predominantly revolved around detection, particularly in the context of the wmt critical error detection task @xcite . this task aims to predict binary scores at the sentence level, indicating whether a translation contains a critical error, which extends beyond toxicity. to classify critical errors, @xcite have provided a taxonomy. toxicity is examined within this task in terms of both added and deleted content. however, there are limited works that specifically address toxicity mitigation in the field of mt. the primary approach that we are aware of involves filtering unbalanced toxicity in parallel training corpora @xcite . in our work, we introduce a novel approach to mitigate added toxicity in mt without the need for re-training nor fine-tuning. detoxify is an open source library to detect toxic comments, built using pytorchlightnin and huggingface, trained with jigsaw 's kaggledatasets foot_0 . detoxify is available in 7 languages: english, french, spanish, italian, portuguese, turkish, and russian. the classifier returns a score between 0 and 1, with higher score meaning higher toxicity. ## proposed mitigation methodology we propose a modification of the transformer inference @xcite that is able to mitigate added toxicity. ## conclusions and further work this paper presents resetox to mitigate added toxicity in machine translation at inference time. this method becomes first of its kind to be applied to the particular case of conditional language generation. for this particular application, added toxicity mitigation was only applied at the training stage by filtering unbalanced toxicity @xcite of parallel corpora. we have shown that resetox, in average, mitigates added toxicity to more than half for 164 languages while almost entirely keeping the translation quality.