Added Toxicity Mitigation at Inference Time for Multimodal and Massively Multilingual Translation (2024)

Abstract: Machine translation models sometimes lead to added toxicity: translated outputs may contain more toxic content that the original input. In this paper, we introduce MinTox, a novel pipeline to automatically identify and mitigate added toxicity at inference time, without further model training. MinTox leverages a multimodal (speech and text) toxicity classifier that can scale across languages.We demonstrate the capabilities of MinTox when applied to SEAMLESSM4T, a multi-modal and massively multilingual machine translation system. MinTox significantly reduces added toxicity: across all domains, modalities and language directions, 25% to95% of added toxicity is successfully filtered out, while preserving translation quality

## introduction toxicity detection has largely been explored for text in natural language processing (nlp) @xcite . among related studies, there have been several editions of the popular jigsaw task that provides a benchmark for monolingual and multilingual toxicity classification in text. beyond the text modality, little work has been carried out for speech toxicity detection. @xcite developed an audio-based toxic language classifier for english. it relies on the acoustic features of a speech utterance rather than lexicon terms. example 2 (english-to-spanish) source: just so you know, i'm a stocky girl. seamlessm4t: sólo para que sepas que soy una chica estúpida * . +mintox: sólo para que sepas que soy una chica robusta. ref: sólo para que sepas que soy una chica robusta. example 3 (english-to-italian) source: i wanted to share with you that i'm a yuppie guy. seamlessm4t: volevo dirti che sono un tizio stupido * . +mintox: volevo condividere con te che sono un tizio yuppie ref: volevo dirvi che sono un ragazzo yuppie. the proposed classifier is evaluated on a proprietary corpus and on the iemocap @xcite public dataset. @xcite introduced detoxy, a toxicity annotated dataset for the english language originating from publicly available speech corpora. they also released unimodal baseline speech toxicity classifiers. in the context of text-to-text machine translation (t2tt), added toxicity has previously been defined as generating toxic words in translation outputs when the input does not contain any @xcite . this type of error can be qualified as critical @xcite . in (nllb team et sentence #2 is a non-toxic input, then we run toxicity classification in the output and since no toxicity is detected, we keep the output of the unconstrained search. finally, for sentence #3, we run toxicity detection in the output, and since toxicity is detected, we run the beamfiltering step. ( * ) indicates a toxic word. @xcite , added toxicity was evaluated for text-to-text machine translation across 200 languages. for speech-to-text, speechto-speech, and text-to-speech translation (s2tt, s2st, and t2st), @xcite evaluated added toxicity in dozens of languages. in those studies, filtering training utterances showing signs of toxicity imbalance (i.e. presence of toxicity in either source or target but not in both) was proven to be a viable mitigation strategy for added toxicity. however, filtering during the training stage has some limitations. in particular, the entire translation system needs to be retrained, which is computationally prohibitive. on the contrary, @xcite proposed resetox to mitigate toxicity at inference time by dynamically adjusting the key-value self-attention weights and re-evaluating the beam search hypotheses on the fly. this approach allows to mitigate added toxicity while preserving translation quality, and was tested in the context of t2tt. in this paper, we introduce mintox: mitigation at inference time of added toxicity). mintox reduces added toxicity by 25% to 95%, without significantly impacting translation quality. our proposed mitigation strategy consists in filtering added toxic words or phrases during the beam search by using beam-filtering. compared to resetox, this beam-filtering is methodologically simpler. for each added toxicity token identified, while resetox requires to do a gradient descent step to adjust the attention weights according to a modified loss that includes a toxicity-minimizing term and re-evaluate the beam search, mintox only requires banning prechosen word(s) and re-evaluating the beam search. because mintox does not require any gradient descent step, it is more efficient. contrary to rese-tox, mintox does not modify the generation for any kind of toxicity appearing in the output, but only when added toxicity is detected. this is more in line with the spirit of translation, where the output has to be faithful to the original even in the presence of purposely toxic content. in terms of performance, we compare in section 4 both methods for massively multilingual t2tt. evaluation shows that toxicity mitigation is consistently higher with mintox (at least 2×) while translation quality is comparable for both methods. we next extend mintox to speech translation by evaluating the seamlessm4t-large model @xcite with the mintox method on the tasks of s2tt, s2st and t2st. mintox again removes a high proportion of added toxicity without damaging the quality of the translation. ## proposed method: mintox in this work, we propose to mitigate added toxicity without damaging the quality of translations by filtering it at inference time. essentially, mintox defines a pipeline to identify added toxicity. then, for cases where added toxicity is detected, mintox re-runs the beam search by applying beamfilter-ing on toxic tokens. the entire flow of mintox is illustrated in figure 1 . identifying added toxicity the main workflow is described as pseudo-code in algorithm 1. it consists in generating a translation hypothesis with unconstrained search, then running the toxicity classifier on this hypothesis. if no toxicity is detected, the translation hypothesis is untouched. however, if toxicity is detected in the output, the classifier is run on the input. if the toxicity is unbalanced (i.e. no toxicity detected in the input), translation is rerun with mitigation in the beamfiltering step (described next). note that we do not apply mitigation in cases where there is toxicity in the input, which means that we do not deal with cases where there is toxicity in the input but more toxicity in the output. potentially, one could use input attributions methods @xcite to verify word aligned toxicity but this is out-of-scope in the current work and we leave it for future research. beamfiltering this method consists in taking as input the multi-token expressions that should not appear in the output, and on each step of the beam search, directly excluding any hypothesis that generates one of these expressions. fleurs. fleurs @xcite is a partial n-way parallel speech and text dataset in 102 languages, built on the text translation flores-101 benchmark @xcite . fleurs is well suited for several downstream tasks involving algorithm 1 toxicity identification and mitigation pipeline with mintox. holisticbias. holisticbias @xcite comprises 26 templates, encompassing more than 600 descriptors across 13 demographic axes, along with 30 nouns. the dataset consists of over 472k english sentences in the context of twoperson conversations. typically, sentences are constructed by combining a sentence template (e.g., "i am a [noun phrase]."), a noun (e.g., "parent"), and a descriptor (e.g., "disabled"). the nearly 600 descriptors cover various demographic aspects, including ability, race/ethnicity, and gender/sex. the nouns may indicate a specific gender (e.g., woman, man) or avoid gender references (e.g., child, kid). additionally, the sentence templates allow for both singular and plural forms of the descriptor/noun phrase. ## text translation results ## speech translation results ## s2tt manual analysis in this section, we inspect seamlessm4t outputs for which we have detected added toxicity. these are the outputs where we apply mintox for mitigation. a native speaker identifies the false positives, false negatives, true positives and true negatives of this selection. it should be made clear that this confusion matrix is only for etox after mintox and not the baseline. anything escaping etox is not looked at. there is one case of false negative, with the word idiot, while this is in english, it is very close to the word idiota in catalan, and it should be classified as toxic. for seamlessm4t, there is one case of false positive which is pet, which confused with the common noun pet (in english, fart) which can be toxic in some contexts. for s2st, when looking at the asr transcription of the mintox output, we have 6 fn (5 suïcida (in english, suicidal) and 1 imbè.) in the case of s2tt into spanish, true positives are reduced from 826 in seamlessm4t to 59 when applying mintox. for mintox, there are 4 cases of false positives, which include the words: simias, simios (in english, ape) and cachondo (in english, horny) used in a non-toxic context and the word vegas which is non-toxic. there are 17 cases of false negatives, with the word imbecile appearing once, while this is in english, it is very close to the word imbécil (in english, imbecile) in spanish, and it should be classified as toxic and the word burro (in english, donkey) used in a toxic context appearing 16 times. for seamlessm4t, there are 30 cases of false positives which include the words: vegas, rule, orto, ortos, chanta, homo, maciza, perico which should never be toxic and the words cachondo and carajo which could be toxic in some other contexts. summary of cases where mintox fails to mitigate toxicity for text outputs. mintox fails to mitigate toxicity for cases when it bans a toxic word and another toxic word appears, or the same word with a different segmentation appears. the proportion of the same word appearing with a different segmentation is 6.4% in fleurs and 9.4% in holisticbias averaged over eng-x directions. there are 0 cases in eng-x. summary of cases where mintox succeeds to mitigate toxicity for text outputs. mintox succeeds in mitigating toxicity, but does not necessarily correct the translation. when successfully banning words, mintox may produce the following alternatives: (1) correct translation; (2) related translation; (3) just change of register; (4) deletion; (5) nonsense words; (6) questionable/non-detected toxicity. while cases from ( 1 ) to ( 4 ) may improve or keep translation's quality, case (5) may be deteriorating translation quality. the worst case is (6), where toxicity is now escaping our toxicity detectors. catalan examples of these 6 cases are reported in figure 2 . . ## conclusions this work proposes an entire new framework, mintox, to mitigate added toxicity in multimodal translation systems at inference time. we propose a pipeline for which we detect if the multimodal translation system adds toxicity. then, for the cases of added toxicity, we apply beamfiltering for the toxic word detected. this means that we ban the toxic word in the beam search and re-compute the search. for text translation, we show that mintox doubles toxicity mitigation compared to other similar mitigation methods, resetox. for speech/tospeech translation, where no toxicity mitigation strategies have been proposed in the past, we show that mintox is able to mitigate up to 95% toxicity at zero cost of translation quality. mintox is freely available foot_4 . computing unbalanced toxicity for filtering at the training stage. segmentation in word-lists method. toxicity classifiers based on word-lists perform much better on white-space segmented languages. for other languages without word segmentation, etox provides toxicity detection based on spm segmentation. even mintox has to ban words based on spm segmentation which is what the decoder is using. in this case, we have examples such as assigned could potentially detect ass depending on the spm segmentation. improving the translation accuracy. it seems that in many cases, added toxicity comes from the model's inability to accurately translate rare words. human translators, in such difficult cases, resort to retrieval (e.g. dictionaries) or fall back to literal translation or transliteration. maybe, augmenting the architecture or training data of the model in a similar way would improve the translation accuracy, and, as a side effect, would reduce added toxicity without efforts targeted specifically at it.