Adversarially Improving NMT Robustness to ASR Errors with Confusion Sets (2022)

Abstract: Neural machine translation (NMT) models are known to be fragile to noisy inputs from automatic speech recognition (ASR) systems. Existing methods are usually tailored for robustness against only homophone errors which account for a small portion of realistic ASR errors. In this paper, we propose an adversarial example generation method based on confusion sets that contain words easily confusable with a target word by ASR to conduct adversarial training for NMT models. Specifically, an adversarial example is generated from the perspective of acoustic relations instead of the traditional uniform or unigram sampling from the confusion sets. Experiments on different test sets with hand-crafted and real-world noise demonstrate the effectiveness of our method over previous methods. Moreover, our approach can achieve improvements on the clean test set.

## introduction neural machine translation (nmt) has been widely used and deployed as a "de facto standard" @xcite . in many application scenarios, nmt models translate sentences generated by automatic speech recognition (asr) systems. although current asr systems have made substantial progress, texts recognized by them still suffer from a variety of recognition errors, i.e., deletion, insertion or substitution of tokens, where substitution errors are the most common errors among them @xcite . these errors will result in severe degradation of translation quality due to the discrepancy between training and test data @xcite . in order to mitigate the negative impact of substitution errors on nmt models, many studies explore external phonetic information as extra representation or training objective. @xcite improve nmt robustness to homophone errors with joint textual and phonetic embeddings. @xcite utilize a gating mechanism to integrate phonetic information into the final output of the encoder to alleviate homophone errors. @xcite exploit a noise detector to convert homophone errors tokens into syllables and use a syllable-aware nmt model to translate the mixed sequences into target texts. these methods are usually designed for dealing with noisy tokens with same or similar pronunciation. however, realistic substitution noises in asr-generated texts are not only limited to homophone errors due to complicated acousticslinguistics relations, as shown in figure 1 . when the correct character '身 (shēn)' is recognized as a non-homophonous character '先 (xiān)' by an asr system, previous methods fail to provide correct translation with the help of external phonetic information, indicating that employing phonetic information is not sufficient to handle realistic asr errors. to tackle this issue, we propose an adversarial example generation method based on confusion sets, where words in a confusion set for a target figure 2 : illustration of the proposed method. the right most part denotes the embedding space for a confusion set. g is the gradient of input token x. gradient descent is performed to change the original embedding x to x. then a token farthest from x is selected for substitution. our key idea is to make the representations of confusing tokens close to those of corresponding golden tokens in the embedding space so as to model the acoustic relations of confusing tokens. to this end, we craft adversarial examples that have weak acoustic relations with original sentences to attack the nmt model according to both the gradient of the source token and the distance between token embeddings. with the generated adversarial examples, we conduct adversarial training to improve the robustness of nmt models against asr errors. to sum up, our contributions are as follows: ## approach we follow previous practice of using adversarial training to improve the robustness of nmt @xcite ) by iteratively adding generated adversarial examples to the training set. in this section, we will introduce our approach (illustrated in figure 2 ) in detail. ## conclusion in this paper, we have presented an adversarial example generation method based on confusion sets to make nmt models robust against real asr errors. the acoustic relations between confusing tokens modeled by our approach can make nmt models more resilient to asr errors. experimental results on two chinese-english text translation tasks and one chinese-english speech translation task prove that the effectiveness of our method. moreover, our method does not require any changes to models. it could be therefore orthogonal and complementary to other methods to further improve the robustness of nmt model. token and ground-truth token embeddings is critical to handle asr errors. moreover, the poor result obtained when λ = 0.0 on the noisy test set indicates that gradient information of the victim model benefits the robustness of nmt to asr noise. we conjecture the addition of nmt gradient information can help generate diversified adversarial examples.