Robust Neural Machine Translation with ASR Errors (2020)

Abstract: In many practical applications, neural machine translation systems have to deal with the input from automatic speech recognition (ASR) systems which may contain a certain number of errors. This leads to two problems which degrade translation performance. One is the discrepancy between the training and testing data and the other is the translation error caused by the input errors may ruin the whole translation. In this paper, we propose a method to handle the two problems so as to generate robust translation to ASR errors. First, we simulate ASR errors in the training data so that the data distribution in the training and test is consistent. Second, we focus on ASR errors on homophone words and words with similar pronunciation and make use of their pronunciation information to help the translation model to recover from the input errors. Experiments on two Chinese-English data sets show that our method is more robust to input errors and can outperform the strong Transformer baseline significantly.

## introduction in recent years, neural machine translation (nmt) has achieved impressive progress and has shown superiority over statistical machine translation (smt) systems on multiple language pairs @xcite . nmt models are usually built under the encoder-decoder architecture where the encoder produces a representation for the source sentence and the decoder generates target translation from this representation word by word @xcite @xcite . now nmt systems are widely used in real world and in many cases they receive as input the result of the automatic speech recognition (asr) system. despite the great success, nmt is subject to orthographic and morphological errors which can be comprehended by human @xcite . due to the auto-regression of decoding process, translation errors will be accumulated along with the generated sequence. once a translation error occurs at the beginning, it will lead to a totally different translation. although asr technique is mature enough for commercial applications, there are still recognition errors in their output. these errors from asr systems will bring about translation errors even totally meaning drift. as the increasing of asr errors, the translation performance will decline gradually @xcite . moreover, the training data used for nmt training is mainly human-edited sentence pairs in high quality and thus asr errors in the input are always unseen in the training data. this discrepancy between training and test data will further degrade the translation performance. in this paper, we propose a robust method to address the above two problems introduced by asr input. our method not only tries to keep the consistency of the training and test data but to correct the input errors introduced by asr systems. we focus on the most widely existent substitution errors in asr results which can be further distinguished into wrong substitution between words with similar pronunciation and wrong substitution between the words with the same pronunciation (known as homophone words). reference this gift is full of affection. trans-hm this gift contains an application. ## background as our method is based on the self-attention based neural machine translation model (transformer) @xcite , we will first introduce transformer briefly before introducing our method. ## the proposed method although asr is mature for commercial applications, there are still recognition errors in the result of asr. the asr recognition errors can be classified into three categories: substitution, deletion and insertion, which are shown in our method aims to improve the robustness of nmt to asr errors. to this end, our method first constructs a training data set which has a similar data distribution with the test data, then makes use of pronunciation information to recover from the sp errors and hm errors. specifically, our method works in a flow of three steps as 1. adding sp errors and hm errors in the training data randomly to simulate asr errors occurring in test; figure 1 illustrate the architecture of our method. note that the above three steps must be cascaded which means we always first try to correct the pronunciation information for sp errors and then use the corrected pronunciation information to play a part in the translation for hm errors. we will introduce the three steps in details in the following sections. ## related work it is necessary to enhance the robustness of machine translation since the asr system carries misrecognized transcriptions over into the downstream mt system in the slt scenario. prior work at- the figure has fallen sharply by almost half. ## conclusion voice input has become popular recently and as a result, machine translation systems have to deal with the input from the results of asr systems which contains recognition errors. in this paper we aim to improve the robustness of nmt when its input contains asr errors from two aspects. one is from the perspective of data by adding simulated asr errors to the training data so that the training data and the test data have a consistent distribution. the other is from the perspective of the model itself. our method takes measures to handle two types of the most widely existent asr errors: substitution errors between the words with similar pronunciation (sp errors) and substitution errors between homophone words (hm errors). for sp errors, we make use of the context pronunciation information to correct the embedding of pinyin words. for hm errors, we use pronunciation information directly to amend the encoding of source words. experiment results prove the effectiveness of our method and the ablation study indicates that our method can handle both the types of errors well. experiments also show that our method is stable during training and more robust to the errors.