Secoco: Self-Correcting Encoding for Neural Machine Translation (2021)

Abstract: This paper presents Self-correcting Encoding (Secoco), a framework that effectively deals with noisy input for robust neural machine translation by introducing self-correcting predictors. Different from previous robust approaches, Secoco enables NMT to explicitly correct noisy inputs and delete specific errors simultaneously with the translation decoding process. Secoco is able to achieve significant improvements over strong baselines on two real-world test sets and a benchmark WMT dataset with good interpretability. We will make our code and dataset publicly available soon.

## introduction neural machine translation (nmt) has witnessed remarkable progress in recent years @xcite . most previous works show promising results on clean datasets, such as wmt news translation shared tasks @xcite . however, inputs in real-world scenarios are usually with a wide variety of noises, which poses a significant challenge to nmt. in order to mitigate this issue, we propose to build a noise-tolerant nmt model with a selfcorrecting encoding (secoco) framework that explicitly models the error-correcting process as a sequence of operations: deletion and insertion. figure 1 demonstrates a simple correcting process that transforms a noisy sequence "abbd" into its correct sequence "abcd" via a deletion and inserting operation. in order to learn desirable operations for noise correction given noisy inputs, we propose a insertion predictor and deletion predictor that predict appropriate deletion and insertion operations respectively. the two predictors work alternatively abbd abd abcd delete 0010 insert 00c figure 1 : an example of the correcting and operation generation process. assume we want to correct a synthesized noisy sequence "abbd" to its correct sequence "abcd". we can apply a deletion "b" operation to the third position (0010) and an insertion "c" operation to the third position (00c). ("abbd","0010") and ("abd","00c") can be regarded as training examples. for training the two predictors, we collect a list of pairs (source sequence, operation sequence) (e.g., ("abbd","0010") shown in figure 1 ) from original training data by randomly deleting or inserting tokens from/to original clean sequences. with these collected training instances, we optimize the insertion and deletion predictors as well as nmt simultaneously in a multi-task learning way. for inference, we propose two different variants for secoco depending on the decoding modes. the first variant is an end-to-end approach like normal nmt decoding where the encoder is implicitly trained with self-correcting information. in this setting, we only predict operations during training and the encoder can have this kind of knowledge. the other variant is iterative editing, which corrects the input gradually and performs translation after the input is unchanged. compared with previous approaches, secoco has two advantages. first, secoco introduces a more explicit and direct way to model the noise correcting process. second, secoco enables an interpretable translation process. with the predicted operation sequence, it is easy to understand how the noisy input is corrected. we conduct experiments on three test sets, including dialogue, speech, and wmt14 en-de tasks. the results show that secoco outper- encoder deletion predictor insertion predictor deletion predictor insertion predictor @xmath0. we omit the translation part here due to the space limit. the right part shows the decoding modes. ## approach our approach is illustrated in figure 2 . the left part of figure 2 demonstrates the encoding module of secoco. the only difference of secoco from standard translation models is the two correcting operation predictors, which generate the operation sequence based on the encoder representation of an input text. the deletion predictor decides which word to be deleted while the insertion predictor decides which word to be inserted into which position. the combination of these two operations is able to simulate arbitrary complex correcting operations @xcite . we illustrate the training data synthesizing process for the two predictors in figure 1 . it is worth noting that for correction that contains several iterations of editing (i.e., deletion or insertion), we sample only one iteration from it. ## related work approaches to the robustness of nmt can be roughly divided into three categories. in the first research line, adversarial examples are generated with black-or white-box methods. the generated adversarial examples are then used to combine with original training data for adversarial training @xcite @xcite @xcite . in the second strand, a wide variety of methods have been proposed to deal with noise in training data @xcite @xcite @xcite @xcite . finally, efforts have been also explored to directly cope with naturally occurring noise in texts, which are closely related to our work. @xcite ; @xcite ; @xcite focus on word spelling errors. @xcite study translation problems caused by speech recognition. @xcite introduce back-translation to generate more natural synthetic data, and employ extra tags to distinguish synthetic data from raw data. @xcite propose a reconstruction method based on one encoder and two decoders architecture to deal with natural noise for nmt. different from ours, most of these works use the synthetic data in a coarsegrained and implicit way (i.e. simply combining the synthetic and raw data). ## conclusions in this paper, we have presented a framework secoco to build a noise-tolerant nmt model with selfcorrecting capability. with the proposed secoco-e2e and secoco-edit methods, secoco exhibits both efficiency and interpretability. experiments and analysis on the three test sets demonstrate that the proposed secoco is able to improve the quality of nmt in translating noisy inputs, and make better use of synthetic data.