Progressive Translation: Improving Domain Robustness of Neural Machine Translation with Intermediate Sequences (2023)

Abstract: Previous studies show that intermediate supervision signals benefit various Natural Language Processing tasks. However, it is not clear whether there exist intermediate signals that benefit Neural Machine Translation (NMT). Borrowing techniques from Statistical Machine Translation, we propose intermediate signals which are intermediate sequences from the “source-like” structure to the “target-like” structure. Such intermediate sequences introduce an inductive bias that reflects a domain-agnostic principle of translation, which reduces spurious correlations that are harmful to out-of-domain generalisation. Furthermore, we introduce a full-permutation multi-task learning to alleviate the spurious causal relations from intermediate sequences to the target, which results from exposure bias . The Minimum Bayes Risk decoding algorithm is used to pick the best candidate translation from all permutations to further improve the performance. Experiments show that the introduced intermediate signals can effectively improve the domain robustness of NMT and reduces the amount of hallucinations on out-of-domain translation. Further analysis shows that our methods are especially promising in low-resource scenarios.

## introduction a spectrum of studies recently arose in natural language processing (nlp), which incorporates intermediate supervision signals into the model by simply converting the intermediate signals into textual sequences and prepending or appending these sequences to the output sequence. it benefits tasks such as math word problems @xcite , commonsense reasoning @xcite , programs execution @xcite , summarisation @xcite , etc. this trend further there she is. that she is. that is she. ## analysis bleu score indicates that the proposed methods can improve domain robustness. in this section, we investigate the reduction of hallucinations and performance on larger datasets of our methods. ## conclusion our results show that our introduced intermediate signals effectively improve the ood performance of nmt. intermediate sequence lex can benefit ood by simply prepending it to the target. ali is more likely to be erroneous during inference than lex, which results in degenerated target due to the spurious causal relationship. our proposed permutation multi-task learning successfully alleviates the problem and manifests the effectiveness of ali. experiments also confirm that the mbr algorithm can further improve the performance by dynamically selecting a consensus translation from all permutations. the human evaluation shows that the proposed methods substantially reduce the number of hallucinations of the out-of-domain translation. experiments on the larger corpus sizes indicate that our methods are especially promising in the low-resource scenarios. our work is the first attempt to complete the puzzle of the study of intermediate signals in nmt, and two new ideas may benefit this study in other areas: 1) thinking intermediate signals from the intermediate structures between the transformation from the input to the output; 2) the permutation multi-task learning, instead of only pre/appending intermediate sequences to the output sequence. the permutation multi-task learning + mbr decoding framework is also a potential solution for any multi-pass generation tasks (e.g. speech translation), which suffer from the error propagation problem. the problem is alleviated with the permutation which disentangles causal relations between intermediate and final results. finally, our work provides a new perspective of data augmentation in nmt, i.e. augmenting data by introducing extra sequences instead of directly modifying the source or target.