Integrating Translation Memories into Non-Autoregressive Machine Translation (2023)

Abstract: Non-autoregressive machine translation (NAT) has recently made great progress. However, most works to date have focused on standard translation tasks, even though some edit-based NAT models, such as the Levenshtein Transformer (LevT), seem well suited to translate with a Translation Memory (TM). This is the scenario considered here. We first analyze the vanilla LevT model and explain why it does not do well in this setting. We then propose a new variant, TM-LevT, and show how to effectively train this model. By modifying the data presentation and introducing an extra deletion operation, we obtain performance that are on par with an autoregressive approach, while reducing the decoding load. We also show that incorporating TMs during training dispenses to use knowledge distillation, a well-known trick used to mitigate the multimodality issue.

## introduction non-autoregressive neural machine translation (nat) has been greatly advanced in recent years @xcite . nat takes advantage from parallel decoding to generate multiple tokens simultaneously and speed up inference. this is often at the cost of a loss in translation quality when compared to autoregressive (ar) models @xcite . this gap is slowly closing and methods based on iterative refinement @xcite @xcite and on connectionist temporal classification @xcite are now reporting bleu scores similar to strong ar baselines. most works on nat focus on the standard machine translation (mt) task, where the decoder starts from scratch, with the exception of @xcite ; @xcite , who use nat to integrate lexical constraints in decoding. however, edit-based nat models, such as the levenshtein transformer (levt) of @xcite , seem to be a natural candidate to perform mt with translation memories (tm). levt is able to iteratively edit an initial target sequence by performing insertion and deletion operations until convergence. this design also matches the concept of using tms in mt, where given a source sentence, we aim to edit a candidate translation retrieved from the tm. this idea has been used for decades in the localization industry and implemented into basic computer-aided translation tools. translators wishing to translate a sentence can benefit from fuzzy matching techniques to retrieve similar segments from the tm. these segments can then be revised, thereby improving productivity and consistency of the translation process @xcite . the retrieval of similar examples from a tm has also proved useful in conventional (ar) neural mt systems; they can be injected into the encoder @xcite or as priming signals in the decoder @xcite to influence the translation process. these studies report significant gains in translation performance in technical domains, where the translation of terms and phraseology greatly benefits from examples found in a tm. our main focus in this work is to develop an improved version of levt suited to the revision part of tm use, where the translation retrieved from tm is modified via edit operations in a nonautoregressive way. we first show that the original levt cannot perform well on this task and explain that this failure is a direct consequence of its training design. we propose to fix this issue with tm-levt, which includes an additional deletion step. next, we propose to further improve the training procedure in two ways: (a) by also including the retrieved candidate translation on the source side, as done in ar tm-based approaches @xcite ; (b) by simultaneously training with empty and non-empty initial target sentences. in our experiments, tm-levt achieves performance that is on par with a strong ar ap-proach on various domains when translating with tms, with a reduced decoding load. we also observe that incorporating an initial translation both on the source and target sides makes knowledge distillation (kd, @xcite useless. this contrasts with standard nat models, which rely on kd to alleviate the multimodality issue @xcite . as far as we know, this work is the first to study nat with tms in a controlled setting. our contributions are the following: (a) we show that the original levt training scheme is not suited to edit similar translations from a tm; (b) we propose a variant of levt, tm-levt with an improved training procedure, which yields performance that are close, or even similar to ar approaches when translating with good tm matches, with a reduced decoding load; (c) we highlight the benefits of multi-task training (with and without tms) to attain the best performance; (d) we discuss the reasons why kd hurts the training of nat with tms. ## conclusion in this paper, we studied ways to augment the levt architecture with tms. our proposal adds an initial deletion operation during training to detect possible unrelated tokens present in tm matches. by copying the tm match both on the source side and on the target side as an initial target sequence, our model vastly outperformed the original levt model and achieved bleu scores approaching those of a strong ar model both when decoding from scratch and when editing a tm match. compared to levt, tm-levt also generates translations that contain less unrelated tokens, and is able to converge in fewer iterations. we also found that training with tms improves nat performance on standard mt. finally, we have tried to combine kd with our approach, concluding that it was more hurting than helpful for tm-based architectures.