Benchmarking Diffusion Models for Machine Translation (2024)

Abstract: Diffusion models have recently shown great potential on many generative tasks.In this work, we explore diffusion models for machine translation (MT).We adapt two prominent diffusion-based text generation models, Diffusion-LM and DiffuSeq, to perform machine translation.As the diffusion models generate non-autoregressively (NAR),we draw parallels to NAR machine translation models.With a comparison to conventional Transformer-based translation models, as well as to the Levenshtein Transformer,an established NAR MT model,we show that the multimodality problem that limits NAR machine translation performance is also a challenge to diffusion models.We demonstrate that knowledge distillation from an autoregressive model improves the performance of diffusion-based MT.A thorough analysis on the translation quality of inputs of different lengths shows that the diffusion models struggle more on long-range dependencies than other models.

## introduction diffusion models have shown promising results in a wide range of generative tasks, such as image generation @xcite ), text-to-speech synthesis @xcite , and robotic control @xcite , but their application to natural language processing (nlp) is still a less explored direction. the last two years have seen various approaches to this @xcite , including discrete (token level) diffusions @xcite and continuous (embedded) diffusions. continuous diffusion models typically generate whole sequences in an iterative and nonautoregressive (nar) manner, and have shown strong results for controllable generative modelling @xcite . they have also been applied to sequence-to-sequence tasks such as open-domain dialog and question generation @xcite . in this work, we focus on machine translation (mt), another sequence-to-sequence task that requires fluent outputs over a vocabulary different from the input and the preservation of semantic meanings of the input sequences. despite potential speed advantages foot_0 , nar translation models tend to lag behind their ar counterparts in translation quality 2 @xcite @xcite as a result of the conditional independence assumption, where output tokens are generated independent of each other. this prompts us to compare diffusion-based mt models to conventional nar mt models. we explore how techniques commonly applied to nar mt models could benefit diffusion-based models. specifically, we seek to answer the following questions: 1) how can we adapt existing diffusion-based text generation models to machine translation? 2) how do these diffusion-based mt models compare to standard ar and nar machine translation models? 3) what are reasons for the performance gap and how can we bridge the gap? ## adapting diffusion models to machine translation machine translation is an instance of the conditional language modeling problem. specifically, it aims to automatically translate text from one source language to another target language, and may be described as modeling the distribution over the target space conditioned by a sequence from the source space. currently, the primary model choice for machine translation is the encoder-decoder architecture, especially the transformer @xcite , where an encoder module first encodes the source sequence, passing the encoding on to the decoder, which autoregressively generates an output sequence conditioned by the source encoding. to this end, formally we describe the probability of a sequence y given the conditioning information x under the transformer model p θ : ## experimental setup dataset and preprocessing we use the german-english text-to-text partition of the covost (wang et al., 2020) dataset and train the models for german-to-english translation. this dataset was chosen due to its comparable size to the experimental setup of @xcite , which used 50k to 98k samples in training. due to the slow decoding process of the autoregressive sampling method, the algorithm 1 autoregressive sampling the source sequence ▷ initialize the translation as the empty word 2: t ← ϵ 3: for @xmath0, . . . , s do 4: j ← s ⊕ (ŝ) ⊕ t 5: pad j up to length s ▷ embed concatenated sequence 6: draw xt ∼ n (0, i d×s ) 10: for @xmath1, . . . , 0 do ▷ with σt+1 as described in equation (1a test set was reduced to a subset of 2010 samples. 7 the dataset statistics are in ## conclusion using sequence-level knowledge distillation we saw a clear improvement in both training speed and model performance of diffusion-based machine translation systems. we believe they benefit from the reduced syntactic conditional entropy of distilled datasets and conclude that they suffer of a form of the multimodality problem, similarly to other nar machine translation systems. based on this, methods employed in other nar models to help them handle multimodality in the data are likely to improve the performance of diffusionbased machine translation approaches. the diffusion-lm-mt model proved capable of expressing the joint density of source and translation implying that with an improved infilling algorithm good conditional densities could be sampled from these models. however, using the infilling algorithm, it was ill-fit to produce high quality samples when used for a sequence to sequence task in our experiments. in a similar manner, the method of autoregressive sampling for diffusion-based translation systems had little impact on the quality of samples. the diffusion-based mt models studied currently struggle with training and inference speed. while some factors contributing to the slow optimization of these models such as the static sequence length may be alleviated, the problem of predicting the initial sequence based of the noisy version remains difficult. inference speed on the other hand can be improved with methods such as ddim @xcite and newer work on diffusion models for the image domain likely could be applied to diffusion language models as well. when using the results of diffusion language models directly without using mbr decoding, the models still fall decidedly behind the transformerbased baselines. at the same time, mbr-decoding does not seem broadly applicable, as long as inference is still as slow as in current models.