Is Encoder-Decoder Redundant for Neural Machine Translation? (2022)

Abstract: Encoder-decoder architecture is widely adopted for sequence-to-sequence modeling tasks. For machine translation, despite the evolution from long short-term memory networks to Transformer networks, plus the introduction and development of attention mechanism, encoder-decoder is still the de facto neural network architecture for state-of-the-art models. While the motivation for decoding information from some hidden space is straightforward, the strict separation of the encoding and decoding steps into an encoder and a decoder in the model architecture is not necessarily a must. Compared to the task of autoregressive language modeling in the target language, machine translation simply has an additional source sentence as context. Given the fact that neural language models nowadays can already handle rather long contexts in the target language, it is natural to ask whether simply concatenating the source and target sentences and training a language model to do translation would work. In this work, we investigate the aforementioned concept for machine translation. Specifically, we experiment with bilingual translation, translation with additional target monolingual data, and multilingual translation. In all cases, this alternative approach performs on par with the baseline encoder-decoder Transformer, suggesting that an encoder-decoder architecture might be redundant for neural machine translation.

## introduction sequence-to-sequence modeling is often approached with neural networks (nns), prominently encoder-decoder nns, nowadays. for the task of machine translation (mt), which is by definition also a sequence-to-sequence task, the default choice of nn topology is also an encoderdecoder architecture. for example, in early works like @xcite , the authors already make the distinction between their convolutional sentence model (encoder) and recurrent language model (decoder) conditioned on the former. in follow-up works like @xcite and @xcite , the concept of encoderdecoder network is further developed. while extensions such as attention @xcite , multi-task learning @xcite , convolutional networks @xcite and selfattention @xcite are considered for sequence-to-sequence learning, the idea of encoding information into some hidden space and decoding from that hidden representation sticks around. given the success and wide popularity of the transformer network @xcite , many works focus on understanding and improving individual components, e.g. positional encoding @xcite , multi-head attention @xcite , and an alignment interpretation of cross attention @xcite . in works that go a bit further and make bigger changes in terms of modeling, e.g. performing round-trip translation @xcite and going from autoregressive to non-autoregressive @xcite , the encoderdecoder setup itself is not really questioned. in the mean time, it is not to say that the field is completely dominated by one approach. because works like the development of direct neural hidden markov model @xcite @xcite , investigation into dropping attention and separate encoding and decoding steps @xcite and going completely encoder-free @xcite do exist, where the default encoder-decoder regime is not directly applied. meanwhile, in the field of language modeling, significant progress is achieved with the wide application of nns. with the progress from early feedforward language models (lms) @xcite , to the successful long short-term memory network lms @xcite , and to the more recent transformer lms @xcite , the modeling capacity of lms nowadays is much more than their historic counterparts. this is es-pecially true when considering some of the most recent extensions, such as large-scale modeling @xcite , modeling very long context @xcite and going from autoregressive modeling to non-autoregressive modeling @xcite . because mt can be thought of as a contextualized language modeling task with the source sentence being additional context, one natural question is if simply concatenating the source and target sentences and train an lm to do translation would work @xcite . this idea is simple and straightforward, but special care needs to be taken about the attention mechanism and source reconstruction. in this work, we explore this alternative approach and conduct experiments in bilingual translation, translation with additional target monolingual data and multilingual translation. our results show that dropping the encoder-decoder architecture and simply treating the task of mt as contextualized language modeling is sufficient to obtain state-of-the-art results in translation. this result has several subtleties and implications, which we discuss in sec.5, and opens up possibilities for more general interfaces for multimodal modeling. ## related work in the literature, few but interesting works exist which closely relate to the idea mentioned above. in @xcite , the authors mention the possibility to use source sentence as context for contextualized language modeling. in @xcite , with the intuition to coordinate the learning of transformer encoder and decoder layer by layer, the authors share the encoder and decoder parameters and learn a joint model on concatenated source and target sentences. however, no explicit source side reconstruction loss is included. similarly, in irie (2020), a small degradation in translation quality is observed when a causal mask is used and no source reconstruction is included. because the masking is critical for correctly modeling the dependencies regarding the concatenated sequence, in @xcite , the authors put special focus on discussing the differences and implications of three types of attention masks. in @xcite , the authors expand upon the idea and propose a two-step decaying learning rate schedule to reconstruct the source sentence to regularize the training process. in that work, the authors show competitive performance compared to transformer baselines in several settings. more recently, in @xcite , the authors also use a languagemodeling-style source side reconstruction loss to regularize the model, and additionally explore the model scaling cross-lingual transfer capabilities. another work that explores the long-context modeling potential of lms is @xcite , where data from domains other than translation is included in model training. @xcite is a more recent addition to this direction of research, where lm as a general interface for multimodal data is investigated. because our focus is in mt, we refer to such a model, where encoder-decoder architecture is dropped and an lm is used to model the concatenation of source and target sentence, as translation language models (tlms foot_0 ). the work by @xcite is probably the most directly related work compared to our work, therefore we believe it is important to highlight the similarities and differences between their work and ours. the core concept of dropping encoder-decoder architecture is similar between @xcite and our work, and competitive performance of tlms compared to encoder-decoder models in various settings is achieved in both works. however, we additionally explore the task of autoencoding in the source side, adding bidirectional-encoder-representations-from-transformers-style (bert) noise @xcite , using alternative learning rate schedules, training mt models with backtranslated (bt) data and doing multilingual training. further, we discuss subtleties and implications associated with the tlm. ## methodology the core concept of tlm is to concatenate the source and the target sentences and treat the translation task as a language modeling task during training. the two majors points of concern are the attention mechanism and the source-side reconstruction loss. in this section, we explain the details related to these two points, and additionally discuss the implications when additional target-side monolingual data or multilingual data is available. ## experiments to verify the performance of tlm compared to the baseline encoder-decoder transformer model, we perform experiments on four machine translation datasets. specifically, we experiment with the international conference on spoken language translation (iwslt) @xcite 2014 germanto-english (de-en), the conference on machine translation (wmt) 2016 english-to-romanian (en-ro) @xcite , 2019 chinese-to-english (zh-en) @xcite datasets. additionally, for multilingual experiments, we create a custom multilingual (multi.) dataset from newscommentary v16 @xcite , performing translation among three languages, german (de), spanish (es), and french (fr), in six direction: de-es, es-de, de-fr, fr-de, es-fr, fr-es foot_2 . for the monolingual data, we sample 5m sentences from the english news crawl monolingual corpus foot_3 . to create synthetic zh-en data, we employ our en-zh transformer model to do back-translation @xcite . the data is pre-processed with the byte pair encoding (bpe) @xcite algorithm. we lowercase the text for de-en and for the other language pairs, we leave the original casing as is. we implement the transformer model and the tlm model with different options such as using different attention masks, shifting versus not shifting the source output, adding or not adding bert-style @xcite noises and different learning rate schedules, in pytorch @xcite . the back-translation and multilingual experiments are done by adding corresponding language tags to the concatenation of source and target sentences. we follow the training and search hyperparameters as closely as possible to the original transformer @xcite paper. note that, when searching with tlm, the entire source sentence until (and including) the target start of @xmath0. the beam search is then carried out only on the target outputs. we report translation performances in bleu @xcite and ter @xcite scores using the multeval tool from @xcite . ## discussions through extensive experiments, we show that the encoder-decoder architecture is not a must to achieve decent translation performance, because an encoder-only tlm is also capable of obtaining comparable performance when carefully tuned. here, we touch upon several important implications and subtleties that come with using tlms. first, although the encoder-decoder architecture is dropped, the cross attention is still existent in the tlm. as shown in fig. 1 , the difference compared to the baseline is that for each target position i, the softmax needs to normalize the attention weights over j + i instead of j. however, because we know the softmax is decent at zeroing out certain positions, e.g. see fig. 1 in @xcite , this should not be a problem. next, although we do not expand on search in this paper, our internal experiments verify that the search with tlm behaves similarly to the baseline. further, one may wonder how separate source and target vocabularies should be handled in case of tlms. here, we note that having separate source and target word embedding matrices is the same as concatenating them in the vocabulary size dimension into a bigger word embedding matrix for tlm. what could pose as a problem is the increased length of the concatenated sequence. this puts extra requirements to the model and its capabilities to model long context dependencies. note that, concatenation may not be the only way to combine the source and target contexts. for instance, in the eager model proposed in press and smith (2018), the authors essentially "stack" instead of "concatenate". moreover, when decoding efficiency is critical, tlm may suffer because a separate decoder is not existent and each translation query goes through the entire network. another limitation is that the source side reconstruction loss considered in this work may also be applied to the transformer baseline, and might change the picture when comparing the two. that said, tlms are undoubtedly exciting models opening new possibilities. for example, with such generative models, generation of synthetic translation pairs from scratch can be easily done. another worth-to-mention application is end-toend speech translation (st). while previous work, e.g. in @xcite , connects the encoder of the automatic speech recognition model and the decoder of the mt model, effectively throwing away 50% of the pre-trained model parameters, tlms can retain all pre-trained parameters and result in more compact end-to-end st models. ## conclusion in this work, we question the long-standing encoder-decoder architecture for neural machine translation. through extensive experiments in various translation directions, considering backtranslation and multilingual translation, we find that an encoder-only model can perform as good as an encoder-decoder model. we further discuss implications and subtleties of such models to motivate further research into more compact models and more general neural network interfaces.