On Target Representation in Continuous-output Neural Machine Translation (2022)

Abstract: Continuous generative models proved their usefulness in high-dimensional data, such as image and audio generation. However, continuous models for text generation have received limited attention from the community. In this work, we study continuous text generation using Transformers for neural machine translation (NMT). We argue that the choice of embeddings is crucial for such models, so we aim to focus on one particular aspect”:” target representation via embeddings. We explore pretrained embeddings and also introduce knowledge transfer from the discrete Transformer model using embeddings in Euclidean and non-Euclidean spaces. Our results on the WMT Romanian-English and English-Turkish benchmarks show such transfer leads to the best-performing continuous model.

## introduction & related work discrete neural models represent the majority of systems used in sequence-to-sequence tasks @xcite . despite the promising advantages of continuous-output models in terms of efficiency and expressivity, literature has awarded them relatively little attention. while past work focuses on continuous training objectives, we remark that the choice of word representations is essential. continuous-output nmt was first studied by @xcite . they study regularized probabilistic loss functions, even though their results show that by far the biggest gain comes from switching to pretrained fasttext @xcite embeddings from word2vec @xcite . @xcite follow up with a study of margin-based losses. however, to the best of our knowledge, there is no comprehensive study on token-level representation and their impact on the continuous nmt performance. in our work, we attempt to fill the gap and give insights about target representation in continuous-output nmt by highlighting an analogy between target representations and the output layer of a discrete model. we propose, as a knowledge transfer strategy, pretraining word representations with a discrete translation model. on two different language pairs, namely romanian-english (@xmath0, we find that this strategy outperforms externally-trained representations, even from massive pretrained language models. moreover, we find, somewhat surprisingly, that high dimensionality not only does not help, but can even substantially hurt, and that taking into account the natural spherical geometry of the cosine objective can lead to better performance with smaller dimensionality. ## continuous-output nmt nmt seeks to translate a sequence of tokens 𝒙 1:𝑁 = (𝑥 1 , ..., 𝑥 𝑁 ) from the source language to a sequence 𝒚 1:𝑇 = (𝑦 1 ,...,𝑦 𝑇 ) in the target language using a neural model: the probabilistic model above is typically implemented by sequence-to-sequence deep neural models @xcite @xcite , using the decomposition in a discrete model, the conditional token probabilities in eq. ( 2 ) are categorical distributions over a fixed vocabulary v tgt , in a continuous model, the output space is not limited to a discrete vocabulary but instead gives mass to the entire space w, and we interpret the notation 𝑝(𝑦 𝑖 |𝒚 1:𝑖-1 ,𝒙) to mean 𝑝(𝒘(𝑦 𝑖 )|𝒚 1:𝑖-1 ,𝒙). a common parametrization uses the cosine similarity, here, the distribution is over a continuous space, so the normalizer is an @xmath1. by a symmetry argument, it can be shown that the normalizer does not depend on 𝒉 and is therefore a constant, yielding the cosine distance loss: the cosine loss is an intuitive choice with a history of use in nlp @xcite . its probabilistic interpretation we give has roots in directional statistics @xcite , and corresponds to a langevin distribution (also known as vmf) with fixed scale. @xcite studied more general langevin distributions for nmt. even though these more flexible formulations provide useful modelling extensions, the impact of the loss seems less than the impact of embeddings. ## experiments we experiment using the publicly available wmt 2016 @xmath2, and the wmt 2018 @xmath3. we compute bleu @xcite using sacrebleu @xcite 1 on newsdev2016 and newstest2016 for both @xmath4. detailed information about data is collected in appendix a. all experiments and implementation are based on fairseq @xcite framework. we use 6-layers transformer base model as a baseline. for continuous model, encoder and decoder embeddings size are set to 512 (they are not initialized with pretrained embeddings), and output layer size depends on the target embeddings dimensionality. we choose the best model checkpoint based on development bleu. for generation, we rely on the top-1 nearest neighbor search (greedy) using cosine similarity, the details are discussed in appendix c. ## conclusion in this work, we investigated the importance of target representations for continuous nmt in two language pairs. we find that our proposed strategy to transfer embeddings from a discrete transformer model outperforms all other embedding choices. we pinpoint the impact of properties like dimensionality and geometry, and provide further insight into the errors made by continuous models. our proposed transfer strategy is effective despite using much less data compared to large pretrained models. we believe that further research into combining external data with mt-transfer embeddings may be necessary for improving continuous model performance. even though our model performance is behind the discrete model, we argue that this work can be seen as a stepping stone for building strong and reliable continuous model for text generation.