The Unreasonable Effectiveness of Random Target Embeddings for Continuous-Output Neural Machine Translation (2024)

Abstract: Continuous-output neural machine translation (CoNMT) replaces the discrete next-word prediction problem with an embedding prediction.The semantic structure of the target embedding space (*i.e.*, closeness of related words) is intuitively believed to be crucial. We challenge this assumption and show that completely random output embeddings can outperform laboriously pre-trained ones, especially on larger datasets. Further investigation shows this surprising effect is strongest for rare words, due to the geometry of their embeddings. We shed further light on this finding by designing a mixed strategy that combines random and pre-trained embeddings, and that performs best overall.

## introduction since text is naturally discrete, i.e., each token in a target sentence is represented by an integer index in the vocabulary, neural machine translation (nmt), as many other language generation tasks, is trained mainly as a discrete-output model with softmax over the full vocabulary followed by the cross-entropy loss. continuous-output neural machine translation (conmt) models, in contrast, are trained to predict the continuous representation based on the distances between vectors. it is an appealing line of study for computational and modeling related reasons @xcite , as well as a reliable test bed for exploring the properties of continuous language spaces that appear in modern deep generative models @xcite . however, conmt introduces its own challenge, namely mapping to and from a continuous space. during training, conmt model requires continuous targets, and while decoding, one needs to map back to the discrete text representation. text mapping to continuous space is widely explored in nlp and can be done using embeddings of tokens, words @xcite @xcite and sentences @xcite . cosine similarity between word embeddings is well correlated with lexical similarity metrics, motivating the use of cosine distance against pre-trained embeddings as an effective training strategy for conmt nearest neighbor beam decoding would in this case include related words and, unlike discrete cross-entropy, the training strategy does not discourage synonyms. previous studies show that the quality of continuous-output models highly depends on the choice of embeddings @xcite @xcite . in general, in conmt the embeddings are pre-trained and fixed: otherwise, making all embeddings equal yields an unwanted global optimum. obtaining pre-trained word embeddings can be computationally expensive, especially if one needs to train an embeddings model from scratch. in this work we randomly initialize target embeddings for continuous-output models and keep them static during training. @xcite applied static random embeddings for text classification model's input; however, to the best of our knowledge, the effect of untrained random target embeddings has not been previously studied in the literature, especially for text-generating tasks such as machine translation. however, we show that random target embeddings perform close to their pre-trained counterpart, and even surpass them on the larger datasets, challenging the assumption that target embeddings must preserve semantic relationships. meaningful structures in target embedding space could help with generalization, but our results suggest that any such benefits are smaller than one might expect, and sensitive to embedding concentration. we hypothesize and bring experimental evidence that conmt performance is negatively impacted when there is too little space around embeddings, i.e., when embeddings are tangled rather than more spread out. our findings on three nmt tasks, namely wmt 2018 @xmath0, wmt 2016 @xmath1, and wmt 2019 @xmath2. strikingly, on the largest dataset (en-de), random embeddings show the largest gain over pre-trained ones. we propose a simple yet efficient combination of random and pre-trained embeddings, and show that it improves model performance in most cases considered. more generally, our findings show that dispersion is an important property of embedding space geometry, and that integrating semantic information should be done with care. ## continuous-output nmt the machine translation task involves learning to map sequences of input tokens 𝒙 = (𝑥 1 , . . . , 𝑥 𝑚 ) to output tokens 𝒚 = (𝑦 1 , . . . , 𝑦 𝑛 ). in standard (discrete) nmt, each step is a multi-class next word prediction task, minimizing: continuous nmt models were first studied by @xcite , who also propose other probabilistic losses and later other marginbased objectives @xcite , with limited gain and at the cost of additional hyperparameters; we therefore focus on the robust cosine objective. we further justify the choice of cosine over maxmargin as an objective function in appendix c.1. on the other hand, the choice of embeddings 𝑬 makes a much larger difference, especially due to the fact that all previous work keeps this parameter frozen: indeed, if it were trainable, equation (2) would have trivial global optima by setting all 𝑬 (𝑡) to the same vector for all 𝑡. with modern transformer architectures, the best performing embeddings overall tend to be the "oracle" output embeddings learned by a pre-trained discrete mt system @xcite . we highlight that the cosine loss is invariant to the norms of both the embeddings and of the decoder hidden state, and therefore we may restrict our modeling problem to the unit sphere. optimizing equation (1) pushes the model 𝒉 away from all tokens different from the "gold" token, even if some other tokens (e.g., synonyms) could otherwise be a good fit. equation ( 2 ) has no such effect, leading to a promise of more diverse generations. an appealing intuition is that synonyms and related words being nearby in embedding space contributes to the performance of conmt and enables such diversity. however in practice, greedy nearest-neighbor lookup is applied, and beam search decoding is not well-studied in the context of conmt. therefore, in this work, we dwell more into the beam search performance for conmt, and compare pre-trained and completely random embeddings. ## random embeddings generation we consider two different distributions from which to sample the |𝑉 | random embeddings. ## experimental setup and data we train conmt systems with pre-trained target embeddings as well as randomly-generated target embeddings. the pre-trained embeddings we use are extracted from a discrete nmt system trained on the same training data, following the setup of @xcite , who found this strategy to outperform other subword-level pre-trained embeddings for conmt. results are reported on three wmt translation tasks: 1 wmt 2016 @xmath3, wmt 2018 @xmath4, the latter including back-translated data. note that for en-tr we use only wmt 2018 training data with 207k training sentences in order to investigae a challenging lower-resource and morphology-rich scenario. data statistics are collected in appendix a. for subword tokenization we used the same sen-tencepiece @xcite ) model for all language pairs, specifically the one used in the mbart multilingual model @xcite . this choice allows for unified preprocessing for all languages we cover. we validate that token-based models performs generally better than word-level models (appendix c.4), even though subwords introduce an additional challenge of predicting subword continuation (appendix c.5). we used the fairseq @xcite ) framework for training our models. baseline discrete models are trained with cross-entropy loss, label smoothing equal to 0.1 and effective batch size 65.5k tokens. both discrete and continuous models are trained with learning rate 5 • 10 -4 , 10k warm-up steps for ro-en and en-de, and 4k for the smaller en-tr dataset. all continuous models are trained with the cosine distance objective in equation (2). we provide all training details in appendix b. we measure translation accuracy using sacre-bleu 2 @xcite and bertscore 3 (zhang et al., 2020). note that bertscore is scaled differently for each language, so the scores cannot be compared across languages. ## additional related work conmt losses. earlier work in conmt suggests loss functions other than cosine, based on modified langevin (a.k.a. von mises-fisher) log-likelihood, or based on max-margin constructions, to perform better @xcite . nevertheless, in preliminary experiments, we find that when using more modern architectures and datasets, such objectives do not outperform the cosine loss. the cosine loss is an instance of langevin log-likelihood with spread 𝜅 = 1 (appendix d), allowing for a theoretically-grounded beam search over sequence likelihood, whereas for max-margin losses it is not clear how to derive a principled beam search. nevertheless, we provide a small set of additional experiments confirming that max-margin losses underperform cosine while showing similar effects in appendix c.1. retrieval-augmented nmt. similarly to conmt, 𝑘-nn mt @xcite @xcite relies on the distance-based retrieval from datastore in decoding time, with cosine similarity and euclidean distance as a popular choice of the similarity measure. even though creation of a datastore and extracting target embeddings are two distinct processes, they both share similar traits and rely on discrete transformer mt system as a source of representations. @xcite argue that quality of 𝑘-nn mt directly depends on the quality of retrieved neighbors contexts from the datastore, and show that 𝑘-nn mt exhibits a related issue with high similarity between unrelated keys. our findings suggests that randomization could provide paths toward improved performance in 𝑘-nn mt. unargmaxability. @xcite point out that standard (discrete) language models can have "unargmaxable" vocabulary items. when using directional modelling (on the unit sphere), unargmaxability is mitigated and only occurs for identical embeddings; however, embeddings that are too close to their neighbors can have very small voronoi sets, leading to the phenomenon we identify in this work, which is problematic in practice for conmt. random perturbations to embeddings might effectively mitigate unargmaxability in discrete models as well. hubness. hubness @xcite @xcite is a phenomenon that impacts nearest-neighbor retrieval ## conclusion our experimental results show that randomly initialized target embeddings can achieve similar performance as pre-trained ones and even surpass them when a sufficiently large amount of data is available. the gap is most pronounced on very rare tokens. we also found that beam @xmath5. we suggest combining random and pre-trained embeddings in attempt to maintain high accuracy on frequent tokens as well as rare tokens. this simple approach proved to be effective for en-tr and ro-en in terms of overall performance. however, more refined ways to combine random embeddings with semantically meaningful anchors may lead to more reliable improvements, and ideally hold the potential to remove the reliance on a pre-trained model entirely. finding the best ways to achieve this potential is an important avenue of future work for conmt and for continuous modeling of language repesentations more broadly.