Neural Machine Translation without Embeddings (2021)

Abstract: Many NLP models operate over sequences of subword tokens produced by hand-crafted tokenization rules and heuristic subword induction algorithms. A simple universal alternative is to represent every computerized text as a sequence of bytes via UTF-8, obviating the need for an embedding layer since there are fewer token types (256) than dimensions. Surprisingly, replacing the ubiquitous embedding layer with one-hot representations of each byte does not hurt performance; experiments on byte-to-byte machine translation from English to 10 different languages show a consistent improvement in BLEU, rivaling character-level and even standard subword-level models. A deeper investigation reveals that the combination of embeddingless models with decoder-input dropout amounts to token dropout, which benefits byte-to-byte models in particular.

## introduction neural nlp models often operate on the subword level, which requires language-specific tokenizers @xcite and subword induction algorithms, such as bpe @xcite . instead, working at the byte level by representing each character as a variable number of unicode (utf-8) bytes, does not require any form of preprocessing, allowing the model to read and predict every computerized text using a single vocabulary of 256 types. while previous work found that byte-level models tend to underperform models based on subword tokens @xcite , byte-based models exhibit an interesting property: their vocabulary is smaller than the number of latent dimensions (256 < d). in this work, we demonstrate that this property allows us to remove the input and output embedding layers from byte-to-byte translation models, and in doing so, improve the models' performance consistently. we replace the dense trainable embedding matrix with a fixed one-hot encoding of the vocabulary as the first and last layers of a standard transformer model. machine translation experiments on 10 language pairs show that byte-to-byte models without an embedding layer achieve higher bleu scores than byte-based models with parameterized embeddings (+0.5 on average), thus closing the performance gap with subword and character models. we observe this result consistently throughout a wide variety of target languages and writing systems. the fact that removing parameters improves performance is counter-intuitive, especially given recent trends in machine learning that advocate for increasingly larger networks. we further investigate why embeddingless models yield better results and find implicit token dropout (commonly referred to as "word dropout") as the main source of that boost. while prior work shows that randomly masking tokens from the decoder input can improve the performance of language generation models @xcite , we find that this effect is amplified when operating at the byte level. overall, our results suggest that, even without additional parameters, byte-based models can compete and potentially outperform subword models, but that they may require alternative optimization techniques to achieve that goal. ## byte tokenization modern software typically represents text using unicode strings (utf-8), which allows one to encode virtually any writing system using a variable number of bytes per token; english characters are typically represented by a single byte, with other writing systems taking two (e.g. arabic), three (e.g. chinese), or four (e.g. emojis) bytes per character. by treating each byte as a separate token, we can encode any natural language text using a single uni- original text будь здоров. subwords (bpe) бу@ дь здо@ ров . characters б у д ь з д о р о в . bytes (utf-8) d0 91 d1 83 d0 b4 d1 8c 20 d0 b7 d0 b4 d0 be d1 80 d0 be d0 b2 2e versal vocabulary of only 256 token types. moreover, byte tokenization obviates the need for any heuristic preprocessing, such as splitting spaces, punctuation, and contractions. figure 1 illustrates subword, character, and byte tokenization. our model is based on the original transformer encoder-decoder @xcite with one main difference: we eliminate the input and output token embedding layers. these layers typically use a common parameter matrix @xmath0. 2 instead, we use a fixed one-hot representation of our byte vocabulary. for instance, the character "r" could be represented as a vector with 1 at dimension 82 and 0 elsewhere. since it is standard practice to use representations of more than 256 dimensions, every possible byte can be represented by such one-hot vectors. to predict the next token for a decoder input of n tokens, we take the output of the last transformer decoder layer, @xmath1, and apply a softmax across each vector's dimensions. formal expressions of the input and output of our model are detailed in figure 2 . omitting the embedding layer reduces the number of parameters by a factor of o(|v | • d). 3 we do add a total of 3 parameters to scale the encoder and decoder's (one-hot) inputs and the decoder's output (before the softmax). we initialize all three with √ d, akin to the constant scaling factor typically applied to the input embedding layer in transformers. despite the reduction in model size, memory figure 2: the main differences between the original encoder-decoder model and the new embeddingless model. @xmath2; p n are the positional embeddings up to length n. in addition to replacing the embedding layers, we also remove the dropout layers on the encoder input and decoder output, since zeroing out entries of one-hot vectors is equivalent to randomly masking out input tokens or deleting significant parts of the model's predicted distribution. the dropout on the decoder input (prefix of the target fed with teacher forcing) remains intact at this point and is applied throughout our main experiments. further analysis shows that decoder input dropout is in fact a significant source of performance gains, which we further investigate in section 6. ## experiments we train byte-tokenized embeddingless models for machine translation and compare them to standard byte, character, and subword-based models on a diverse set of languages. we adopt a standard experimental setup that was designed and tuned for the subword baseline and limits our hyperparameter tuning to dropout probabilities. ## results are embeddings essential? the results show that it is indeed possible to train embeddingless machine translation models that perform competitively. the performance gaps between models with different tokenization schemes are relatively small. except for vietnamese, the difference between the embeddingless model and the best embeddingbased model is always under 1 bleu. in the most controlled setting, where we compare byte-based models with and without learnable embeddings, models without embeddings consistently achieve higher bleu scores in 19 of 20 cases (and an equal score for ru-en), with a boost of about 0.5 bleu on average. when compared to models based on character embeddings, the embeddingless byte-to-byte approach yields higher bleu scores in 17 out of 20 cases, though the average difference is quite small in practice (0.3 bleu). is subword tokenization superior to bytes or characters? previous work in machine translation shows that subword models consistently outperform character or byte-based models @xcite @xcite . however, our results indicate that this is not necessarily the case. when translating from english to a foreign language, the original direction of the iwslt dataset, embeddingless byte-to-byte models achieve performance that is equal or better than subword embedding models' in 8 out of 10 cases. we observe a different trend when translating into english, where subword models surpass other models for every source language; the fact that moses is a particularly good tokenizer for english -and less so for other languages -is perhaps related to this phenomenon. whereas prior work proposed closing the performance gap by adding layers to the basic architecture, under the assumption that character-based models lack capacity or expressiveness, our results show that actually removing a component from the model can improve performance under certain conditions. it is possible that character and byte-based transformer models encounter an optimization issue rather than one of capacity or expressivity. ## analysis why does removing the embedding matrix improve the performance of byte-based models? as mentioned in section 3, the embeddingless models do not use dropout on the encoder input and decoder output, but do apply dropout on the decoder input while training. since the embeddingless decoder's inputs are fixed one-hot vectors, using dropout implicitly drops out complete tokens. in prior work, token dropout ("word dropout") has been shown to have a consistently positive effect @xcite . we, therefore, rerun our experiments while embedding-based models embed-less subword char byte byte @xmath3.33 +0.53 +0.42 +0.62 @xmath4.69 +0.67 +0.92 +0.83 given the benefits of token dropout on the baseline models, we re-evaluate the results from section 5, while allowing for token dropout as a potential hyperparameter. ## related work there is prior work on replacing language-specific tokenizers with more universal tokenization approaches. @xcite shows how character n-gram embeddings can be effectively trained by segmenting text using a stochastic process. sentencepiece @xcite tokenizes raw unicode strings into subwords using bpe @xcite or unigram lm @xcite . byte bpe @xcite extends senten-cepiece to operate at the byte level. while this approach is indeed more language-agnostic than heuristic tokenizers, it does suffer from performance degradation when no pre-tokenization (e.g. splitting by whitespace) is applied. 6 moreover, the assumption that subword units must be contiguous segments does not hold for languages with non-concatenative morphology such as arabic and hebrew. character and byte-based language models @xcite treat the raw text as a sequence of tokens (characters or bytes) and do not require any form of preprocessing or word tokenization, and @xcite even demonstrate that byte-based language models can perform comparably to word-based language models on the billion-word benchmark @xcite . although earlier results on lstm-based machine translation models show that character tokenization can outperform subword tokenization @xcite , recent literature shows that the same does not hold for transformers @xcite @xcite . to narrow the gap, recent work suggests using deeper models @xcite or specialized architectures @xcite . our work deviates from this trend by removing layers to improve the model. this observation contests the leading hypothesis in existing literature -that the performance gap results from reduced model capacity -and suggests that the problem may be one of optimization. ## conclusions this work challenges two key assumptions in neural machine translation models: the necessity of embedding layers, and the superiority of subword tokenization. experiments on 10 different languages show that, despite their ubiquitous usage, competitive models can be trained without any embeddings by treating text as a sequence of bytes. our investigation suggests that different tokenization methods may require revisiting the standard optimization techniques used with transformers, which are primarily geared towards sequences of english subwords.