Local Byte Fusion for Neural Machine Translation (2023)

Abstract: Subword tokenization schemes are the dominant technique used in current NLP models. However, such schemes can be rigid and tokenizers built on one corpus may not adapt well to other parallel corpora. It has also been observed that in multilingual corpora, subword tokenization schemes oversegment low-resource languages, leading to a drop in translation performance. An alternative to subword tokenizers is byte-based tokenization, i.e., tokenization into byte sequences using the UTF-8 encoding scheme. Byte tokens often represent inputs at a sub-character granularity, i.e., one character can be represented by a span of byte tokens. This results in much longer byte sequences that are hard to interpret without aggregating local information from multiple byte tokens. In this paper, we propose a Local Byte Fusion (LOBEF) method for byte-based machine translation—utilizing byte n-gram and word boundaries—to aggregate local semantic information. Extensive experiments on multilingual translation, zero-shot cross-lingual transfer, and domain adaptation reveal a consistent improvement over vanilla byte-based models. Further analysis also indicates that our byte-based models are parameter-efficient and perform competitive to subword models.

## introduction multilingual neural machine translation (nmt) has proven effective to transfer knowledge learned from a high-resource language to a low-resource language. however, existing multilingual nmt models still rely on a pre-built subword tokenizer (e.g., bpe @xcite , senten-cepiece @xcite ) to tokenize a sentence into a sequence of subword units. this has two drawbacks. first, once the tokenizer is fixed, we lose the flexibility of changing the word tokenization if we aim to fine-tune the nmt model on another parallel corpus of interest for adaptation. second, when a subword tokenizer is built on unbalanced multilingual data, word tokens from a lowresource language are usually under-represented, resulting in over-segmentation of a word into many single characters. a recent study @xcite measures over-segmentation by the fertility score of a subword scheme which indicates how many subwords a whole word is broken down into. it shows a negative correlation between the fertility score and the performance of multilingual models on the over-segmented languages. although character-based models have often been proposed as a solution to these problems @xcite @xcite , they come with their own tradeoffs related to the significant overhead of processing long character sequences during training and inference @xcite . besides, these models still adopt a fixed vocabulary of characters, leading to the same issue as a fixed subword tokenizer for adaptation. another point of difference is that in character-based methods, the vocabulary still consists of one unique embedding for each character under consideration. in byte-based approaches, the tokens are at a sub-character granularity and the model has to figure out how to combine bytes for different languages. recently, there has been renewed interest in character-based models that adopt a byte tokenization scheme @xcite @xcite -tokenization of texts into utf-8 byte tokens. although these byte-based models have shown competitive performance to subword models on multilingual nlu benchmarks @xcite , their performance on multilingual generation, especially on multilingual nmt is still underexplored. although @xcite recently demonstrate the effectiveness of byte tokenization for bilingual machine translation, a comprehensive study of such bytebased methods on the multilingual paradigm across a wide variety of languages and domains is still to remedy these issues, we propose two variants of local byte fusion (lobef) 1 techniques that explicitly aggregate byte tokens to learn character-/word-level representations for byte-based nmt models. our first variant utilizes four n-gram convolutional layers to aggregate bytes for learning character-level information, and our second variant utilizes word boundaries to aggregate a span of bytes for learning word-level context. we conduct extensive experiments to compare our methods with the vanilla byte-based model and the embeddingless model from @xcite in a multilingual translation setting. our many-toone translation results show that aggregating local information in earlier layers encourages the model to capture local information for seven source languages, yielding an average gain of up to 1.4 bleu over the vanilla byte-based nmt model while performing competitively with subword models. we further demonstrate the effectiveness of lobef on the zero-shot/few-shot cross-lingual transfer and cross-domain adaptation settings, showing the flexibility of byte-based nmt models over subword baselines when fine-tuning is required for data adaptation. additionally, our method also improves over vanilla byte-based nmt models for adaptation. our contributions are as follows: ## local byte fusion for languages that do not exclusively use the english character set, encoding them often requires more than one byte. vanilla byte-based models can only implicitly aggregate character-level or wordlevel representations for these languages, potentially resulting in poor interpretability and suboptimal results in multilingual settings. hence, we propose two fusion techniques that encourage models to explicitly aggregate character-level and word-level information from byte sequences. we also adopt byte sequences as inputs and outputs for our model, and use vanilla transformer as the backbone. as we focus on multilingual encoding in this work, we only modify the encoder, and adopt the same decoder architecture from (shaham and levy, 2021). note that a more sophisticated design of the decoder will also involve a special design of decoding algorithms @xcite which goes beyond the scope of this work. besides, to conduct a more comprehensive study, we also consider the case where we retain embedding layers for the encoder and decoder of the byte-based model. this implies that instead of one-hot representations for the byte sequences, we can learn dense vector representations. since the vocabulary size of all byte tokens is 256, this does not amount to adding a significant number of extra parameters. ## results in this section, we detail the results of the various experiments and discuss their implications. ## discussion and analysis next, we further present a qualitative analysis of byte-based models and our proposed variants. we use the compare-mt toolkit @xcite to holistically analyze how the outputs of these models differ and what aspects different models excel at. we compare the multilingual nmt models ( §5.1) on the german-english translation as a sample language pair for the analysis. specifically, we group all source words in the test sentences into buckets by the word fertility score (figure 4a ) and the word length in terms of characters (figure 4b ). recall that the word fertility score measures how many subword units a word is broken into, and we use the bpe tokenizer used in section 5.1. we evaluate byte-based models (i.e., byte, byte-wsf, byte-ncf) using one-hot representations on each group in terms of source word translation accuracy. as german is a high-resource language, most german words (45%) have a fertility score of 1 implying that they are not segmented by the bpe tokenizer, and all byte-based methods perform comparable on these words. we find that the byte-ncf method performs better than the other two byte-based methods on oversegmented words (as indicated by the accuracy on words with fertility scores above 4). we also find that the byte-ncf method outperforms other methods on translating long words (depicted by the accuracy on words with length greater than 15 characters). comparing to the word-based (byte-wsf) or sentence-level full self-attention (byte), we hypothesize that this is a result of encoding a smaller sequence length when using the convolution fusion operation, reducing the pressure of byte-based models to capture too much information from a span of byte tokens. character-level models applying neural models directly on character sequences has been extensively studied @xcite @xcite @xcite @xcite @xcite . characteraware methods were mainly developed by the use of word boundaries and convolutions over characters @xcite @xcite @xcite . however, for machine translation, character-based nmt models @xcite @xcite still suffer from a high computational overhead of encoding and decoding much longer sequences. tokenization-free methods recent attempts have focused on using unicode or utf-8 encoding scheme to remove pre-built subword tokenizers from preprocessing. these byte-based methods have achieved promising results in terms of accuracy and speed-up in several multilingual language understanding tasks @xcite @xcite or bilingual translation tasks @xcite , while their application to multilingual or cross-lingual/domain settings is still underexplored. ## conclusion we propose a local byte fusion (lobef) method with two variants-one employing convolutions on byte n-grams and the other utilizing word-based self-attention restricted to word boundaries. we show that these two fusion variants improve upon vanilla byte-based models indicating that neural machine translation models benefit from the explicit aggregation of local semantic information for characters or words at lower layers of neural networks. our experiments show that both one-hot and dense versions of byte-based models perform competitively on multilingual machine translation and even beat subword baselines on multiple language pairs. we also conduct an investigation of the effectiveness of byte-based techniques in both zero-/few-shot cross-lingual transfer and domain adaptation settings, and find that they outperform subword models by a large margin.