Integrating Multi-scale Contextualized Information for Byte-based Neural Machine Translation (2024)

Abstract: Subword tokenization is a common method for vocabulary building in Neural Machine Translation (NMT) models. However, increasingly complex tasks have revealed its disadvantages. First, a vocabulary cannot be modified once it is learned, making it hard to adapt to new words. Second, in multilingual translation, the imbalance in data volumes across different languages spreads to the vocabulary, exacerbating translations involving low-resource languages. While byte-based tokenization addresses these issues, byte-based models struggle with the low information density inherent in UTF-8 byte sequences. Previous works enhance token semantics through local contextualization but fail to select an appropriate contextualizing scope based on the input. Consequently, we propose the Multi-Scale Contextualization (MSC) method, which learns contextualized information of varying scales across different hidden state dimensions. It then leverages the attention module to dynamically integrate the multi-scale contextualized information. Experiments show that MSC significantly outperforms subword-based and other byte-based methods in both multilingual and out-of-domain scenarios. Code can be found in https://github.com/ictnlp/Multiscale-Contextualization.

## introduction in neural machine translation (nmt) systems, subword tokenization has been the most common and effective method to mitigate the out-of-vocabulary (oov) problem. however, both bpe @xcite and sentencepiece @xcite ) fix the word segmentation rule or vocabulary once they have learned them on the initial corpus, making it difficult to ensure adaptation to new corpora. this is worsened in out-of-domain scenarios. additionally, in multilingual scenarios with data imbalance, subword vocabularies tend to focus on high-resource languages, overlooking low-resource ones. this imbalance can cause an increase in oov cases or over-segmentation of texts, harmful to translation model performance. byte-based method is able to solve these problems with few embedding parameters and has aroused extensive researches @xcite @xcite @xcite . in byte-based models, text is converted into byte sequences according to utf-8 encoding, with each byte as a token within the vocabulary. they generally use a vocabulary with a maximum size of 256 but can adapt to imbalanced scenarios like multilingual translation and out-of-domain adaptation. however, a feature of utf-8 encoding hinders conventional transformer model @xcite from adapting well to byte-based vocabulary: a single character may correspond to 1 to 4 utf-8 bytes. the number is 1 for english characters, but arabic and many asian languages require multiple bytes to represent a single character. therefore, sometimes a single byte does not have a determined meaning; it requires the integration of local information to encode its semantics. to address that, various methods have been proposed for integrating local contextual information. su4mt @xcite learns contextual information with an attentive semantic fusion layer, but requires accurate segmentation. megabyte @xcite segments a sentence into blocks of 4 and simply concatenates the tokens. charformer @xcite segments a sentence 4 times with block sizes ranging from 1 to 4 each, and employs mean-pooling to perform local integration. the weighted-sum of 4 results yields the final output. lobef @xcite proposed byte-ncf, replacing mean-pooling with convolutional neural networks (cnns) for better performance. then, n contextualizing functions g(•) are applied to these parts respectively. a simple and effective structure for local contextualization is the 1-d convolution neural network (cnn). therefore, we leverage cnns with different kernel size k to control the contextualization scope. since different dimensions are contextualized with various granularities, our model can realize multi-scale contextualization. to preserve the original information, g(•) is also allowed to be a "passthrough" function, which directly returns the input without any operations. in equation ( 1 ), g i (•, k) means the contextualization function for group i, and k is the kernel size. here, @xmath0. next, the contextualized vector parts xi are calculated by g i (x i , k). finally, they are concatenated to form x, which acts as the input of the mha module. preliminary experiments on cnn configurations guided the best structure for the whole model. first, padding on the left side deteriorates model performance, so the cnns are set to pad on both sides, as shown in figure 1 . second, applying msc to transformer decoder layers causes a discrepancy between training and testing, when a token's right side tokens are not yet generated. as a result, msc is only applied to encoder layers. it is worth noting that the kernel size k is recommended to be an odd number or zero, otherwise, manual zero-padding is required to keep the output length the same as the input. empirically, it is 2 9795 better to choose k from {0, 1, 3, 5, 7}. ## experiments we experiment with two multilingual datasets and a domain-adaptation dataset to investigate the performances and properties of the msc approach and other byte-based language models. ## conclusions in this paper, we make two primary contributions. firstly, we show when byte-based models outperforms subword-based models and when they don't. in massively multilingual translation scenarios which involves a wide range of languages, byte-based models exhibit clear superiority, particularly for low-resource languages. in limited language numbers with sufficient training data, bytebased models lags behind subword-based models. secondly, we introduce a multi-scale contextualization (msc) method which enhances the adaptability of byte-based models to diverse inputs. by dynamically integrating multi-scale contextualized information, msc outperforms other byte-based models in generally all languages.