M o CE : Adaptive Mixture of Contextualization Experts for Byte-based Neural Machine Translation (2025)

Abstract: Byte-based machine translation systems have shown significant potential in massively multilingual settings. Unicode encoding, which maps each character to specific byte(s), eliminates the emergence of unknown words, even in new languages, enabling broad language scalability. However, byte-level tokenization results in sequences that are hard to interpret due to limited semantic information per byte. Local contextualization has proven effective in assigning initial semantics to tokens, improving sentence comprehension. Nevertheless, variations in encoding rules across languages necessitate an adaptive approach for effective contextualization. To this end, we propose Adaptive MultiScale-Headed Attention (Ada-MSHA), adaptively selecting and mixing attention heads, which are treated as contextualization experts. This enhances the flexibility of contextualization scales and improves the potential to discover a better strategy than previous methods. Experiment results show that our method outperforms existing methods without extensive manual adjustment of hyper-parameters and surpasses subword-based models with fewer parameters in Ted-59 dataset.

## introduction neural machine translation (nmt) is a consistently hot research topic, and recent years have seen the growing significance of multilingual language modeling @xcite . the selection of tokenization and vocabulary is critical to multilingual language models, which plays an important role in vectorization of texts and discretization of predicted hidden states. while some models @xcite use large vocabularies to ensure word coverage, others @xcite opt for byte fallback strategy. these approaches allow them to completely avoid unknown words with a smaller vocabulary size. byte-based models like @xcite ; @xcite ; @xcite convert all words into utf-8 byte, which further reduces the vocabulary size to about 256. this strategy also reduces the size of the embedding table, saving parameters and accelerating token embedding and inference. besides, it eliminates the unknown-word problem and can be easily generalized to massively multilingual scenarios. empirical study @xcite has also shown the performance superiority of byte-based mnt models. however, the drawbacks of byte-based models are obvious, most notably that an individual byte struggles to convey a specific semantic meaning. therefore, various contextualization methods @xcite have been proposed to alleviate this problem. megabyte @xcite reassembles byte streams into groups of four, constructing group representations by concatenating their hidden states. charformer @xcite and lobef @xcite employ local-contextualization techniques to encode bytes, with charformer using mean-pooling and lobef using convolutional neural networks (cnns). msc @xcite argues that a byte should contribute to multiple neighboring contexts, necessitating a multi-scale contextualization approach. to this end, msc groups hidden state dimensions and assigns cnns with different kernel sizes to each group. although msc provides an effective framework for modeling multi-scale contextualization and achieves state-of-the-art performance, it suffers from the limitation of manually predefined scales. this reduces the model's ability to generalize to multilingual scenarios, particularly in massively multilingual machine translation, which may in-volve over 50 languages. under utf-8 rule, a character may convert to 1 to 4 bytes, depending on the language. this leads to varying requirements of contextualization scale for different languages. however, once msc decides the contextualization scales, they are unchangeable for any input. to address this, we leverage the concept of mixture of experts (moe) @xcite and propose mixture of contextualization experts (moce), which can adaptively determine cnn kernel sizes based on each input text. specifically, we modify multi-head attention to propose adaptive multiscale-headed attention (ada-msha) module. this proposed attention allows each head to be locally contextualized and the contextualization scales are adaptive to the input. instead of predefined scales that msc uses, moce dynamically combines different scales with model needs. the flexibility of contextualization scales is therefore significantly enhanced, resulting in better performance. additionally, we prove that given language id as prior knowledge benefits the scale selection. experiment results on two massively multilingual translation datasets, ted-59 and opus-100, demonstrate our proposed method outperforms other byte-based translation models with similar parameter use. compared with the subword-based model, moce requires fewer parameters while performing better in ted-59 dataset. ## method in this section, we first introduce multiscale-headed attention, a way to model multi-scale contextualization. then we extend it to adaptive multiscale-headed attention, the core module of moce. we also propose how to leverage language id information to further improve our method. ## experiment settings to verify the effectiveness of our proposed approach, we experiment on two massively multilingual translation datasets with over 50 languages. we mainly compare with byte-based models but also try the subword-based transformer model. ## conclusions in this work, we propose an adaptive multiscale-headed attention approach for effective local contextualization. it conducts contextualization on attention heads within multi-head attention. next, we leverage the concept of moe to achieve adaptive selection of contextualization functions. extensive experiments demonstrate the superiority of ada-msha over other byte-based models and disclose why it performs well.