Byte-based Multilingual NMT for Endangered Languages (2022)

Abstract: Multilingual neural machine translation (MNMT) jointly trains a shared model for translation with multiple language pairs. However, traditional subword-based MNMT approaches suffer from out-of-vocabulary (OOV) issues and representation bottleneck, which often degrades translation performance on certain language pairs. While byte tokenization is used to tackle the OOV problems in neural machine translation (NMT), until now its capability has not been validated in MNMT. Additionally, existing work has not studied how byte encoding can benefit endangered language translation to our knowledge. We propose a byte-based multilingual neural machine translation system (BMNMT) to alleviate the representation bottleneck and improve translation performance in endangered languages. Furthermore, we design a random byte mapping method with an ensemble prediction to enhance our model robustness. Experimental results show that our BMNMT consistently and significantly outperforms subword/word-based baselines on twelve language pairs up to +18.5 BLEU points, an 840% relative improvement.

## introduction neural machine translation (nmt) has achieved great success and dominates recent research on translation tasks in both academic and industry studies @xcite @xcite . in particular, multilingual neural machine translation (mnmt) has been shown to benefit low-resource language translation by jointly training mnmt models with high-resource languages @xcite . however, most existing nmt/mnmt models are based on word or subword tokenization, which has three main problems. first, language-specific tokenizers, such as bpe @xcite , may introduce inaccurate segmentations @xcite . second, out-of-vocabulary (oov) words/subwords are still unavoidable and hurt translation performance. third, some languages show a decrease in translation quality with multilingual training due to specific characteristics of the language variety, a problem known as the representation bottleneck @xcite . it essentially limits the improvement of transfer learning from high-resource nmt, like chinese-english @xcite to the low-resource nmt, like aymara-spanish. recently, byte-based nmt models show comparable performance to word/subword-based models @xcite . because modern byte encoding systems such as utf-8 have only 256-byte entries in total, i.e., 0x00 to 0xff, the unified byte tokenization in all languages obviates the traditional preprocessing of language-specific subword tokenization and restricts the vocabulary to a fixed and small one. therefore, the 256-byte-sized utf-8 vocabulary avoids the oov issues. despite the advantages of byte tokenization, the byte encoding has not been investigated in multilingual nmt yet, to the best of our knowledge. in this paper, we show that byte tokenization can be naturally applied to mnmt systems with great advantages. given the unified encoding of byte tokenization in all languages, byte encoding is able to address the representation bottleneck problem in mnmt systems effectively. for example, figure 1 shows the overlaps of subword and byte vocabularies among multiple languages. we notice with the growth of the language number, there is almost no overlap of word vocabularies, while the byte vocabulary still has a large overlap. this large overlap of byte vocabulary enables enhanced knowledge sharing among different languages and can help the model learn more generalized representations for these languages. taking the low resource as a common property for most endangered languages, ## related work multilingual neural machine translation word and subword-level tokenizations are widely used in natural language processing, including nmt/mnmt. morishita et al. @xcite propose to incorporate hierarchical subword features to improve neural machine translation. massively multilingual nmt models are proposed by @xcite and @xcite . they are trained on massive language pairs and show a strong and positive impact on low-resource languages. however, these models tend to have representation bottlenecks @xcite , due to large vocabulary size and large diversity of training languages. two mnmt systems @xcite are proposed to solve this problem by modifying the model architectures, adding special constraints on training, or designing more complicated preprocessing methods. pan et al. @xcite adopt the contrastive learning scheme in many-to-many mnmt. tan et al. @xcite propose a distillation based approach to boost the accuracy of mnmt systems. however, these word/subword-based models still need complex preprocessing steps such as data augmentation or special model architecture design. byte tokenizaiton recently, byte tokenization methods are proposed to address the oov problems in word/subword-based models. ruiz et al. @xcite compare character-based and byte-based nmt systems and show that byte-based systems converge faster. wang et al. @xcite combine subwords tokenization with byte encoding and propose a bytelevel bpe (bbpe). shaham and levy @xcite propose embeddingless byte-to-byte machine translation by replacing the token embedding layer in subword-based models with one-hot encoding for bytes. however, among these models, byte-level mnmt is still not studied, and the randomness of byte tokenization as we discussed above is not investigated. therefore, different from the previous work, we mainly focus on byte-based mnmt, while simultaneously considering the randomness of bytes and endangered languages. ## experimental results in the main paper, we report the bleu scores of all cases. the chrf for endangered languages in cases 3-5 can be found in appendix c. ## conclusion multilingual neural machine translation has been successful in enhancing low-resource languages because of knowledge sharing. to address the representation bottleneck in existing subword-based multilingual translation systems, we propose a bytebased mnmt model, bmnmt with the transformer architecture. to improve the model generalizability and robustness, we further design an ensemble prediction method with random byte encoding. our experimental results show that bm-nmt can alleviate the representation bottleneck and has a stronger generalization ability compared with subword-based mnmt. meanwhile, bmnmt with ensemble prediction improves the transla-tion performance and robustness on endangered language translation tasks. extending our bytebased method to large scale models and datasets is promising and can improve model performance, which will be our future work.