Improving Embedding Transfer for Low-Resource Machine Translation (2023)

Abstract: Low-resource machine translation (LRMT) poses a substantial challenge due to the scarcity of parallel training data. This paper introduces a new method to improve the transfer of the embedding layer from the Parent model to the Child model in LRMT, utilizing trained token embeddings in the Parent model’s high-resource vocabulary. Our approach involves projecting all tokens into a shared semantic space and measuring the semantic similarity between tokens in the low-resource and high-resource languages. These measures are then utilized to initialize token representations in the Child model’s low-resource vocabulary. We evaluated our approach on three benchmark datasets of low-resource language pairs: Myanmar-English, Indonesian-English, and Turkish-English. The experimental results demonstrate that our method outperforms previous methods regarding translation quality. Additionally, our approach is computationally efficient, leading to reduced training time compared to prior works.

## introduction neural machine translation (nmt) systems have revolutionized the field of natural language processing (nlp), offering remarkable performance gains. extensive studies @xcite @xcite have consistently demonstrated that nmt systems trained on substantial parallel corpora yield exceptional results. however, low-resource machine translation (lrmt) remains a significant obstacle in the nlp domain. the need for more training data presents a formidable hurdle in training accurate and robust machine translation systems, particularly for languages with limited resources. unfortunately, many languages fall into this category and require increased availability of parallel corpora for practical machine translation training. as a result, researchers have dedicated their efforts to developing innovative methods to enhance machine translation quality for low-resource languages. the challenge of lrmt has sparked considerable research interest in recent years @xcite @xcite , leading to innovative approaches to tackle the issue. transfer learning, unsupervised learning, and active learning techniques are some of the methods that have been explored, all showing promising results in enhancing translation quality for low-resource languages. in particular, transfer learning has emerged as a highly effective and straightforward approach for the lrmt task. it has significantly improved translation model performance by leveraging pre-trained high-resource language models. in essence, this approach involves transferring knowledge from a high-resource parent model to a lowresource child model, resulting in a remarkable enhancement in the latter's efficacy. overall, transfer learning is a highly efficacious and practical technique that holds immense potential in improving machine translation for low-resource languages. the parent-child transfer learning framework, initially introduced by @xcite , has been a vital breakthrough in improving the lrmt task. several studies have optimized the technique's effectiveness by transferring additional information from the parent model's embedding layer through different means. for instance, @xcite and @xcite proposed using a shared vocabulary, while @xcite suggested a cross-lingual token mapping method. @xcite emphasized the importance of aligning the vocabulary before embedding transfer, which led to notable improvements. recently, @xcite have taken this work a step further by duplicating aligned sub-word embeddings, improving transferable parent-child nmt. these techniques have improved the transfer learning effect and enhanced the lrmt task's performance. this study introduces a new method to enhance the parent-child transfer framework by transferring the embedding layer from the parent to child models. the previous work by @xcite only partially transferred word embeddings from the parent model for words with identical forms. meanwhile, @xcite used both aligned multilingual and morphologically-identical sub-words for embedding transfer, which may lead to inconsistencies. our new approach overcomes the existing limitations in earlier works @xcite and tends to optimize the embedding transfer process. specifically, it involves projecting tokens from parent and child models into a shared semantic space, then computing their semantic similarity measure. this way, each token in the embedding layer of the child model can be represented using the relevant pre-trained embeddings of the related tokens in the parent model, leading to enhanced embedding transfer accuracy. we validated our approach by conducting comprehensive experiments on three benchmark datasets, myanmar-english, indonesian-english, and turkish-english. the results from the experiments showed that our approach not only outperformed the existing state-of-the-art methods but also reduced the training effort, thus proving its effectiveness and efficiency. in short, our contributions revolve around two key points: introducing a new approach to transferring token embeddings from the parent to child model by measuring their semantic similarity within the same semantic space and validating its effectiveness and efficiency through meticulous experiments on benchmark datasets. ## related work transfer learning has been proven effective for nmt under low-resource conditions. @xcite pioneered the transferable parent-child framework, significantly improving bleu scores across various low-resource languages. their method involved training a high-resource language pair as a parent model and using the trained weights to initialize a child model. the child model was then trained on a limited parallel corpus of a low-resource language pair. however, this approach overlooked a significant challenge: the vocabulary mismatch between parent and child models. subsequent research endeavors have tackled this challenge with determination and perseverance. @xcite advocated for using a shared vocabulary between parent and child models, as it has proven advantageous. however, it comes with a catch: the parent model needs prior knowledge of the child's language during training. this can be limiting and may only sometimes be feasible. to overcome this obstacle, @xcite proposed a universal vocabulary strategy for transfer learning. this approach involves simultaneously training sub-word tokens across multiple languages and using romanisation for languages with non-latin scripts. while this method is promising, it may only work for some languages in real-world scenarios. additionally, it could result in overly aggressive and sub-optimal subword segmentation for unseen languages. in another direction, several studies @xcite @xcite have utilized bilingual word embedding alignment as an approach to initialize the embedding layer. @xcite proposed a simple yet effective method that improves word-by-word translation of cross-lingual embeddings using only monolingual corpora without resorting to back-translation. @xcite , on the other hand, utilized careful parameter initialization, denoising effects of language models, and automatic generation of parallel data through iterative back-translation. @xcite demonstrated effective techniques for transferring a pre-trained nmt model to a new, unrelated language that lacks shared vocabularies. their approach involved mitigating vocabulary mismatches through cross-lingual word embeddings, training a more language-agnostic encoder through artificial noise injection, and generating synthetic data from pretraining data without back-translation. recently, @xcite conducted a study to investigate the effects of various strategies for transferring token embeddings between parent and child models. the study found that aligning the vocabulary before transferring the embeddings is essential for practical performance improvements. however, their approach only involved partial token matching, where morphologically-identical tokens were duplicated embeddings while the rest were randomly assigned embeddings. subsequently, @xcite attempted to address this limitation by copying token embeddings among aligned multilingual tokens, enabling the transfer of embeddings for morphologically-identical and elaborately-aligned tokens. however, duplicating embeddings for the same token across different languages may only sometimes be appropriate as it could result in different meanings @xcite . furthermore, using distinct techniques to transfer embeddings for morphologically-similar and morphologicallydissimilar token types may lead to inconsistency. therefore, this paper presents a unified and comprehensive approach to transfer embeddings by projecting all tokens in the same semantic space and considering their relationships. by doing so, we can overcome the existing limitations of previous approaches and ensure consistency in transferring embeddings for morphologically-similar and morphologically-dissimilar token types. ## conclusion this paper introduced a new method to improve embedding transfer for the child model in the lrmt task by leveraging trained token embeddings in the parent model's high-resource vocabulary. by projecting all tokens of the child and parent models into a shared semantic space, it helps easily calculate the semantic similarity measure between tokens, thereby creating high-quality embeddings of the tokens in the child model's low-resource vocabulary with the softmax technique. our approach is then thoroughly evaluated on the three benchmark low-resource datasets: myanmar-english, indonesian-english, and turkish-english. the experimental results indicate that our method yields stable improvements in translation quality on all the datasets. our approach is also computationally efficient, resulting in a reduction in training time consumption compared to baseline models. in future work, we will continue to enhance the embedding transfer technique since it is vital to improving the lrmt task in terms of effectiveness and efficiency.