Sub-Word Alignment is Still Useful: A Vest-Pocket Method for Enhancing Low-Resource Machine Translation (2022)

Abstract: We leverage embedding duplication between aligned sub-words to extend the Parent-Child transfer learning method, so as to improve low-resource machine translation. We conduct experiments on benchmark datasets of My-En, Id-En and Tr-En translation scenarios. The test results show that our method produces substantial improvements, achieving the BLEU scores of 22.5, 28.0 and 18.1 respectively. In addition, the method is computationally efficient which reduces the consumption of training time by 63.8%, reaching the duration of 1.6 hours when training on a Tesla 16GB P100 GPU. All the models and source codes in the experiments will be made publicly available to support reproducible research.

## introduction low-resource machine translation (mt) is challenging due to the scarcity of parallel data and, in some cases, the absence of bilingual dictionaries @xcite @xcite . unsupervised, multilingual and transfer learning have been proven effective in the low-resource mt tasks, grounded on different advantages (section 2). in this paper, we follow aji et al. (2020)'s work to utilize cross-language transfer learning, of which the "parent-child" transfer framework is first proposed by @xcite . in the parent-child scenario, a parent mt model and a child mt model are formed successively, using the same neural network structure. in order to achieve the sufficient warm-up effect from scratch, the parent is trained on high-resource language pairs. further, the child inherits the parent's properties (e.g., inner parameters and embedding layers), and it is boosted by the fine-tuning over low-resource language pairs. one of the distinctive contributions in aji et al. (2020)'s * corresponding author. we attempt to extend aji et al. (2020)'s work by additionally duplicating embedding information among the aligned multilingual sub-words. it is motivated by the assumption that if the duplication between morphologically-identical subwords contributes to cross-language transference, the duplication among any other type of equivalents is beneficial in the same way, such as that of the aligned sub-words, most of which are likely to be morphologically-dissimilar but semanticallysimilar (or even exactly the same). in our experiments, both the parent and child mt models are built with the transformer-based @xcite encoder-decoder architecture (section 3.1). we use the unigram model from sentencepiece @xcite for tokenizing, and carry out sub-word alignment using eflomal (section 3.2). on the basis, we develop a normalized element-wise embedding aggregation method to tackle the many-to-one embedding duplication for aligned sub-words (section 3.3). the experiments show that our method achieves substantial improvements without using data augmentation. ## related work the majority of previous studies can be sorted into 3 aspects in terms of the exploited learning strategies, including unsupervised, multilingual and transfer learning. in this paper, we extend aji et al. (2020)'s work, transferring embedding information not only among the morphologically-identical sub-words but the elaborately-aligned sub-words. ## results and analysis due to unavoidable errors in the sub-word alignment, the utilization of a single aligned sub-word for embedding duplication easily results in performance degradation. aggregating and normalizing embeddings of all possible aligned sub-words help to overcome the problem. figure 1 shows the nmt performance obtained when the i-th top-ranked aligned sub-word is exclusively used for transfer, as well as the aggregation of top-i sub-words is used. it can be found that the latter model almost always outperforms the former model. we compare the training time consumption of all experiments, the result is shown in obviously, the time that mean-pc consumes during training is less than other models. in the scenario of tr-en mt, the training duration is even shortened from 4.49 hours (i.e., about 269 minutes) to 2.14, compared to the baseline model. most probably, it is caused by the transferring of a larger number of sub-word embeddings during training. in other word, mean-pc actually transfers not only morphologically-identical sub-words but the aligned ones. this contributes more to the avoidance of redundant learning over sub-word embeddings. all in all, mean-pc is less time-consuming when producing substantial improvements. ## conclusion we enhance transferable parent-child nmt by duplicating embeddings of aligned sub-words. the experimental results demonstrate that the proposed method yields substantial improvements for all the considered mt scenarios (including my-en, id-en and tr-en). more importantly, we successfully reduce the training duration. the efficiency can be improved with the ratio of about 50% at best. additional survey in the experiments reveals that phonetic symbols can be used for transfer learning between the languages belonging to different families. for example, the phonologies of hamburger in german and burmese are similar (hámburger vs hambhargar). in the future, we will study bilingual embedding transfer of phonologically-similar words, so as to further improve low-resource nmt.