Leveraging M andarin as a Pivot Language for Low-Resource Machine Translation between C antonese and E nglish (2024)

Abstract: Cantonese, the second most prevalent Chinese dialect after Mandarin, has been relatively overlooked in machine translation (MT) due to a scarcity of bilingual resources. In this paper, we propose to leverage Mandarin, a high-resource language, as a pivot language for translating between Cantonese and English. Our method utilizes transfer learning from pre-trained Bidirectional and Auto-Regressive Transformer (BART) models to initialize auxiliary source-pivot and pivot-target MT models. The parameters of the trained auxiliary models are then used to initialize the source-target model. Based on our experiments, our proposed method outperforms several baseline initialization strategies, naive pivot translation, and two commercial translation systems in both translation directions.

## introduction cantonese is estimated to have 86.6 million native speakers @xcite , primarily spoken in hong kong, macau, guangdong, guangxi, and various overseas chinese communities @xcite . originating from the same language family as mandarin, cantonese shares numerous vocabulary and grammatical similarities with its high-resource counterpart. however, despite these resemblances, the linguistic disparities between cantonese and mandarin are substantial enough to render them mutually unintelligible @xcite . consequently, the feasibility of leveraging mandarin resources to process cantonese text is widely questioned @xcite . despite the popularity of cantonese, there has been limited effort for developing a quality translation system for cantonese. as of the time of writing, google translate has yet to support cantonese. in contrast to mandarin, parallel corpora for cantonese are extremely scarce, presenting significant challenges in training neural machine translation (nmt) models for cantonese. researchers have proposed various strategies to address low-resource nmt. a technique that has been shown to be effective is to involve a pivot language. in pivot-based translation, the source sentence is first translated into a pivot language, which is then translated to the target language @xcite . despite the simplicity, this method has a few disadvantages. namely, it requires two translation models for decoding, equivalently doubling the number of parameters as well as the latency; errors from the source-pivot translation may also propagate into the final prediction. as such, studies have been investigating how to directly train a source-target model with the help of the pivot language, such as using the encoder from the source-pivot model and the decoder from the pivot-target model to initialize the source-target model @xcite . the choice of the pivot language is vital to the translation quality @xcite . prior research typically selected the pivot language based on the relatedness between source and pivot languages, and the availability of bilingual language resources @xcite . for cantonese-english translation, mandarin is an obvious choice, as it is closely related to cantonese and there is an abundance of mandarin-english parallel corpora. however, to the best of our knowledge, no prior studies have examined using mandarin as a pivot language for translating between cantonese and english. in this paper, we aim to bridge the research gap by providing empirical evidence that the usage of mandarin can improve the translation performance between cantonese and english. in particular, we use pre-trained cantonese, mandarin, and english bidirectional and auto-regressive transformer (bart; @xcite ) models to initial-ize source-pivot and pivot-target translation models. the trained source-pivot and pivot-target translation models are then used to initialize the desired source-target translation model. ## background in this section, we highlight some of the linguistic differences between cantonese and mandarin. ## proposed method our proposed method is largely based on the transfer learning approach by @xcite . a limitation of their approach is the requirement for a large amount of source-pivot and pivot-target parallel data to train the auxiliary models. however, given the scarcity of cantonese parallel data, it is challenging to train a robust source-pivot model entirely from scratch. to address this issue, we also transfer parameters from pre-trained bart models to the source-pivot and pivot-target models, leveraging the data efficiency of pre-trained language models. we will illustrate our method in terms of cantonese to english translation, but as our experiments will demonstrate, the method is equally effective in the reverse direction. the core steps of our method are as follows (figure 1 ): ## data in this section, we describe the data used for pretraining and fine-tuning. ## experiment in this section, we outline the baselines chosen for comparison. these baselines are selected to address the following research questions: our baselines include different initialization strategies for the yue-en model. in particular, we choose to initialize the yue-en encoder using the encoder from one of the followings: zh bart, yue bart or yue-zh model, and the yue-en decoder using the decoder from one of the followings: en bart or zh-en model. these encoder and decoder initialization strategies are fully crossed, resulting in six conditions. additionally, we include a baseline where all parameters in the yue-en model are initialized randomly from n (0, 0.02), following the configuration used by @xcite . moreover, we include pivot translation as a baseline, where source sentences are decoded twice: first through the yue-zh model, and then through the zh-en model. the encoders and decoders are yue-zh and zh-en models are initialized from pre-trained bart models, and are trained on yue-zh and zh-en parallel data respectively. finally, we compared our proposed method to two existing translation platforms that support translation between cantonese and english: azure ai translator 16 and baidu fanyi 17 . to examine whether the same approach would work for translation from english into cantonese, we also repeat the experiments for english to cantonese, using mandarin as a pivot. the translation directions in the auxiliary models are adjusted accordingly. for all training and inference, we use one nvidia geforce rtx 3090 gpu with a batch of 64. we use the adamw optimizer ( hutter, 2019) with a constant learning rate of 3e-5. to speed up training, the models are trained in half (16-bit) precision. we use a maximum of 500k training steps for pre-training yue bart, and 10 training epochs for fine-tuning source-pivot, pivottarget and source-target models. we randomly select 5% of the parallel corpora as validation sets. the final models are selected based on validation loss. for the source-target model, we additionally select 10% of the parallel corpora to use as the test set. for decoding, we use beam-search with a beam size of 4. the translation performance is measured by the bleu score @xcite . ## results in this section, we present the bleu score of our experiments. examples of translation results are analyzed in appendix a. for each example, our model's translation is compared to that from azure ai translator and baidu fanyi. ## conclusion in this paper, we experiment pivot-based transfer learning as a way to improve the quality of lowresource cantonese-english translation. our approach involves transferring the parameters of pretrained yue, zh, and en bart models to auxiliary source-pivot and pivot-target nmt models. these auxiliary models are then fine-tuned with parallel data. finally, the parameters of the source-pivot encoder and pivot-target decoder are transferred to the desired source-target model. our results demonstrate significant improvements over randomly initialized models, demonstrating the benefit of transfer learning. moreover, transferring from models that are trained for related tasks (mt in auxiliary models versus text infilling in barts) and languages (cantonese versus mandarin) can further enhance the translation performance. additionally, by training a single source-target model, we reduce the problem of error propagation in naive pivot translation. finally, our model also outperforms two existing commercial translation systems, baidu fanyi and azure ai translator. examples of translation results reveal that our model is better than both understanding and generating cantonese idioms. future work can explore the potential of using mandarin to generate synthetic yue-en parallel data by, for example, translating mandarin sentences in zh-en parallel data into cantonese.