Low-Resource Neural Machine Translation: A Case Study of C antonese (2022)

Abstract: The development of Natural Language Processing (NLP) applications for Cantonese, a language with over 85 million speakers, is lagging compared to other languages with a similar number of speakers. In this paper, we present, to our best knowledge, the first benchmark of multiple neural machine translation (NMT) systems from Mandarin Chinese to Cantonese. Additionally, we performed parallel sentence mining (PSM) as data augmentation for the extremely low resource language pair and increased the number of sentence pairs from 1,002 to 35,877. Results show that with PSM, the best performing model (BPE-level bidirectional LSTM) scored 11.98 BLEU better than the vanilla baseline and 9.93 BLEU higher than our strong baseline. Our unsupervised NMT (UNMT) results also refuted previous assumption n (Rubino et al., 2020) that the poor performance was related to the lack of linguistic similarities between the target and source languages, particularly in the case of Cantonese and Mandarin. In the process of building the NMT system, we also created the first large-scale parallel training and evaluation datasets of the language pair. Codes and datasets are publicly available at https://github.com/evelynkyl/yue_nmt .

## introduction there are over 85 million cantonese speakers around the globe, and it is the de facto spoken language in hong kong, macau, and the canton region in china @xcite . the language is also deemed the most influential and well-known variety of chinese languages after mandarin @xcite ); nevertheless, cantonese has rather limited linguistic resources. while there are varying sizes of cantonese-english corpora, such as hong kong hansards (legislative council of the hong kong special administrative region, 2022) and hong kong laws parallel text @xcite , the latter of which contains nearly 3 million parallel sentences between the two languages, the same cannot be said for the pair of cantonese and mandarin. english and cantonese share very few linguistic features, and are considered distant languages. on the contrary, cantonese and mandarin are typologically similar in that they share more linguistic features such as grammatical structures and basic lexical items than cantonese does with english @xcite . as such, our work aims to take advantage of the typological similarities between the two languages and investigate whether the similarities would enable decent translation quality despite having a limited amount of training data. the existing cantonese (hong kong variant) -mandarin corpora are quite small and mostly in the domain of conversational transcripts and social media (luke and wong, @xcite . this can be further demonstrated by the dependency treebank built by @xcite , which consists of only 13,918 words/tokens, as compared to 285,000 in mandarin in universal dependencies (ud; @xcite . most state-of-theart (sota) deep learning algorithms require a large amount of data to perform well. it holds true especially for more complex tasks, such as machine translation (mt), question answering, and neural text generation @xcite . as a consequence, most of these complex tasks are not commonly applied to cantonese. language, however, is the core of one's cultural identity @xcite . in light of that, the main goal of this paper is to benchmark different mandarin to cantonese nmt approaches to pave the way for future research on cantonese nmt systems. the contributions of the paper include providing the first baseline of cantonese nmt and the first large training and evaluation parallel dataset of the language pair. our hypothesis is that creating an mt system with a high-resource, typolog-ically close language might produce decent translation outputs. if that is the case, the limited resources of cantonese can be improved by utilizing the mt system, hence enabling implementations of nlp systems with better performance for the low-resource language. ## linguistic considerations of cantonese and mandarin most chinese texts encountered in nlp is in mandarin, largely due to its high availability in linguistic resources. nearly all cantonese speakers can read and write in written mandarin, and it is conventionally preferred in academic and legal settings to write in written mandarin to convey a sense of formality @xcite . as a consequence, there is very little cantonese text data available, which, in turn, makes cantonese seldom included in a majority of the nlp research and systems @xcite . nevertheless, cantonese and mandarin are typologically similar languages, where they have a similar grammatical system and share basic lexical items such as times, numbers, and personal pronouns that are identical in orthography @xcite . even though the two languages are closely related, there are, indeed, a plethora of linguistic differences. the most notable one is the phonological systems, in which their similarities are minimal in terms of sound inventory and intonation @xcite . on the aspects of syntactic structure, the main difference between the two languages is their word orders, where cantonese allows a more flexible word ordering compared to @xcite . furthermore, there are distinctive grammatical features in cantonese that do not exist in mandarin @xcite , including, but not limited to, post-verbal elements, structural particles, directional verbs, definiteness, and aspect markers. in terms of lexical dissimilarity, there are seven to eight thousand distinct words and expressions in cantonese that are written in a different character from any mandarin words, or that carry a different meaning from the mandarin words of similar forms @xcite . these distinct words attribute almost one third of the total vocabulary in cantonese, and half of them are commonly used in daily conversation among cantonese speakers. take a parallel sentence pair from the ud data as an example to illustrate the differences and simi-larities between the two languages. sentence 1 denotes its expression in cantonese, while sentence 2 refers to the sentence in mandarin. (1) 嗰時啲 that time's cd舖 cd shops 仲多過 even more than 而家啲 now's 七十一。 7-11. "there were more cd shops at that time than the 7-11 (convenience stores) we have now." (2) 那時候 at that time 唱片店 cd shops 比現在 compared to now 七十一 7-11 還要多。 even more. as can be observed, the lexical tokens between the two sentences are quite different, with only four words (in bold) in overlap and three of them being a numerical item and one being a timing word. on the contrary, the syntactic structures are roughly similar, with word order differences such as the placement of time, subject, and comparison expression. the distinctive lexical, syntactic, and phonological differences result in the language pair being mutually unintelligible @xcite . consequently, transforming from mandarin to cantonese should be treated as a translation task. ## methodology and experimental setup we implemented the following nmt systems on the direction of mandarin to cantonese in the experiment: 1. word-based bidirectional lstm model with general attention mechanism as the baseline (bilst m ), 2. word-based (1) + fine-tuning as a strong baseline (bilst m t ), 3. word-based (2) + psm (bilst m t + psm), 4. bpe-based fine-tuned bilstm + psm (bilst m bpe+t + psm), 5. word-based transformer (t rans w + psm), 6. bpe-based transformer (t rans bpe + psm), 7. unsupervised nmt via language model pretraining and transfer learning with adapters (relm adap + psm) ## results the unmt system with pre-trained lm scored 1.85 on sacrebleu (+0.61 points compared to the vanilla baseline) and is the second-worst performing model in the experiment. the strong baseline (model 2, fine-tuned vanilla nmt) outperforms it by 1.44 bleu even with only 1k sentence pairs. ## analysis effect of corpus size bitext mining improved the model performance substantially (+9.08 bleu, bilst m t + psm compared to bilst m t ) with merely some minor changes in the training parameters in view of the increased data size. it shows that this technique is successful in assisting model learning and thus improving its performance by increasing the size of the training data. ## limitations translation systems are prone to making generalizations based on the frequency of gender-role, race, religion, and other stereotypes occurrences in the datasets. one typical example is "man is to programmer as woman is to homemaker" @xcite . the cantonese-mandarin ud parallel treebank used in this study was sourced from a television show, which might contain stereotypes in the dialogues. besides, the bitext mined sentence pairs were sourced from the wikipedia sites of cantonese and mandarin. given that wikipedia is an open-source community where everyone can contribute, its content could be vulnerable to social injustice and stereotypes as well. their presence in the training data, if any, would reinforce the stereotypes in the translation system. one way to mitigate such potential issues is by treating it as a domain adaptation problem, as recommended by saunders and byrne (2020). in terms of evaluation, the main automated metric in this study is sacrebleu. using only one metric, however, is not able to provide a full picture of the model performance and its translation quality. although we used manual analysis along with sacrebleu, having a non-matching based metric such as bertscore @xcite or sim-ile @xcite would be helpful in evaluating the contextual similarity between the input and the translation. ## conclusion and future work in this paper, we presented the first benchmark of various nmt approaches for cantonese. due to the minimal amount of training data, the baseline models failed to produce intelligible results. we alleviated this issue by using parallel sentence mining as data augmentation and have increased the training data size from ∼1k to ∼36k. it resulted in a tremendous boost in performance (+9.08 bleu) and produced higher-quality translations. additionally, we provided a large parallel training and evaluation dataset of cantonese and mandarin for future research. one of the interesting findings in this paper is that our transformer mt systems performed worse than the bilstm systems in terms of sacrebleu. this is reasonable given the large amount of data required by transformer-based models and the limited amount of training data. what is more intriguing is that using varied word representations in an nmt system leads to very different results. we found that bpe-level models generally perform better. the bpe-level transformer model produces more comprehensible translations despite having a lower bleu score than the two bilstm models. we hypothesize that this is because of the evaluation metric's (bleu) architecture favoring models with a copying tendency. besides the supervised models, we also implemented an unsupervised nmt with lm pre-training. it is, however, among the worst-performing models, in spite of the large amount of training data in comparison with the rest of the models. future work can be dedicated to different approaches to improve the performance of mandarin-cantonese nmt systems. while this study has investigated the direction of mandarin to cantonese as a way to alleviate the lower resource in cantonese, our next step would include both translation directions as well. in addition, one could explore various approaches to mitigate the severe oov issue, such as applying jyutping romanization of the characters @xcite or using bbpe @xcite or unigram language modeling @xcite rather than bpe as the subword tokenization technique. danielle saunders and bill byrne. 2020. reducing gender bias in neural machine translation as a domain adaptation problem. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 7724-7736, online. association for computational linguistics.