Exploring Methods for Building Dialects- M andarin Code-Mixing Corpora: A Case Study in T aiwanese Hokkien (2022)

Abstract: In natural language processing (NLP), code-mixing (CM) is a challenging task, especially when the mixed languages include dialects. In Southeast Asian countries such as Singapore, Indonesia, and Malaysia, Hokkien-Mandarin is the most widespread code-mixed language pair among Chinese immigrants, and it is also common in Taiwan. However, dialects such as Hokkien often have a scarcity of resources and the lack of an official writing system, limiting the development of dialect CM research. In this paper, we propose a method to construct a Hokkien-Mandarin CM dataset to mitigate the limitation, overcome the morphological issue under the Sino-Tibetan language family, and offer an efficient Hokkien word segmentation method through a linguistics-based toolkit. Furthermore, we use our proposed dataset and employ transfer learning to train the XLM (cross-lingual language model) for translation tasks. To fit the code-mixing scenario, we adapt XLM slightly. We found that by using linguistic knowledge, rules, and language tags, the model produces good results on CM data translation while maintaining monolingual translation quality.

## introduction code-switching or code-mixing (cm), which stands for using more than one language in one conversation or sentence, often occurs in multilingual societies. because of the rapid development of social media, cm has become more prevalent in the past decade, making it be a new challenge in natural language processing (nlp). although mandarin is the dominant language in taiwan, taiwanese hokkien has nearly as many speakers as mandarin @xcite . taiwanese tend to mix dialects and mandarin in daily communication, creating code-mixed languages such as taiwanese hokkien-mandarin or hakka-mandarin. compared with hokkien, taiwanese hokkien integrates japanese phrases and culture due to historical factors, and gradually evolved into a different dialect. unlike other cm languages based on recognized writing systems, such as spanish-english, hindi-english, or bahasa rojak, there were no official writing systems for the dialects in taiwan until the government built one in the 21st century. therefore, there is nearly no corpus for hokkien-mandarin or other taiwanese code-mixed languages. under this circumstance, code-mixing-based nlp tasks (cm tasks) are even more difficult to address in comparison with other monolingual nlp tasks. the lack of resources @xcite , originating from having no formal writing system for taiwanese hokkien in the past, makes it hard to make breakthroughs in dialect-related cm tasks. researchers are often stuck because of lacking corpus to develop deep learning models. furthermore, not having a writing system increases the possibility of a language's vanishing @xcite . therefore, creating a code-mixed corpus is vital for stepping into the cm-related nlp realm and it also helps to protect the dialects. after creating a codemixed corpus, more nlp tasks such as machine translation can be developed. pre-trained language models have achieved outstanding performance in many nlp tasks. language models @xcite @xcite have become the mainstream and needful portion in most nlp areas. however, pre-trained language models rely on large-scale corpora, which is a challenge for low-resource languages @xcite . transfer learning is a possible solution because it uses the knowledge from high-resource tasks to improve performance on the related task. it can reduce the amount of required training data and widely improve the effectiveness when solving low-resource problems, especially in translation task @xcite @xcite . some research further focused on machine translation, and showed that multilingual models can generalize monolingual inputs to code-switching sentences @xcite without being specifically trained to learn the representations of cm languages. in this paper, we take hokkien-mandarin, a code-mixed language, as the research target because of its large population of speakers in taiwan. due to the shortage of hokkien-mandarin corpus, we have done several tasks to overcome the low-resource challenge. first, we proposed a method for hokkien word segmentation via a mandarin tokenizer toolkit. the method would not be affected by the language morphology and can maintain the syntax structure of hokkien. it can be seen as the first linguistics-based solution without training a language-specific word segmentation model. then, we used the hokkien word segmentation tool to synthesize a hokkien-mandarin code-mixed corpus for further use. after that, we proposed a hokkien-mandarin cross-lingual language model and achieved good performance on hokkien-mandarin cm translation and maintained the monolingual translation result at the same time. our main contributions are as follows: (1) we proposed a method of implementing hokkien word segmentation. (2) we presented a parallel corpus of 76,013 hokkien-mandarin cm sentences and 75,150 non-parallel cm data. (3) we built a hokkien-mandarin cm translation model through the cross-lingual model. there are two methods to represent the writing system of taiwanese hokkien, logograms, and phonography. a logogram is a written character that represents a word or morpheme. in contrast, phonography is an orthography in which the graphemes correspond to the phonemes of the language. the only logogram writing system is written taiwanese hokkien (wth), which is entirely made of mandarin characters (hàn-jī). wth uses the morpheme and meaning of conventional mandarin characters instead of their phone to create characters for taiwanese hokkien. wth has an official standard for the writing system and is now taught in schools in taiwan. on the other hand, there are various phonography writing systems such as poj (peh-ōe-jī), tai-lo and han-romanization mixed script (han-lo). ## hokkien-mandarin cm dataset hokkien is one of the most popular dialects in taiwan @xcite , and switching between hokkien and mandarin is very common. however, to the best of our knowledge, there is no code-mixed dataset for hokkien and mandarin. establishing a cm dataset is a challenge we have to overcome. in our study, we collect two types of data: monolingual data in mandarin and hokkien separately, and parallel data in hokkien-mandarin. for the mandarin corpus, we collect the latest mandarin corpus from wikipedia. we also gather taiwan news as a corpus from 2018 to 2019. we collect 2.2 gb of data for training a mandarin language model. for hokkien and mandarin parallel corpus, we used icorpus foot_1 and the example sentences from moe's dictionary of frequently-used taiwan ## hokkien language model and translation system once we synthesize the code-mixing corpus, we can intuitively apply the data to various nlp tasks. in this study, we chose to apply our cm dataset to the translation task. the main advantage of developing a translation model is that we can translate cm sentences to monolingual sentences and use them as input data for other nlp applications without retraining cm-based nlp models. previous research @xcite has pointed out that multilingual models can generalize well on cm data. we believe that since bilingual people can use knowledge of one language to aid the learning of the other, they can identify the language to which the vocabulary belongs in a code-mixed sentence. also, understand the meaning of the sentence without learning the sentence structure and grammar of the code-mixed language. considering the size of our corpus, we use xlm as our multilingual model architecture. following xlm, we build a vocabulary dictionary at the character level, including all mandarin and hokkien characters, roman numerals, and english letters, for a total of 26,780 characters. same as our cm dataset, we distinguish mandarin and hokkien characters by appending the _@ symbol when building the vocabulary set. we also keep the same special token as xlm. at last, we applied two methods to xlm to verify our hypothesis, dynamic language identification (dli) mechanism and transfer learning. ## experiment for the dataset, taking the diversity in cm habits into consideration, we sample some data from the hokkien corpus and synthesize them into the cm data randomly. we shuffle all the sentences and split them into training, validation, and test sets in a ratio of 8:1:1 in each parallel corpus. moreover, a few parts of our hokkien corpus which can be found corresponding to mandarin translation were reserved for testing. there are 823 sentences in total, and we use them to synthesize the cm preserved assessment dataset (pad) by the same method. pad is not used in any pre-training stage of the xlm model. the cmi and spf in pad data are 0.537 and 0.3, respectively. our evaluation metrics are bleu @xcite and bert-score @xcite . all experiments were done three times and we report the average score. in the model configuration, we set the dimension of embedding to 768. the rest of the configuration follows the vanilla xlm. all models are trained on nvidia 3090 gpu. it took 7 days to train zh-mlm and about 1 to 3 days for the rest of the models. our experiment results show that using dli brings a positive influence on most configurations of the model. furthermore, transfer learning in the clm and mlm stages significantly improves performance. it demonstrates that continuous training with our cm data enables monolingual language models to provide better performance when applied to cm translation tasks. for more details and discussions in our experiment, please refer to a.2. ## conclusion in this paper, we introduce the hokkien background and the cm phenomenon between dialects and mandarin in taiwan. we proposed a hokkien-mandarin cm dataset based on the linguist theory and the hokkien grammar rules. we proposed a solution to the hokkien word segmentation through a linguistics-based toolkit, articut. based on the x-bar theory, we can avoid the negative impact of morphology on the sino-tibetan languages. we simply modify the language embedding mechanism and use transfer learning in the xlm model, which performs well on both cm and monolingual translation. we again prove that under a fully trained language model and well-defined language identity, the cross-lingual model can generalize the knowledge to a cm sentence without special training. we also verify the feasibility of linguistic-based background knowledge as a low-resource language solution. after developing the cm corpora and translation systems, it can be further extended to any existing monolingual task. meanwhile, we can use them to generate speech data, and train a cm speech recognition model.