Breaking the Representation Bottleneck of C hinese Characters: Neural Machine Translation with Stroke Sequence Modeling (2022)

Abstract: Existing research generally treats Chinese character as a minimum unit for representation. However, such Chinese character representation will suffer two bottlenecks: 1) Learning bottleneck, the learning cannot benefit from its rich internal features (e.g., radicals and strokes); and 2) Parameter bottleneck, each individual character has to be represented by a unique vector. In this paper, we introduce a novel representation method for Chinese characters to break the bottlenecks, namely StrokeNet, which represents a Chinese character by a Latinized stroke sequence (e.g., “凹 (concave)” to “ajaie” and “凸 (convex)” to “aeaqe”). Specifically, StrokeNet maps each stroke to a specific Latin character, thus allowing similar Chinese characters to have similar Latin representations. With the introduction of StrokeNet to neural machine translation (NMT), many powerful but not applicable techniques to non-Latin languages (e.g., shared subword vocabulary learning and ciphertext-based data augmentation) can now be perfectly implemented. Experiments on the widely-used NIST Chinese-English, WMT17 Chinese-English and IWSLT17 Japanese-English NMT tasks show that StrokeNet can provide a significant performance boost over the strong baselines with fewer model parameters, achieving 26.5 BLEU on the WMT17 Chinese-English task which is better than any previously reported results without using monolingual data. Code and scripts are freely available at https://github.com/zjwang21/StrokeNet.

## introduction neural machine translation (nmt) has become the mainstream paradigm in machine translation recently @xcite @xcite . with rich bilingual parallel corpora, nmt achieves state-of-the-art performance on multiple translation benchmarks. in chinese nmt tasks, the chinese character has been the minimum representation unit for a long time. however, such representation perhaps might not be the best choice for chinese nmt due to the two following representation bottlenecks. the first is the learning bottleneck. the representation learning of chinese does not fully utilize its rich internal features. latin languages have rich information in words like affixes. actually, chinese also has this kind of internal information. a chinese character usually contains one radical (rarely has two) and several other radical-like components @xcite . characters with the radical "扌" commonly are verbs. the characters "扎(tie)", "拉(pull)", "打(hit)", "扔(throw)" and "提(carry)" all have the meaning of acting with hands because they have the same radical "扌". latin languages can easily learn this internal information by subword modeling while chinese cannot if just taking character as the minimum unit into consideration, which limits the representation capability of nmt models. the second is the parameter bottleneck. in chinese nlp models, the parameters used for chinese word representation can be a huge number. in largescale cross-lingual pre-trained language models like xlm-r @xcite , mbart @xcite and mt5 @xcite , chinese tokens account for a very unbalanced proportion of the vocabulary. for instance, the vocabulary of xlm-r and mbart is learned from the corpus of 100 languages, resulting in 250k subword tokens, of which chinese tokens account for nearly 20k. besides, in these models, non-latin languages like chinese have to learn embedding individually. it is difficult for them to compress the size of vocabulary by sharing subwords as latin languages do. in such an overparameterized scenario, nmt models might meet a serious over-fitting issue. to break the above two representation bottlenecks of chinese, in this paper we introduce stro-kenet, a novel representation method for chinese characters. specifically, strokenet transforms each chinese character to its corresponding stroke sequence. then strokenet transforms each stroke to a lowercased latin character by a predefined rule. with this transformation, a chinese character is represented by a latinzied stroke sequence, looking like an english word. the similar chinese characters that share the same radical will also share the same affixes in their latinized representation. as strokenet represents chinese as latinized stroke sequence, we now can implement several powerful methods, which are previously not applicable to non-latin languages, on chinese nmt tasks. strokenet can learn subword vocabulary @xcite on the latinized stroke sequence to break the learning bottleneck. through this technique, strokenet can benefit from internal features such as radicals to enhance representation learning. furthermore, strokenet can share subword embedding between source and target languages (press and wolf, 2017), leading to a significant parameter reduction and overcoming model over-fitting. finally, the ciphertext-based data augmentation @xcite , a powerful technique in latin languages can be added into strokenet to achieve performance boost in several nmt tasks. these techniques can take full advantage of the rich internal information brought by latinized strokes. we conduct experiments on widely-used nist chinese-english, wmt17 chinese-english and iwslt17 japanese-english nmt tasks. the results show that strokenet provides a significant performance boost over strong baselines using fewer model parameters. we achieve a new state-of-theart result of 26.5 bleu on the wmt17 chinese-english task, with an increment of 2.1 bleu over the scaling transformer baseline @xcite . our main contributions are as follows: ## strokenet to break the learning and parameter bottlenecks of chinese character representation, we propose strokenet that maps a chinese character into a latinized sequence, and apply it to nmt tasks. figure 1 shows the overall framework of strokenet. ## evaluation we aim to answer the research questions through the following experiments: ## analysis effect of bpe merge operations strokenet benefits from the subword modeling technique. to explore how it works in strokenet, we conduct experiments on the nist zh-en benchmark, applying different numbers of bpe merge operations. we conduct experiments on 20k, 30k, 40k, and 50k merge operations. results are detailed in figure 2. 30k merge operations appear to be the best choice. for the validation set, the translation performance reaches the highest at 30k. for the test set, it gradually decreases as the number of merge operations increases. we see variations of less than 0.7 bleu in the dev set and less than 0.4 bleu in the test set as the number of bpe merge operations changes. and large improvements over the vanilla baseline are observed regardless of the number of bpe merge operations with at least +2.2 bleu for both the validation and test sets. the results show that strokenet is robust to the number of bpe merge operations. learning curves to explore how strokenet performs better than the vanilla baseline, we draw the learning curves during training. we train strokenet and the vanilla baseline for both 50 epochs on the nist zh-en. the validation bleu during training is shown in figure 3 . 6 in strokenet, the bleu on the validation set rises faster in the early period and finally achieves higher bleu than the vanilla baseline. with the latinized stroke level representation and the application of powerful techniques in latin languages, nmt models with strokenet learn faster and better than vanilla models. the rich internal information in chinese characters becomes readily available for strokenet to learn. the results prove that strokenet successfully breaks the representation learning bottleneck, showing its positive effects on chinese nmt tasks. ## conclusion in this paper we introduce strokenet, a novel technique for chinese nmt tasks using latinized stroke sequence of chinese characters. strokenet breaks the representation learning bottleneck and the parameter bottleneck in chinese nmt tasks, which requires no external data and significantly outperforms several strong prior works. we show that representing chinese characters in stroke level works well on nmt tasks to bring more internal structure information. we demonstrate that it is possible to implement popular and powerful techniques designed for latin languages in chinese nmt tasks. we conduct several analyses on the effects of these latin language techniques, proving they bring an obvious performance boost to stro-kenet. overall, strokenet is a simple and effective approach for chinese nmt tasks and yields strong results in both high-source and low-source settings. future work includes applying strokenet to other language generation tasks @xcite .