C hinese Character Decomposition for Neural MT with Multi-Word Expressions (2021)

Abstract: Chinese character decomposition has been used as a feature to enhance Machine Translation (MT) models, combining radicals into character and word level models. Recent work has investigated ideograph or stroke level embedding. However, questions remain about different decomposition levels of Chinese character representations, radical and strokes, best suited for MT. To investigate the impact of Chinese decomposition embedding in detail, i.e., radical, stroke, and intermediate levels, and how well these decompositions represent the meaning of the original character sequences, we carry out analysis with both automated and human evaluation of MT. Furthermore, we investigate if the combination of decomposed Multiword Expressions (MWEs) can enhance the model learning. MWE integration into MT has seen more than a decade of exploration. However, decomposed MWEs has not previously been explored.

## introduction neural machine translation (nmt) @xcite @xcite has recently replaced statistical machine translation (smt) @xcite @xcite as the state-of-the-art for machine translation (mt). however, research questions still remain, such as how to deal with out-ofvocabulary (oov) words, how best to integrate linguistic knowledge and how best to correctly translate multi-word expressions (mwes) @xcite @xcite . for oov word translation for european languages, substantial improvements have been made in terms of rare and unseen words by incorporating sub-word knowledge using byte pair encoding (bpe) @xcite . however, such methods cannot be directly applied to chinese, japanese and other ideographic languages. integrating sub-character level information, such as chinese ideograph and radicals as learning knowledge has been used to enhance features in nmt systems @xcite @xcite . @xcite , for example, explain that the meaning of some unseen or low frequency chinese characters can be estimated and translated using radicals decomposed from the chinese characters, as long as the learning model can acquire knowledge of these radicals within the training corpus. chinese characters often include two pieces of information, with semantics encoded within radicals and a phonetic part. the phonetic part is related to the pronunciation of the overall character, either the same or similar. for instance, chinese characters with this two-stroke radical, 刂 (tí dāo páng), ordinarily relate to knife in meaning, such as the chinese character 劍 (jiàn, sword) and multi-character expression 鋒利 (fēnglì, sharp). the radical 刂 (tí dāo páng) preserves the meaning of knife because it is a variation of a drawing of a knife evolving from the original bronze inscription (fig. 4 in appendices). not only can the radical part of a character be decomposed into smaller fragments of strokes but the phonetic part can also be decomposed. thus there are often several levels of decomposition that can be applied to chinese characters by combining different levels of decomposition of each part of the chinese character. as one example, figure 1 shows the three decomposition levels from our model and the full stroke form of the above mentioned characters 劍(jiàn) and 鋒(fēng) . to date, little work has been carried out to investigate the full potential of these alternative levels of decomposition of chinese characters for the purpose of machine translation (mt). in this work, we investigate chinese character decomposition, and another area related to chinese characters, namely chinese mwes. we firstly investigate translation at increasing levels of decomposition of chinese characters using underlying radicals, as well as the additional chinese character strokes (corresponding to ever-smaller units), breaking down characters into component parts as this is likely to reduce the number of unknown words. then, in order to better deal with mwes which have a common occurrence in general contexts @xcite , and working in the opposite direction in terms of meaning representation, we investigate translating larger units of chinese text, with the aim of restricting translation of larger groups of chinese characters that should be translated together as one unit. in addition to investigating the effects of decomposing characters we simultaneously apply methods of incorporating mwes into translation. mwes can appear in chinese in a range of ways, such as fixed (or semi-fixed) expressions, metaphor, idiomatic phrases, and institutional, personal or location names, amongst others. in summary, in this paper, we investigate: (i) the degree to which chinese radical and stroke sequences represent the original word and character sequences that they are composed of; (ii) the difference in performance achieved by each decomposition level; (iii) the effect of radical and stroke representations in mwes for mt. furthermore, we offer: ## related work chinese character decomposition has been explored recently for mt. for instance, @xcite and @xcite , considered radical embeddings as additional features for @xmath0. @xcite tested a range of encoding models including word+character, word+radical, and word+character+radical. this final setting with word+character+radical achieved the best performance on a standard nist foot_1 mt evaluation data set for @xmath1. furthermore, zhang and matsumoto (2018) applied radical embeddings as additional features to character level lstm-based nmt on @xmath2. none of the aforementioned work has however investigated the performance of decomposed character sequences and the effects of varied decomposition degrees in combination with mwes. subsequently, @xcite developed bidirectional english ⇔ japanese, english ⇔ chinese and chinese ⇔ japanese nmt with word, character, ideograph (the phonetics and semantics parts of characters are separated) and stroke levels, with experiments showing that the ideograph level was best for @xmath3, while the stroke level was best for @xmath4. although their ideograph and stroke level setting replaced the original character and word sequences, there was no investigation of intermediate decomposition performance, and they only used bleu score for automated evaluation with no human assessment involved. this gives us inspiration to explore the performance of intermediate level embedding between ideograph and strokes for the mt task. ## chinese character decomposition in this section, we introduce a character decomposition approach and the extraction tools which we apply in this work (code will be publicly available). we utilize the open source ids dictionary foot_2 which was derived from the chise (character information service environment) project foot_3 . it is comprised of 88,940 chinese characters from cjk (chinese, japanese, korean script) unified level-3: … full-stroke: ideographs and the corresponding decomposition sequences of each character. most characters are decomposed as a single sequence, but characters can have up to four possible decomposed representations. the reason for this is that the character can come from different resources, such as chinese hanzi (g, h, t for mainland, hong kong, and taiwan), japanese kanji (j), korean hanja (k), and vietnamese chunom (v), etc. 5 even though they have the same root of hanzi, the historical development of languages and writing systems in different territories has resulted in certain degrees of variation in their appearance and stroke order. for instance, (且, qiě) vs (目, mù) from the second character example in figure 2 . figure 2 shows example characters that have two different decomposition sequences. in our experiments, when there is more than one decomposed representation of a given character, we choose the chinese mainland decomposition standard (g) for the model, since the corpora we use correspond best to simplified chinese as used in mainland china. the examples in figure 2 also show the general construction and corresponding decomposition styles of chinese characters, such as left-right, up-down, inside-outside, and embedded amongst others. to obtain a decomposition level l representation of chinese character α, we go through the ids file l times. each time, we search the ids file character list to match the newly generated smaller sized characters and replace them with decomposed representation recursively. ## nmt experiments we test the various levels of decomposed chinese and chinese mwes using publicly available data from the wmt-2018 shared tasks chinese to en- ## evaluation in order to assess the performance of each model employing a different meaning representation in terms of decomposition and mwes, we carried out both automatic evaluation using bleu @xcite in fig. 3 , and human evaluation (direct assessment) of the outputs of the system. since decomposition level 3 yields generally higher scores than the other two levels, we also applied decomposition of mwes to level 3 and concatenated the bilingual glossaries to the training. we used the models with the most learning steps, 180k, and run human evaluation on the amazon mechanical turk crowd-sourcing platform, foot_5 including the strict quality control measures of @xcite . direct assessment scores for systems were calculated as in @xcite by firstly computing an average score per translation before calculating the overall average for a system from its average scores for translations. significance tests in the form of wilcoxon rank-sum test are then applied to score distributions of the latter to identify systems that significantly outperform other systems in the human evaluation. results of the direct assessment human evaluation are shown in we also discovered that the decomposed models generated fewer system parameters for the neural nets to learn, which potentially reduces computational complexity. for instance, the total trainable variable size of the character sequence baseline model is 89,456,896, while this number decreased to 80,288,000 and 80,591,104 respectively for the rxd3 and rxd2 models (a 10.25% drop for rxd3). as mentioned by @xcite , in nlp tasks the total number of possible words is so large that the word sequence models have to operate on an extremely high-dimensional and sparse discrete space. the decomposition model reduced the overall size of possible tokens for the model to learn, which is more space efficient. for the automatic and human evaluation results, where decomposition level 2 achieved a surprisingly lower score than the other levels, error analysis revealed an important insight. while level 1 decomposition encoded the original character sequences into radical representations, and this typically contains semantic and phonetic parts of the character, and level 3 gives a deeper decomposition of the character such as the stroke level pieces with sequence order. in contrast, however, level-2 decomposition appears to introduce some intermediate characters that mislead model learning. ally constructed from fewer strokes than the original root character, but can be decomposed from it. as in figure 1 , from decomposition level 2, we get new characters 从 (cóng) and 王 (wáng) respectively from 劍 (jiàn, sword) and 鋒 (fēng, edge/sharp point), but they have no direct meaning from their father characters, instead meaning "from" and "king" respectively. in summary, decomposition level-2 tends to generate some intermediate characters that do not preserve the meaning of the original root character's radical, nor those of the strokes, but rather smaller sized independent characters with fewer strokes that result in other meanings. ## conclusions and future work in this work, we examined varying degrees of chinese character decomposition and their effect on chinese to english nmt with attention architecture. to the best of our knowledge, this is the first work on detailed decomposition level of chinese characters for nmt, and decomposition representation for mwes. we conducted experiments for decomposition levels 1 to 3; we had a look at level 4 decomposition and it appears similar to level 3 sequences. we publish our extraction toolkit free for academic usage. we conducted automated evaluation with the bleu metric, and crowd sourced human evaluation with the direct assessment (da) methodology. our conclusion is that the chinese character decomposition levels 1 and 3 can be used to represent or replace the original character sequence in an mt task, and that this achieves similar performance to the original character sequence model in our nmt setting. however, decomposition level 2 is not suitable to represent the original character sequence in meaning at least for mt. we leave it to future work to explore the performance of different decomposition levels in other nlp tasks. another finding from our experiments is that while adding bilingual mwe terms can both increase character and decomposed level mt score according to the automatic metric bleu, the human evaluation shows no statistical significance between them. significance testing using automated evaluation metrics will be carried out in our future work, such as meteor @xcite , and lepor @xcite , in addition to bleu. we will consider different mwe integration methods in future and reduce the training set to investigate the differences in low-resource scenarios (5 million sentence pairs for training set were used in this work). we will also sample a set of the testing results and conduct a human analysis regarding the mwe translation accuracy from different representation models. we will further investigate different strategies of combining several level of decompositions together and their corresponding performances in semantic representation, such as mt task. the ids file we applied to this work limited the performance of full stroke level capability, and we will look for alternative methods to achieve full-stroke level character sequence extraction for nlp tasks investigation. we also attach the translations of these two sentences by four other models. with regard to the first sentence mwes, all the four models translate san francisco mall correctly as ref and rxd3 beating base model. in terms of the second sentence mwes, basemwe and rxd2 drop out the mwe 近日 (jìnrì, recently) as base model, and all the four models drop out the translation of mwe 楼梯间 (lóutījiān, stairwell). src 28 岁 厨师 被 发现 死 于 旧⾦金金⼭山 ⼀一家 商场 近⽇日 刚 搬 ⾄至 旧⾦金金⼭山 的 ⼀一位 28 岁 厨师 本周 被 发现 死 于 当地 ⼀一家 商场 的 楼梯间 。 ref 28 @-@ year @-@ old chef found dead at san francisco mall a 28 @-@ year @-@ old chef who had recently moved to san francisco was found dead in the stairwell of a local mall this week .