2kenize: Tying Subword Sequences for C hinese Script Conversion (2020)

Abstract: Simplified Chinese to Traditional Chinese character conversion is a common preprocessing step in Chinese NLP. Despite this, current approaches have insufficient performance because they do not take into account that a simplified Chinese character can correspond to multiple traditional characters. Here, we propose a model that can disambiguate between mappings and convert between the two scripts. The model is based on subword segmentation, two language models, as well as a method for mapping between subword sequences. We further construct benchmark datasets for topic classification and script conversion. Our proposed method outperforms previous Chinese Character conversion approaches by 6 points in accuracy. These results are further confirmed in a downstream application, where 2kenize is used to convert pretraining dataset for topic classification. An error analysis reveals that our method’s particular strengths are in dealing with code mixing and named entities.

## introduction chinese character (or script) conversion is a common preprocessing step for chinese nlp practitioners @xcite . traditional chinese (tc) and simplified chinese (sc) are the two standardized character sets (or scripts) for written chinese. tc is predominantly used in taiwan, hong kong, and macau, whereas sc is mainly adopted in mainland china and sc characters are simplified versions of tc characters in terms of strokes and parts. therefore, chinese nlp practitioners apply script converters foot_0 to translate the dataset into their desired language. this is especially useful for tc nlp practitioners because tc is less widely used and under-resourced as compared to sc. converting from tc to sc is generally straightforward because there are one-to-one correspondences between most of the characters, so conversion can be performed using mapping tables @xcite . however, conversion from sc to tc is an arduous task as some sc characters can be mapped to more than one tc character depending on the context of the sentence. a detailed analysis by @xcite shows that sc to tc conversion is a challenging and crucial problem, as 12% of sc characters have one-to-many mappings to tc characters. our experiments show that current script converters achieve sentence accuracy results of 55-85% ( §3). another issue is that varying tokenization would lead to different results as chinese is an unsegmented language, see despite this being an important task, there is a lack of benchmarks, foot_2 which implies that this problem is understudied in nlp. in this study, we propose 2kenize, a subword segmentation model which jointly considers simplified chinese and forecasting traditional chinese constructions. we achieve this by constructing a joint simplified chinese and traditional chinese language model based viterbi tokenizer. performing mapping disambiguation based on this tokenization method improves sentence accuracy by 6 points as compared to off-the-shelf converters and supervised models. our qualitative error analysis reveals that our method's particular strengths are in dealing with code-mixing and named entities. additionally, we address the issue of a lack of benchmark datasets by constructing datasets for script conversion and tc topic classification. ## 2kenize: joint segmentation and conversion we employ subword tokenization, as it addresses the issue of rare and unknown words @xcite and has been shown advantageous for the language modelling of morphologicallyrich languages @xcite . this achieves improvements in accuracy for neural machine translation (nmt) tasks and has now become a prevailing practice @xcite . the most widelyutilized method is byte pair encoding @xcite ), a compression algorithm that combines frequent sequences of characters, which results in rare strings being segmented into subwords. unigram @xcite and bpe-drop @xcite use subword ambiguity as noise, as well as stochastically-corrupted bpe segmentation to make it less deterministic. for nmt tasks generally, subword segmentation is seen as a monolingual task and applied independently on source and target corpora. we hypothesize that translation tasks, and specifically conversion tasks, as investigated here, would have a bet-ter performance if segmentation were performed jointly. hence, in this section, we describe our proposed method 2kenize, which jointly segments by taking the source and its approximate target sentences into account. this motivates the main idea of this paper: we propose 2kenize which jointly considers the source sentence and its corresponding target conversions by doing lookaheads with mappings. ## extrinsic evaluation an accurate script converter should produce a less erroneous dataset, which should in turn improves the accuracy of the downstream tasks. in this section, we demonstrate the effect of script conversion on topic classification tasks to examine this assumption. we also study the impact of tokenization and pooling on the accuracy of topic classification. we apply the converter to the language modelling corpus (wikitext), then train a classifier for informal and formal topic classification on that translated data. this allows us to measure the performance of the converter compared to other ones for a specific downstream task. ## takeaways and open questions the contributions of our work are: