Sub-Character Tokenization for C hinese Pretrained Language Models (2023)

Abstract: Tokenization is fundamental to pretrained language models (PLMs). Existing tokenization methods for Chinese PLMs typically treat each character as an indivisible token. However, they ignore the unique feature of the Chinese writing system where additional linguistic information exists below the character level, i.e., at the sub-character level. To utilize such information, we propose sub-character (SubChar for short) tokenization. Specifically, we first encode the input text by converting each Chinese character into a short sequence based on its glyph or pronunciation, and then construct the vocabulary based on the encoded text with sub-word segmentation. Experimental results show that SubChar tokenizers have two main advantages over existing tokenizers: 1) They can tokenize inputs into much shorter sequences, thus improving the computational efficiency. 2) Pronunciation-based SubChar tokenizers can encode Chinese homophones into the same transliteration sequences and produce the same tokenization output, hence being robust to homophone typos. At the same time, models trained with SubChar tokenizers perform competitively on downstream tasks. We release our code and models at https://github.com/thunlp/SubCharTokenization to facilitate future work.

## introduction large-scale transformer-based pretrained language models (plms) @xcite @xcite @xcite , in which tokenization * equal contribution. † corresponding authors. plays a fundamental role, have achieved great success in recent years and attracted wide research interest. the most popular type of tokenization adopted by plms is sub-word tokenization, such as byte pair encoding (bpe) @xcite , word-piece @xcite , and unigram language model segmentation @xcite . recent chinese plms such as cpm @xcite adopt this kind of sub-word tokenization. apart from sub-word tokenization, many other chinese plms adopt a simple character tokenizer (chartokenizer for short) that treats every single chinese character as a token @xcite @xcite . however, we believe that both of these existing tokenizers are sub-optimal for chinese. this is based on the observation that chinese has unique linguistic characteristics: to assess the effectiveness of our proposed method, we train a series of bert-style plms using the existing and proposed tokenizers. we evaluate these models on over ten datasets of various downstream natural language understanding (nlu) tasks. through extensive evaluation, we find that models trained with subchar tokenizers match models trained with character and subword tokenizers on downstream task performance. more importantly, subchar tokenizers have two major advantages compared to existing tokenizers: the increased use of combination tokens leads to significantly decreased length of the tokenized sequences. for example, on the iflytek long text classification dataset, with the same vocabulary size as the chartokenizers, subchar tokenizers can achieve as much as 40% length reduction on the tokenized output. such length reduction can significantly speed up both pretraining and finetuning. we believe that our work is an important step towards more tailored techniques for languages beyond just english by effectively integrating the unique linguistic characteristics of the language @xcite . ## method in this section, we describe our proposed sub-char tokenization in detail. we break it down into two steps: 1) chinese character encoding; 2) vocabulary construction based on the encoded sequences. ## experiment setup in this section, we introduce our baselines, datasets, and experiment settings. ## experiment results in this section, we present the experiment results and the main findings. we not only evaluate on a wide range of common chinese nlu datasets, but also perform robustness evaluation on both synthetic and real-world noisy data. ## analysis in this section, we conduct various analyses to better understand the working mechanisms of sub-char tokenization, including illustrations of the efficiency improvement and ablations on different components of our tokenization pipeline. ## conclusion in this work, we propose sub-character tokenization and conduct comprehensive experiments to illustrate its advantage over existing tokenization methods. compared to treating each individual character as a token (chartokenizer) or directly running sub-word tokenization on the raw chinese text (sub-word tokenizer), our subchar tokenizers not only perform competitively on downstream nlu tasks, but, more importantly, they can be much more efficient and robust. we conduct a series of ablation and analysis to understand the reasons why subchar tokenizers are more efficient, as well as the impact of linguistic and nonlinguistic encoding. given the advantages of our subchar tokenizers, we believe that they are better alternatives to all existing chinese tokenizers, especially in applications where efficiency and robustness are critical. it is possible that our approach can be useful for other morphologically poor languages and more complicated methods could be developed based on subchar tokenization for even better performance. we leave these interesting directions for future exploration. on a broader level, our work makes an important attempt in developing more tailored methods for a language drastically different from english with promising results. we believe that this is a crucial future direction for the community given the language diversity in the world. we hope that our work can inspire more such work in order to benefit language technology users from different countries and cultures.