ZEN : Pre-training C hinese Text Encoder Enhanced by N-gram Representations (2020)

Abstract: The pre-training of text encoders normally processes text as a sequence of tokens corresponding to small text units, such as word pieces in English and characters in Chinese. It omits information carried by larger text granularity, and thus the encoders cannot easily adapt to certain combinations of characters. This leads to a loss of important semantic information, which is especially problematic for Chinese because the language does not have explicit word boundaries. In this paper, we propose ZEN, a BERT-based Chinese text encoder enhanced by n-gram representations, where different combinations of characters are considered during training, thus potential word or phrase boundaries are explicitly pre-trained and fine-tuned with the character encoder (BERT). Therefore ZEN incorporates the comprehensive information of both the character sequence and words or phrases it contains. Experimental results illustrated the effectiveness of ZEN on a series of Chinese NLP tasks, where state-of-the-art results is achieved on most tasks with requiring less resource than other published encoders. It is also shown that reasonable performance is obtained when ZEN is trained on a small corpus, which is important for applying pre-training techniques to scenarios with limited data. The code and pre-trained models of ZEN are available at https://github.com/sinovation/ZEN .

## introduction pre-trained text encoders @xcite @xcite @xcite have drawn much attention in natural language processing (nlp), because state-of-theart performance can be obtained for many nlp tasks using such encoders. in general, these encoders are implemented by training a deep neural model on large unlabeled corpora. although the use of big data brings success to these pre-trained encoders, it is still unclear whether existing encoders have effectively leveraged all useful information in the corpus. normally, the pre-training procedures are designed to learn on tokens corresponding to small units of texts (e.g., word pieces for english, characters for chinese) for efficiency and simplicity. however, some important information carried by larger text units may be lost for certain languages when we use a standard encoder, such as bert. for example, in chinese, text semantics are greatly affected by recognizing valid n-grams foot_0 . this means a pre-trained encoder can potentially be improved by incorporating such boundary information of important n-grams. recently, there are studies adapting bert for chinese with word information, yet they are limited in maintaining the original bert structure, augmented with learning from weakly supervised word information or requiring external knowledge. as an example, a representative study in @xcite proposed to use the whole-word masking strategy to mitigate the limitation of word information. they used an existing segmenter to produce possible words in the input sentences, and then trained a standard bert on the segmented texts by masking whole words. @xcite proposed to perform both entity-level and phraselevel masking to learn knowledge and information from the pre-training corpus. however, their approaches are limited in the following senses. first, both methods rely on the word masking strategy so that the encoder can only be trained with existing word and phrase information. second, similar to the original bert, the masking strategy results in the mis-match of pretraining and fine-tuning, i.e., no word/phrase information is retained when the encoders are applied to downstream prediction tasks. third, incorrect word segmentation or entity recognition results cause errors propagated to the pre-training process and thus may negatively affected the generalization capability of the encoder. in this paper, we propose zen, a chinese (z) text encoder enhanced by representing n-grams, which provides an alternative way to improve character based encoders (e.g., bert) by using larger text granularity. to train our model, one uses an n-gram lexicon from any possible sources such as pre-defined dictionaries and n-gram lists extracted via unsupervised approaches. such lexicon is then mapped to training texts, and is used to highlight possible combinations of characters that indicate likely salient contents during the training process. our model then integrate the representations of these n-gram contexts with the character encoder. similarly, the fine-tune process on any task-specific dataset further enhances zen with such n-gram representations. an important feature of our method is that while the model explicitly takes advantage of n-gram information, the model only outputs character-level encodings that is consistent with bert. therefore downstream tasks are not affected. zen extends the original bert structure and explicitly incorporates representations of large granular texts into it, which is different from (and complementary to) previous methods that relied on weak supervision such as masking. 3 to mitigate the error propagation, in the n-gram encoder, we use the attention mechanism to dynamically weigh n-grams so that those truly useful n-grams are emphasized while noisy ones are less learned with low weights. in addition, the n-gram vocabulary is collected from an extra large corpus, and it can be easily adapted to any source from different domains to ensure incorporating the most important n-grams before training zen. our experiments follow the standard procedure, i.e., training zen on the chinese wikipedia dump and fine-tune it on several chinese downstream nlp tasks. experiment results demonstrate its validity and effectiveness where state-of-the-art performance is achieved on many tasks using the ngrams, which are automatically learned from the training data other than external or prior knowledge. in particular, our method outperforms some existing encoders trained on much larger corpora. 3 although the character encoder may still use masking as a learning objective, the encoded n-grams are explicitly ## zen the overall architecture of zen is shown in figure 1 , where the backbone model (character encoder) is bert foot_1 @xcite , enhanced by n-gram information represented by a multi-layer encoder. since the basis of bert is well explained in previous studies @xcite , in this paper, we focus on the details of zen, by explaining how n-grams are processed and incorporated into the character encoder. ## analysis we analyze zen with several factors affecting its performance. details are illustrated in this section. ## related work representation learning of text attracts much attention in recent years, with the rise of deep learning in nlp @xcite @xcite . there are considerable interests in representing text with contextualized information @xcite @xcite @xcite . following this paradigm, pre-trained models have been proposed and are proven useful in many nlp tasks @xcite @xcite @xcite . in detail, such models can be categorized into two types: autoregressive and autoencoding encoders. the former models behave like normal language models that predict the probability distributions of text units following observed texts. these models, such as gpt @xcite and gpt2 @xcite , are trained to encode a uni-directional context. differently, the autoencoding models, such as bert @xcite and xlnet @xcite , leverage bidirectional context, and encode texts by reconstructing the masked tokens in each text instance according to their context from both sides. particularly for chinese, many enhanced pretrain models are proposed that can utilize wordlevel information in one way or another because words carry important linguistic information. for example, ernie 1.0 @xcite adopted a multi-level masking strategy performed on different level of texts; its improved version, ernie 2.0 @xcite used continual pre-training strategy which is benefited from multi-task learning with more parameters in the model. recently, bert-wwm @xcite enhanced chinese bert with a simple masking of whole-words. in addition, there are other recent studies that enhanced bert for chinese language processing, such as optimizing training via special optimization techniques @xcite or from prior knowledge @xcite . all the studies revealed that processing on larger granularity of text is helpful in chinese, which is consistent with previous findings in many chinese nlp tasks @xcite @xcite . compared to the aforementioned studies, zen thus provides an alternative solution that explicitly encodes n-grams into character-based encoding, rather than through weak supervision, i.e., masking, to incorporate word/phrase information. ## conclusion and future work in this paper, we proposed zen, a pre-trained chinese text encoder enhanced by n-gram representations, where different combinations of characters are extracted, encoded and integrated in training a backbone model, i.e., bert. in zen, given a sequence of chinese characters, n-grams are extracted and their information are effectively incorporated into the character encoder. different from previous work, zen provides an alternative way of learning larger granular text for pre-trained models, where the structure of bert is extended by another transformer-style encoder to represent the extracted n-grams for each input text instance. experiments on several nlp tasks demonstrated the validity and effectiveness of zen, where stateof-the-art results were obtained from them while zen is built upon the bert base model and requires less training data and no knowledge from external sources compared to other existing chinese text encoders. experiments of zen on small corpus also showed its efficiency and capability of being able to learn with limited data. further analyses revealed the factors affecting zen's performance, where the quality of the n-gram lexicon and the number of n-grams used for each input are more important than the training epoch. note that zen employs a different method to incorporate word (n-gram) information, which could be complementary to some other previous approaches. therefore, it is potentially beneficial to combine it with other character-encoding approaches. for future work, we plan to enlarge zen as well as apply it to other languages that lack white-space tokenization and compare different ngram extraction methods, e.g., obtaining n-grams by byte pair encoding @xcite or wordpiece tokenization @xcite .