MVP - BERT : Multi-Vocab Pre-training for C hinese BERT (2021)

Abstract: Despite the development of pre-trained language models (PLMs) significantly raise the performances of various Chinese natural language processing (NLP) tasks, the vocabulary (vocab) for these Chinese PLMs remains to be the one provided by Google Chinese BERT (CITATION), which is based on Chinese characters (chars). Second, the masked language model pre-training is based on a single vocab, limiting its downstream task performances. In this work, we first experimentally demonstrate that building a vocab via Chinese word segmentation (CWS) guided sub-word tokenization (SGT) can improve the performances of Chinese PLMs. Then we propose two versions of multi-vocab pre-training (MVP), Hi-MVP and AL-MVP, to improve the models’ expressiveness. Experiments show that: (a) MVP training strategies improve PLMs’ downstream performances, especially it can improve the PLM’s performances on span-level tasks; (b) our AL-MVP outperforms the recent AMBERT (CITATION) after large-scale pre-training, and it is more robust against adversarial attacks.

## introduction the pre-trained language models (plms), including bert @xcite and its variants @xcite , have been proven beneficial for many natural language processing (nlp) tasks, such as text classification, question answering @xcite , natural language inference (nli) @xcite and relation extraction @xcite , on english, chinese and many other languages. although they bring impressive improvements for chinese nlp tasks, most chinese plms still use the vocabulary (vocab) provided by google chinese bert @xcite . google chinese * contact: 52205901018@stu.ecnu.edu.cn. bert is a character (char) based model since it splits the chinese characters with blank spaces. in the pre-bert era, a part of the literature on chinese natural language processing (nlp) first do chinese word segmentation (cws) to divide the text inputs into sequences of words and use a wordbased vocab in nlp models @xcite . there are many arguments on which vocab a chinese nlp model should adopt. the advantages of char-based models are apparent. first, char-based vocab is smaller, thus reducing the model size. second, it does not rely on cws, thus avoiding word segmentation error, which can directly result in performance gain in span-based tasks such as named entity recognition (ner). third, char-based models are less vulnerable to data sparsity or the presence of out-of-vocab (oov) words and thus less prone to over-fitting @xcite . however, word-based model has its advantages. first, it will result in shorter sequences than char-based counterparties, thus are faster. second, words are less ambiguous, thus helping models learn the semantic meanings of words. third, with a word-based model, exposure biases may be reduced in text generation tasks @xcite . another branch of literature tries to balance the two by combining word-based embedding with charbased embedding @xcite . this article tries to strike a balance between the char-based and word-based models and provides alternative approaches for pre-training chinese plms. we experiment on two approaches to build a vocab for chinese plms: (1) following @xcite , separate the chinese chars with white spaces, and then learn a sub-word tokenizer (denote as char); (2) first segment the sentences with a cws toolkit like jieba 1 , and then learn a sub-word tokenizer (denoted as sgt); (3) do cws and keep the high-frequency words as tokens and low-frequency words will be tokenized by sgt (denoted as seg). see figure 1 for their workflow of processing an input sentence. the experiments show that sgt is best suited for plms. inspired by the previous work that incorporates multiple vocabularies (vocabs) or naturally combines multiple vocabs @xcite @xcite , we also investigate a series of strategies, which we will call multi-vocab pre-training (mvp) strategies. the first version of mvp incorporates a hierarchical structure to combine the char-based vocab and word-based vocab. from the viewpoint of model forward pass, chinese characters' embeddings are aggregated to form the vector representations of multi-gram words or tokens, which are fed into transformer encoders. then the word-based vocab will be used in masked language model (mlm) training. the second version of mvp (denoted as al-mvp) is to employ an additional vocab to form an auxiliary loss term in mlm, enhancing the plm's ability to capture the contextual information. extensive experiments and ablation studies are conducted. we select bpe implemented by sentencepiece foot_1 as the sub-word tokenization model, and albert @xcite (tiny and base model) as our plms. pre-training is done on chinese wikipedia corpus foot_2 (c-1), and a larger corpus we collect (c-2). the mvp strategies are compared on a series of chinese benchmark datasets, two of which are sentence classification (cls) tasks, two are named entity recognition (ner) tasks, and the remaining two are machine reading comprehension (mrc) tasks. the experimental results reveal the following take-aways: 1) combining cws and sub-word tokenization yields the best vocab for chinese plms; 3) mvp strategies can improve a single-vocab model on all three types of tasks. we now summarize the following contributions in this work. ## related work since @xcite , a large amount of literature on pre-trained language models appear and push the nlp community forward with a speed that has never been witnessed before. @xcite is one of the earliest plms that learns contextualized representations of words. gpts @xcite and bert @xcite take advantage of transformer @xcite . gpts are uni-directional and make predictions on the input text in an auto-regressive manner, and bert is bi-directional and makes predictions on the whole or part of the input text. at its core, what makes bert so powerful are the pre-training tasks, i.e., mask language modeling (mlm) and next sentence prediction (nsp), where the former is more important than the latter. since bert, a series of improvements have been proposed. the first branch of literature improves the model architecture of bert. albert @xcite makes bert more light-weighted by embedding factorization and progressive cross-layer parameter sharing. zaheer et al. (2020) improve bert's performance on longer sequences by employing sparser attention. the second branch of literature improves the training of bert. @xcite stabilize and improve the training of bert with a larger corpus. more work has focused on new language pre-training tasks. albert @xcite introduce sentence order prediction (sop). struct-bert @xcite designs two novel pretraining tasks, word structural task and sentence structural task, to learn better representations of tokens and sentences. ernie 2.0 @xcite proposes a series of pre-training tasks and applies continual learning to incorporate these tasks. electra @xcite has a gan-style pre-training task for efficiently utilizing all tokens in pre-training. our work is closely related to this literature branch by designing a series of novel pretraining objectives by incorporating multiple vocabularies. our proposed method is off-the-shelf and can be easily incorporated with other pre-training tasks. another branch of literature looks into the role of words in pre-training. although not mentioned in @xcite , the authors propose whole word masking in their open-source repository, which is effective for pre-training bert. in span-bert @xcite ), text spans are masked in pre-training, and the learned model can substantially enhance the performances of span selection tasks. it is indicated that word segmentation is vital for chinese plms. cui et al. (2019) and sun et al. (2019) both show that masking tokens in the units of natural chinese words instead of single chinese characters can significantly improve chinese plms. liu et al. ( 2019 ) apply cws to build a vocab that can improve chinese-english translation performance. ambert @xcite propose to leverage vocabs of different granularity in encoding sentences and improve the pre-training. in this work, compared to literature, our contributions are: (a) we find that cws and sub-word tokenization can improve the pre-trained models' performances on downstream tasks. (b) we propose mvp pre-training tasks, which are proven to improve the expressiveness of pre-trained models and downstream performances. ## our methods this section presents our methods for rebuilding the vocab for chinese plms and introducing our series of mvp strategies. ## conclusions in this work, we propose a series of novel pretraining methods called mvps, which leverage multiple vocabularies in the language model pretraining. to select the vocabs for mvp pre-training, we first conduct experiments to validate sgt, which combines chinese word segmentation and sub-word tokenization, works best for the chinese language model pre-training. we then use experiments to show that our proposed mvp methods can achieve better performances than ambert with less computational resources. also, we show our mvp method can improve the pre-trained model's robustness against adversarial attacks.