“Is Whole Word Masking Always Better for C hinese BERT ?”: Probing on C hinese Grammatical Error Correction (2022)

Abstract: Whole word masking (WWM), which masks all subwords corresponding to a word at once, makes a better English BERT model. For the Chinese language, however, there is no subword because each token is an atomic character. The meaning of a word in Chinese is different in that a word is a compositional unit consisting of multiple characters. Such difference motivates us to investigate whether WWM leads to better context understanding ability for Chinese BERT. To achieve this, we introduce two probing tasks related to grammatical error correction and ask pretrained models to revise or insert tokens in a masked language modeling manner. We construct a dataset including labels for 19,075 tokens in 10,448 sentences. We train three Chinese BERT models with standard character-level masking (CLM), WWM, and a combination of CLM and WWM, respectively. Our major findings are as follows: First, when one character needs to be inserted or replaced, the model trained with CLM performs the best. Second, when more than one character needs to be handled, WWM is the key to better performance. Finally, when being fine-tuned on sentence-level downstream tasks, models trained with different masking strategies perform comparably.

## introduction bert @xcite is a transformer-based pretrained model, whose prosperity starts from english language and gradually spreads to many other languages. the original bert model is trained with character-level masking (clm). 1 a certain percentage (e.g. 15%) of tokens in the input se-quence is masked and the model is learned to predict the masked tokens. it is helpful to note that a word in the input sequence of bert can be broken into multiple wordpiece tokens @xcite . 2 for example, the input sentence "she is undeniably brilliant" is converted to a wordpiece sequence "she is un ##deni ##ably brilliant", where "##" is a special prefix added to indicate that the token should be attached to the previous one. in this case the word "undeniably" is broken into three wordpieces {"un", "##deni", "##ably"}. in standard masked language modeling, clm may mask any one of them. in this case, if the token "##ably" is masked, it is easier for the model to complete the prediction task because "un" and "##deni" are informative prompts. to address this, whole word masking (wwm) masks all three subtokens (i.e., {"un", "##deni", "##ably"}) within a word at once. for chinese, however, each token is an atomic character that cannot be broken into smaller pieces. many chinese words are compounds that consisting of multiple characters @xcite . 3 for example, "手机" (cellphone) is a word consisting of two characters "手" (hand) and "机" (machine). here, learning with wwm would lose the association among characters corresponding to a word. in this work, we introduce two probing tasks to study chinese bert model's ability on characterlevel understanding. the first probing task is character replacement. given a sentence and a position where the corresponding character is erroneous, the task is to replace the erroneous character with the correct one. the second probing task is character insertion. given a sentence and the positions where a given number of characters should be inserted, the task is to insert the correct characters. we leverage the benchmark dataset on grammatical error correction @xcite and create a dataset including labels for 19,075 tokens in 10,448 sentences. we train three baseline models based on the same text corpus of 80b characters using clm, wwm, and both clm and wwm, separately. we have the following major findings. (1) when one character needs to be inserted or replaced, the model trained with clm performs the best. moreover, the model initialized from roberta @xcite and trained with wwm gets worse gradually with more training steps. (2) when more than one character needs to be handled, wwm is the key to better performance. (3) when evaluating sentence-level downstream tasks, the impact of these masking strategies is minimal and the model trained with them performs comparably. ## our probing tasks in this work, we present two probing tasks with the goal of diagnosing the language understanding ability of chinese bert models. we present the tasks and dataset in this section. the first probing task is character replacement, which is a subtask of grammatical error correction. given a sentence @xmath0, x 2 , ..., x i , ..., x n } of n characters and an erroneous span @xmath1, i + 1, ..., i + k] of k characters, the task is to replace es with a new span of k characters. the second probing task is character insertion, which is also a subtask of grammatical error correction. given a sentence @xmath2, x 2 , ..., x i , ..., x n } of n characters, a position i, and a fixed number k, the task is to insert a span of k characters between the index i and i + 1. we provide two examples of these two probing tasks with @xmath3. for the character replacement task, the original meaning of the sentence is "these are all my ideas". due to the misuse of a character at the 7th position, its meaning changed significantly to "these are all my attention". our character replacement task is to replace the misused character "主" with "注". for the character insertion task, what the writer wants to express is "human is the most important factor. however, due to the lack of one character between the 5th and 6th position, its meaning changed to "human is the heaviest factor". the task is to index: 1 2 3 4 5 6 7 8 9 index: 1 2 3 4 5 6 7 8 input: 人 类 是 最 重 的 因 素 (en: human is the heaviest factor.) (en: human is the most important factor.) ## experiments in this section, we first describe the bert-style models that we examined, and then report numbers. ## related work we describe related studies on chinese bert model and probing of bert, respectively. the authors of bert @xcite provided the first chinese bert model which was trained on chinese wikipedia data. on top of that, @xcite trained roberta-wwm-ext with wwm on extended data. @xcite further trained a chinese electra model and macbert, both of which did not have [mask] tokens. elec-tra was trained with a token-level binary classification task, which determined whether a token was the original one or artificially replaced. in macbert, [mask] tokens were replaced with synonyms and the model was trained with wwm and ngram masking. ernie @xcite was trained with entity masking, similar to wwm yet tokens corresponding to an entity were masked at once. language features are considered in more recent works. for example, ambert @xcite and lattice-bert @xcite both take word information into consideration. chinese-bert @xcite utilizes pinyin and glyph of characters. probing aims to examine the language understanding ability of pretrained models like bert when model parameters are clamped, i.e., without being fine-tuned on downstream tasks. @xcite study how well pretrained models learn factual knowledge. the idea is to design a natural language template with a [mask] token, such as "the wife of barack obama is @xcite .". if the model predicts the correct answer "micheal obama", it shows that pretrained models learn factual knowledge to some extent. similarly, @xcite study how pretrained models learn commonsense knowledge and @xcite examine on tasks that require symbolic understanding. @xcite propose to probe chinese bert models in terms of linguistic and world knowledge. ## conclusion in this work, we present two chinese probing tasks, including character insertion and replacement. we provide three simple pretrained models dubbed ours-clm, ours-wwm, and ours-clm-wwm, which are pretrained with clm, wwm, and a combination of clm and wwm, respectively. ours-wwm is prone to lose the association between words and result in poor performance on probing tasks when one character needs to be inserted or replaced. moreover, wwm plays a key role when two or more characters need to be corrected.