AMBERT: A P RE - TRAINED L ANGUAGE M ODEL WITH M ULTI -G RAINED T OKENIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Pre-trained language models such as BERT have exhibited remarkable performances in many tasks in natural language understanding (NLU).
The tokens in the models are usually ﬁne-grained in the sense that for languages like English they are words or sub-words and for languages like Chinese they are characters.
In English, for example, there are multi-word expressions which form natural lexical units and thus the use of coarse-grained tokenization also appears to be reasonable.
In fact, both ﬁne-grained and coarse-grained tokenizations have advantages and disadvantages for learning of pre-trained language models.
In this paper, we propose a novel pre-trained language model, referred to as AMBERT (A Multi-grained BERT), on the basis of both ﬁne-grained and coarse-grained tokenizations.
For English, AMBERT takes both the sequence of words (ﬁnegrained tokens) and the sequence of phrases (coarse-grained tokens) as input after tokenization, employs one encoder for processing the sequence of words and the other encoder for processing the sequence of the phrases, utilizes shared parameters between the two encoders, and ﬁnally creates a sequence of contextualized representations of the words and a sequence of contextualized representations of the phrases.
Experiments have been conducted on benchmark datasets for Chinese and English, including CLUE, GLUE, SQuAD and RACE.
The results show that AMBERT outperforms the existing best performing models in almost all cases, particularly the improvements are signiﬁcant for Chinese.
We also develop a version of AMBERT which performs equally well as AMBERT but uses about half of its inference time.
I NTRODUCTION
Pre-trained models such as BERT, RoBERTa, and ALBERT (Devlin et al., 2018; Liu et al., 2019; Lan et al., 2019) have shown great power in natural language understanding (NLU).
The Transformerbased language models are ﬁrst learned from a large corpus in pre-training, and then learned from labeled data of a downstream task in ﬁne-tuning.
With Transformer (Vaswani et al., 2017), pre-training technique, and big data, the models can effectively capture the lexical, syntactic, and semantic relations between the tokens in the input text and achieve the state-of-the-art performances in many
NLU tasks, such as sentiment analysis, text entailment, and machine reading comprehension.
In BERT, for example, pre-training is mainly conducted based on mask language modeling (MLM) in which about 15% of the tokens in the input text are masked with a special token [MASK], and the goal is to reconstruct the original text from the masked text.
Fine-tuning is separately performed for individual tasks as text classiﬁcation, text matching, text span detection, etc.
Usually, the tokens in the input text are ﬁne-grained; for example, they are words or sub-words in English and characters in Chinese.
In principle, the tokens can also be coarse-grained, that is, for example, phrases in
English and words in Chinese.
There are many multi-word expressions in English such as ‘New
York’ and ‘ice cream’ and the use of phrases also appears to be reasonable.
It is more sensible to use words (including single character words) in Chinese, because they are basic lexical units.
In fact, all existing pre-trained language models employ single-grained (usually ﬁne-grained) tokenization.
Previous work indicates that the ﬁne-grained approach and the coarse-grained approach have both pros and cons.
The tokens in the ﬁne-grained approach are less complete as lexical units but their representations are easier to learn (because there are less token types and more tokens in training data), while the tokens in the coarse-grained approach are more complete as lexical units but their representations are more difﬁcult to learn (because there are more token types and less tokens in training data).
Moreover, for the coarse-grained approach there is no guarantee that tokenization (segmentation) is completely correct.
Sometimes ambiguity exists and it would be better to retain all possibilities of tokenization.
In contrast, for the ﬁne-grained approach tokenization is carried out at the primitive level and there is no risk of ‘incorrect’ tokenization.
For example, Li et al.
(2019) observe that ﬁne-grained models consistently outperform coarsegrained models in deep learning for Chinese language processing.
They point out that the reason is that low frequency words (coarse-grained tokens) tend to have insufﬁcient training data and tend to be out of vocabulary, and as a result the learned representations are not sufﬁciently reliable.
On the other hand, previous work also demonstrates that masking of coarse-grained tokens in pre-training of language models is helpful (Cui et al., 2019; Joshi et al., 2020).
That is, although the model itself is ﬁne-grained, masking on consecutive tokens (phrases in English and words in Chinese) can lead to learning of a more accurate model.
In Appendix A, we give examples of attention maps in BERT to further support the assertion.
In this paper, we propose A Multi-grained BERT model (AMBERT), which employs both ﬁnegrained and coarse-grained tokenizations.
For English, AMBERT extends BERT by simultaneously constructing representations for both words and phrases in the input text using two encoders.
Specifically, AMBERT ﬁrst conducts tokenization at both word and phrase levels.
It then takes the embeddings of words and phrases as input to the two encoders.
It utilizes the same parameters across the two encoders.
Finally it obtains a contextualized representation for the word and a contextualized representation for the phrase at each position.
Note that the number of parameters in AMBERT is comparable to that of BERT, because the parameters in the two encoders are shared.
There are only additional parameters from multi-grained embeddings.
AMBERT can represent the input text at both word-level and phrase-level, to leverage the advantages of the two approaches of tokenization, and create richer representations for the input text at multiple granularity.
We conduct extensive experiments to make comparison between AMBERT and the baselines as well as alternatives to AMBERT, using the benchmark datasets in English and Chinese.
The results show that AMBERT signiﬁcantly outperforms single-grained BERT models with a large margin in both Chinese and English.
In English, compared to Google BERT, AMBERT achieves 2.0% higher
GLUE score, 2.5% higher RACE score, and 5.1% more SQuAD score.
In Chinese, AMBERT improves average score by over 2.7% in CLUE.
Furthermore, a simpliﬁed version AMBERT with only the ﬁne-grained encoder can preform much better than the single-grained BERT models with a similar amount of inference computation.
We make the following contributions in this work.
• Study of multi-grained pre-trained language models, • Proposal of a new pre-trained language model called AMBERT as an extension of BERT, which makes use of multi-grained tokens and shared parameters, • Empirical veriﬁcation of AMBERT on the English and Chinese benchmark datasets GLUE, SQuAD, RACE, and CLUE.
R ELATED WORK
There has been a large amount of work on pre-trained language models.
ELMo (Peters et al., 2018) is one of the ﬁrst pre-trained language models for learning of contextualized representations of words in the input text.
Leveraging the power of Transformer (Vaswani et al., 2017), GPTs (Radford et al., 2018; 2019) are developed as unidirectional models to make prediction on the input text in an autoregressive manner, and BERT (Devlin et al., 2018) is developed as a bidirectional model to make prediction on the whole or part of the input text.
Mask language modeling (MLM) and next sentence prediction (NSP) are the two tasks in pre-training of BERT.
Since the inception of BERT, a number of new models have been proposed to further enhance the performance of it.
XLNet
(Yang et al., 2019) is a permutation language model which can improve the accuracy of MLM. RoBERTa
(Liu et al., 2019) represents a new way of training more reliable BERT with a very large amount of data.
ALBERT (Lan et al., 2019) is a light-weight version of BERT, which shares parameters across layers.
StructBERT (Wang et al., 2019) incorporates word and sentence structures into BERT for learning of better representations of tokens and sentences.
ERNIE2.0 (Sun et al., 2020) is a variant of BERT pre-trained in multiple tasks with coarse-grained tokens masked.
ELECTRA (Clark et al., 2020) has a GAN-style architecture for efﬁciently utilizing all tokens in pre-training.
It has been found that the use of coarse-grained tokens is beneﬁcial for pre-trained language models.
Devlin et al.
(2018) point out that ‘whole word masking’ is effective for training of BERT.
It is also observed that whole word masking is useful for building a Chinese BERT (Cui et al., 2019).
In
ERNIE (Sun et al., 2019b), entity level masking is employed as a strategy for pre-training and proved to be effective for language understanding tasks (see also (Zhang et al., 2019)).
In SpanBERT (Joshi et al., 2020), text spans are masked in pre-training and the learned model can substantially enhance the accuracies of span selection tasks.
It is indicated that word segmentation is especially important for Chinese and a BERT-based Chinese text encoder is proposed with n-gram representations (Diao et al., 2019).
All existing work focuses on the use of single-grained tokens in learning and utilization of pre-trained language models.
In this work, we propose a general technique of exploiting multigrained tokens for pre-trained language models and apply it to BERT.
O UR M ETHOD : AMBERT
In this section, we present the model, pre-training, and ﬁne-tuning of AMBERT.
We also make a discussion on alternatives of AMBERT.
[Figure]
of creating multi-grained representations.
The input is a sentence in English and output is the overall representation of the sentence.
There are two encoders for processing the sequence of ﬁne-grained tokens and the sequence of coarsegrained tokens respectively.
The ﬁnal contextualized representations of ﬁne-grained tokens and coarse-grained tokens are denoted as r x 0 , r x 1 , · · · , r xm and r z 0 , r z 1 , · · · , r zn respectively.
Figure 1 gives an overview of AMBERT.
AMBERT takes a text as input.
Tokenization is conducted on the input text to obtain a sequence of ﬁne-grained tokens and a sequence of coarse-grained tokens.
AMBERT has two encoders, one for processing the ﬁne-grained token sequence and the other for processing the coarse-grained token sequence.
Each of the encoders has exactly the same architecture as that of BERT (Devlin et al., 2018) or Transformer encoder (Vaswani et al., 2017).
The two encoders share the same parameters at each corresponding layer, except that each has its own token embedding parameters.
The ﬁne-grained encoder generates contextualized representations from the sequence of ﬁne-grained tokens through its layers.
In parallel, the coarse-grained encoder generates contextualized representations from the sequence of coarse-grained tokens through its layers.
AMBERT outputs a sequence of contextualized representations for the ﬁne-grained tokens and a sequence of contextualized representations for the coarse-grained tokens.
AMBERT is expressive in that it learns and utilizes contextualized representations of the input text at both ﬁne-grained and coarse-grained levels.
The model retains all possibilities of tokenizations and automatically learns the attention weights (importance) of representations of multi-grained tokens.
AMBERT is also efﬁcient through sharing of parameters between the two encoders.
The parameters represent the same ways of combining representations, no matter whether representations are those of ﬁne-grained tokens or coarse-grained tokens.
3. P RE -T RAINING
Pre-training of AMBERT is mainly conducted on the basis of mask language modeling (MLM), at both ﬁne-grained and coarse-grained levels.
Next sentence prediction (NSP) is not essential as indicated in many studies after BERT (Lan et al., 2019; Liu et al., 2019).
We only use NSP in our experiments for comparison purposes).
Let ˆ x denote the sequence of ﬁne-grained tokens with some of them being masked, and ¯ x denote the masked ﬁne-grained tokens.
Let ˆ z denote the sequence of coarse-grained tokens with some of them being masked, and ¯ z denote the masked coarse-grained tokens.
Pre-training is deﬁned as optimization of the following function,
[Equation]
where m i takes 1 or 0 as values and m i = 1 indicates that ﬁne-grained token x i is masked, m denotes the total number of ﬁne-grained tokens; n j takes 1 or 0 as values and n j = 1 indicates that coarse-grained token z j is masked, n denotes the total number of coarse-grained tokens; and θ denotes parameters.
3. F INE -T UNING
In ﬁne-tuning of AMBERT for classiﬁcation, the ﬁne-grained encoder and coarse-grained encoder create special [CLS] representations, and both representations are used for classiﬁcation.
Finetuning is deﬁned as optimization of the following function, which is a regularized loss of multi-task learning, starting from the pre-trained model,
[Equation]
where x is the input text, y is the classiﬁcation label, r x 0 and r z 0 are the [CLS] representations of ﬁne-grained encoder and coarse-grained encoder, [ a , b ] denotes concatenation of vectors a and b ,
λ is coefﬁcient, and (cid:107)(cid:107) 2 denotes L2 norm.
The last term is based on agreement regularization (Brantley et al., 2019), which forces agreement between the predictions ( ˜ y x and ˜ y z ).
Similarly, ﬁne-tuning of AMBERT for span detection can be carried out, in which the representations of ﬁne-grained tokens are concatenated with the representations of corresponding coarse-grained tokens.
The concatenated representations are then utilized in the task.
3. A LTERNATIVES
We can consider two alternatives to AMBERT, which also rely on multi-grained tokenization.
We refer to them as AMBERT-Combo and AMBERT-Hybrid and make comparisons of them with AMBERT in our experiments.
AMBERT-Combo has two individual encoders, an encoder (BERT) working on the ﬁne-grained token sequence and the other encoder (BERT) working on the coarse-grained token sequence, without parameter sharing between them.
In learning and inference AMBERT-Combo simply combines the output layers of the two encoders.
Its ﬁne-tuning is similar to that of AMBERT.
AMBERT-Hybrid has only one encoder (BERT) working on both the ﬁne-grained token sequence and the coarse-grained token sequence.
It creates representations on the concatenation of two sequences and lets the representations of the two sequences interact with each other at each layer.
Its pre-training is formalized in the following function,
[Equation]
where the notations are the same as in (1).
Its ﬁne-tuning is the same as that of BERT.
E XPERIMENTS
We make comparisons between AMBERT and the baselines including ﬁne-grained BERT and coarse-grained BERT, as well as the alternatives including AMBERT-Combo and AMBERT-Hybrid, using benchmark datasets in both Chinese and English.
The experiments on the alternatives can also be seen as ablation study on AMBERT.
4. D ATA FOR P RE -T RAINING
For Chinese, we use a corpus consisting of 25 million documents (57G uncompressed text) from
Jinri Toutiao 1 .
Note that there is no common corpus for training of Chinese BERT.
For English, we use a corpus of 13.9 million documents (47G uncompressed text) from Wikipedia and OpenWebText (Gokaslan & Cohen, 2019).
Unfortunately, BookCorpus, one of the two corpora in the original paper for English BERT, is no longer publicly available.
The characters in the Chinese texts are naturally taken as ﬁne-grained tokens.
We conduct word segmentation on the texts and treat the words as coarse-grained tokens.
We employ a word segmentation tool based on a n-gram model.
Both tokenizations exploit WordPiece embeddings (Wu et al., 2016).
There are 21,128 characters and 72,635 words in the vocabulary of Chinese.
The words in the English texts are naturally taken as ﬁne-grained tokens.
We perform coarse-grained tokenization on the English texts in the following way.
Speciﬁcally, we ﬁrst calculate the n-grams in the Wikipedia documents using KenLM (Heaﬁeld, 2011).
We next build a phrase-level dictionary consisting of phrases whose frequencies are sufﬁciently high and whose last words highly depend on their previous words.
We then employ a left-to-right search algorithm to perform phrase-level tokenization on the texts.
There are 30,522 words and 77,645 phrases in the vocabulary of English.
4. E XPERIMENTAL SETUP
We make use of the same parameter settings for the AMBERT and BERT models.
All models in this paper are ‘base-models’ having 12 layers of encoder.
It is too computationally expensive for us to train the models as ‘large models’ having 24 layers.
The hyper-parameters are basically the same as those in the original BERT paper (Devlin et al., 2018), which are given in Appendix C.
The optimizer is Adam (Kingma & Ba, 2014).
To enhance efﬁciency, we use mixed precision for all the models.
Training is carried out on Nvidia V-100.
The numbers of GPUs used for training are from 32 to 64, depending on the model sizes.
In pre-training of the AMBERT models, in total 15% of the coarse-grained tokens are masked, which is the same proportion for the BERT models.
To retain consistency, the masked coarse-grained tokens are also masked as ﬁne-grained tokens.
In ﬁne-tuning, we use the same hyper-parameters as those in the original papers of the baselines, and all the hyper-parameters are given in Appendix C.
4. C HINESE T ASKS 4.3. B ENCHMARKS
We use the benchmark datasets, Chinese Language Understanding Evaluation (CLUE)
(Xu et al., 2020) for experiments in Chinese.
CLUE contains six classiﬁcation tasks, that are TNEWS,
IFLYTEK and CLUEWSC2020, AFQMC, CSL and CMNLI 2 , and three reading comprehension tasks which are CMRC2018, ChID and C 3 .
The details of all the benchmarks are shown in Appendix B. Data augmentation is also performed for all models in the tasks of TNEWS, CSL and
CLUEWSC2020 to achieve better performances (see Appendix D for detailed explanation).
4.3. E XPERIMENTAL R ESULTS
We compare AMBERT with the BERT baselines, including the BERT model released from Google, referred to as Google BERT, and the BERT model trained by us, referred to as Our BERT, including character based (ﬁne-grained) and word based (coarse-grained) models.
Case study in Appendix E.
Table 1 shows the results of the classiﬁcation tasks.
AMBERT improves average scores of the
BERT baselines by about 1.0% and also works better than AMBERT-Combo and AMBERT-Hybrid.
The results of Machine Reading Comprehensive (MRC) tasks are shown in Table 2.
AMBERT improves average scores of the BERT baselines by over 3.0%.
Our BERT (word) performs poorly in
CMRC2018.
This is probably because the results of word segmentation are not accurate enough for the task.
AMBERT-Combo and AMBERT-Hybrid are on average better than single-grained BERT models.
AMBERT further outperforms both of them.
[Table] [Table] We also compare AMBERT with the state-of-the-art models at the leader board of CLUE 3 .
The base models, whose parameters are fewer than 200M, are trained with different datasets and procedures
[Table]
and thus the comparisons should only be taken as references.
Note that the settings of the base models are the same as that of Xu et al.
Table 3 shows the results.
The average score of
AMBERT is higher than all the other models.
We conclude that multi-grained tokenization is very helpful for pre-trained language models and the design of AMBERT is reasonable
[Table]
[Table]
4. E NGLISH T ASKS 4.4. B ENCHMARKS
The General Language Understanding Evaluation (GLUE) benchmark (Wang et al., 2018) is a collection of nine NLU tasks.
Following BERT (Devlin et al., 2018), we exclude the task WNLI for the reason that results of different models on this task are undifferentiated.
In addition, three machine reading comprehensive tasks are also included, i.e., SQuAD v1.1, SQuAD v2.0, and RACE.
The details of English benchmarks can be found in Appendix B.
4.4. E XPERIMENTAL R ESULTS
We compare AMBERT with the BERT models on the tasks in GLUE.
The results of Google BERT are from the original paper (Devlin et al., 2018), and the results of Our BERT are obtained by us.
From Table 4 we can see the following trend. 1) Multi-grained models particularly AMBERT can achieve better results than single-grained models.
Among the multi-grained models, AMBERT performs best with fewer parameters and less computation.
Case study is given in Appendix E.
We also make comparison on the SQuAD tasks.
The results of Google BERT are either from the papers (Devlin et al., 2018;
Yang et al., 2019) or from our runs with the ofﬁcial code.
From Tablewe make the following conclusions.
1) in SQuAD, AMBERT outperforms Google BERT with a large margin.
Our BERT (word) generally performs well and Our BERT (phrase) performs poorly in the span detection tasks.
In RACE, AMBERT performs best among all the baselines for both development set and test set.
3) AMBERT is the best multi-grained model. [Table]
[Table] We compare AMBERT with the state-of-the-art models in both GLUE 4 and MRC.
The results of baselines, in Table
[Table]
are either reported in published papers or re-implemented by us with HuggingFace’s Transformer (Wolf et al., 2019).
We use the provided implementation in HuggingFace’s Transformer, without additional data augmentation or question-answering module 5 . Note that AMBERT outperforms all the models on average 6 without using training techniques such as bigger batches and dynamic masking.
[Table]
Although AMBERT can make signiﬁcant improvements over single-grained models, the computation time is doubled. To enhance the inference speed of AMBERT, we develop a simpliﬁed version of it, referred to as AMBERT-Single, in which the two encoders are pre-trained and ﬁne-tuned in learning and only the single-grained encoder is utilized in inference.
Note that it has the same amount of inference computation as BERT. We conduct experiments on CLUE/GLUE/SQuADs/RACE with AMBERT-Single.
The results on the development sets are shown in Table 7.
We conclude that, a) for the English tasks, AMBERT-Single achieves similar results as AMBERT and outperforms “Our BERT (Single)” with a large margin using the same amount of inference time; b) for the Chinese tasks, AMBERT-Single is slightly worse than AMBERT and performs much better than “Our BERT (Single)”.
Therefore, in practice, one can train an
AMBERT with two encoders and use only one of them in inference, i.e., AMBERT-Single.
[Table]
4. R EGULARIZATION IN F INE - TUNING
Table 8 shows the results of using different values as regularization coefﬁcients in ﬁne-tuning on the development sets of CLUE, GLUE and RACE.
It appears that for most tasks the use of regularization is necessary.
For simplicity, we did not use the best value of coefﬁcient for each task and instead we adopt 0.0 for RACE and 1.0 for the other tasks
[Table]
[Table]
4. D ISCUSSIONS
We further investigate the reason that AMBERT is superior to AMBERT-Combo.
Figure 2 shows the distances between the [CLS] representations of the ﬁne-grained encoder and coarse-grained encoder in AMBERT-Combo and AMBERT after pre-training, in terms of cosine dissimilarity (one minus cosine similarity) and normalized Euclidean distance.
One can see that the distances in AMBERTCombo are larger than the distances in AMBERT in the tasks.
We perform the assessment using the data in the other tasks and ﬁnd similar trends.
The results indicate that the representations of ﬁne-grained encoder and coarse-grained encoder are closer in AMBERT than in AMBERT-Combo.
These are natural consequences of using AMBERT and AMBERT-Combo, whose parameters are respectively shared and unshared across encoders.
It implies that the higher performances by AMBERT is due to its parameter sharing, which can use less parameters to learn and represent similar ways of combining tokens no matter whether they are ﬁne-grained or coarse-grained.
An intuitive explanation is that the ways of combining representations of ﬁne-grained tokens and the ways of combining representations of coarse-grained tokens “in the same contexts” are exactly the same.
[Table] We also examine the reasons that AMBERT works better than AMBERT-Hybrid, while both of them exploit multi-grained tokenization.
Figure 3 shows the attention weights of ﬁrst layers in AMBERT and AMBERT-Hybrid, as well as the single-grained BERT models, after pre-training.
In AMBERTHybrid, the ﬁne-grained tokens attend more to the corresponding coarse-grained tokens and as a result the attention weights among ﬁne-grained tokens are weakened.
In contrast, in AMBERT the attention weights among ﬁne-grained tokens and those among coarse-grained tokens are intact.
It appears that attentions among single-grained tokens (ﬁne-grained ones and coarse-grained ones) play important roles in downstream tasks.
[Figure]
To answer the question why the improvements by AMBERT on Chinese are larger than on English in the same pre-training settings, we further make an analysis.
We tokenize 10,000 randomly selected
Chinese sentences in CLUE with our Chinese word tokenizer.
As shown in Table 9, the average proportion of words is 51.5%, which indicates that about half of the tokens are ﬁne-grained and half are coarse-grained in Chinese.
We also tokenize 10,000 randomly selected English sentences in GLUE with our English phrase tokenizer.
The average proportion of phrases is only 13.1%, which means that there are much less coarse-grained tokens than ﬁne-grained tokens in English.
Therefore, we postulate that for Chinese it is necessary for a model to process the language at both ﬁne-grained and coarse-grained levels.
AMBERT indeed has the capability.
[Figure]
[Table]
[Table]
TNEWS/
[Table] [Table] [Table] [Table]
TNEWS/
[Table] [Table]  C ONCLUSION In this paper, we have proposed a novel pre-trained language model called AMBERT, as an extension of BERT.
AMBERT employs multi-grained tokenization, that is, it uses both words and phrases in English and both characters and words in Chinese.
With multi-grained tokenization, AMBERT learns in parallel the representations of the ﬁne-grained tokens and the coarse-grained tokens using two encoders with shared parameters.
Experimental results have demonstrated that AMBERT signiﬁcantly outperforms BERT and other models in NLU tasks in both English and Chinese.
AMBERT increases average score of Google BERT by about 2.7% in Chinese benchmark CLUE.
AMBERT improves Google BERT by over 3.0% on a variety of tasks in English benchmarks GLUE, SQuAD (1.1 and 2.0), and RACE.
We also develop AMBERT-Simple which performs equally well as AMBERT with about half of inference time.
As future work, we plan to study the following issues: 1) to investigate model acceleration methods in learning of AMBERT, such as sparse attention (Child et al., 2019; Kitaev et al., 2020; Zaheer et al., 2020) and synthetic attention (Tay et al., 2020); 2) to apply the technique of AMBERT into other pre-trained language models such as XLNet; 3) to employ AMBERT in other NLU tasks.
A A TTENTION MAPS FOR SINGLE - GRAINED MODELS
We construct ﬁne-grained and coarse-grained BERT models for English and Chinese, and examine the attention maps of the models using the BertViz tool (Vig, 2019).
Figure A shows the attention maps of the ﬁrst layer of ﬁne-grained models for several sentences in English and Chinese.
One can see that there are tokens that improperly attend to other tokens in the sentences.
For example, in the English sentences, the words “drawing”, “new” and “dog” have high attention weights to “portrait”, “york”, and “food”, respectively, which are not appropriate.
For example, in the Chinese sentences, the chars “ 拍 ”, “ 北 ”, “ 长 ” have high attention weights to “ 卖 ”, “ 京 ”, “ 市 ”, respectively, which are also not reasonable.
(It is veriﬁed that the bottom layers at BERT mainly represent lexical information, the middle layers mainly represent syntactic information, and the top layers mainly represent semantic information (Jawahar et al., 2019).)
Ideally a token should only attend to the tokens with which they form a lexical unit at the ﬁrst layer.
This cannot be guaranteed in the ﬁnegrained BERT model, however, because usually a ﬁne-grained token may belong to multiple lexical units (i.e., there is ambiguity).
Figure 5 shows the attention maps of the ﬁrst layer of coarse-grained models for the same sentences in English and Chinese.
In the English sentences, the words are combined into the phrases of “drawing room”, “york minister”, and “dog food”.
The attentions are appropriate in the ﬁrst two sentences, but it is not in the last sentence because of the incorrect tokenization.
Similarly, in the Chinese sentences, the high attention weights of words “ 球 拍 (bat)” and “ 京 城 (capital)” are reasonable, but that of word “ 市 长 (mayor)” is not.
Note that incorrect tokenization is inevitable.
B D ETAILED DESCRIPTIONS FOR THE BENCHMARKS B. C HINESE T ASKS
TNEWS is a text classiﬁcation task in which titles of news articles in TouTiao are to be classiﬁed into 15 classes.
IFLYTEK is a task of assigning app descriptions into 119 categories.
CLUEWSC2020, standing for the Chinese Winograd Schema Challenge, is a co-reference resolution task.
AFQMC is a binary classiﬁcation task that aims to predict whether two sentences are semantically similar.
CSL uses the Chinese Scientiﬁc Literature dataset containing abstracts and their keywords of papers and the goal is to identify whether given keywords are the original keywords of a paper.
CMNLI is based on translation from MNLI (Williams et al., 2017), which is a large-scale, crowd-sourced entailment classiﬁcation task.
CMRC
(Cui et al., 2018) makes use of a span-based dataset for
Chinese machine reading comprehension.
ChID (Zheng et al., 2019) is a large-scale Chinese IDiom cloze test.
C 3 (Sun et al., 2019a) is a free-form multiple-choice machine reading comprehension for
Chinese.
B. E NGLISH T ASKS
CoLA (Warstadt et al., 2019) contains English acceptability judgments drawn from books and journal articles on linguistic theory.
SST-2 (Socher et al., 2013) consists of sentences from movie reviews and human annotations of their sentiment.
MRPC (Dolan & Brockett, 2005) is a corpus of sentence pairs automatically extracted from online news sources, and the target is to identify whether a sentence pair is semantically equivalent.
STS-B (Cer et al., 2017) is a collection of sentence pairs and the task is to predict similarity scores.
QQP is a collection of question pairs and requires models to recognize semantically equivalent ones.
MNLI (Williams et al., 2017) is a crowd-sourced collection of sentence pairs with textual entailment annotations.
QNLI (Wang et al., 2018) is a question-answering dataset consisting of question-paragraph pairs, where one of the sentences in the paragraph contains the answer to the corresponding question.
RTE (Bentivogli et al., 2009) comes from a series of annual textual entailment challenges.
C H YPER - PARAMETERS C. H YPER - PARAMETERS IN PRE - TRAINING
We adopt the standard hyper-parameters of BERT in pre-training of the models except batch sizes which are tuned to make our ﬁne-grained BERT models comparable to the Google BERT models.
Table 10 shows the hyper-parameters in our Chinese AMBERT and English AMBERT.
Our BERT models and alternatives of AMBERT (AMBERT-Combo and AMBERT-Hybrid) all use the same hyper-parameters in pre-training.
[Table]
[Figure]
[Table] [Figure] [Table]
[Figure]
C. H YPER - PARAMETERS IN F INE - TUNING
For the Chinese tasks, since all the original papers do not report detailed hyper-parameters in ﬁnetuning of the baseline models
[Figure]
we uniformly use the same hyper-parameters as shown in Tableexcept training epoch, because AMBERT and AMBERT-Combo have more parameters and need more training to get converged.
We choose the training epochs for all models when the performances on development sets stop to improve.
As for the English tasks
[Figure]
Table 12 shows all the hyper-parameters in ﬁne-tuning of the models.
We adopt the best hyper-parameters in the original papers for the baselines.
Moreover, for AMBERT ‡ , we also tune learning rate ([1e-5, 2e-5, 3e-5]) and batch size ([16, 32]) for GLUE with the same method in RoBERTa (Liu et al., 2019).
[Table]
D D ATA A UGMENTATION
To enhance the performance [Table] we conduct data augmentation for the three Chinese classiﬁcation tasks of TNEWS, CSL, and CLUEWSC2020.
In TNEWS, we use both keywords and titles.
In CSL, we concatenate keywords with a special token “ ”.
In CLUEWSC2020, we duplicate a few instances having pronouns in the training data such as “ 她 (she)”.
[Table]
E C ASE STUDY
WORD/PHRASE represents Our BERT word/phrase.
In English the tokens in the same phrase are concatenated with “ ”, and in Chinese phrases are split with “/”
[Table]
[Table]
[Table]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Table]
[Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Table]
来 / ， / 是 / 因 为 / 现 实 /
[Table]
平 / 淡 /
[Table]
[Table]
现 实 是 如 此 平 淡 ， 以 致 于 虚 拟 现 实 技 术 业 务 得 到 了 刺 激 。 ( 现 实 / 是 / 如 此 / 平 淡 / ， / 以 致 / 于 / 虚 拟 / 现 实 / 技 术 / 业 务 / 得 到 / 了 / 刺 激 / 。 )
We also qualitatively study the results of BERT and AMBERT, and ﬁnd that they support our claims (cf., Section 1) very well.
Here, we give some random examples from the entailment tasks (QNLI and CMNLI) in Table 13.
One can have the following observations.
The ﬁne-grained models (e.g.,
Our BERT word) cannot effectively use complete lexical units such as “Doctor Who” and “ 打 死 ” (sentence pairs 1 and 5), which may result in incorrect predictions.
The coarse-grained models (e.g., Our BERT phrase), on the other hand, cannot effectively deal with incorrect tokenizations, for example, “the blind” and “ 格 式 ” (sentence pairs 2 and 6).
3) AMBERT is able to make effective use of complete lexical units such as “sister station” in sentence pair 4 and “ 员 工 / 工 人 ” in sentence pair 7, and robust to incorrect tokenizations, such as “used to” in sentence pair 3. 4) AMBERT can in general make more accurate decisions on difﬁcult sentence pairs with both ﬁne-grained and coarse-grained tokenization results.