C har BERT : Character-aware Pre-trained Language Model (2020)

Abstract: Most pre-trained language models (PLMs) construct word representations at subword level with Byte-Pair Encoding (BPE) or its variations, by which OOV (out-of-vocab) words are almost avoidable. However, those methods split a word into subword units and make the representation incomplete and fragile. In this paper, we propose a character-aware pre-trained language model named CharBERT improving on the previous methods (such as BERT, RoBERTa) to tackle these problems. We first construct the contextual word embedding for each token from the sequential character representations, then fuse the representations of characters and the subword representations by a novel heterogeneous interaction module. We also propose a new pre-training task named NLM (Noisy LM) for unsupervised character representation learning. We evaluate our method on question answering, sequence labeling, and text classification tasks, both on the original datasets and adversarial misspelling test sets. The experimental results show that our method can significantly improve the performance and robustness of PLMs simultaneously.

## introduction unsupervised pre-trained language models like bert @xcite and roberta @xcite have achieved surprising results on multiple nlp benchmarks. these models are pre-trained over large-scale open-domain corpora to obtain general language representations and then fine-tuned for specific downstream tasks. to deal with the large vocabulary, these models use byte-pair encoding (bpe) @xcite or its variations as the encoding method. instead of whole words, bpe performs statistical analysis of the training corpus and split the words into subword units, a hybrid between character-and word-level representation. even though bpe can encode almost all the words in the vocabulary into wordpiece tokens without oov words, it has two problems: 1) incomplete modeling: the subword representations may not incorporate the fine-grained character information and the representation of the whole word; 2) fragile representation: minor typos can drastically change the bpe tokens, leading to inaccurate or incomplete representations. this lack of robustness severely hinders its applicability in real-world applications. we illustrate the two problems by the example in figure 1 . for a word like backhand, we can decompose its representation at different levels by a tree with a depth of 3: the complete word at the first layer, the subwords at the second layer, and the last characters. bpe only considers representations of subwords on the second layer and misses the potentially useful information at the first and last layer. furthermore, if there is noise or typo in the characters (e.g., missing the letter 'k'), the subwords and its number at the second layer will be changed at the same time. models relying purely on these subword representations thus suffer from this lack of robustness. we take the conll-2003 ner development set as an example. nearly 28% of the nouns words will be split into more than one subword with bert tokenizer. when we randomly remove a character from figure 1: the internal structure tree of backhand, which has two subwords: back,hand. if the letter k is removed, the subwords will change to b, ach, and. the noun words in the dataset like the example in figure 1 , about 78% of the words will be tokenized into completely different subwords, and 77% of the words have a different number of subwords. if we focus on the leaf nodes in the example, we can find that the difference of the two trees is only one leaf. so we extend the pre-trained language models by integrating character information of words. there are two challenges for character integration: 1) how to model character information for whole words instead of subwords; 2) how to fuse the character representations with the subwords information in the original pre-trained models. we propose a new pre-training method charbert (bert can also be replaced by other pre-trained models like roberta) to solve these problems. instead of the traditional cnn layer for modeling the character information, we use the context string embedding @xcite to model the word's fine-grained representation. we use a dual-channel architecture for characters and original subwords and fuse them after each transformer block. furthermore, we propose an unsupervised character learning task, which injects noises into characters and trains the model to denoise and restores the original word. the main advantages of our methods are: 1) character-aware: we construct word representations from characters based on the original subwords, which greatly complements the subword-based modeling. 2) robustness: we improve not only the performance but also the robustness of the pre-trained model; 3) model-agnostic: our method is agnostic to the backbone plm like bert and roberta, so that we can adapt it to any transformer-based plm. in summary, our contributions in this paper are: to handle the large vocabularies in natural language corpora, most plms process the input sequence in subword units by bpe @xcite instead of whole words, split a word into subwords by the byte pair encoding compression algorithm. the size of bpe vocabulary usually ranges from 10k-100k subword units, most of which are unicode characters. @xcite introduce another implementation that uses bytes instead of unicode characters as the base subword units, allowing bpe to encode any input sequence without oov words with a modest vocabulary size (50k). ## methodology in this section, we present the overall framework of charbert and its submodules, including the model architecture in section 3.2, the character encoder in section 3.3, the heterogeneous interaction module in section 3.4, the new pre-training task in section 3.5, and the fine-tuning method in section 3.6. ## experiments in this section, we present the pre-training details of charbert and the fine-tuning results on three kinds of tasks: question answering, sequence labeling, and text classification. furthermore, we construct three character attack test set from those tasks and evaluate the robustness of charbert. ## analysis in this section, we conduct some experiments on conll-2003 ner task with the test set to further analyze the 'incomplete modeling' and 'fragile representation' problems. in the end, we compare the contextual word embeddings generated by bert and charbert with a feature-based method. ## conclusion in this paper, we address the important limitations of current plms: incomplete modeling and lack of robustness. to tackle these problems, we proposed a new pre-trained model charbert by injecting the character-level information into plms. we construct the representations from characters by sequential gru layers and use a dual-channel architecture for the subword and character. furthermore, we propose a new pre-training task nlm for unsupervised character representation learning. the experimental results show that charbert can improve both the performance and robustness of pre-trained models. in the future, we will extend charbert to other languages to learn cross-lingual representations from character information. we believe that charbert can bring even more improvements to morphologically rich languages like arabic, where subwords cannot adequately capture the morphological information. on the other hand, we will extend charbert to defense other kinds of noise, e.g., word-level, sentence-level noise, to improve the robustness of plms comprehensively.