Does C hinese BERT Encode Word Structure? (2020)

Abstract: Contextualized representations give significantly improved results for a wide range of NLP tasks. Much work has been dedicated to analyzing the features captured by representative models such as BERT. Existing work finds that syntactic, semantic and word sense knowledge are encoded in BERT. However, little work has investigated word features for character languages such as Chinese. We investigate Chinese BERT using both attention weight distribution statistics and probing tasks, finding that (1) word information is captured by BERT; (2) word-level features are mostly in the middle representation layers; (3) downstream tasks make different use of word features in BERT, with POS tagging and chunking relying the most on word features, and natural language inference relying the least on such features.

## introduction large scale pre-trained models such as bert @xcite have been widely used in nlp, giving improved results in a wide range of downstream tasks, including dependency parsing @xcite , summarization @xcite and reading comprehension @xcite . to better understand the reason behind their effectiveness, a natural question is what knowledge can be learned during the self-supervised pre-training stage. some previous works prove that bert effectively captures syntactic information @xcite , semantic information @xcite , commonsense knowledge @xcite , and factual knowledge @xcite without fine-tuning on task-specific datasets, which explains its effectiveness on related tasks. bert has also gained success in chinese nlp @xcite @xcite . however, relatively little work investigate knowledge learned by chinese bert. different from english, chinese sentences are written as sequences of characters without explicit word boundary. yet chinese bert is character-based, where the contextualized representations are built on each character. for traditional chinese nlp, word segmentation is considered an essential pre-processing step @xcite . in neural modeling, @xcite argue that character-based model consistently outperforms word-based model for chinese, because the former can alleviate the over-fitting and out-of-vocabulary issue. in this paper, we investigate whether chinese bert encodes word structure features. we aim to answer the following three research questions. first, how much word information is captured by character-based chinese bert? second, out of 12 representation layers of a bert encoder, which layers encode the most word-level features? third, the connection between word-level features embedded in bert representations and the performance of downstream tasks such as named entity recognition (ner) and natural language inference (nli). intuitively, some tasks rely on word features more than other tasks. by analyzing word knowledge inside bert after fine-tuning on each task, we can gain empirical evidence on how the task is solved by bert. we take two main approaches for analyzing bert. first, bert is based on transformer @xcite , a multi-layer multi-head self-attention-network architecture, where the embedding of each character is calculated by neural attention @xcite over all the characters in a sequence. inspired by @xcite , we look into the attention weight distribution of each head in each layer, in order to understand whether some attention head exhibit salient word-level patterns in the attention targets. second, we take chinese word segmentation as a probing task @xcite , training a linear classifier based on the bert representation at each transformer layer, so that word-level information contained in the layer can be quantified through segmentation performance. results on two chinese datasets with varying segmentation standards consistently show that word information is captured by bert representation. there exist attention heads that focus on the start and end characters in each word, word unigrams and bigrams, as well as word boundary patterns. in addition, word information is captured mostly in the middle layers of transformer, allowing light-weight probing layers to achieve segmentation f1 score around 90% on both segmentation datasets. finally, we find that different chinese tasks require different levels of word information, with fine-tuning on tasks such as pos tagging and chunking significantly improving the probing task, while fine-tuning on tasks such as nli significantly decreasing probing accuracies. to our knowledge, we are the first to investigate word structure knowledge in chinese bert. our code has been released at https://github.com/ylwangy/bert_zh_analysis . ## bert bert @xcite consists of multi-layer transformer @xcite blocks. formally, given a sentence @xmath0, c 2 , ..., c n , each c i is first transformed into input vector e i by summing up token embeddings e c i , segment embeddings se c i , which distinguish the sentence location, and position embeddings pe c i , which indicates character position: the vectors e 1 , ..., e @xmath1, which consists of l layers. now for each layer, denote the input as e. e is then transformed into vectors for queries q m , keys k m , and values v m via linear mappings, {q m , k m , v m } ∈ r n×d k : ## attention distribution analysis we analyze the distribution of attention weight α in eq. 3 across different layers and heads. specifically, we compute the attention weight from characters to several specific characters as well as the attention distribution from characters to surrounding words. the structures are illustrated by figure 1 . 2 for each character, we first investigate its attention weights to several specific characters, including the character itself, the previous character, the next character, and two special tokens [cls] and [sep], as shown in figure 1(a) . formally, the attention from c i to c j in a certain head is defined as in eq. 3: similarly, for each c j that is the last character of some word w k , we consider the attention weights to the first character of the current word and the last character of the previous word: figure 1(b) shows one example, where w 1 and w 2 are two adjacent words, c 1 , c 2 are the first and last character of the word w 1 , and c 3 , c 5 are the first and last character of the word w 2 , respectively. for this example, we consider the attentions α c 1 →c 2 , α c 3 →c 5 , α c 1 →c 3 (eq. 7), and α c 2 →c 1 , α c 5 →c 3 , α c 5 →c 2 (eq. 8). in addition to character-to-character attention weights, we consider character-to-word attention by taking the average of attention weights to characters that belong to specific words. formally, we define the attentions from a character c j in word w k to it previous, current and next words as follows: ## probing task ## conclusion we investigated the capability of chinese bert for capturing the word structure using two different methods. first, analyzing the attention distribution for different patterns, we find that some of the attention heads can capture the word structure implicitly. second, using a word segmentation probing task for the contextualized representation inside the model, we find that a simple linear classifier performs well in the middle layers. by using our probing method, we find evidence that different chinese tasks rely on different degrees of word information, with nli relying the least on word features.