Enhancing Pre-trained C hinese Character Representation with Word-aligned Attention (2020)

Abstract: Most Chinese pre-trained models take character as the basic unit and learn representation according to character’s external contexts, ignoring the semantics expressed in the word, which is the smallest meaningful utterance in Chinese. Hence, we propose a novel word-aligned attention to exploit explicit word information, which is complementary to various character-based Chinese pre-trained language models. Specifically, we devise a pooling mechanism to align the character-level attention to the word level and propose to alleviate the potential issue of segmentation error propagation by multi-source information fusion. As a result, word and character information are explicitly integrated at the fine-tuning procedure. Experimental results on five Chinese NLP benchmark tasks demonstrate that our method achieves significant improvements against BERT, ERNIE and BERT-wwm.

## introduction pre-trained language models (plm) such as elmo @xcite , bert @xcite , ernie @xcite , bert-wwm @xcite and xlnet @xcite have been proven to capture rich language information from text and then benefit many nlp applications by simple fine-tuning, including sentiment classification @xcite , natural language inference @xcite , named entity recognition @xcite and so on. generally, most popular plms prefer to use attention mechanism @xcite to represent the natural language, such as word-to-word self-attention for english. unlike english, in chinese, words are not separated by explicit delimiters. since without word boundaries information, it is * corresponding author intuitive to model characters in chinese tasks directly. however, in most cases, the semantic of a single chinese character is ambiguous. for example, the character "拍" in word "球拍 (bat)" and "拍卖 (auction)" has entirely different meanings. moreover, several recent works have demonstrated that considering the word segmentation information can lead to better language understanding, and accordingly benefits various chinese tasks @xcite @xcite @xcite . all these factors motivate us to expand the character-level attention mechanism in chinese plms to represent the semantics of words foot_0 . to this end, there are two main challenges. (1) how to seamlessly integrate the segmentation information into character-based attention module of plm is an important problem. (2) gold-standard segmentation is rarely available in the downstream tasks, and how to effectively reduce the cascading noise caused by chinese word segmentation (cws) tools @xcite is another challenge. in this paper, we propose a new architecture, named multi-source word aligned attention (mwa), to solve the above issues. (1) psycholinguistic experiments @xcite have shown that readers are likely to pay approximate attention to each character in one chinese word. drawing inspiration from such findings, we introduce a novel word-aligned attention, which could aggregate attention weight of characters in one word into a unified value with the mixed pooling strategy @xcite . (2) for reducing segmentation error, we further extend our word-aligned attention with multi-source segmentation produced by various segmenters and deploy a fusion function to pull together their disparate outputs. as shown in ## experiments 3.1 experiments setup to test the applicability of the proposed mwa attention, we choose three publicly available chinese pre-trained models as the basic encoder: bert, ernie, and bert-wwm. in order to make a fair comparison, we keep the same hyper-parameters (such maximum length, warm-up steps, initial learning rate, etc.) as suggested in bert-wwm @xcite for both baselines and our method on each dataset. we run the same experiment for five times and report the average score to ensure the reliability of results. besides, three popular cws tools: thulac @xcite , ictclas @xcite and hanlp @xcite are employed to segment sequence. the experiments are carried out on five chinese nlp tasks and six public benchmark datasets: sentiment classification (sc): we adopt chnsenticorp foot_2 and weibo-100k sentiment dataset foot_3 in this task. chnsenticorp dataset has about 10k sentences, which express positive or negative emotion. weibo-100k dataset contains 1.2m microblog sentence pair matching (spm): we use the most widely used dataset lcqmc @xcite in this task, which aims to identify whether two questions are in a same intention. natural language inference (nli): this task is to exploit the contexts of text and concern inference relationships between sentences. xnli @xcite is a cross-language language understanding dataset; we only use the chinese language part of xnli to evaluate the language understanding ability. and we processed this dataset in the same way as ernie @xcite did. machine reading comprehension (mrc): mrc is a representative document-level modeling task which requires to answer the questions based on the given passages. drcd @xcite is a public span-extraction chinese mrc dataset, whose answers are spans in the document. we implement our model with pytorch @xcite , and all baselines are converted weights into pytorch version. all experiments employ modified adam @xcite as optimizer with 0.01 weight decay and 0.1 warmup ratio. all pre-trained models are configured to 12 layers and 768 hidden dimension. the detail settings are shown in ## conclusion in this paper, we develop a novel multi-source word aligned attention model (referred as mwa), which integrates word segmentation information into character-level self-attention mechanism to enhance the fine-tuning performance of chinese plms. we conduct extensive experiments on five nlp tasks with six public datasets. the proposed approach yields substantial improvements compared to bert, bert-wwm and ernie, demonstrating its effectiveness and universality. furthermore, the word-aligned attention can also be applied to english plms to bridge the semantic gap between the whole word and the segmented word-piece tokens, which we leave for future work.