R o C h B ert: Towards Robust BERT Fine-tuning for C hinese (2022)

Abstract: Despite of the superb performance on a wide range of tasks, pre-trained language models (e.g., BERT) have been proved vulnerable to adversarial texts. In this paper, we present RoChBERT, a framework to build more Robust BERT-based models by utilizing a more comprehensive adversarial graph to fuse Chinese phonetic and glyph features into pre-trained representations during fine-tuning. Inspired by curriculum learning, we further propose to augment the training dataset with adversarial texts in combination with intermediate samples. Extensive experiments demonstrate that RoChBERT outperforms previous methods in significant ways: (i) robust – RoChBERT greatly improves the model robustness without sacrificing accuracy on benign texts. Specifically, the defense lowers the success rates of unlimited and limited attacks by 59.43% and 39.33% respectively, while remaining accuracy of 93.30%; (ii) flexible – RoChBERT can easily extend to various language models to solve different downstream tasks with excellent performance; and (iii) efficient – RoChBERT can be directly applied to the fine-tuning stage without pre-training language model from scratch, and the proposed data augmentation method is also low-cost.

## introduction the emergence of pre-trained language models (plms) has revolutionized natural language processing (nlp) to a new era. as a result, largescale plms like bert @xcite have become the mainstream models for various downstream tasks including text classification @xcite , question answering @xcite and machine translation @xcite , and have drastically boosted their performance. in the chinese domain, a wide variety of classical nlp tasks also benefit from the bert-based plms. despite the impressive performance, bertbased models have been proved to be vulnerable to maliciously generated adversarial texts @xcite @xcite . meanwhile, in the real-world scenarios, adversaries usually generate obfuscated texts, i.e., manually crafted adversarial texts to bypass online security-sensitive systems, which has posed severe physical threats to the deployed systems. in contrast to the alphabet languages such as english, the meaning of individual chinese character can be implied from its pronunciation and glyph. thus, substituting characters with others similar in characteristics can hardly bring impact to the context understanding. hence, unlike replacing words with synonyms to mislead english models, adversaries targeting chinese models prefer substituting characters with others sharing similar pronunciation or glyph, as illustrated in figure 1 . several defense methods such as adversarial training (at) @xcite and adversarial detection @xcite have been proposed recently. however, most of them focus on solving english adversarial texts. based on chinese characteristics, some chinese-specific methods are proposed. for example, chinesebert @xcite incor-porates chinese pronunciation and glyph features into model pre-training, and it has achieved sota performance on many chinese nlp tasks. but chi-nesebert needs to be pre-trained from scratch and its number of parameters is quite large. it has to query the corresponding pinyin and glyph of input characters at each inference, which makes the training and prediction very slow. to solve the challenges above, we propose rochbert, a lightweight and flexible method to strengthen robustness for chinese bert-based models. firstly, we generate an updated adversarial graph based on advgraph @xcite to capture the phonetic and glyph relationships between characters, which are commonly exploited by adversarial texts. as current advgraph only incorporates 3,000 commonly used characters and fails to capture certain glyph relationships, we update it to cover more characters and more glyph relationships. then, we use adversarial graph to learn the representation of characters based on chinese pronunciation and glyph. the graph mainly serves as embedding weights within rochbert. meanwhile, we leverage the target model's hidden states of last layer as pre-trained representation. these two kinds of embedding will then be fused in the fine-tuning of target model with specified downstream nlp tasks. secondly, we design a novel data augmentation method inspired by curriculum learning @xcite for better fusion, which is proved to be more efficient than the traditional at. specifically, we add both intermediate and adversarial texts into training datasets, which is very computationally efficient and will not decrease the accuracy on benign texts compared with traditional at. to the best of our knowledge, this is the first work to strengthen the robustness of pre-trained chinese bert-based models during fine-tuning. our contributions are summarized as follows: ## related work pre-trained language model. the applications of plms have achieved great success on various downstream nlp tasks and avoid training a new model from scratch. bert is first introduced to learn universal language representations via masked language model objective and next sentence prediction task, which is then improved in the following works such as roberta @xcite and albert @xcite . in the chinese nlp domain, @xcite proposed chinsebert, which incorporates both the glyph and pinyin features of chinese characters into language model pre-training, and also gained sota performance on many chinese nlp tasks. however, these works mainly focus on improving the model performance on benign texts, and the effort of enhancing model robustness is fairly limited. ## methodology given a benign input text @xmath0, a classifier f has learned the mapping f : @xmath1, which maps x from the feature space x to the label space y as f f (x) = y. the adversary intend to obtain an adversarial text x * by adding some perturbations ∆x on x with ∆x @xmath2, and deceive f into making a wrong prediction on x * , i.e., f f (x * ) = ŷ and @xmath3. in this paper, we aim to improve the inherent robustness of f so to get a more robust model f such that f can resist the adversarial text x * , i.e., f f (x * ) = y. overview. as illustrated in figure 2 , we first generate an updated adversarial graph based on phonetic and glyph relationships between all commonly used chinese characters. then, we adopt node2vec @xcite to learn the representation of each character x i , which will then be used as the node embedding. for x i in text x, its node embedding is concatenated and passed through feature extraction module. simultaneously, x is also fed into the target plm. we utilize the hidden states of last layer as the pre-trained representation of x. after concatenating pre-trained representation and feature embedding, we use a multimodal fusion module to further fuse the information from two channels. finally, the fused representation can be used for most of the downstream tasks. moreover, to better enhance the fusion process, we have designed an efficient data augmentation method, which makes full use of adversarial texts and helps the model further understand which characters may be similar in pronunciation and glyph. ## conclusion in this paper, we propose rochbert, a lightweight defense method to strengthen robustness for pre-trained chinese bert-based models. first, we update the current adversarial graph and learn the feature representation of characters based on chinese pronunciation and glyph. then rochbert fuse semantic and feature embeddings to fine-tune the target model with specified downstream nlp task, e.g., classification. to enhance fusion, we further introduce an augmentation method inspired by curriculum learning. we evaluate rochbert against several attack algorithms, the results show that rochbert greatly strengthen the model robustness without compromising their normal generalization. in addition, rochbert can be used to improve most chinese bert-based models and it is computationally efficient.