R o CB ert: Robust C hinese Bert with Multimodal Contrastive Pretraining (2022)

Abstract: Large-scale pretrained language models have achieved SOTA results on NLP tasks. However, they have been shown vulnerable to adversarial attacks especially for logographic languages like Chinese. In this work, we propose RoCBert: a pretrained Chinese Bert that is robust to various forms of adversarial attacks like word perturbation, synonyms, typos, etc. It is pretrained with the contrastive learning objective which maximizes the label consistency under different synthesized adversarial examples. The model takes as input multimodal information including the semantic, phonetic and visual features. We show all these features areimportant to the model robustness since the attack can be performed in all the three forms. Across 5 Chinese NLU tasks, RoCBert outperforms strong baselines under three blackbox adversarial algorithms without sacrificing the performance on clean testset. It also performs the best in the toxic content detection task under human-made attacks.

## introduction large-scale pretrained models, by finetuning on sufficient annotated data, have been able to approach or even surpass human performance on many benchmark testsets @xcite @xcite @xcite . however, even pretrained with huge amounts of text, the models are still vulnerable under adversarial attacks like synonyms, word deletion/swapping, misspelling, etc @xcite @xcite . these adversarial examples occur frequently in the real-world scenario and can be made either naturally (e.g., typos) or maliciously (e.g., to avoid auto detection of toxic content) 1 . the lack of robustness with them can easily lead to large performance drop when testing in the noisy real-world traffic. the issue is particularly outstanding for logographic languages like chinese since the attack can be either with the glyph character, pinyin (the romanized phonetic representations) or a combination of them @xcite @xcite . we show some examples in current research works usually adopt two ways to defend adversarial attacks: (1) run spell checking to correct the written errors before feeding to the prediction model @xcite @xcite , and (2) adversarial training, which adds adversarial example to the training data @xcite @xcite . for the former, chinese spell checking itself is even a more difficult task because it requires the model to accurately recover the original text. any tiny errors of the spell checking can lead to unpredicted model behaviors. for the latter, it is hard for the model to adapt to all adversarial variants only in the finetuning stage, especially when the training data is sparse @xcite . to address the above challenges, we propose rocbert, a robust chinese bert pretrained with the contrastive learning objective by maximizing the label consistency under various adversarial examples. the adversarial examples are synthesized from an algorithm that encapsulates common types of attacks. we also consider combinatorial attacks where multiple types of attacks can be added on top of each other, which has never been considered in previous research. to defend attacks in all levels, we incorporate multimodal information into the encoder. the phonetic and visual features are inserted into one self-attention layer then dynamically fused in later layers. across 5 standard nlu tasks and one toxic content detection task, we show the pretrained model achieves new sotas under various adversarial attackers. in short, our contribution are (1) we propose pretraining a robust chinese bert with adversarial contrastive learning, such that the model can perform well on not only clean testbeds, but also adversarial examples. (2) the model is pretrained with synthesized adversarial examples covering combinations of semantic, phonetic and visual attacks. it takes as input multimodal features to handle all levels of possible attacks. (3) the pretrained model outperforms strong baselines across 5 nlu tasks and 1 toxic content detection task under various adversarial attackers. (4) we perform an extensive ablation studies for pretraining options and have a wide comparison with popular defending methods, which we hope will benefit future research. ## related work adversarial attack there have been a lot of works showing the vulnerability of nlp models under adversarial examples @xcite @xcite , which are understandable by humans yet lead to significant model prediction drops. there are usually two types of attacks: (1) semantic equivalent replacement, which can be synthesized by replacing words based on vector similarity @xcite , wordnet synonyms @xcite , masked prediction from pretrained models @xcite @xcite , etc. (2) noise injection, which can be synthesized by adding/deleting/swapping words @xcite @xcite , replacing words with phonetically or visually similar ones @xcite . for logographic languages like chinese, the noise can be much more complex as it can be injected on both the glyph characters or romanized pinyins @xcite . adversarial defense the most common way of adversarial defense is adversarial training, which simply appends synthesized adversarial examples into the training data @xcite . nonetheless, it relies only on the limited labeled training data. in contrast, the proposed rocbert is pretrained on billions of text and can better adapted to diverse adversarial variants. another popular way is to first remove the noise with off-the-shelf spell checkers, then feed the corrected text into the model @xcite . however, chinese spell checking requires fully recovering the correct text and current model performances are far from satisfactory @xcite @xcite . any tiny error in the spell checking process can lead to unpredicted model behaviors. it also incurs significant latency to model prediction. rocbert does not add additional latency and can perform well even if fully recovery is difficult due to its consistency-maximization pretraining objective. there have also been works on pretraining more robust models through virtual adversarial training and noise regularization @xcite @xcite , but they perform poorly on man-made attacks. ## multimodal contrastive pretraining with the above-mentioned algorithm to sample adversarial examples, we can pretrain the model with the multimodal contrastive learning objective. ## conclusion we present rocbert: the first pretrained chinese language model that is robust under various forms of adversarial attacks. it is pretrained with the multimodal contrastive learning objective and achieves the best performance on 5 chinese nlu tasks un-der three different attacking algorithms without negative effects on clean testsets. it also significantly outperforms the others in the toxic content detection task. extensive ablation studies are provided to benefit future research.