Enhance Robustness of Sequence Labelling with Masked Adversarial Training (2020)

Abstract: Adversarial training (AT) has shown strong regularization effects on deep learning algorithms by introducing small input perturbations to improve model robustness. In language tasks, adversarial training brings word-level robustness by adding input noise, which is beneficial for text classification. However, it lacks sufficient contextual information enhancement and thus is less useful for sequence labelling tasks such as chunking and named entity recognition (NER). To address this limitation, we propose masked adversarial training (MAT) to improve robustness from contextual information in sequence labelling. MAT masks or replaces some words in the sentence when computing adversarial loss from perturbed inputs and consequently enhances model robustness using more context-level information. In our experiments, our method shows significant improvements on accuracy and robustness of sequence labelling. By further incorporating with ELMo embeddings, our model achieves better or comparable results to state-of-the-art on CoNLL 2000 and 2003 benchmarks using much less parameters.

## introduction deep neural network (dnn) based methods have shown great success in various natural language processing (nlp) tasks, such as text classification, sentiment analysis, machine translation, and sequence labelling @xcite @xcite . however, some studies @xcite have illustrated that usually dnn models are not robust enough to input noise. even adding tiny perturbations to the input could lead to a large increase in loss or mislabelling. to avoid such kind of errors and improve model robustness, adversarial training (at) @xcite was proposed to force models to give consistent predictions even with perturbations. adversarial training is an approach to improve the robustness and generalization of models, by training models on original examples as well as adversarial examples. unlike adversarial attack, which aims to find the weakness of models, the objective of adversarial training is to improve the model robustness. consequently, adversarial attack tends to find the worst adversarial examples, while adversarial training favors adversarial examples which can improve the model robustness and accuracy, not necessarily the worst ones. despite the success of adversarial training (at) on text tasks such as text classification @xcite and part-of-speech (pos) tagging @xcite , its gains on other sequence labelling tasks, such as named entity recognition and chunking, are not significant @xcite . at generates adversarial examples by introducing small but worst-case perturbations to the input embeddings. it is equivalent to replacing words with their close neighbors in embedding space. while this idea improves token-level robustness by ensuring small changes in embedding space would not shift model predictions, replacement by far-away or out-of-vocabulary words is not recoverable by at. suppose "i went to massachusetts by car" is one sentence in the training set and "massachusetts" is labelled as location. if "massachusetts" is replaced by "connecticut", at is likely to handle it since their embeddings should be close. however, if it is replaced by "kiyomizu-dera" (a temple in kyoto, japan), at is unlikely to correctly label this word, since their embeddings might be far away due to lack of co-occurrence. but in this sentence, a human is able to reason its label to location easily by recognizing the contextual phrase "went to" or "by car". consequently, contextual information is crucial to improve model robustness in context level. yet, robustness upon contextual information is not augmented in adversarial training, and thus its gains to sequence labelling tasks are not that significant and can be further improved. to incorporate context-level robustness into adversarial training, we propose a new approach, named masked adversarial training (mat). mat applies word masks or substitutions (details in method section) when computing loss from adversarial examples, which forces the model to predict the right labels with no word information or wrong information. for example, if "massachusetts" in the sentence "i went to massachusetts by car" is masked or replaced by an irrelevant word such as "pineapple", in order to make the right prediction with such noise, the model has to learn more contextual information from "went to" or "by car". such approach would enhance model robustness since it relies more about contextual information. we evaluate mat on two sequence labelling tasks, named entity recognition (ner) and chunking, which did not exhibit great improvement by using conventional adversarial training in previous literature @xcite . in the experiments, we demonstrate that mat significantly improves performance on top of at for those two sequence labelling tasks, and can achieve comparable or better performance than state-of-the-art. also further analysis indicates that mat improves generalization over unseen words and unseen patterns. ## related work some previous work investigated various approaches to improve contextual robustness during training, such as word dropout @xcite , cross-view training @xcite and masked language model for bert @xcite . word dropout @xcite directly drops some words from input and forces the model to make the same predictions, which simulates that situation where model has not seen some words in training. cross-view training @xcite takes auxiliary predictions from neighbor lstm neurons for each direction and forces them to predict the same label as the current neuron. hence, cross-view training is equivalent to a stricter word dropout which drops all words before/after the current word. such dropouts could be treated as augmenting the training data with pieces of input. but in most cases, pieces of input are not valid natural language and thus would create a discrepancy between training and test. masked language model @xcite smooths this inconsistency by applying replacement of tokens for some data while masking the rest (equivalent to word dropout). however, the replacement in masked language model is randomly chosen from the full vocabulary, but the substitution in real scenarios follows some distribution (e.g. replacing "massachusetts" with a location name is more likely than an animal name), which is not considered in masked language model. hence, we propose a mask mechanism which is similar to masked language model but has a new substitute selection pipeline to address the concern about substitution probability. we apply it on adversarial examples to force the model to learn more contextual information when optimizing the adversarial loss. our new approach combines word-level and context-level robustness and achieves superior performance in our experiments. ## evaluation two sequence labelling tasks are evaluated with "slot-f1" metric, the same as evaluation metrics in @xcite @xcite . considering the relatively small size of the test sets, mean and standard deviation across 5 runs over different random seeds are reported for comparisons. ## conclusion in our experiments, we have shown that mat significantly improves model robustness and generalization on sequence labelling tasks, especially for unseen words or patterns. for the two tasks used in this paper, our approach achieves better or comparable performance to current state-of-the-art with much smaller models. this model architecture is adaptable for all sequence labelling problems and the contextual information brought by mat has potential benefits for other language tasks.