S eq VAT : Virtual Adversarial Training for Semi-Supervised Sequence Labeling (2020)

Abstract: Virtual adversarial training (VAT) is a powerful technique to improve model robustness in both supervised and semi-supervised settings. It is effective and can be easily adopted on lots of image classification and text classification tasks. However, its benefits to sequence labeling tasks such as named entity recognition (NER) have not been shown as significant, mostly, because the previous approach can not combine VAT with the conditional random field (CRF). CRF can significantly boost accuracy for sequence models by putting constraints on label transitions, which makes it an essential component in most state-of-the-art sequence labeling model architectures. In this paper, we propose SeqVAT, a method which naturally applies VAT to sequence labeling models with CRF. Empirical studies show that SeqVAT not only significantly improves the sequence labeling performance over baselines under supervised settings, but also outperforms state-of-the-art approaches under semi-supervised settings.

## introduction while having achieved great success on various computer vision and natural language processing tasks, deep neural networks, even state-of-the-art models, are usually vulnerable to tiny input perturbations @xcite . to improve the model robustness against perturbations, @xcite proposed to train neural networks on both original training examples and adversarial examples (examples generated by adding small but worst-case perturbations to the original examples). this approach, named adversarial training (at), has been reported to be highly effective on image classification @xcite , text classification @xcite , as well as sequence labeling @xcite . however, at is limited to a supervised scenario, which uses the labels to compute adversarial losses. to make use of unlabeled data, virtual adversarial training (vat) was proposed to extend at to semi-supervised settings @xcite . unlike at which treats adversarial examples as new training instances that have the same labels as original examples, vat minimizes the kl divergence between estimated label distribution of original examples and that of adversarial examples. in this manner, both labeled and unlabeled data can be used in training to improve accuracy and robustness. as a semi-supervised learning algorithm, vat was reported to be effective on both image @xcite ) and text classifications @xcite . moreover, a recent study @xcite conducted comprehensive comparisons on various popular semisupervised learning algorithms. vat turned out to be the most effective one. despite its success in classification tasks, vat has not shown similar effectiveness in sequence labeling tasks. in the conventional classification task, the model learns a mapping between a sentence (sequence of tokens) and a label. nevertheless, in sequence labeling task, the target function becomes a mapping from a sequence of tokens to a sequence of labels. to apply vat on sequence labeling, @xcite proposed to use a softmax layer on the top of token representations to obtain label probability distributions for each token. in this fashion, vat could take kl divergence between tokens at the same position of the original sequence and the adversarial sequence as the adversarial losses. this approach shows marginal improvements over baseline models on several benchmarks, but fails to achieve comparable performance as other stateof-the-art models @xcite @xcite . although the approach above applies vat on the entire sequence, it locally normalizes the label probability per token and assumes all transitions between labels have equal possibilities. but in sequence labeling tasks, label transition probabilities are not always the same. for example, a song name is more likely to appear after a singer name, compared to a travel company. to incorporate label transitions into sequence models, @xcite proposed conditional random field (crf). crf models the probability distribution of the whole label sequence given the input sequence, instead of yielding a label probability distribution for each token. it takes account of both token features and transition features. most state-of-the-art sequence labeling models apply a crf on top of token representations as a decoder. such neural-crf models usually outperform models without crf @xcite @xcite . to apply the conventional vat on a model with crf, one can calculate the kl divergence on the label distribution of each token between the original examples and adversarial examples. however, it is sub-optimal because the transition probabilities are not taken into account. to better address these issues, we proposed se-qvat, a variant of vat that can be used along with crf. our evaluation demonstrates that seq-vat brings significant improvements in supervised settings, rather than marginal improvements reported from previous vat-based approaches clark et al.. in the semi-supervised settings, seqvat also outperforms many widely used methods such as self-training (st) @xcite and entropy minimization (em) @xcite , as well as the state-of-the-art semisupervised sequence labeling algorithm, cross-view training (cvt) @xcite . ## evaluation all sequence labeling tasks are evaluated with "slot-f1" metric, which is used in @xcite @xcite . ## conclusion in this paper, we propose a crf compatible vat training algorithm and demonstrate that sequence labeling tasks can greatly benefit from it. our proposed method, seqvat, has strong effects to improve model robustness and accuracy on supervised sequence labeling tasks. in addition, seqvat is also highly effective in semi-supervised settings and outperforms traditional semi-supervised algorithms (st and em) as well as a state-of-the-art approach (cvt). overall, our approach is highly effective for chunking, ner and slot filling, and can be easily extended to solve other sequence labeling problems in both supervised and semi-supervised settings.