S eq M ix: Augmenting Active Sequence Labeling via Sequence Mixup (2020)

Abstract: Active learning is an important technique for low-resource sequence labeling tasks. However, current active sequence labeling methods use the queried samples alone in each iteration, which is an inefficient way of leveraging human annotations. We propose a simple but effective data augmentation method to improve label efficiency of active sequence labeling. Our method, SeqMix, simply augments the queried samples by generating extra labeled sequences in each iteration. The key difficulty is to generate plausible sequences along with token-level labels. In SeqMix, we address this challenge by performing mixup for both sequences and token-level labels of the queried samples. Furthermore, we design a discriminator during sequence mixup, which judges whether the generated sequences are plausible or not. Our experiments on Named Entity Recognition and Event Detection tasks show that SeqMix can improve the standard active sequence labeling method by 2.27%–3.75% in terms of F_1 scores. The code and data for SeqMix can be found at https://github.com/rz-zhang/SeqMix .

## introduction many nlp tasks can be formulated as sequence labeling problems, such as part-of-speech (pos) tagging @xcite , named entity recognition (ner) @xcite , and event extraction @xcite . recently, neural sequential models @xcite @xcite have shown strong performance for various sequence labeling task. however, these deep neural models are label hungrythey require large amounts of annotated sequences to achieve strong performance. obtaining large amounts of annotated data can be too expensive for practical sequence labeling tasks, due to tokenlevel annotation efforts. active learning is an important technique for sequence labeling in low-resource settings. active sequence labeling is an iterative process. in each iteration, a fixed number of unlabeled sequences are selected by a query policy for annotation and then model updating, in hope of maximally improving model performance. for example, @xcite ; @xcite select query samples based on data uncertainties; @xcite compute model-aware similarity to eliminate redundant examples and improve the diversity of query samples; @xcite ; @xcite use reinforcement learning to learn query policies. however, existing methods for active sequence labeling all use the queried samples alone in each iteration. we argue that the queried samples provide limited data diversity, and using them alone for model updating is inefficient in terms of leveraging human annotation efforts. we study the problem of enhancing active sequence labeling via data augmentation. we aim to generate augmented labeled sequences for the queried samples in each iteration, thereby introducing more data diversity and improve model generalization. however, data augmentation for active sequence labeling is challenging, because we need to generate sentences and token-level labels jointly. prevailing generative models @xcite are inapplicable because they can only generate word sequences without labels. it is also infeasible to apply heuristic data augmentation methods such as context-based words substitution @xcite , synonym replacement, random insertion, swap, and deletion @xcite , paraphrasing @xcite or back translation @xcite , because label composition is complex for sequence labeling. directly using these techniques to manipulate tokens may inject incorrectly labeled sequences into training data and harm model performance. we propose seqmix, a data augmentation method for generating sub-sequences along with their labels based on mixup @xcite . under the active sequence labeling framework, se-qmix is capable of generating plausible pseudo labeled sequences for the queried samples in each iteration. this is enabled by two key techniques in seqmix: (1) first, in each iteration, it searches for pairs of eligible sequences and mixes them both in the feature space and the label space. (2) second, it has a discriminator to judge if the generated sequence is plausible or not. the discriminator is designed to compute the perplexity scores for all the generated candidate sequences and select the low-perplexity sequences as plausible ones. we show that seqmix consistently outperforms standard active sequence labeling baselines under different data usage percentiles with experiments on named entity recognition and event detection tasks. on average, it achieves 2.95%, 2.27%, 3.75% f 1 improvements on the conll-2003, ace05 and webpage datasets. the advantage of seqmix is especially prominent in low-resource scenarios, achieving 12.06%, 8.86%, 16.49% f 1 improvements to the original active learning approach on the above three datasets. our results also verify the proposed mixup strategies and the discriminator are vital to the performance of seqmix. ## conclusion we propose a simple data augmentation method seqmix to enhance active sequence labeling. by performing sequence mixup in the latent space, se-qmix improves data diversity during active learning, while being able to generate plausible augmented sequences. this method is generic to different active learning policies and various sequence labeling tasks. our experiments demonstrate that seqmix can improve active learning baselines consistently for ner and event detection tasks; and its benefits are especially prominent in low-data regimes. for future research, it is interesting to enhance seqmix with language models during the mixup process, and harness external knowledge for further improving diversity and plausibility.