Unsupervised Paraphrasing Consistency Training for Low Resource Named Entity Recognition (2021)

Abstract: Unsupervised consistency training is a way of semi-supervised learning that encourages consistency in model predictions between the original and augmented data. For Named Entity Recognition (NER), existing approaches augment the input sequence with token replacement, assuming annotations on the replaced positions unchanged. In this paper, we explore the use of paraphrasing as a more principled data augmentation scheme for NER unsupervised consistency training. Specifically, we convert Conditional Random Field (CRF) into a multi-label classification module and encourage consistency on the entity appearance between the original and paraphrased sequences. Experiments show that our method is especially effective when annotations are limited.

## introduction supervised training for named entity recognition (ner) requires token level annotations, which are time consuming and more expensive to obtain than the sequence level annotations commonly used for classification tasks. due to the scarcity of labeled data, various semi-supervised approaches have been investigated for training in low-resource scenarios, i.e., only a small amount of labeled data is available @xcite . unsupervised consistency training is a common approach to semi-supervised learning in ner. it encourages prediction consistency between the original and the augmented examples, by leveraging the availability of a larger amount of unlabeled data. recently, @xcite proposed the unsupervised data augmentation (uda), which substitutes traditional token-wise perturbations with higher quality data augmentation, e.g., paraphrasing via back-translation. uda achieves state-of-the-art results on a wide variety of classification tasks with only tens or hundreds of labeled examples, and even sometimes matching the performance of supervised training with a much larger (fullyannotated) dataset. in the case of ner, due to the difficulty of obtaining token-level annotations, it is of interest to extend uda for ner models whose predictions are (token-level) sequences instead of single (sentence-level) labels. more recently, @xcite augmented unlabeled samples for ner by randomly replacing a portion of input tokens with outputs from a language model, thus constraining the model predictions to be invariant to the replacement operation. there are two problems with this approach: i) there is no guarantee that the type of entity (label) will remain unchanged after replacement; and ii) the newly generated context from replacement is constrained by length of the original sequence, which restricts the quality of augmentation. in fact, @xcite suggests that there exists strong correlation between the quality of the augmentation and the performance of consistency training. in this paper, we explore the use of paraphrasing as a means for higher quality data augmentation for unsupervised consistency training in ner. compared with token replacement, the key difficulty of using paraphrasing is that the alignment of tokens between the original and paraphrased sequence is unclear. however, since paraphrasing does not change the substance of the text, we can expect a paraphrase to contain the same entities as in the original sequence. so motivated, instead of relying on token-level consistency, we encourage consistency on the occurrence of entities between predictions on the original and paraphrased sequences. in doing so, we convert the conditional random field (crf) @xcite predictor of ner into a binary multi-label classification module indicating the occurrence of each entity label, e.g., location (loc) or person (per). experimental results show that our method outperforms token replacement and other semi-supervised learning approaches when annotations are scarce. ## related work in addition to token replacement discussed above @xcite al., 2020), şahin and steedman (2019); dai and adel (2020) also investigated on randomly swapping tokens or text-spans in the input sequence as augmentation. however, such methods may be problematic for languages that rarely have inflectional morphemes, such as english, where words follow strict ordering @xcite . therefore, we are not considering swap-based methods in our experiments. other semi-supervised approaches for ner include cvt @xcite which regularizes model predictions to be invariant when masking-out parts of the input data. recently, @xcite proposed an adapted version of virtual adversarial training with crf, outperforming cvt on ner tasks. in the experiments, we show that our method can achieve better performance than both token replacement and seqvat in low-resource scenarios. ## conclusion in this paper, we explored the use of paraphrasing as data augmentation strategy in unsupervised consistency training for ner. experiments show that our approach outperforms token replacement and another state-of-the-art semi-supervised learning approach in low-resource scenarios.