Handling Rare Entities for Neural Sequence Labeling (2020)

Abstract: One great challenge in neural sequence labeling is the data sparsity problem for rare entity words and phrases. Most of test set entities appear only few times and are even unseen in training corpus, yielding large number of out-of-vocabulary (OOV) and low-frequency (LF) entities during evaluation. In this work, we propose approaches to address this problem. For OOV entities, we introduce local context reconstruction to implicitly incorporate contextual information into their representations. For LF entities, we present delexicalized entity identification to explicitly extract their frequency-agnostic and entity-type-specific representations. Extensive experiments on multiple benchmark datasets show that our model has significantly outperformed all previous methods and achieved new start-of-the-art results. Notably, our methods surpass the model fine-tuned on pre-trained language models without external resource.

## introduction in the context of natural language processing (nlp), the goal of sequence labeling is to assign a categorical label to each entity word or phrase in a text sequence. it is a fundamental area that underlies a range of applications including slot filling and named entity recognition. traditional methods use statistical models. recent approaches have been based on neural networks @xcite @xcite @xcite @xcite @xcite and they have made great progresses in various sequence labeling tasks. however, a great challenge to neural-networkbased approaches is from the data sparsity problem @xcite . specifically in the context of sequence labeling, the majority of entities in test dataset may occur in training corpus few times or are absent at all. in this paper, we refer this phenomenon particularly to rare entity problem. it is different from other types of data sparsity problems such as the lack of training data for lowresource language @xcite , as this rare entity problem is more related to a mismatch of entity distributions between training and test, rather than the size of training data. we present an example of the problem in the rare entities can be categorized into two types: out-of-vocabulary (oov) for those test set entities that are not observed in the training set, and low frequency (lf) for those entities with low frequency (e.g., fewer than 10) occurrences in the training set. without proper processing, rare entities can incur the following risks when building a neural network. firstly, oov terms may act as noise for inference, as they lack lexical information from training set @xcite . secondly, it is hard to obtain high-quality representations on lf entities @xcite . lastly, high occurrences of oov and lf entities expose distribution discrep-ancy between training and test, which mostly leads to poor performances during test. in general, there are two existing strategies attempting to mitigate the above issues: external resource and transfer learning. the external resource approach, for example @xcite , uses external knowledge such as part-of-speech tags for ner or additional information from intent detection for slot filling. however, external knowledge such as part-of-speech tag is not always available for practical applications and open source taggers such as @xcite may perform poorly for cross-domain annotations. character or n-gram feature are mainly designed to deal with morphologically similar oov words. the transfer learning approach, such as using elmo @xcite and bert @xcite , fine-tunes pre-trained models on the downstream task @xcite . nevertheless, it is not directly addressing problems such as entity distribution discrepancy between training and test. moreover, our proposed methods surpass these methods without resorting to external resources nor large pre-trained language models. this paper proposes novel techniques that enable sequence labeling models to achieve state-of-theart performances without using external resource nor transfer learning. these are local context reconstruction enables oov entities to be related to their contexts. one key point is applying variational autoencoder to model this reconstruction process that is typically a one-to-many generation process. delexicalized entity identification aims at extracting frequency-agnostic and entity-type-specific representation, therefore reducing the reliance on high-frequency occurrence of entities foot_0 . it uses a novel adversarial training technique to achieve this goal. both methods use an effective random entity masking strategy. we evaluate the methods on sequence labeling tasks on several benchmark datasets. extensive experiments show that the proposed methods significantly outperform previous models by a large margin. detailed analysis indicates that the proposed methods indeed alleviate the rare entity problem. notably, without using any external knowledge nor pre-trained models, the proposed methods surpass the model that uses fine-tuned bert. ## background given an input sequence @xmath0, x 2 , • • • , x n ] with n tokens, the sequence labeling task aims at learning a functional mapping to obtain a target label sequence @xmath1, y 2 , • • • , y n ] with equal length. in the following, we briefly introduce a typical method for sequence labeling and review related techniques we use in deriving our model. ## the model we illustrate the overall framework of the proposed model in figure 1 . its baseline sequence labeling module is described in section 2.1. we describe the details of local context reconstruction in sec. 3.1 and delexicalized entity identification in sec. 3.2, together with an example to illustrate them in figure 2. we denote parameters in sec. 2.1 as θ rnn and θ emb , respectively, for its rnn and matrix to obtain 𝑁(0, 1) ℎ ( on fares with [sos] [eos] on fares with ℎ ) ℎ * ℎ + kl divergence indianapolis with fares on monday local context representation oov masking [unk] log 𝜎 𝜇 posterior gaussian mlp reparameterization trick minus feature (a) vae for local context reconstruction. pooling on monday morning b-date morning delexicalization mlp discriminator delexicalized material -∇ # 𝑓 adversarial training cross entropy +∇ # 𝑓 pooling clone binary classification (b) at for delexicalized entity identification. embedding. parameters in sec. 3.1 and sec. 3.2 are each denoted as θ lcr and θ d . ## training algorithm notice that the model has three modules with their own objectives. we update their parameters jointly using algorithm 1. the algorithm first improves discriminator θ d to identify delexicalized items. it then updates θ lcr and θ rnn with joint optimization j vae and j at to improve θ rnn to fool the discriminator. as vae optimization of j vae has posterior collapse problem, we adopt kl cost annealing strategy and word dropout techniques @xcite . finally, the algorithm updates both of θ rnn and θ emb in bi-lstm+crf by gradient ascent according to eq. ( 2 ). note that θ lcr shares the same parameters with θ rnn and θ emb . during experiments, we also find it is beneficial to have a few epochs of pretraining of parameters θ rnn and θ emb with optimization of eq. ( 2 ). ## experiments this section compares the proposed model against state-of-the-art models on benchmark datasets. ## analysis it is noteworthy that the substantial improvements by the model are obtained without using external resources nor large pre-trained models. keys to its success are local context reconstruction and delexicalized entity identification. this section reports our analysis of these modules. ## related work neural sequence labeling has been an active field in nlp, and we briefly review recently proposed approaches related to our work. slot filling and ner neural sequence labeling has been applied to slot filling @xcite @xcite and ner @xcite @xcite @xcite . for slot filling, multi-task learning for joint slot filling and intent detection has been dominating in the recent literature, for example @xcite . the recent work in @xcite employs a collaborative memory network to further model the semantic correlations among words, slots and intents jointly. for ner, recent works use explicit architecture to incorporate information such as global context @xcite or conduct optimal architecture searches @xcite . the best performing models have been using pre-training models on large corpus @xcite or incorporating fine-tuning on existing pre-trained models @xcite such as bert @xcite . external resource this approach to handle rare entities includes feature engineering methods such as incorporating extra knowledge from part-ofspeech tags @xcite or character embeddings @xcite . extra knowledge also includes tags from public tagger @xcite . multi-task learning has been effective in incorporating additional label information through multiple objectives. joint slot filling and intent detection have been used in @xcite @xcite . joint part-ofspeech tagging and ner have been used in @xcite . transfer learning this approach refers to methods that transfer knowledge from highresources to low-resources @xcite or use models pretrained on large corpus to benefit downstream tasks @xcite . the most recent work in @xcite applies adversarial training that uses a resourceadversarial discriminator to improve performances on low-resource data. ## conclusion we have presented local context reconstruction for oov entities and delexicalized entity identification for low-frequency entities to address the rare entity problem. we adopt variational autoencoder to learn a stochastic reconstructor for the reconstruction and adversarial training to extract frequency-agnostic and entity-type-specific features. extensive experiments have been conducted on both slot filling and ner tasks on three benchmark datasets, showing that sequence labeling using the proposed methods achieve new state-of-the-art performances. importantly, without using external knowledge nor fine tuning of large pretrained models, our methods enable a sequence labeling model to outperform models fine-tuned on bert. our analysis also indicates large potential of further performance improvements by exploiting oov and lf entities.