Learning from Miscellaneous Other-Class Words for Few-shot Named Entity Recognition (2021)

Abstract: Few-shot Named Entity Recognition (NER) exploits only a handful of annotations to iden- tify and classify named entity mentions. Pro- totypical network shows superior performance on few-shot NER. However, existing prototyp- ical methods fail to differentiate rich seman- tics in other-class words, which will aggravate overfitting under few shot scenario. To address the issue, we propose a novel model, Mining Undefined Classes from Other-class (MUCO), that can automatically induce different unde- fined classes from the other class to improve few-shot NER. With these extra-labeled unde- fined classes, our method will improve the dis- criminative ability of NER classifier and en- hance the understanding of predefined classes with stand-by semantic knowledge. Experi- mental results demonstrate that our model out- performs five state-of-the-art models in both 1- shot and 5-shots settings on four NER bench- marks. We will release the code upon accep- tance. The source code is released on https: //github.com/shuaiwa16/OtherClassNER.git.

## introduction named entity recognition (ner) seeks to locate and classify named entities from sentences into predefined classes @xcite . humans can immediately recognize new entity types given just one or a few examples @xcite . although neural ner networks have achieved superior performance when provided large-scale of training examples @xcite , it remains a non-trivial task to learn from limited new samples, also known as few-shot ner @xcite . traditional ner models, such as lstm+crf @xcite , fail in few-shot settings. they calculate the transition probability matrix based on statistics, which requires a large number of data for optimization. recently, prototypical network @xcite shows potential on fewshot ner. the basic idea is to learn prototypes for each predefined entity class and an other class, then classify examples based on which prototypes they are closest to @xcite . most existing studies focus on the predefined classes and leverage the label semantic to reveal their dependency for enhancement @xcite . however, they ignore the massive semantics hidden in the words of other class (o-class for short). in this paper, we propose to learn from o-class words, rather than using predefined entity classes only, to improve few-shot ner. in fact, o-class contains rich semantics and can provide stand-by knowledge for named entity identification and disambiguation. as shown in figure 1 (a), if we can detect an undefined class consisting of references to named entities (such as pronouns), then due to their interchangeability @xcite , we will obtain prior knowledge for named entity identification. for example, newton can be replaced with he or professor in s 2 and s 3 . if we can detect additional classes, including he and professor, we will have more evidence about where newton may appear. in addition, if we can detect an undefined class that composed of action (o 1 ), we may cap-ture underlined relations between different named entities, which is important evidence when distinguishing the named entity type @xcite . nevertheless, it is challenging to detect related undefined classes from o class words due to two reasons: 1) miscellaneous semantics. o-class contains miscellaneous types of words. based on our observations, although there are massive related yet undefined classes, the noise maybe even more, such as function and stop words. these noisy classes have little or negative impacts on the identification of target entities. therefore, how to distinguish noise from task-related classes is a key point. 2) lack of golden label. we neither have the labeled examples nor the metadata of each undefined class. the zero-shot methods (pushp and srivastava, 2017) fail in this case, since they need metadata (such as class name and class description) as known information. unsupervised clustering methods also cannot meet quality requirements as shown in our experiment. to handle the issues, we propose the mining undefined classes from other-class (muco) model to leverage the rich semantics to improve few-shot ner. instead of a single prototype, we learn multiple prototypes to represent miscellaneous semantics of o-class. figure 1( @xcite shows the difference between our method and previous methods. to distinguish task-related undefined classes without annotations, we leverage weakly supervised signals from predefined classes and propose a zero-shot classification method called zeroshot miner. the main idea is inspired by transfer learning in prototypical network. prototypical network can be quickly adapted to new class b when pre-training on related base class a. the underlined reason is that if two classes (a and b) are task-related, when we make examples in a class to cluster in the space, the examples in b class also tend to cluster in the space, even without explicit supervision on class b @xcite . based on this phenomenon, we first perform prototype learning on predefined classes to cluster words in predefined classes, and then regard words in o-class that also tend to cluster as the undefined classes. specifically, we train a binary classification to judge whether clustering occurs between any two of the words. after that, we label the found undefined classes back into sentences to jointly recognize predefined and undefined classes for knowledge transfer. our contributions can be summarized as follows: ## related work few-shot ner aims to recognize new categories with just a handful of examples @xcite . four groups of methods are adopted to handle the low-resource issue: knowledge enhanced, cross-lingual enhanced, crossdomain enhanced, and active learning. knowledgeenhanced methods exploit ontology, knowledge bases or heuristics labeling @xcite @xcite as side information to improve ner performance in limited data settings, which suffer from knowledge low-coverage issue. cross-lingual @xcite and cross-domain enhanced methods @xcite respectively use labeled data from a counterpart language or a different domain as external supervised signals to avoid overfitting. when the language or domain discrepancy is large, these two methods will inevitably face the problem of performance degradation @xcite . active learning methods @xcite explicitly expand corpus by selecting the most informative examples for manual annotation, which need extra human-laboring. different from previous methods, we focus on mining the rich semantics in the o class to improve few-shot ner. ## methodology figure 2 illustrated the architecture of the proposed muco model. muco is composed of two main modules: undefined classes detection detects multiple undefined classes hidden in o class to fully exploit the rich semantics in o class. joint classification jointly classifies the undefined classes and predefined classes, so as to leverage the stand-by semantic knowledge in undefined classes to enhance the understanding of predefined classes. ## conclusion in this paper, we propose mining undefined classes from other-class (muco) to utilize the rich semantics in o class to improve few-shot ner. specifically, we first leverage weakly supervised signals from predefined classes to detect undefined classes from o classes. then, we perform joint classification to exploit the stand-by semantic knowl-