Leveraging Type Descriptions for Zero-shot Named Entity Recognition and Classification (2021)

Abstract: A common issue in real-world applications of named entity recognition and classification (NERC) is the absence of annotated data for the target entity classes during training. Zero-shot learning approaches address this issue by learning models from classes with training data that can predict classes without it. This paper presents the first approach for zero-shot NERC, introducing novel architectures that leverage the fact that textual descriptions for many entity classes occur naturally. We address the zero-shot NERC specific challenge that the not-an-entity class is not well defined as different entity classes are considered in training and testing. For evaluation, we adapt two datasets, OntoNotes and MedMentions, emulating the difficulty of real-world zero-shot learning by testing models on the rarest entity classes. Our proposed approach outperforms baselines adapted from machine reading comprehension and zero-shot text classification. Furthermore, we assess the effect of different class descriptions for this task.

## introduction named entity recognition and classification (nerc) is the task of identifying spans of text corresponding to named entities and classifying these spans from a set of pre-defined entity classes. a prevalent issue for many real-world applications is that annotated data does not readily exist. this motivates the focus on the zero-shot setting @xcite , where annotated data is not available for the classes of interest. instead, information available from observed classes must be transferred to unseen target classes. recently zero-shot approaches making use of textual representations to represent entity classes * work done when author was working at google. were explored for entity linking (el) @xcite and named entity typing (net) @xcite , which are similar to the nerc subtask of named entity classification (nec). however, no previous work has addressed the task of zero-shot nerc, which additionally requires the detection of which tokens make up an entity in addition to its type, i.e. named entity recognition (ner). this paper is the first to study zero-shot nerc, by leveraging entity type descriptions. the task is illustrated in figure 1 . during testing, the input is a sentence and a set of target entity classes. each accompanied by its description, and the goal is to recognize and classify entities in these target classes. descriptions contain crucial information for the task. given as input "shantou harbour, a natural river seaport, opens to the south china sea." and a class facility in figure 1 , using a description "names of human-made structures: infrastructure (streets, bridges), [...]" a connection between facility and shantou harbour can be made without having seen an annotated example in training. while using descriptions enables us to predict entity classes unseen in training, nerc poses the additional challenge of modelling the negative class (non-entity tokens) as its definition includes different entity classes and tokens in training and testing. it is possible that words observed as non-entities during training belong to one of the test classes, as seen in figure 1 : both huaqiao park, in training, and shantou harbour, during testing, are entities of the class facility, however, huaqiao park is labelled as a non-entity in the former. based on this insight we propose several architectures for nerc based on cross-attention between the sentence and the entity type descriptions using transformers @xcite combined with pre-training @xcite . we explore modelling the negative class by (i) using a description for the negative class, (ii) modelling the negative class directly, (iii) modelling the negative class using the representations generated for the classes corresponding to types. for evaluation we introduce zero-shot adaptations to two real-world nerc datasets with distinct properties: the ontonotes @xcite as well as the highly domain-specific med-mentions dataset @xcite . the adaptations adhere to recommendations to zeroshot evaluation @xcite by evaluating models on the rarest classes while ensuring that all class sets are disjoint. our best model achieves a macro f 1 of 0.45 on ontonotes-zs and 0.38 on medmentions-zs, outperforming a state-ofthe-art mrc model for nerc @xcite and an adapted zero-shot text classification model @xcite . an analysis on the classification and recognition task in isolation highlights the importance of the description choice, finding that annotation guidelines result in higher scores than the class name itself or wikipedia passages. ## zero-shot nerc in nerc, given a sentence @xmath0, ..., w n of length n and a description d c for each class @xmath1, we predict a sequence of labels ŷ ∈ (c ts ) n , with n being the length of the sentence. we model the task as multiclass classification, which despite ignoring the sequential struc-ture of the output, it has been found to be competitive @xcite . thus, we predict the correct class for each token w at position t: arg max @xmath2, w t , d c ), using a suitable function f modelling the semantic affinity between w t and d c in the context of s. the parameters of f need to be learned without annotated data for c ts , but with annotated data and descriptions for the training c tr classes. to model f we focus on the use of crossattention @xcite in the form of a transformer encoder @xcite . for each type description d c , the cross-attention encoder (x-enc) generates a vector representation v t,@xmath3: with v t,@xmath4,@xmath5. the value o t,c indicates how likely is that token w t belongs to entity class c. in order to be able to recognize entities in addition to classifying them, the scores for each token o t,c 1 ; ...; o t,c k are concatenated with a score for belonging to the negative class o t,neg , corresponding to not belonging to any of the types considered: with o @xmath6. obtaining a good estimate for this score is a key challenge in performing zero shot nerc and we discuss it in the next section. we then select the class with the highest score probability after applying a softmax operation: we label this model sequential multiclass crossattention model (smxm). referring to the initial example, cross-attention enables shantou harbour to attend to infrastructure in the type description of the class facility, generating a representation for this token based on the type description in the context of the sentence. ## related work state-of-the-art approaches to nerc include the bidirectional lstm-crf @xcite , and more recently models based on the pretrained transformer architectures, e.g. bert @xcite . these methods are unsuitable for zero-shot learning, with exception to the explored baselines in this paper @xcite . apart from nerc, manually de-fined class descriptions have also been explored for relation classification @xcite ) who pose the task as one of textual entailment. @xcite use descriptions for zero-shot net, however, similar to a previous attempt by @xcite , they use the underlying hierarchy to only include unseen classes in the leaves of the hierarchy to reduce the relevant unseen classes to only two or three. the only work on zero-shot word sequence labelling @xcite explores the transfer from labels on a sentence level objective (e.g. sentiment analysis) to a token or phrase-based annotation, similar to @xcite . @xcite label their approach zero-shot named entity recognition, however, they focus on recognizing unseen entities not entity classes. finally, @xcite focused on few-shot nerc using prototypical networks @xcite . they tested their model in the zero-shot setting, but concluded that their approach is not suitable for zero-shot learning as the results on ontonotes were too low. ## conclusions & future work this paper explored the task of zero-shot nerc with entity type descriptions to transfer knowledge from observed to unseen classes. we addressed the zero-shot nerc specific challenge that the not-an-entity class is not well defined by proposing a multiclass architecture that uses class-aware encoding to model the negative class. the models were evaluated based on zero-shot adaptations of the ontonotes and medmentions dataset. the results show that the proposed model outperforms strong baselines and further indicate that highquality entity descriptions (i.e. annotation guidelines) are an effective way to transfer knowledge from observed to unseen classes. future work will aim to incorporate the dependencies between the labels predicted.