MZET : Memory Augmented Zero-Shot Fine-grained Named Entity Typing (2020)

Abstract: Named entity typing (NET) is a classification task of assigning an entity mention in the context with given semantic types. However, with the growing size and granularity of the entity types, few previous researches concern with newly emerged entity types. In this paper, we propose MZET, a novel memory augmented FNET (Fine-grained NET) model, to tackle the unseen types in a zero-shot manner. MZET incorporates character-level, word-level, and contextural-level information to learn the entity mention representation. Besides, MZET considers the semantic meaning and the hierarchical structure into the entity type representation. Finally, through the memory component which models the relationship between the entity mention and the entity type, MZET transfers the knowledge from seen entity types to the zero-shot ones. Extensive experiments on three public datasets show the superior performance obtained by MZET, which surpasses the state-of-the-art FNET neural network models with up to 8% gain in Micro-F1 and Macro-F1 score.

## introduction named entity typing (net) is the task of inferring semantic types for the given named entity mentions in utterances. for instance, given an entity mention "john" in the utterance "john plays piano on the stage". the goal for net is to infer that "john" is a pianist or a musician, and a person. standard net approaches @xcite @xcite ) only consider a tiny set of coarse-grained types, and discard fine-grained types with a different level of granularity. in recent years, fine-grained named entity typing (fnet) @xcite @xcite @xcite continues to draw researchers' attention, because it can provide additional information that benefits a lot of downstream tasks like relation extraction @xcite , entity linking @xcite , and question answering @xcite . however, with the ever-growing number of entity types especially for fine-grained ones, it is difficult and expensive to collect sufficient annotations per category and retrain the whole model. therefore, a zero-shot paradigm is welcomed in fnet to handle the increasing number of unseen types. the task we deal with in this paper is named zero-shot fine-grained named entity typing (zfnet), which is to detect the unseen fine-grained entity types that have no labeled data available. learning generalizable representations for entity mentions and types is essential for the zfnet task. previous works learn these representations either from hand-crafted features @xcite , or pre-trained word embeddings @xcite . these methods are insufficient and inefficient when challenged by poly-semantic, ambiguity, or even the newly-emerged mentions. the most recent works @xcite learn more informative but resource-costing representations by assembling the exterior wikipedia knowledge base. with the learned representations for entity mentions and entity types, most of the existing zero-shot fnet methods @xcite project them into a shared semantic space. the shared space is learned through minimizing the distance between entity mentions and its corresponding seen entity types. in the prediction phase, testing entity mentions are classified to the nearest unseen entity types based on the assumption that the learned distance measurement also works for unseen types. these methods' ability to transfer knowledge from seen types to unseen types is limited since they do not explicitly build connections between seen types and unseen types. in this work, we propose the memory augmented zero-shot fnet model (mzet) to tackle the aforementioned problems. mzet is designed to automatically extract the multi-information integrated mention representations and structure-aware semantic type representations with a large-scale pre-trained language model @xcite . to effectively transfer knowledge from seen types to unseen types, mzet regards seen types as memory components and explicitly models the relationships between seen types and unseen types. intuitively, we want to mimic the way how humans learn new concepts. humans learn new concepts by comparing the similarities and differences between new concepts and old concepts stored in our memory. in summary, the main contributions of mzet are as follows. 1) we propose the memory augmented zero-shot fnet model (mzet) that can be trained in an end-to-end fashion. mzet extracts multiinformation integrated mention representations and structure-aware semantic type representations without additional augmented data sources. 2) mzet regards seen types as memory components and explicitly models the relationships between seen types and unseen types to effectively transfer knowledge to new concepts. 3) mzet outperforms existing zero-shot fnet models significantly on the zero-shot fine-grained, coarse-grained, and hybrid-grained named entity typing over three benchmark datasets. ## problem definition we begin by formalizing the problem of zero-shot fine-grained named entity typing (zfnet). for a given entity mention x, the task of named entity typing (net) is to identify the type y for x. suppose we have a training type set y @xmath0, y s 2 , ..., y s ds } with d s seen types. there are a large number of labeled examples available for these seen types, d @xmath1, y i ), @xmath2, 2, ..., |d tr |} with y @xmath3. the task of zfnet is to classify a new mention which belongs to one of the unseen fine-grained entity types y @xmath4, y u 2 , ..., y u du }, where d u is the number of unseen fine-grained entity types and y seen ∩ y @xmath5. ## the proposed model the overview of the proposed mzet framework is illustrated in figure 1 . specifically, mzet consists of three components: 1) zero-shot memory network that identifies entity types for entity mentions, introduced in sec.3.1; 2) mention processor which extracts representation for entity mentions, detailed in sec.3.2; 3) label processor which obtains type representation, depicted in sec.3.3. ## related work fnet is a long-standing task in natural language processing @xcite . most of the proposed fnet methods are based on a distant supervisor, but diverse in classification architectures. ling et al. @xcite propose multi-label and multi-class multilayer perceptron model assigns each mention of the corresponding label tags. naka et al. @xcite type newly emerging out-of knowledge base entities by a fine-grained typing system and harnesses relational paraphrase with type signatures for probabilistic weight computation. del et al. @xcite designs a system, finet, with the help from wordnet @xcite . @xcite propose afet for automatic fine-grained entity typing with hand-crafted features and label embedding from the hierarchical type path. shim et al. @xcite and anan et al. @xcite adopt attentive neural network models for fnet. attention information and contextural embedding are proposed to enhance fnet performance. those methods develope, from hand-crafted features to neural network learned fea-tures, to allow fine-grained typing system fancy, automatic and effective. but their architectures can not apply to new and unseen entity types. to handle unseen types, zero-shot learning @xcite is introduced for named entity typing. several works @xcite propose to solve unseen entity typing with clustering. these works cluster mentions and propagate type information from representative mentions to unseen types. another direction is to construct a shared space for linking the seen and unseen data. these models @xcite @xcite map the mention and label embedding into a shared latent space, then estimate the closeness score for each mention-label pair. most of the existing zero-shot fnet methods limit the model's flexibility with considerable auxiliary resources or pre-prepared hand-crafted features. in the perspective of entity type representations, most recent researchers @xcite @xcite obtain informative entity type representations by assembling related wikipedia pages. their performances are decent yet resourcecosting. others @xcite @xcite @xcite exploit typical pre-trained semantic label embedding, which is easily-applied but pale in performance. apart from various methods for entity type representation, mention reion approaches are also evolving recently. @xcite and @xcite ultilize the pre-prepared hand-crafted features, while others @xcite embed the mention by some pre-trained wording embedding methods @xcite . but, these methods are insufficient and inefficient when challenged by polysemantic, ambiguity, or even the newly-emerged mention. ## conclusions in this paper, we propose an end-to-end neural network, mzet, that enables zero-shot fine-grained named entity typing. it extracts comprehensive representations concerning word and character, mention, mention's context, and raw label text without auxiliary information. it adopts the memory network to gather the representations for zero-shot paradigm. extensive experiments on three public datasets show prominent performances obtained by mzet, which surpasses the state-of-the-art neural network models for zero-shot fnet.