MANNER : A Variational Memory-Augmented Model for Cross Domain Few-Shot Named Entity Recognition (2023)

Abstract: This paper focuses on the task of cross domain few-shot named entity recognition (NER), which aims to adapt the knowledge learned from source domain to recognize named entities in target domain with only a few labeled examples. To address this challenging task, we propose MANNER, a variational memory-augmented few-shot NER model. Specifically, MANNER uses a memory module to store information from the source domain and then retrieve relevant information from the memory to augment few-shot task in the target domain. In order to effectively utilize the information from memory, MANNER uses optimal transport to retrieve and process information from memory, which can explicitly adapt the retrieved information from source domain to target domain and improve the performance in the cross domain few-shot setting. We conduct experiments on English and Chinese cross domain few-shot NER datasets, and the experimental results demonstrate that MANNER can achieve superior performance.

## introduction named entity recognition (ner) is a fundamental nlp task that aims at classifying mention spans into entity types. previous works mainly study the ner task in a supervised setting @xcite @xcite . however, supervised learning requires largescale annotated datasets, which can be difficult to obtain in some scenarios (e.g., annotating biomedical named entities always requires domain expertise @xcite ). in this paper, we focus on a more practical and challenging setting in realworld applications, namely cross domain few-shot ner -given a source domain with sufficient labeled data and a target domain with a few labeled * work done when interning at alibaba damo academy. † yong jiang is the corresponding author. 1 our code is publicly available at: https://github. com/alibaba-nlp/manner source domain ner model !! %: jimmy won a medal &: [b-person] [o] [o] [o] … ' @xmath0 a task in source domain target domain %: einstein was born in german. &: [b-scientist] [o] [o] [o] [o] figure 1 : illustration of the cross domain few-shot ner task, where the ner model is first pretrained on a set of tasks (each task has a support set, e.g., s s and a query set, e.g., q s ) in source domain and then adapted to a few-shot task in target domain with a few labeled data. recent work demonstrated that learning prototype representations for each label class could be effective to address few-shot tasks @xcite , and this idea has also been applied to fewshot ner tasks @xcite @xcite . specifically, when dealing with a few-shot task in the target domain, these models learn prototypes for each entity type based on a few labeled data available in the support set and then assign labels to tokens in the query set by measuring their distances to the prototypes. however, since there are only a few labeled examples for each entity type in the support set, the prototypes obtained from the support set only may not be accurate and representative, leading to the suboptimal and unstable performance of prototype-based few-shot ner models @xcite . to this end, we propose a variational memory-augmented cross domain few-shot ner model, abbreviated as manner. it introduces an external memory module that utilizes information from the source domain to augment the support set in the target domain, so as to learn more accurate prototypes. the basic idea of introducing the memory module is that the entity type information from the source domain can provide additional background knowledge for learning prototypes in the target domain @xcite . for example, in figure 1 , the information of the entity type "person" in the source domain can provide guidance for recognizing the entity type "scientist" in the target domain. specifically, manner stores token representations of entity types from the source domain in a memory module. for each entity type in the target domain, manner first retrieves the most similar entity types from the memory and then leverages the retrieved information to learn prototype for the entity type. one critical issue when using the memory module is how to utilize the information from the memory to augment few-shot tasks in the target domain. recent research indicates that the performance of memory-augmented methods which directly use neural networks to fuse information from the memory and the task @xcite , is suboptimal when dealing with cross domain tasks @xcite , such as our cross domain fewshot ner task. this is because the knowledge (i.e., entity type information) of the source domain stored in the memory can be inconsistent with that of the target domain. therefore, in cross domain few-shot ner tasks where the entity types of the source domain and target domain are disjoint, directly utilizing the information retrieved from the memory may be suboptimal. actually, we empirically found that this could degrade the model performance (see § 4.2). to address this problem, we take inspiration from domain adaption and leverage optimal transport @xcite to retrieve and process information from the memory. one benefit of using optimal transport is that we can adapt the retrieved information from the source domain to the target domain via the optimal transport plan. this adaption process helps alleviate the inconsistency problem between the two domains. our contributions can be summarized as follows: ## preliminaries in this section, we formalize the cross domain fewshot ner task, and provide a brief introduction to optimal transport, which serves as the foundation of our model. task formulation. ner is a sequence labeling task, where each token in the sequence is assigned a label representing an entity class or "o" (not an entity). in this paper, we focus on a practical setting of ner, namely corss domain few-shot ner @xcite , where the ner model is first pretrained on data-sufficient source domain(s) d s and then tranferred to target domain(s) d t with only a few labeled examples. formally, we denote a sentence and its labels as @xmath1, x 2 , . . . , x n } and @xmath2, y 2 , . . . , y n }, respectively. following previous works @xcite , we adopt the episode learning paradigm in this paper, where we first pretrain the model on a set of tasks d @xmath3, q s )} from the source domain and then adapt the model to another set of tasks d @xmath4, q t )} from the target domain. each task consists of a support set @xmath5, y (i) )} n ×k @xmath6, and a query set @xmath7, y (j) )} n ×k ′ @xmath8, where n denotes the number of entity types in a task, k and k ′ denote the number of few-shot samples that belong to each entity type in the support set and the query set, respectively. given a task in the target domain, the goal of our model is to predict the labels of sentences in the query set after adapting the model to the task with its support set (i.e., finetuning the model with the support set). figure 1 provides an illustration of the cross domain few-shot ner task. optimal transport. cross domain few-shot ner task can be considered as a domain adaption task. optimal transport (ot) is a widely used method to solve the domain adaption tasks in the field of computer vision @xcite @xcite . specifically, ot is a metric that measures the distance ## methodology the overall framework of our manner is shown in figure 2 . in this section, we first introduce the details of manner in §3.1, and then introduce the pretraining of manner on the source domain in §3.2. we finally introduce how to adapt the model to the target domain in §3.3. ## related work few-shot ner. recently, few-shot ner has received growing interest. previous works mainly address few-shot ner with meta-learning methods @xcite @xcite @xcite . these methods build few-shot models either upon prototypical network @xcite , which learns prototypes for entity types @xcite @xcite @xcite , or maml @xcite , which adapts the model parameters to few-shot tasks through inner-update on the support set @xcite . another line of work adopts the transfer learning paradigm, where they first learn a feature extractor on the source domain and then transfer the pretrained model to the target domain @xcite @xcite . these methods make predictions through the nearest neighbor inference (wiseman and stratos, 2019). in addition, some recent works focus on the two-stage few-shot ner model @xcite @xcite , where they decompose the ner task into two-subtasks: span detection and entity typing. moreover, prompt-based techniques @xcite @xcite have also been proposed to address few-shot ner tasks. in contrast, manner stores the information from the source domain in the memory, which is then used to augment few-shot task in target domain. memory. memory-augmented methods have been widely studied in the field of computer vision @xcite @xcite @xcite . particularly, @xcite propose to augment neural network with neural turing machine @xcite for few-shot learning, which enables quickly encoding and retrieving new information. @xcite further introduce a memory controller to select the minimum samples to be stored in the memory. memory-augmented methods have also been successfully applied in nlp tasks, such as question answering @xcite ), text classification @xcite ), text generation @xcite and slot tagging @xcite . compared with above methods, our model utilizes optimal transport to adapt the retrieved memory to the target domain instead of using neural networks, which is more effective. optimal transport. in domain adaption, optimal transport is a widely used method to transport data from the source domain to the target do-main @xcite @xcite @xcite . theoretical guarantees have been provided in @xcite to justify the use of ot in domain adaption. in @xcite , they propose to transport features from the source domain to the target domain through a barycentric mapping. however, they only consider transporting feature distributions. in contrast, some works propose to align the joint distributions of features and labels in source and target domains @xcite . ## conclusion this paper proposes manner to handle the cross domain few-shot ner task. manner uses a memory module to store information from the source domain, which is then leveraged to augment few-shot task in the target domain. to effectively utilize the information from the memory, manner uses optimal transport to retrieve and process information from the memory, which enables explicitly adapting the retrieved information to the target domain and improve the performance in the cross domain few-shot setting. experimental results on both english and chinese few-shot ner datasets show that manner can achieve superior performance over existing methods.