L ight NER : A Lightweight Tuning Paradigm for Low-resource NER via Pluggable Prompting (2022)

Abstract: Most NER methods rely on extensive labeled data for model training, which struggles in the low-resource scenarios with limited training data. Existing dominant approaches usually suffer from the challenge that the target domain has different label sets compared with a resource-rich source domain, which can be concluded as class transfer and domain transfer. In this paper, we propose a lightweight tuning paradigm for low-resource NER via pluggable prompting (LightNER). Specifically, we construct the unified learnable verbalizer of entity categories to generate the entity span sequence and entity categories without any label-specific classifiers, thus addressing the class transfer issue. We further propose a pluggable guidance module by incorporating learnable parameters into the self-attention layer as guidance, which can re-modulate the attention and adapt pre-trained weights. Note that we only tune those inserted module with the whole parameter of the pre-trained language model fixed, thus, making our approach lightweight and flexible for low-resource scenarios and can better transfer knowledge across domains. Experimental results show that LightNER can obtain comparable performance in the standard supervised setting and outperform strong baselines in low-resource settings.

## introduction named entity recognition (ner) has been a fundamental task of research within the natural language processing (nlp) community. mostly, the ner task is formulated as a sequence classification task, aiming to assign the labels to each entity in the input sequence. and those entity labels are all based on pre-defined categories, such as location, organization, person. the current mature methodologies for handling ner is using pre-trained language models (plms) @xcite equipped with several ner paradigms to perform extensive training process on large corpus, such as label-specific classifier paradigm (lc) @xcite , machine reading comprehension paradigm (mrc) @xcite and unified generative paradigm (bartner @xcite ). unfortunately the resulting models are highly associated with seen categories, which often explicitly memorizing entity values @xcite , partially because the output layers require a consistent label set between training and testing. note that these models require to build a new model from scratch to adapt to a target domain with new entity categories, thus, achieving unsatisfactory performance when the target labeled data is limited. unfortunately, this problem is prevalent in realworld application scenarios and draws attention to a challenging but practical research problem: lowresource ner, where the model is built to quickly identify new entities in a completely unseen target domain with only a few supporting samples in the new domain. overall, low-resource ner @xcite @xcite ) mainly faces two issues as shown in figure 1: (1) class transfer. entity categories can be different across rich-resource settings and low-resource settings. for example, source news domain contains the entity categories with "person" ,"location" , etc, while target movie domain adds new categories with "rating_average" and "actor". in this situation, the current mainstream method such as lc, mrc and bartner have to refactor a new model and train it from scratch, which is expensive, and unrealistic for realworld settings. (2) domain transfer. compared with rich-resource settings, the low-resource setting may have a different textual domain. intuitively, the sentence in news domain and atis domain contain the different grammar style and allegorical theme, which is not trivial to transfer the model fully trained in the source domain to target domain with few examples. to address the issue of class transfer, we first reformulate the ner task from sequence labeling to a generative framework with a unified learnable verbalizer to realize class transfer. considering different categories involve varying numbers of words as their descriptions, vanilla mainstream method that assign single classifier for entity may lose important label semantic information. thus, we propose to construct a unified learnable verbalizer based on generative framework. different from bartner @xcite ) that has extra mlp layer in encoder and classifier in decoder, our method only contain the original architecture of pre-trained generative model by constructing constructing a unified learnable verbalizer for entity. therefore, our approach can directly leverage any new or complicated entity types without modifying the network structure. recently, prompt-tuning @xcite @xcite has emerged to become surprisingly effective for the model adaptation of plms, especially in the low-resource setting. however, the prompt-tuning relies on reformulating the paradigm of downstream tasks into new tasks similar to mlm pre-training, which is not efficient for sequence labeling tasks such as ner. inspired by the success of prompt-learning @xcite in domain adaptation, we propose a lightweight tuning paradigm with pluggable guidance for ner (light-ner) to tackle the downsides of domain transfer. specifically, we propose to incorporate learnable parameters into the self-attention layer in lms and regard the parameters as knowledgeable guidance. in particular, we explore lightweight tuning with the pluggable guidance module to urge it to learn domain transfer ability and condition it at inference time. in light of the limits of the existing techniques, we are interested in building a lightweight tuning framework fot low-resource ner with pluggable prompting. notably, the modules in lightner are extremely coupled and indispensable to each other. it is precisely because we design a generative model equipped with a decoupling space to solve the issue class transfer that the pluggable guidance module can realize domain knowledge transfer with lightweight tuning. in a nutshell, lightner consists of the following contributions: ## conclusion and future work in this paper, we propose a lightweight tuning paradigm for low-resource ner via pluggable prompting (lightner), which can accomplish the class transfer and domain transfer for low-resource ner without modifying the plm's parameters and architecture. note that we only tune pluggable guidance module with the whole parameter of the plms fixed, thus, making our approach lightweight and flexible for low-resource scenarios and can better transfer knowledge across domains and tasks. experimental results reveal that lightner can obtain competitive results in the rich-resource setting and outperform baselines in the low-resource setting. in the future, we plan to leverage knowledge graphs to enhance the pluggable guidance module for better knowledge transfer performance.