PCBERT : Parent and Child BERT for C hinese Few-shot NER (2022)

Abstract: Achieving good performance on few-shot or zero-shot datasets has been a long-term challenge for NER. The conventional semantic transfer approaches on NER will decrease model performance when the semantic distribution is quite different, especially in Chinese few-shot NER. Recently, prompt-tuning has been thoroughly considered for low-resource tasks. But there is no effective prompt-tuning approach for Chinese few-shot NER. In this work, we propose a prompt-based Parent and Child BERT (PCBERT) for Chinese few-shot NER. To train an annotating model on high-resource datasets and then discover more implicit labels on low-resource datasets. We further design a label extension strategy to achieve label transferring from high-resource datasets. We evaluated our model on Weibo and the other three sampling Chinese NER datasets, and the experimental result demonstrates our approach’s effectiveness in few-shot learning.

## introduction ner is a fine-grained sequence labeling task, a slight change in each token will significantly impact the model results. a big challenge of ner is to enhance the performance in low-resource scenarios. there are some prior works @xcite @xcite that demonstrate that transfer learning can improve the model performance. however, they all rely on similar semantic distribution between source and target datasets, and both datasets should contain rich annotated data. a significant difficulty of few-shot or zero-shot ner is the lack of annotated labels in practical application. another challenge of chinese ner is the implicit word boundary, which makes it difficult for the model to distinguish the entity boundary. the lexicon-based approach is a standard solution to solve the above issue. but the performance of traditional lexicon-based models in chinese few-shot ner is still unsatisfactory. recently, prompt-tuning @xcite on the pre-trained language models (plms) has been thoroughly considered for low-resource scenarios because the prompt-tuning process is highly consistent with the target task. previous work @xcite @xcite has demonstrated that prompt-tuning can more effectively enhance the model performance on few-shot ner compared with fine-tuning. however, when the semantic distribution is quite different, using prompt-tuning for semantic transfer learning will decrease model performance, which implies the semantic transfer is unsuitable for ner in the above situation. besides, the implicit boundaries of chinese words make the size of the prompt template uncertain and require a higher ability to judge its boundary. moreover, using inappropriate prompt construction engineering on chinese few-shot ner datasets can not improve model performance effectively but increases training time. in this work, we introduce an enhanced lexicon feature and a prompt-based label transfer approach to address the above issues. we leverage the lexicon feature to enhance chinese word boundary distinction ability in few-shot ner datasets. we further design a label extension strategy to achieve label transferring from high-resource datasets. we propose a parent and child bert(pcbert) model powered by a label lexicon adapter and a prompttuning component to integrate the lexicon feature and the implicit label feature. and it is worth noting that our implementation with a transformer encoder is more efficient than some decoding template-based approaches. we evaluated our model on weibo @xcite and the other three samplings of chinese ner datasets, and the experimental result demonstrates our approach's effectiveness in few-shot learning. our model outperforms other related work in all experiments and achieves state-of-the-art f1 scores on weibo. the contributions of this work can be summarized as follows: ## method in this paper, we propose a two-stage model named pcbert for chinese few-shot ner, which consists of the parent and the child component. both components are implemented with bert @xcite , and we defined them as p-bert and c-bert, respectively. the overall model structure of pcbert is illustrated in figure 1 . the p-bert is a prompt-based model to extract the implicit label extension features in the target dataset; the c-bert is a lexicon-based model inspired by the lebert @xcite and further incorporates multi-label features of each lexicon. in the first stage, the p-bert pre-trains on the label extension dataset. then the p-bert is set to be frozen in the second stage, providing label extension features to fine-tune the c-bert. the structure and functionality are described in the following.