Formulating Few-shot Fine-tuning Towards Language Model Pre-training: A Pilot Study on Named Entity Recognition (2022)

Abstract: Fine-tuning pre-trained language models is a common practice in building NLP models for various tasks, including the case with less supervision. We argue that under the few-shot setting, formulating fine-tuning closer to the pre-training objective shall be able to unleash more benefits from the pre-trained language models. In this work, we take few-shot named entity recognition (NER) for a pilot study, where existing fine-tuning strategies are much different from pre-training. We propose a novel few-shot fine-tuning framework for NER, FFF-NER. Specifically, we introduce three new types of tokens, “is-entity”, “which-type” and “bracket”, so we can formulate the NER fine-tuning as (masked) token prediction or generation, depending on the choice of the pre-training objective. In our experiments, we apply to fine-tune both BERT and BART for few-shot NER on several benchmark datasets and observe significant improvements over existing fine-tuning strategies, including sequence labeling, prototype meta-learning, and prompt-based approaches. We further perform a series of ablation studies, showing few-shot NER performance is strongly correlated with the similarity between fine-tuning and pre-training.

## introduction pre-trained language models @xcite @xcite have the ability to capture extensive semantic and syntactic information in text. such prior understanding of language offers a solid foundation to downstream tasks, therefore, fine-tuning language models have been widely applied in various nlp applications, especially when the task-specific annotations are limited (a.k.a. few-shot setting) @xcite . while many few-shot strategies have been explored, there is no established systematic principle * corresponding author. on designing these strategies. intuitively, given the same amount of task-specific annotations, a pretraining aligned fine-tuning strategy should more easily extract beneficial knowledge encoded during pre-training. therefore, we hypothesize that the performance of few-shot fine-tuning is tied to the similarity between the fine-tuning strategy and the pre-training objective. to verify this hypothesis, we take the named entity recognition (ner) task for a pilot study, since the common fine-tuning strategies for ner are much different from popular pre-training tasks. specifically, the most common fine-tuning paradigm for ner is to train a sequence labeling model on top of a pre-trained language modelfor each token in a sentence, simultaneously decide whether it belongs to an entity and the type of the entity among given entity types. as an example showing the difference, the commonly used pretrained language model bert has a pre-training goal to disambiguate some masked tokens by predicting the correct token (masked language modeling). while in sequence labeling, each token (representation) is to decide two different pieces of information @xcite . following our hypothesis, we propose a fewshot fine-tuning framework for ner, fff-ner as shown in figure 1 . at the core is decoupling span detection and type prediction, since they are essentially two quite different tasks, which is not easy to learn altogether in few-shot fine-tuning. we introduce two new types of tokens that we dub as "is-entity" and "which-type" to perform the two tasks individually. with the help of these two types of tokens, we consider each instance not a single sentence but a sentence-span pair and ask the model to predict whether the span is an entity and, if it is, the type of the entity. the two tokens are placed around the span. by introducing another type of token, bracket, around the span and the two special tokens, the sentence highlights the span in consider- bert-like bart-like more similar to pre-training less similar to pre-training [sos] w1 bpad b1 b2 b3 b4 b5 bpad b1 b[b] b2 b3 b4 w2 w3 w4 w5 [eos] w1 [b] w2 w3 w4 b[e] [e] b! ! b[t] [t] b5 w5 bpad [eos] 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 positional embedding spatial embedding word embedding source embeddings target embeddings ## our framework fff-ner in this section, we introduce our framework fff-ner, and specifically talk about how to adapt it for a masked language modeling and sequence to sequence pre-trained model. ## conclusions in this work, we draw attention to an intuitive principle for few-shot fine-tuning designs: the closer the fine-tuning task is to the pre-training task, the better the performance. we empirically prove this by designing a few-shot fine-tuning framework for ner, fff-ner. our framework is not constrained to a specific architecture of language models and can be easily extended to distinct pre-trained models (e.g., bert and bart in our experiments). our model outperforms existing fine-tuning strategies including sequence labeling, prototype metalearning, and prompt-based approaches on the standard benchmark for few-shot ner. through a series of ablations, we also show that if we manually remove ingredients in our framework that make the fine-tuning task similar to pre-training, the performance drops. we see future works in two directions. one is to incorporate label semantics into the bert based fff-ner, so that even less supervision can suffice. the other is to verify our principle of few-shot fine-tuning on other downstream tasks.