Good Examples Make A Faster Learner: Simple Demonstration-based Learning for Low-resource NER (2022)

Abstract: Recent advances in prompt-based learning have shown strong results on few-shot text classification by using cloze-style templates. Similar attempts have been made on named entity recognition (NER) which manually design templates to predict entity types for every text span in a sentence. However, such methods may suffer from error propagation induced by entity span detection, high cost due to enumeration of all possible text spans, and omission of inter-dependencies among token labels in a sentence. Here we present a simple demonstration-based learning method for NER, which lets the input be prefaced by task demonstrations for in-context learning. We perform a systematic study on demonstration strategy regarding what to include (entity examples, with or without surrounding context), how to select the examples, and what templates to use. Results on in-domain learning and domain adaptation show that the model’s performance in low-resource settings can be largely improved with a suitable demonstration strategy (e.g., a 4-17% improvement on 25 train instances). We also find that good demonstration can save many labeled examples and consistency in demonstration contributes to better performance.

## introduction neural sequence models have become the de facto approach for named entity recognition (ner) and have achieved state-of-the-art results on various ner benchmarks @xcite @xcite . however, these data-hungry models often rely on large amounts of labeled data manually annotated by human experts, which are expensive and slow to collect @xcite , especially for specialized domains (e.g., research papers). to improve ner performance on low-resource (label scarcity) settings, prior works seek auxiliary supervisions, such as entity dictionary @xcite @xcite and labeling rules @xcite , to either augment humanlabeled data with pseudo-labeled data, or incorporate meta information such as explanation @xcite @xcite , context @xcite , and prompts @xcite to facilitate training. however, such methods have the following challenges: (1) human efforts to create auxiliary supervisions (e.g., dictionaries, rules, and explanations); (2) high computational cost to make predictions. for example, @xcite shows effectiveness on entity type prediction given the entity span by constructing a prompt with the structure "[entity span] is [mask]". however, when the entity span is not given, cloze-style prompts need to be constructed over all the entity candidates in the sentence with the structure "[entity candidate] is @xcite " to make a prediction @xcite . such bruteforce enumerations are often expensive. in this paper, we propose demonstration-based learning @xcite , a simple-yet-effective way to incorporate automatically constructed auxiliary supervision. the idea was originally proposed in prompt-based learning to show some task examples before the cloze-style template so that the model can better understand and predict the masked slot @xcite . this paper proposes modified version of demonstrationbased learning for ner task. instead of reformatting the ner task into the cloze-style template, we augment the original input instances by appending automatically created task demonstrations and feed them into pre-trained language models (ptlms) so that the model can output improved token representations by better understandings of the tasks. unlike existing efforts which require additional human labor to create such auxiliary supervisions, our model can be automatically constructed by picking up proper task examples from the train data. moreover, unlike approaches that need to change the format of token classification into cloze-style mask-filling prediction which can neglect latent relationships among token labels, our approach can be applied to existing token classification module in a plug-and-play manner (see figure 1 (a) vs (b)). we investigate the effectiveness of task demonstration in two different low-resource settings: (1) in-domain setting which is a standard ner benchmark settings where the train and test dataset come from the same domain; and (2) domain-adaptation setting which uses sufficient labeled data in source domain to solve new tasks in a target domain. here, we study which variants of task demonstration are useful to train an accurate and label-efficient ner model and further explore ways to adapt the source model to target domain with a small amount of target data. we propose two ways of automatic task demonstration construction: (1) entity-oriented demonstration selects an entity example per entity type from train data to construct the demonstration. it allows the model to get a better sense of entity type by showing its entity example; and (2) instance-oriented demonstration retrieves instance example similar to input sentence in train data. it allows the model to get a better sense of the task by showing similar instances and their entities. we show extensive experimental results on conll03, ontonotes 5.0 (generic domain), and bc5cdr (biomedical domain) over 3 different templates and 5 selection/retrieval strategies for task demonstrations. for entity-oriented demonstration, we present 3 selection strategies to choose appropriate entity example per entity type: (1) random randomly selects entity example per entity type; (2) popular selects the entity example which occurs the most per entity type in the train data; and (3) search selects the entity example per entity type that shows the best performance in the development set. and for instanceoriented demonstration, we present 2 retrieval strategies to choose appropriate instance example (sbert @xcite vs. bertscore @xcite ). our findings include: (1) good demonstration can save many labeled examples to reach a similar level of performance in low-resource settings. our approach consistently outperforms standard fine-tuning by up to 3 points in terms of f1 score (p-@xmath0.02); (2) demonstration becomes more effective when we also provide context. for example, not only showing 'fischler is per', but also the sentence that contains 'fischler' as person, such as 'france backed fischler's proposal'; and (3) consistency in demonstration contributes to better performance. our experiments show that using consistent demonstration for all instances rather than varying per instance lead to better performance 2 related works ner with additional supervision recent attempts addressing label scarcity have explored various types of human-curated resources as auxiliary supervision. one of the research lines to exploit such auxiliary supervision is distant-supervised learning. these methods use entity dictionaries @xcite @xcite or labeling rules @xcite to generate noisylabeled data for learning a ner model. although these approaches largely reduce human efforts in annotation, the cross-entropy loss may make the model be overfitted to the wrongly labeled tokens due to noisy labels @xcite . another line of research is incorporating such auxiliary supervision during training and inference in a setting of supervised learning. these approaches usually incorporate external information that is encoded including pos labels, syntactic constituents, dependency relations @xcite , explanations @xcite @xcite , retrieved context @xcite and prompts @xcite . demonstration-based learning providing a few training examples in a natural language prompt has been widely explored in autoregressive lms @xcite . such prompt augmentation is called demonstrationbased learning @xcite . this is designed to let prompt be prefaced by a few examples before it predicts label words for [mask] in the clozestyle question. recent works on this research line explore a good selection of training examples @xcite and permutation of them as demonstration @xcite . ## problem definition in this section, we introduce basic concepts of named entity recognition, standard fine-tuning for sequence labeling, and domain adaptation for sequence labeling. we then formally introduce our goal -generating task demonstration and then developing a learning framework that uses them to improve ner models. ## demonstration-based ner in this work, we focus on how to create effective task demonstration x to elicit better token representations for x, and then we propose an efficient learning framework that can be improved by the effect of [x; x]. this section introduces the concepts of demonstration-based learning, and provides details of the approach. here, we study example sampling strategies and templates to construct the demonstration (sec 4.1) and how we can train the ner model with the demonstration (sec 4.2). ## experimental results we first compare the overall performance of all baseline models and our proposed framework with the amount of training data 25 and 50 to show the impact of our approach in a low-resource scenario, assuming a task that needs to be annotated from scratch. then, we show performance analysis to show the effectiveness of our approach and whether the model really learns from the demonstration. ## conclusion in this paper, we propose demonstration-based learning for named entity recognition. specifically, we present entity-oriented demonstration and instance-oriented demonstration and show that they successfully guide the model towards better understandings of the task in low-resource settings. we observe that entity-oriented demonstration is more effective than instance-oriented demonstration, and search strategy consistently outperforms all other variants. moreover, we find that consistent demonstration for all the instances is crucial to the superior performance of our approach. we believe that our work provides valuable cost reduction when domain-expert annotations are too expensive and opens up possibilities for future work in automatic demonstration search for few-shot named entity recognition.