A Comparative Study of Pre-trained Encoders for Low-Resource Named Entity Recognition (2022)

Abstract: Pre-trained language models (PLM) are effective components of few-shot named entity recognition (NER) approaches when augmented with continued pre-training on task-specific out-of-domain data or fine-tuning on in-domain data. However, their performance in low-resource scenarios, where such data is not available, remains an open question. We introduce an encoder evaluation framework, and use it to systematically compare the performance of state-of-the-art pre-trained representations on the task of low-resource NER. We analyze a wide range of encoders pre-trained with different strategies, model architectures, intermediate-task fine-tuning, and contrastive learning. Our experimental results across ten benchmark NER datasets in English and German show that encoder performance varies significantly, suggesting that the choice of encoder for a specific low-resource scenario needs to be carefully evaluated.

## introduction pre-trained language models (plm) have been shown to be very effective few-shot learners for a wide range of natural language processing tasks @xcite , as they capture semantically and syntactically rich representations of text via self-supervised training on large-scale unlabeled datasets @xcite . recent research in few-shot named entity recognition (ner) has leveraged such representations, e.g. for metric learning on taskspecific out-of-domain foot_0 data @xcite , optionally augmented by continued pre-training with distantly supervised, indomain data @xcite . however, there has been no systematic comparison of the ner performance of such representations in low-resource scenarios without task-specific out-of-domain data and very limited in-domain data; a prevalent setting in many practical applications. in this paper we conduct a comparative study to answer the following research questions: how well do representations learnt by different pre-trained models encode information that benefits these lowresource scenarios? what can we observe for different categories of encoders, such as encoders trained with masked language modeling, versus encoders that are additionally fine-tuned on downstream tasks, or optimized with contrastive learning? how do they perform across different datasets and languages? we present an evaluation framework inspired by few-shot learning to evaluate representations obtained via different pre-training strategies, model architectures, pre-training data, and intermediate-task fine-tuning in low-resource ner scenarios of varying difficulty (see figure 1 ). we find that the choice of encoder can have significant effects on low-resource ner performance, with f1 scores differing by up to 25% between encoders, and simply picking an encoder of the bert family at random will usually not yield the best results for a given scenario. we observe that while bert in general performs adequately, albert and roberta outperform bert by a large margin in many cases, with albert being especially strong in very low-resource settings with only one available labeled example per class. the main contributions of this study are: (1) a systematic performance evaluation of a wide range of encoders pre-trained with different strategies, such as masked language modeling, taskspecific fine-tuning, and contrastive learning on the task of low-resource named entity recognition; figure 1 : encoder-readout evaluation framework. for each of the n classes, we randomly sample k support tokens including their sentence context, and an unlabeled query token with sentential context. the encoder f θ (•) provides an embedding (or representation) for each token, and the readout module g(•) assigns a class to a query token by comparing its representation q j to the representations {z 1 , . . . , z n ×k } of the support tokens. depending on the readout approach, the c-th class in s is represented either by its prototype embedding (as shown in the example) or by its set of associated token embeddings, e.g. for nearest neighbor classification. in this example q 1 representing merkel would be assigned the class per based on the closest class prototype embedding (red circle). ## encoder evaluation framework to simulate low-resource ner scenarios of varying difficulty, we draw inspiration from the evaluation of few-shot learning methods. we first give a formal definition of the few-shot ner task, and then introduce the encoder evaluation framework itself. ## experiments we illustrate the evaluation framework using a representative set of encoders pre-trained with different strategies. we then give details of the readout approaches, the datasets we used, and all other experimental settings.