Impact of Training Instance Selection on Domain-Specific Entity Extraction using BERT (2022)

Abstract: State of the art performances for entity extraction tasks are achieved by supervised learning, specifically, by fine-tuning pretrained language models such as BERT. As a result, annotating application specific data is the first step in many use cases. However, no practical guidelines are available for annotation requirements. This work supports practitioners by empirically answering the frequently asked questions (1) how many training samples to annotate? (2) which examples to annotate? We found that BERT achieves up to 80% F1 when fine-tuned on only 70 training examples, especially on biomedical domain. The key features for guiding the selection of high performing training instances are identified to be pseudo-perplexity and sentence-length. The best training dataset constructed using our proposed selection strategy shows F1 score that is equivalent to a random selection with twice the sample size. The requirement of only a small number of training data implies cheaper implementations and opens door to wider range of applications.

## introduction information extraction (ie) is the process of turning unstructured texts into structured data @xcite , and is one of the most widely used natural language processing (nlp) tasks in industrial applications. named entity recognition (ner) is an ie task of tagging entities in text with their corresponding types. most existing ner methods require either handcrafted features, and/or a large number of annotated examples @xcite , both of which are labor intensive. recent advances in transformers @xcite and bert @xcite changed the landscape for many nlp tasks. significant performance gain can be achieved by fine-tuning language models on a small number of training examples due to transfer learning. as a result, the pipeline of annotating -fine-tuning becomes com-mon practice. following this pipeline, the first step for each use case is to annotate application specific data. it is therefore beneficial to estimate in advance how many training samples need to be annotated, as well as which samples to annotate. this work answers these two frequently asked questions through empirical studies on the ner task. specifically, we repeatedly down-sample benchmark datasets and fine-tune bert models for the downstream task of token classification. two benchmark datasets (1) general domain @xcite and (2) biomedical domain bc5cdr @xcite are used in this study. in summary, our main contributions are: ## experimental setting the goal of the experiments is to answer beforementioned questions on how many and which training samples to annotate for the named entity extraction task. we repeatedly down-sample benchmark ner datasets and compared model performances finetuned on different number of training examples and different samples. two datasets of different domains, and two bert models pretrained on different datasets are used in this study. ## results we first establish a baseline using the full dataset, which also serves as an upper bound. next, we compare the f1 scores for each dataset for different random sample-sizes and for the training subset selected using our proposed method. finally, we conclude the analysis with a recommended workflow for training instance selection. ## conclusions it can be shown that domain-specific pre-trained bert performs well even when fine-tuned only on small amounts of training samples. initial increase in amount of data leads to large performance gain before saturating at around 200 training examples. for small data sizes, the f1 scores of different random samples vary greatly. a sampling strategy is proposed in this work which uses kernel density estimate to balance the instance selection between pseudo-perplexity and sentence length. the f1 scores of bert models fine-tuned on training sets constructed using our method are equivalent to the same model fine-tuned on a random sample using twice as many training examples. this work provides practical guidelines for annotation requirements, namely, data size and sampling strategy. given the reduced number of training instances needed due to sampling optimisation, data annotation becomes less expensive and can be achievable in more use cases.