impact of sample selection on in-context learning for entity extraction from scientific writing (2023)

Abstract: Prompt-based usage of Large Language Models (LLMs) is an increasingly popular way to tackle many well-known natural language problems. This trend is due, in part, to the appeal of the In-Context Learning (ICL) prompt set-up, in which a few selected training examples are provided along with the inference request. ICL, a type of few-shot learning, is especially attractive for natural language processing (NLP) tasks defined for specialised domains, such as entity extraction from scientific documents, where the annotation is very costly due to expertise requirements for the annotators. In this paper, we present a comprehensive analysis of in-context sample selection methods for entity extraction from scientific documents using GPT-3.5 and compare these results against a fully supervised transformer-based baseline. Our results indicate that the effectiveness of the in-context sample selection methods is heavily domain-dependent, but the improvements are more notable for problems with a larger number of entity types. More in-depth analysis shows that ICL is more effective for low-resource set-ups of scientific information extraction

## introduction extracting relevant information from scientific documents plays a crucial role in improving methods for organising, indexing, and querying the vast amount of existing literature @xcite @xcite . however, annotating datasets for scientific information extraction (ie) is a laborious and costly process that requires the expertise of human experts and the development of annotation guidelines. in recent years, large language models (llms) have demonstrated remarkable performance on various natural language processing (nlp) tasks @xcite @xcite , including entity extraction from scientific documents @xcite , and also for leveraging reported scientific knowledge in downstream data science applications @xcite . these models, such as gpt-3 @xcite and llama @xcite , with billions of parameters and pre-trained on vast amounts of data, have showcased impressive capabilities to tackle tasks in a zero-or few-shot learning by leveraging in-context learning (icl) @xcite . in icl, models are provided with a natural language prompt consisting of three components: a format, a set of training samples (input-label pairsdemonstrations), and a test sentence. llm outputs the predictions for a given test input without updating its parameters. the main advantage of icl is its ability to use the pre-existing knowledge of the language model and generalise from a small number of context-specific samples. however, icl has been shown to be sensitive to the provided samples and randomly selected samples have been shown to introduce significant instability and uncertainty to the predictions @xcite @xcite . this issue can be alleviated by optimising the selection of the in-context samples @xcite @xcite . icl sample selection methods can be divided into 2 categories: (1) the methods for choosing samples from the train set (e.g., the kate method @xcite ), and (2) finding the best prompts by generating samples (e.g., the perplexity method @xcite , sg-icl @xcite ). these methods can significantly reduce the need for extensive human annotation and allow llms to adapt to various domains and tasks. we rely on the survey of icl @xcite ) and delimit the methods for sample selection, from the inference stage of icl. our aim is to provide a comprehensive analysis of these methods for se-lecting samples from the train set as part of icl for entity extraction from scientific documents. most of the methods have been applied with prompt generation (i.e., to select the best generated sample). here, we use the methods only for sample selection from the training set of the dataset for entity extraction from scientific documents and compare their effectiveness for this problem. we also propose the use of the influence method @xcite in an oracle setting, to provide a best-case scenario to compare against. we investigate the in-context sample selection methods (see §3) and evaluate the methods adapted for entity extraction problem on 5 entity extraction datasets: ade, measeval, scierc, stem-ecr, and wlpc, each covering a different scientific subdomain or text modality (see §4.1 for dataset overview). our experiments show that while fully supervised finetuned plms are still the gold standard when training data can be sourced, choosing the right samples for icl can go a long way in improving the effectiveness of icl for scientific entity extraction (see §5.1). our experiments demonstrate an improvement potential of 7.56% on average across all experiments, when comparing the oracle method (the influence method) to the random sample selection baseline, and 5.26% when using the best-performing method in a test setting (kate). moreover, our evaluations show that our main conclusions hold in a simulated low-resource setting (see §5.2). finally, our extensive experiments allow us to synthesise some prescriptive advice for other nlp researchers and practitioners tackling scientific entity extraction (see § 5.5). ## related work by increasing the size of both the model and the corpus, llms have demonstrated the capability of icl, which uses pre-trained language models for new tasks without relying on gradient-based training @xcite . in various tasks, such as inference (ibid), machine translation @xcite , question answering @xcite , table-to-text generation @xcite and semantic parsing @xcite , the icl use of llms mentioned by @xcite has been shown to be on par with supervised baselines in terms of effectiveness. other studies have found, however, that icl does not always lead to better results than finetuning. previous studies investigating icl for ie are very limited @xcite . @xcite evaluate the performance of icl on biomedical ie tasks, named entity recognition (ner) and relation extraction (re). in addition, @xcite apply an entityaware demonstration using the knn sample selection method @xcite for re. to the best of our knowledge, our work is one of the first attempts for ie from scientific documents that present a comprehensive analysis of in-context sample selection methods for the problem with detailed analysis. ## methods in this section, we describe the icl sample selection methods for entity extraction from scientific documents. first, we describe the icl approach in section 3.1 and then introduce the sample selection methods in section 3.2. ## conclusion in this paper, we explore the in-context sample selection methods for icl entity extraction from scientific documents. since entity extraction is a crucial step in ie from scientific documents, we analyse the methods in detail using several datasets from different subdomains, and with different entity types. the experimental results show that the baseline model, finetuned roberta, still achieves the best results for this problem on 4 of 5 datasets. however, the in-context sample selection methods appear to be more robust to the train set data availability and achieve similar results to using a full train set when only a small annotated training set is used for the problem, yielding significantly better results than the baseline model in this low-resource setup. our work aims to extract entity spans using llm with icl. we focus on simple in-context sample selection methods based on similarity, perplexity, relevancy, and influence, and use gpt-3.5 as llm in icl. however, there are several alternative llms pre-trained on different domains, that could be more aligned with the task of scientific entity extraction. as future work, we hope to add a comparative dimension to our work by using these llms, since the icl behaviour of llms can change depending on their scale and pretraining. we also plan to explore the performance of the in-context sample ordering methods @xcite , which are shown to impact the icl effectiveness as well.