Thinking about GPT -3 In-Context Learning for Biomedical IE ? Think Again (2022)

Abstract: Large pre-trained language models (PLMs) such as GPT-3 have shown strong in-context learning capabilities, which are highly appealing for domains such as biomedicine that feature high and diverse demands of language technologies but also high data annotation costs. In this paper, we present the first systematic and comprehensive study to compare the few-shot performance of GPT-3 in-context learning with fine-tuning smaller (i.e., BERT-sized) PLMs on two representative biomedical information extraction (IE) tasks: named entity recognition and relation extraction. We follow the true few-shot setting to avoid overestimating models’ few-shot performance by model selection over a large validation set. We also optimize GPT-3’s performance with known techniques such as contextual calibration and dynamic in-context example retrieval. However, our results show that GPT-3 still significantly underperforms compared to simply fine-tuning a smaller PLM. In addition, GPT-3 in-context learning also yields smaller gains in accuracy when more training data becomes available. More in-depth analyses further reveal issues of in-context learning that may be detrimental to IE tasks in general. Given the high cost of experimenting with GPT-3, we hope our study provides helpful guidance for biomedical researchers and practitioners towards more practical solutions such as fine-tuning small PLMs before better in-context learning is available for biomedical IE.

## introduction given the overwhelming pace of biomedical research and clinical text production, transforming large amounts of biomedical text into structured information has become increasingly important for researchers and practitioners alike. in recent years, pre-trained language models (plms), both generaldomain and biomedicine-specific ones, have remarkably boosted performance on biomedical information extraction (ie) tasks @xcite @xcite @xcite . the latest round of plms such as gpt-3 @xcite , megatron-turing nlg @xcite , the switch transformer @xcite , among others, feature hundreds of billions of parameters and have achieved impressive performance in many nlp tasks using in-context learning-a new few-shot learning paradigm first introduced by @xcite . in-context learning allows plms to use their natural language generation capabilities to solve any task almost like how humans would-by completing a piece of text or prompt. this paradigm allows large plms to solve various nlp problems without updating their parameters, potentially resulting in massive savings in both data annotation and engineering overhead compared with standard model training. even more impressively, gpt-3 in-context learning yields competitive performance against fully supervised baselines in many nlp tasks by adding only a handful of demonstrative examples in the prompt @xcite . the variety of potential biomedical information extraction applications, the high cost of biomedical annotations, and the complexity of model training make in-context learning particularly appealing for biomedical applications. to investigate its practicality, we present the first systematic and comprehensive comparative study of gpt-3 in-context learning and bert-sized @xcite plm fine-tuning in the few-shot setting on named entity recognition (ner) and relation extraction (re), two representative and highly valued biomedical ie tasks. for consistency and comprehensiveness, we use all the biomedical ner and re tasks compiled in the blurb benchmark @xcite . we operate under the true few-shot setting introduced by @xcite to avoid overestimating models' few-shot performance via model selection over a large validation set. we optimize gpt-3's in-context learning performance for biomedical information extraction by leveraging multiple recent techniques. firstly, inspired by studies that show the importance of optimal prompt selection @xcite @xcite , we formulate a prompt structure which allows us to construct prompt designs and select optimal ones systematically. secondly, similar to @xcite , we introduce a k-nearest neighbor (knn) module for in-context example retrieval. finally, for ner, we also use logit biases to ensure that the generated tokens are from the input sentence; for re, we use contextual calibration @xcite to reduce contextual bias. even when equipped with these latest techniques, which indeed improve gpt-3's performance as shown in ablation studies, we find that fine-tuning bert-sized plms substantially outperforms gpt-3 in-context learning across all biomedical information extraction datasets when using the same small training set (e.g., 100 labeled examples). we also find that fine-tuning small plms yields a more reliable return in terms of data annotation: as training data size increases, fine-tuning performance steadily improves while in-context learning performance lags behind. in-depth analyses further reveal that in-context learning struggles with the null class, e.g., sentences that contain no named entity (for ner) or entity pairs that hold none of the target relations (for re), which is likely detrimental to ie tasks in general. in summary, our findings suggest that fine-tuning plms is still a more cost-effective option than gpt-3 in-context learning for biomedical ie tasks, at least before qualitatively better methods for in-context learning are discovered. ## approach in this section, we describe the two paradigms we explored under the true few-shot setting for ner and re: bert-sized plm fine-tuning and gpt-3 in-context learning. ## related work in-context learning. gpt-3 in-context learning @xcite has been found to be competitive against supervised baselines in a broad range of tasks including text classification, natural language inference, machine translation, question answering, table-to-text generation and semantic parsing @xcite @xcite . many techniques have been introduced to bolster its performance by removing biases through calibration @xcite as well as by optimizing prompt retrieval @xcite @xcite , prompt ordering @xcite and prompt design @xcite . previous work exploring gpt-3's in-context learning performance for information extraction tasks is limited. @xcite evaluate smaller gpt-3 models on a modified slot filling task in which all examples have at least one entity of interest. additionally, @xcite evaluate the in-context learning performance of gpt-2 on open-domain ner datasets, modified to keep a specific ratio of empty to non-empty examples. our prompt design for biomedical ner draws heavily from both of these works. as far as we know, our work is among the first to comprehensively evaluate gpt-3's in-context learning performance on ie tasks. prompt design. apart from work on in-context learning, several other research directions study how to reformulate nlp tasks as language generation tasks. schick and schütze ( 2021 ) reformulate text classification and natural language inference tasks using a diverse set of manually constructed cloze-style templates as prompts to improve fewshot learning in smaller pretrained language models. @xcite explore a similar setting but leverage an external language model to generate such templates. both of these demonstrate the importance of using a variety of prompt designs. in a related direction, huguet cabot and navigli (2021) achieve state-of-the-art performance on relation extraction benchmarks by reformulating it as an end-to-end sequence-to-sequence task. in the biomedical domain, several works @xcite @xcite follow the multi-task sequence-to-sequence paradigm introduced by @xcite and outperform previous methods on many tasks such as side effect extraction, ner, re, natural language inference and question answering. our prompt design is heavily inspired by many of these efforts to reformulate ie tasks as sequence-to-sequence tasks. true few-shot learning. @xcite argue that previous work overestimates the few-shot learning abilities of plms by using large validation sets for model and prompt selection. this setting has been adopted by many works in this direction in an effort to more accurately estimate few-shot performance @xcite @xcite . ## conclusions in this work, we explored the potential of gpt-3 incontext learning for the high impact task of biomedical information extraction (ie). given that such a paradigm would provide significant advantages for biomedical ie applications, we spent considerable efforts exploring available techniques that have been proven effective for other in-context learning settings. we showed, however, that current techniques do not enable gpt-3 in-context learning to surpass bert-sized plm fine-tuning on a range of benchmark datasets for biomedical ner and re. additionally, we discussed some potentially general limitations of in-context learning in biomedical ie to be explored in future work: its difficulty in handling the null class, such as entityless ner examples and vacuous relation examples for re. apart from posing this question for further study, we hope our work provides helpful guidance to biomedical researchers and practitioners towards more promising and cost-effective tools for low-resource ie such as small plm fine-tuning or perhaps even directly fine-tuning gpt-3.