Large Language Models as Instructors: A Study on Multilingual Clinical Entity Extraction (2023)

Abstract: In clinical and other specialized domains, data are scarce due to their confidential nature. This lack of data is a major problem when fine-tuning language models. Nevertheless, very large language models (LLMs) are promising for the medical domain but cannot be used directly in healthcare facilities due to data confidentiality issues. We explore an approach of annotating training data with LLMs to train smaller models more adapted to our problem. We show that this method yields promising results for information extraction tasks.

## introduction clinical notes contain the interactions between the patient and healthcare staff. professionals record their impressions, observations, and various medical procedures performed. despite the computerization of clinical documents, notes should remain fairly expressive and in a free format to save time for healthcare personnel and allow for the description of unusual situations @xcite . moreover, a large amount of crucial information is exclusively contained in clinical notes. according to a study by @xcite , approximately 80% of patient phenotypes (a set of observable biological and physical characteristics that can characterize a disease) are present only in free text. these documents are difficult to use without advanced methods such as deep learning in nlp. the use of such methods requires the collection and annotation of a significant amount of medical data. however, @xcite proposes the term "dataset debt," highlighting that learning data in the biomedical field is poorly accessible, poorly documented, and opaque as to its reusability in a commercial or a hospital context. according to the article, only 13% of the 167 analyzed datasets are accessible and downloadable, 22% use a standard structured format, and 40% are in the public domain. in recent years, large language models (llms) have proved their ability to perform a wide range of tasks with high accuracy in a zero-shot or a few-shots contexts. this trend holds great potential for clinical nlp, as preliminary results show promise for information extraction tasks @xcite . however, the clinical domain presents unique challenges due to the confidential and linguistically specific nature of its data, which can make collection and annotation time-consuming and expensive. using llms for efficient information extraction without training data could be attractive, but it raises confidentiality concerns. the model deployment should be controllable, and the model's predictions should evolve to fit a specific and changing annotation guideline. most multilingual llms are not freely available @xcite @xcite , to the best of our knowledge, only bloom is opensource and deployable in a custom infrastructure. the computing resources to use these models remains challenging for healthcare establishments. one approach to address these issues is to distil llms into a smaller model via weak supervision. weak supervision has recently gained community attention because it alleviates the annotation task. this technique refers to annotating datasets using rule-based, heuristic, dictionary extraction or more advanced methods and then training the smaller model on this dataset. in the same way, knowledge distillation aims to transfer knowledge from a master model to a student model. it has often been used to compress large-scale models to improve memory footprint and the inference speed @xcite . moreover, student models trained through knowledge distillation can be more easily monitored and versioned. hosting them increases the healthcare centre's sovereignty, and they become more compliant with existing privacy policies, as input data or predictions don't leave the building. ## motivation and contributions we study the use of llms in the knowledge distillation technique via weak supervision in the multilingual clinical domain, especially in clinical entity extraction. we extend the @xcite study in the sense that we propose an in-depth study of the use of instructgpt-3 to annotate a training dataset. our work 1 mainly aims to compare the annotation quality using weak supervision tasks on a smaller model (figure 1 ). finally, we propose to combine annotations provided by instructgpt-3 and the dictionary extraction method. this takes form in these contributions: ## related works weak supervision deep learning approach has achieved remarkable success in several domains beyond nlp @xcite . however, the main bottleneck is collecting massively annotated data. to address this issue, weak supervision replaces ground-truth annotation with automatic annotation based on heuristic rules, gazetteers or constraints linguistic rules to address. some techniques called distant supervision exploit semantic links from knowledge bases or ontologies @xcite . @xcite proposes an iterative self-training method to combine classic weak supervision and inference of the learning model to extract entities not covered by the initial heuristic rules. in the clinical domain, weak supervision has already been used for specific use cases @xcite @xcite . ## limitation one limitation of our study is the small size of the test set, which may impact the generalizability of our results. additionally, we restrained our work on clinical entity extraction; in future work, we would investigate more in several tasks using the e3c temporality layer to cover a task of name entity recognition and relation extraction tasks. finally, the e3c guidelines have been designed for clinical entity extraction and entity-linking via umls entities. after the first step of manual annotation, some spans of the entities have been modified to fit as close as possible to the semantical concepts found in umls @xcite . for instance, clinical entities could be split into separate disorder concepts, and the extent of a disorder candidate could be reduced to fit with a concept. these biases could induce additional difficulties in finding the correct span for a given model. ## conclusion our results demonstrate that the knowledge distillation with instructgpt-3 outperforms the dictionary supervision for extracting clinical entities. we show that mixing these approaches to build a training dataset brings diversity to the annotations and improves the distilled model performance. weak-supervision approach with llms is relatively promising for creating a training dataset. this reduces the annotation cost and, at the same time, focuses the manual annotation on the test set, which is one of the most prominent parts of highstake domains like healthcare. furthermore, the interest of the approach is also to fine-tune a small to medium-sized lm that may be used locally without the leak of confidential medical data and with a reduced energy cost. in a low-resource context, such as basque, llms offer a competitive alter-native to the classic weak supervision technique, which requires linguistic resources. furthermore, we aim to investigate advanced techniques to combine various annotations by incorporating confidence measures from the different predictions. using other llms predictions and ensemble, the difference could be pertinent because the annotation diversity can improve a model's performance, as we observed on s m onosilver (figure 4 ). additionally, we will consider utilising performance metrics (such as recall and precision) to decide which type of annotations (begin or innertokens) to retain for each prediction method. finally, adapting cot or generated knowledge @xcite for clinical entity extraction could improve llm's precision. to our knowledge, none of these techniques has been adapted to clinical information retrieval. we could craft a prompt with different annotation steps through different examples. at each annotation step, we describe a precise instruction and its result. for example, incorporating the three steps of the e3c annotation into the prompt could help encourage the llm to better adhere to the guideline.