Get the Best out of 1 B LLM s: Insights from Information Extraction on Clinical Documents (2024)

Abstract: While the popularity of large, versatile language models like ChatGPT continues to rise, the landscape shifts when considering open-source models tailored to specific domains. Moreover, many areas, such as clinical documents, suffer from a scarcity of training data, often amounting to only a few hundred instances. Additionally, in certain settings, such as hospitals, cloud-based solutions pose privacy concerns, necessitating the deployment of language models on traditional hardware, such as single GPUs or powerful CPUs. To address these complexities, we conduct extensive experiments on both clinical entity detection and relation extraction in clinical documents using 1B parameter models. Our study delves into traditional fine-tuning, continuous pre-training in the medical domain, and instruction-tuning methods, providing valuable insights into their effectiveness in a multilingual setting. Our results underscore the importance of domain-specific models and pre-training for clinical natural language processing tasks. Furthermore, data augmentation using cross-lingual information improves performance in most cases, highlighting the potential for multilingual enhancements.

## introduction in the last few years the deep learning revolution has produced significant changes in information extraction (ie) from clinical text. pre-trained large language models (llms) based on attention and transformer architectures (e.g., bert, t5, etc.) have become popular mainly due to their superior performance with respect to traditional machine learning approaches. fine-tuning on downstream tasks has been the standard approach used to transfer general pre-trained knowledge to specific tasks of interest, including information extraction from clinical documents. in addition to fine-tuning, continuous pre-training @xcite ) has shown to be effective to adapt a llm to the medical domain or to a specific set of languages. recently, very large language models have further increased both the amount of data used in the pre-training phase, and the complexity of the model parameters. the resulting models (e.g., gpt-4) achieve high performance with few-shot or even zero-shot incontext learning techniques (i.e., prompting) @xcite . finally, instruction-tuning @xcite has emerged as a powerful approach to align pre-trained llms to human expectations for a number of natural language processing (nlp) and conversational tasks, further improving usability and performance of llms. although there is a clear trend towards large language models with general purpose conversational abilities (e.g., chatgpt), when the choice is constrained to open source models for a domainspecific downstream task (more than often in a lowresource setting), the current landscape of solutions is rather restricted. in addition, there are good reasons to constraint application solutions to small models, particularly because they are computationally manageable, avoiding the need of expensive hardware or to move sensitive data on the cloud. given the above considerations, there is a lack of consensus on what would be the best solution. with the aim of shedding light in the current llm landscape, in this paper we investigate how available small llms perform on fine-tuning, continuous pre-training and instruction-tuning on information extraction from clinical documents. we consider llms that are available open source, are within the range of 1b parameters, and that are available in several versions, allowing to investigate the impact of multilinguality and instructiontuning. our experiments encompass english and italian datasets for clinical entity detection, and italian and spanish for relation extraction, addressing two primary research questions: in addition to core experiments on small llms, we conducted additional experiments aiming at assessing the role of data augmentation on the same models and tasks. data augmentation is a common practice to boost performance, and we are interested in either merging or translating datasets of different languages, as they are becoming more and more available, although in limited amounts. the primary contributions of the paper include: (i) comparing fine-tuning, continuous pre-training, and instruction-tuning of the same pre-trained model on two nlp tasks, a novel comparison to our knowledge; (ii) investigating the relations between continuous pre-training on languages and continuous pre-training on a specialized domain, suggesting a promising research direction; (iii) demonstrating that, through accurate parameter optimization, language models with 1b parameters remain competitive, although absolute performance was not the primary focus; and (iv) indicating that languagebased data augmentation enhances performance in our low-resource setting. ## core experiments in this section, we present core experiments comparing four model versions across two information extraction tasks (ner and re) in clinical documents with limited training data and across three different languages. ## results and discussion in this section we present the results of the core experiments on the two tasks, ner and re. ## data augmentation experiments here we present additional results on two tasks obtained through data augmentation on the core models discussed in section 3. our aim is to explore potential correlations between the core and augmented models. ## conclusions in a world dominated by large language models, this work delves into the efficacy of smaller, domain-specific models in the context of finetuning, continuous pre-training, and instructiontuning on clinical information extraction tasks. our findings suggest that, while general-purpose instruction-tuning offers versatility, it may not always be as effective as continuous pre-training in domain-specific tasks. we observed instances where instruction-tuning (flan-t5) yielded competitive results, but its performance varied across languages and domains. while models like medmt5 designed for the medical domain outperform general-purpose counterparts in ner and re, we find that instruction-tuning varies in effectiveness across languages and domains, emphasizing the importance of domain-specific continuous pre-training. this highlights the need for careful consideration when selecting the most suitable approach for a particular nlp task, weighting factors such as data availability, domain specificity, and computational resources. in a landscape where bigger is often seen as better, our work emphasizes the value of smaller, versatile models in scenarios prioritizing data privacy and traditional hardware. in our future work, we plan to assess parameteroptimized strategies such as peft, lora, qlora, and llama-adapter for training larger models on traditional hardware efficiently. this exploration aims to advance model scalability while considering computational constraints, particularly in resource-limited environments.