Parameter-Efficient Fine-Tuning of LL a MA for the Clinical Domain (2024)

Abstract: Adapting pretrained language models to novel domains, such as clinical applications, traditionally involves retraining their entire set of parameters. Parameter-Efficient Fine-Tuning (PEFT) techniques for fine-tuning language models significantly reduce computational requirements by selectively fine-tuning small subsets of parameters. In this study, we propose a two-step PEFT framework and evaluate it in the clinical domain. Our approach combines a specialised PEFT adapter layer designed for clinical domain adaptation with another adapter specialised for downstream tasks. We evaluate the framework on multiple clinical outcome prediction datasets, comparing it to clinically trained language models. Our framework achieves a better AUROC score averaged across all clinical downstream tasks compared to clinical language models. In particular, we observe large improvements of 4-5% AUROC in large-scale multilabel classification tasks, such as diagnoses and procedures classification. To our knowledge, this study is the first to provide an extensive empirical analysis of the interplay between PEFT techniques and domain adaptation in an important real-world domain of clinical applications.

## introduction large language models (llms) have consistently achieved state-of-the-art performance across various nlp tasks. however, while these models exhibit impressive generalisation abilities, they often struggle to perform in specialised domains such as clinical applications, primarily due to the absence of domain-specific knowledge. the complexity of medical terminology and the presence of incomplete sentences in clinical notes contribute to this challenge @xcite . unfortunately, studies have indicated that even llms 1 the code is accessible via https://github.com/ aryopg/clinical_peft . pretrained with datasets comprising biomedical publications still exhibit suboptimal performance when applied to downstream clinical applications, particularly when compared to llms pretrained with clinical notes @xcite @xcite . this observation suggests that there are intrinsic nuances specific to the clinical context that can only be effectively captured if llms undergo pretraining using clinical datasets. the current approach of adapting pretrained llms to the clinical domain typically involves fine-tuning the entire model parameters @xcite @xcite @xcite . however, due to the rapid increase in the size of llms, such a practice demands extensive computational resources, which may not be readily accessible to all researchers. consequently, this challenge will further exacerbate the disparity between the resource-rich and resource-constrained research institutions @xcite . to address the substantial computational demands, studies have proposed various parameter-efficient fine-tuning (peft) techniques. these techniques present a practical solution by finetuning a small subset of additional parameters while keeping the remaining pretrained parameters fixed. as a result, this strategy significantly alleviates the computational burden while achieving comparable performance to that of full fine-tuning. in this study, we propose a two-step peft framework (see figure 1 ). firstly, we introduce clinical llama-lora, a low-rank adaptation @xcite peft adapter built upon the opensource large language model meta ai (llama) @xcite . then, we introduce downstream llama-lora, which is trained on top of the pretrained clinical llama-lora. downstream llama-lora is specifically designed for clinical downstream tasks. the fusion of the two adapters achieves better performance in clinical nlp downstream tasks compared to clinically trained llms while considerably reducing the computational requirements. this study presents the following contributions: ## conclusions in this study, we propose a two-step peft framework. we introduce clinical llama-lora, a lora (hu et al., 2022) adapter built upon llama @xcite . then, we introduce downstream llama-lora, a task-specific adapter that is trained on top of the pretrained clinical llama-lora. the fusion of the two adapters achieves an auroc score of 76.07% macro-averaged across all clinical nlp downstream tasks, which represents a 3.37% improvement over the best-performing clinical llm. our proposed framework achieves improvement in performance while reducing the computational requirements, which is suited for clinical settings that are often constrained by their computational power.