L lama MTS : Optimizing Metastasis Detection with Llama Instruction Tuning and BERT -Based Ensemble in I talian Clinical Reports (2024)

Abstract: Information extraction from Electronic Health Records (EHRs) is a crucial task in healthcare, and the lack of resources and language specificity pose significant challenges. This study addresses the limited availability of Italian Natural Language Processing (NLP) tools for clinical applications and the computational demand of large language models (LLMs) for training. We present LlamaMTS, an instruction-tuned Llama for the Italian language, leveraging the LoRA technique. It is ensembled with a BERT-based model to classify EHRs based on the presence or absence of metastasis in patients affected by Breast cancer. Through our evaluation analysis, we discovered that LlamaMTS exhibits superior performance compared to both zero-shot LLMs and other Italian BERT-based models specifically fine-tuned on the same metastatic task. LlamaMTS demonstrates promising results in resource-constrained environments, offering a practical solution for information extraction from Italian EHRs in oncology, potentially improving patient care and outcomes.

## introduction electronic health records (ehrs) represent the principal data source for hospital centers, housing invaluable information regarding medical histories, treatments, examinations, disease progression and symptoms of a patient. however, efficiently extracting this data with high accuracy and minimal computational resources presents a growing challenge, particularly in the context of the italian language. while solutions specialized in the clinical domain are readily available for the english language @xcite @xcite , the exploration of similar solutions for the italian language remains limited, with only a handful of alternatives @xcite . consequently, our objective is to investigate novel approaches that could be implemented in real-world clinical contexts, to extract specific outcomes from italian textual data. for this purpose, we searched for methods that enable fine-tuning of large language models for specific tasks while minimizing computational resource consumption. recent studies have showcased the effectiveness of implementing instruction-tuning on pre-trained large language models @xcite @xcite , also leveraging techniques such as lora @xcite . through this approach, the number of trainable parameters is reduced, and the model is trained to respond to specific instructions provided during training. in this paper, we introduce llamamts (figure 1 ), a fine-tuned llama model, through the lora instruction tuning technique. our model is designed to identify the presence of tumoral metastasis by analyzing ehrs from patients diagnosed with breast cancer. llama was fine-tuned by using as base model camoscio @xcite , which is a llama adapter for the italian language, trained on the italian translation of the stanford alpaca dataset @xcite . to further enhance model performance, we employed an ensemble approach by incorporating a bertbased model fine-tuned on the same classification task. additionally, to allow the model to learn from entire ehrs (which may exceed the maximum token limit allowed by llama during training), we implemented text summarization on both the training and testing datasets. this enabled information extraction from shorter and more concise texts, reducing the noise that long texts may cause. to evaluate llamamts performances, we compare it with several benchmarks, exploring zeroshot llms configurations and fine-tuning known bert-based model for text classification. results show that our approach, which leverages instruction-tuning and model ensembling, outperforms all the other baselines on our metastatic classification task. ## method our methodology involves 1) the selection of the data corpus for fine-tuning, 2) the summarization of ehrs to obtain shorter texts, 3) the instruction tuning of an existing large language model on the metastatic classification task and 4) the ensemble of the obtained instruction-tuned model with a bertbased model fine-tuned on the same task. ## experiments we started by generating the instruction-tuned model on the metastatic classification task, using the summarized ehrs as training data. we then compared the performance of the instruction-tuned model with several baseline methods, including bert-based approaches fine-tuned on our classification task and large language models implemented in a zero-shot environment. finally, we applied the ensemble techniques between the instruction-tuned model and the bertbased fine-tuned model with the highest performance in order to obtain the final llamamts classifier. ## conclusions the instruction-tuning allowed us to specialize an existing large language model on a medical classification task in an optimized fine-tuning environment, using the lora approach. our study shows that llamamts, which is a fne-tuned llm using lora, has higher performance metrics when compared to the base model camoscio and to other existing approaches that involve conversational llms and bert-based checkpoints ( with this work, we extended advanced nlp techniques on clinical ehr data, automating processes through the usage of powerful language models, trained in the italian language, on a specific classification task, for the extraction of the tumor metastasis information from ehrs. we proposed an approach that is easily portable to other kinds of outcomes, for extracting information not necessarily available in a structured format, from textual ehrs. furthermore, the instruction-tuning approach enables fine-tuning large language models in reasonable time frames, leveraging mid-range computational resources.