i H ealth- C hile-1 at RRG 24: In-context Learning and Finetuning of a Large Multimodal Model for Radiology Report Generation (2024)

Abstract: This paper presents the approach of the iHealth-Chile-1 team for the shared task of Large-Scale Radiology Report Generation at the BioNLP workshop, inspired by progress in large multimodal models for processing images and text. In this work, we leverage LLaVA, a Visual-Language Model (VLM), composed of a vision-encoder, a vision-language connector or adapter, and a large language model able to process text and visual embeddings. We achieve our best result by enriching the input prompt of LLaVA with the text output of a simpler report generation model. With this enriched-prompt technique, we improve our results in 4 of 5 metrics (BLEU-4, Rouge-L, BertScore and F1-RadGraph,), only doing in-context learning. Moreover, we provide details about different architecture settings, fine-tuning strategies, and dataset configurations.

## introduction the task of radiology report generation (rrg) from medical imaging through deep neural networks is an active area of research @xcite . for one thing, addressing and solving this task can help radiologists in identifying anomalies from one or more input images, as well as save them time on administrative chores like typing text reports. thus, doctors can spend more time with patients rather than clinical software @xcite . there have been several methods introduced in recent years to address this task but only recently the progress in opensource multimodal generative systems has opened the room for improving performance by integrating different modalities (text and images) in the same model. in this article, we describe our work leveraging the multimodal model llava @xcite to address this task. ## https://huggingface.co/dcampanini there are several options to leverage llava for this challenge, such as utilizing the original version llava-1.0 @xcite , the clinically finetuned version llava-med @xcite , as well as the newest version llava-1.5 @xcite . due to hardware limitations, in this challenge, we used the language model component with 7 billion parameters (llama 1.0 and vicuna) and we tested several configurations focusing on the findings generation task. in this document, we describe details of several configurations tested, including different vision encoders (clip and biomedclip), vl projector (matrix and mlp) and language model for text decoding (llama1.0 and vicuna-7b). among our findings, we highlight that integrating the output of another method as input context for llava resulted in our best version for the challenge. ## experiments and results the first result in from our experiments with model-1.0, we see that considering only mimic-cxr we have good enough results comparable to using mimic-cxr + chexpert, and consistently outperforming the same model-1.0 trained with the complete challenge datasets ( when we make use of biomedclip @xcite we apply the enriched-prompt technique to the model with the best f1-radgraph, which is the model-1.1 clip-336 vicuna1.5-7b, the result of employing this procedure is an improvement in bleu-4, rouge-l, bert-score, and f1-radgraph, but a big fall in f1-chexbert ( another consequence of implementing an enriched prompt is the generation of shorter findings in comparison with those generated by the other model versions and by the classifier plus template sentences. in ## conclusion in this work, we performed an analysis of different model architectures based on llava-med, we conclude that using the best possible vision-encoder, and llm we can improve some specific aspects of the system, such as the nlp overlapping (bleu-4 and rouge-l) or the more classification related metrics (f1-chexbert), nevertheless to see more consistent results we suggest that more quality data is needed, particularly for alignment (stage 1). moreover, since the promising results in f1-chexbert obtained with biomedclip is convenient to develop a vision-encoder custom to x-ray images. finally, the enriched-prompt techniques show auspicious results. it can work as a guide for the llm, it shows good metrics when we calculate bleu-4, rouge-l, bertscore, and f1-radgraph, but it should be complemented with an accurate classifier system to improve the f1-chexbert.