AIRI at RRG 24: LL a V a with specialised encoder and decoder (2024)

Abstract: We present a new approach to generating the ‘Findings’ and ‘Impression’ sections in the chest X-rays radiology reports, developed as part of the shared radiology task at BioNLP 2024. By integrating a DINOv2 vision encoder trained on medical data with specialized biomedical large language model using the LLaVA framework, our method addresses complex medical semantics and diverse findings in imaging. We use datasets from PadChest, BIMCV-COVID19, CheXpert, OpenI, and MIMIC-CXR. The evaluation metrics demonstrate our method’s effectiveness and the potential for automating the generation of radiology reports.

## introduction the automatic generation of radiology reports from chest x-rays is a challenging and significant task in the field of biomedical natural language processing (bionlp). the growing volume of medical imaging data and the limited number of radiologists necessitate the development of robust automated systems to assist in report generation. such systems not only have the potential to improve clinical workflow efficiency but also to ensure consistency and comprehensiveness in radiological interpretations. in recent years, advancements in deep learning and natural language processing have paved the way for innovative approaches to tackle this task. the new approaches typically involve the integration of convolutional neural networks (cnns) or visual transformers for image feature extraction with recurrent neural networks (rnns) or transformers for text generation @xcite . despite the progress, challenges such as capturing complex medical semantics, handling diverse imaging findings, and ensuring the clinical accuracy of generated reports remain. this paper explores a new method for generating the findings and impression sections of radiology reports from chest x-rays. our approach is to combine a vision encoder, self-trained on medical data, with specialized biomedical llm for text generation, using llava framework. this work was done as a part of radiology report generation ahared task at bionlp 2024 workshop @xcite using the data provided by the organizers. the metrics were calculated using the vilmedic platform @xcite . ## evaluation metrics in the evaluation of radiology report summarization systems, several metrics are commonly used to assess the performance and accuracy of the generated summaries. these metrics ensure that the summaries produced by the models are not only syntactically and semantically correct but also factually accurate. the metrics used in this competition where bleu @xcite , rouge @xcite , bertscore @xcite , f1-chexbert @xcite , and f1-radgraph @xcite . ## methods and results we used the llava model @xcite with a dinov2 encoder @xcite and openbio-llm-8b (ankit pal, 2024) as a text decoder. the whole pipeline was implemented using huggingface's transformers @xcite and trl (von werra et al., 2020) libraries. for image encoding we used a dinov2 model with the following parameters: vit-base 14, initialized from torch.hub's dinov2 vitb14 for text generation, we used openbiollm-8b, an open-source language model designed specifically for the biomedical domain. we used the same model for generating both impression and findings, using different prompts: either "write findings for this x-ray." or "write impression for this x-ray.". we used the following system prompt, inspired by llava-med @xcite :"you are a large language and vision assistant. you are designed to assist human with a variety of medical visual content and clinical research tasks using natural language. follow the instructions carefully and provide clinically valid answers." our results on hidden test sets are presented in