MAIRA at RRG 24: A specialised large multimodal model for radiology report generation (2024)

Abstract: This paper discusses the participation of the MSR MAIRA team in the Large-Scale Radiology Report Generation Shared Task Challenge, as part of the BioNLP workshop at ACL 2024. We present a radiology-specific multimodal model designed to generate radiological reports from chest X-Rays (CXRs). Our proposed model combines a CXR-specific image encoder RAD-DINO with a Large Language Model (LLM) based on Vicuna-7B, via a multi-layer perceptron (MLP) adapter. Both the adapter and the LLM have been fine-tuned in a single-stage training setup to generate radiology reports. Experimental results indicate that a joint training setup with findings and impression sections improves findings prediction. Additionally, incorporating lateral images alongside frontal images when available further enhances all metrics. More information and resources about MAIRA can be found on the project website: http://aka.ms/maira.

## introduction an impactful application of natural language generation in the medical field involves the creation of support systems that interpret patient x-ray images and produce a draft report detailing the clinical findings in these images. such systems have the potential to enhance and expedite radiology reporting workflows. in this regard, a shared task challenge for radiology report generation has been organised as part of the acl 2024 bionlp workshop 1 (rrg24; @xcite . this shared task challenge uses the first large-scale collection of radiology report generation datasets based on 1 https://stanford-aimi.github.io/rrg24/ mimic-cxr @xcite , chexpert @xcite , padchest @xcite , bimcv-covid19 @xcite , and @xcite datasets. this paper covers the participation of the maira (multimodal ai for radiology applications) team at microsoft research in this challenge. we build on the architecture and training approach from our earlier work maira-1 @xcite . this approach combines a cxrspecific image encoder (rad-dino, pérez-garcía et al. ( 2024 )) with a pretrained llm (vicuna-7b 1.5, @xcite ) via an adapter which is an mlp of 4 layers. both llm and adapter are finetuned in a single stage for the task of radiology report generation, while the image encoder is pretrained following the self-supervised dinov2 approach @xcite . for this competition, we produce a variant of maira-1 which is trained only on public data, and further extended it to incorporate lateral images. we share the following outcomes: the image encoder is trained using the images from the mimic-cxr @xcite , chexpert @xcite , padchest (bustos et al., 2020), nih-cxr @xcite , and brax @xcite datasets. both frontal and lateral view images were used. the dataset statistics by source are available in ## experiments we perform experiments in four settings as defined in ## conclusion we have presented maira-rrg24, a radiologyadapted large multimodal model based on the maira-1 architecture @xcite with a rad-dino-like (pérez-garcía et al., 2024) image encoder, trained exclusively with the data available for the rrg24 challenge @xcite . it ex-hibits competitive performance in both lexical and clinical metrics. it benefits from a domain-specific image encoder, a joint training setup for findings and impression prediction leveraging lateral images when available.