Overview of the R ad S um23 Shared Task on Multi-modal and Multi-anatomical Radiology Report Summarization (2023)

Abstract: Radiology report summarization is a growing area of research. Given the Findings and/or Background sections of a radiology report, the goal is to generate a summary (called an Impression section) that highlights the key observations and conclusions of the radiology study. Recent efforts have released systems that achieve promising performance as measured by widely used summarization metrics such as BLEU and ROUGE. However, the research area of radiology report summarization currently faces two important limitations. First, most of the results are reported on private datasets. This limitation prevents the ability to reproduce results and fairly compare different systems and solutions. Secondly, to the best of our knowledge, most research is carried out on chest X-rays. To palliate these two limitations, we propose a radiology report summarization (RadSum) challenge on i) a new dataset of eleven different modalities and anatomies pairs based on the MIMIC-III database ii) a multimodal report summarization dataset based on MIMIC-CXR enhanced with a brand-new test-set from Stanford Hospital. In total, we received 112 submissions across 11 teams.

## introduction the radiology report documents and communicates crucial findings in a radiology study. a standard radiology report usually consists of a background section that describes the exam and patient information, a findings section, and an impression section @xcite . in a typical workflow, a radiologist first dictates the detailed findings into the report and then summarizes the salient findings into the more concise impression section based also on the condition of the patient. automating this summarization task is critical because the impression section is the most important part of a radiology report, and manual summarization can be time-consuming and error-prone. despite its importance, recent studies @xcite @xcite or challenges @xcite on new automated radiology report summarization systems solely focus on chest x-ray or sometimes, omit the modality and anatomy concerned in the used radiology reports @xcite . in addition, while existing models are optimized to generate summaries achieve high performance on the rouge metric @xcite , this does not guarantee factually correct summaries @xcite . to palliate these two limitations, we propose a challenge with two brand new pre-processed datasets of new modalities (mr and ct), anatomies (chest, head, neck, sinus, spine, abdomen, pelvis) and institutions (stanford hospital). we further use a new metric, called f 1 radgraph @xcite to evaluate the factual completeness and correctness of generated radiology impressions. ct abd/pelv ct chest ct head 15,989 12,786 31,402 ct spine mr head ct neck 5,517 7,313 1,140 ct sinus mr spine mr abdomen 1,267 2,821 1,061 mr neck mr pelvis 230 253 the challenge took place on vilmedic @xcite , a modular framework for vision and language multimodal research in the medical field. this library contains reference implementations of state-of-the-art medical vision and language architectures but also hosts ai challenges. ## datasets two summarization datasets were proposed for the shared task. the mimic-iii summarization dataset contains 11 different anatomy-modality pairs @xcite and the mimic-cxr summarization dataset contains findings and impression sections from chest x-ray studies paired with chest x-rays images. in addition to the official mimic-cxr test-set, we are also releasing a brand new out-of-institution test-set from the stanford hospital. for each report, we extract the findings and impression section. however, the finding section is not always stated as such. with the help of one board-certified radiologist, and for each modality/anatomy pair, we create a mapping of the section header that acts as "findings". as an example, for ct head, findings could be referred as "non-contrast head ct", "ct head", "ct head without contrast", "ct head without iv contrast", "head ct", "head ct without iv contrast" or "cta head". this "findings" mapping contains up to 537 candidate sections for our whole dataset. we also discarded reports where multiple studies are pooled in the same radiology reports, leading to multiple intricate observations in the impression section. we release our mapping as well as the code to recreate the dataset from scratch). the final dataset consists of a train, validation and test splits of respectively 59,320, 7,413 and 13,057 findings-impression pairs. in the scope of this challenge, the test split has been split in two: one public test-set of 6,526 samples and one hidden test-set (participants don't have access to the ground-truth impressions) of 6,531 samples. ## metrics we proceed to evaluate the submitted systems using the bleu @xcite and rougel metrics @xcite . a few other metrics were used to score the factual correctness of the generated impressions: f 1 chexbert @xcite this score uses chexbert @xcite , a transformer-based model trained to output abnormalities of chest xrays given a radiology report as input. f 1 chexbert is the f1-score between the prediction of chexbert over the generated report ŷ and the corresponding reference y. the f1 score is calculated over the 5 main observations to be consistent with @xcite . this metric is suitable for the mimic-cxr summarization dataset exclusively. bertscore @xcite leveraging radgraph @xcite annotation scheme and model to design f-score style score that measures the consistency and completeness of generated radiology reports compared to reference reports based on observation and anatomy entities. ## results it's worth noting that the f 1 radgraph metric doesn't necessarily correspond with the rankings based on f 1 chexbert on the mimic-cxr dataset. this indicates that each metric offers a unique perspective on the factual correctness of the generated summaries. ## system descriptions knowlab this solution is a comparison of state-of-the-art generative language models in generating high-quality summaries from radiology reports. a two-stage fine-tuning approach was introduced for utilizing knowledge learnt from different datasets. in particular, authors first compared the fine-tuning results using mimic-iii with sota generative pre-trained language models including bart and t5 models as well as their biomedical variants biobart and scifive. then they did a second round of fine-tuning with mimic-cxr using the bart and t5 models fine-tuned from mimic-iii, by freezing the last two layers during training. they evaluated the performance of our method using a variety of metrics, including bleu, rouge, bertscore, chexbert, and radgraph. our results revealed the potentials of different models in summarizing radiology reports and demonstrated the effectiveness of the two-stage fine-tuning approach. ## conclusion we proposed a radiology report summarization (radsum23) challenge with two brand new preprocessed datasets of new modalities (mr and ct), anatomies (chest, head, neck, sinus, spine, abdomen, pelvis) and institutions (stanford hospital). we further use a new metric, called f 1 radgraph @xcite to evaluate the factual completeness and correctness of generated radiology impressions. our competition gathered 112 submissions across 11 teams. the final leaderboard can be accessed at the following address: https://vilmedic.app/misc/bionlp23/ leaderboard/ foot_0 . the complete data of this competition (datasets and submissions from participants) can be downloaded at https://vilmedic.app/ misc/bionlp23/sharedtask/ .