QIAI at MEDIQA 2021: Multimodal Radiology Report Summarization (2021)

Abstract: This paper describes the solution of the QIAI lab sent to the Radiology Report Summarization (RRS) challenge at MEDIQA 2021. This paper aims to investigate whether using multimodality during training improves the summarizing performances of the model at test-time. Our preliminary results shows that taking advantage of the visual features from the x-rays associated to the radiology reports leads to higher evaluation metrics compared to a text-only baseline system. These improvements are reported according to the automatic evaluation metrics METEOR, BLEU and ROUGE scores. Our experiments can be fully replicated at the following address: https://github.com/jbdel/vilmedic .

## introduction radiology report summarization is a growing area of research. given the findings and background sections of a radiology report, the goal is to generate a summary (called an impression section in radiology reports) that highlights the key observations and conclusion of the radiology study. automating this summarization task is critical because the impression section is the most important part of a radiology report, and manual summarization can be time-consuming and error-prone. this paper describes the solution of the qiai lab sent to the radiology report summarization (rrs) challenge at mediqa 2021 @xcite . this challenge aims to promote the development of clinical summarization models that generate radiology impression statements by summarizing textual findings written by radiologists. since for most reports, the associated x-rays are available, we aim to evaluate if incorporating visual features from x-rays helps our systems for the report summarization task. this task could be defined as multimodal radiology report summarization (mrrs) as depicted in figure 1 . ## data collection the training set consists of 91,544 examples taken from the mimic-cxr v2.0 dataset @xcite . each training example is a free-text chest radiology report that contains the background, findings and impression sections. two validation sets, each with 2,000 reports were used. one validation set was collected from mimic, and the other was collected from the indiana university chest x-rays report dataset (indiana-university). the test set contains 300 reports from the indiana university dataset, and 300 reports from stanford university school of medicine. all report sections were tokenized using the stanford corenlp tokenizer @xcite . ## model this section describes the two architectures that will be bench-marked in the result section. we start by describing the text-based monomodal architecture at section 3.1. this model only takes as input the findings section and outputs the impression section (the summary). in section 3.2, we incorporate visual information into the monomodal architecture to make it multimodal. ## settings both monomodal and multimodal architectures use a 2-layered bi-directional gru for the encoder, and 1-layered gru for the decoder. each gru has a hidden size of 320 units and our embeddings are of size 200. we apply dropout of 0.4 on the source embeddings x t , 0.5 on the source annotations h and 0.5 on the bottleneck b t . we chose adam (kingma and ba, 2014) as the optimizer with a learning rate of 0.0004 and batch size 64. model parameters are initialized using the he initialization method @xcite . we evaluate the model performance using the rouge-2 f1 metrics @xcite , which is commonly used for evaluating machine summarization task. we stop training when the rouge score does not improve for 10 evaluations on the validation set. in the experiment section, we also report the meteor (banerjee and lavie, 2005) and bleu metrics @xcite . in the scope of this paper, we only use the findings section as input to our models and discard the background section. ## experiments the experiments are carried out as follows: the first one is dictated by the challenge as defined in the second setting consists of injecting 1500 out of the 2000 indiana-dev samples into the training set. we keep the remaining 500 for development. this setting allows more training homogeneity compared to regular-split, we refer to it as the mix-split; figure 2 and 3 depict the results of the best scoring configurations for the monomodal and multimodal models on the development sets. each results is obtained by using beam-search with width varying from 8 to 12. finally, e5 means results are from an ensemble of 6 trained models (i.e. model ensembling). a few observations can be made. first, three of four best scoring models (highlighted in bold) is the multimodal variant. each time, the multimodal model is using the allv interaction. it means that injecting the visual features from the x-rays in both the encoder and the decoder improves summarization. secondly, the only instance where the monomodal variant is better is on the indiana-dev set using the regular-split. one could hypothesize that the multimodal model is sensitive to distribution shift; indeed no indiana samples (and therefore indiana x-rays) are in the training set for this configuration. though using model ensembling seems to mitigates the performance drop, it is still lower that the monomodal baseline. finally, we underline the rouge scores from systems that are significantly different (p-@xmath0.05) than the baseline mono models using the approximate randomization test of multeval @xcite . the underlined scores are all from multimodal systems. 6 related work though relatively new, a few previous work can be denoted in the field of radiology report summarization. zhang et al. (2018) first studied the problem of automatic generation of radiology impressions by summarizing textual radiology findings, and showed that an augmented pointergenerator model achieves high overlap with human references. this model has been extended with an ontologyaware pointer-generator and showed improved summarization quality (macavaney et al., 2019). rl-based approaches have been investigated by li et al. ( 2018 ) and @xcite . more recently, @xcite developed a general framework where the evaluation of the factual correctness of a generated summary is done by factchecking it automatically against its reference using an information extraction module. to our knowledge, this work is the first attempt to use multimodality for radiology report summarization.