Improving Radiology Summarization with Radiograph and Anatomy Prompts (2023)

Abstract: The impression is crucial for the referring physicians to grasp key information since it is concluded from the findings and reasoning of radiologists. To alleviate the workload of radiologists and reduce repetitive human labor in impression writing, many researchers have focused on automatic impression generation. However, recent works on this task mainly summarize the corresponding findings and pay less attention to the radiology images. In clinical, radiographs can provide more detailed valuable observations to enhance radiologists’ impression writing, especially for complicated cases. Besides, each sentence in findings usually focuses on single anatomy, such that they only need to be matched to corresponding anatomical regions instead of the whole image, which is beneficial for textual and visual features alignment. Therefore, we propose a novel anatomy-enhanced multimodal model to promote impression generation. In detail, we first construct a set of rules to extract anatomies and put these prompts into each sentence to highlight anatomy characteristics. Then, two separate encoders are applied to extract features from the radiograph and findings. Afterward, we utilize a contrastive learning module to align these two representations at the overall level and use a co-attention to fuse them at the sentence level with the help of anatomy-enhanced sentence representation. The experimental results on two benchmark datasets confirm the effectiveness of the proposed method, which achieves state-of-the-art results.

## introduction a radiology report of an examination is used to describe normal and abnormal conditions with one medical image and two important text sections: findings and impression. the findings section is a free-text description of a clinical radiograph (e.g., † corresponding author. chest x-ray), providing the medical image's detailed observations. meanwhile, the impression is a more concise statement about critical observations summarized from the findings, images and the inference from radiologists and provides some clinical suggestions, such that in practice, clinicians prefer to read the impression to locate the prominent observations and evaluate their differential diagnoses. however, writing impressions is time-consuming and in high demand, which draws many researchers to focus on automatic impression generation (aig) to alleviate the workload of radiologists @xcite @xcite @xcite . for example, @xcite @xcite propose to extract medical ontologies and entities from findings and then utilize graph neural networks (gnns), dual encoder, or reinforcement learning to integrate this knowledge into general sequence-to-sequence models for promoting aig. yet, most existing studies mainly focus on fully using findings to produce impressions and pay rare attention to medical radiography. owing to the fact that some diseases tend to have similar observations, they are difficult to get a clear diagnosis only depending on the textual statements. in this situation, most radiologists usually consider both the image and findings to make a more accurate clinical suggestion in impressions. besides, many approaches have been proposed for radiology report generation and have achieved considerable success @xcite , whose goal is to generate the findings based on a given medical image, further showing the value of knowledge in the medical image. in radiology reports, each findings can be regarded as a text representation of the corresponding medical image, and meanwhile, each image is a visual representation of the findings such that these two modal data can be effectively aligned. therefore, we propose a task that integrates the images and anatomy-enhanced findings for impression generation. according to communication with radiologists, each sentence in the findings focuses on single anatomy, so the sentence-level representation should be easier to align to a certain anatomical region of the image. to enhance such a process, we first construct some rules under the guidance of radiologists and utilize these rules to extract the main anatomies from each sentence. then we put these anatomies at the beginning of the sentence to emphasize anatomy information. next, we use a visual extractor to extract visual features from the radiology image and apply a transformerbased text encoder to embed the corresponding findings. afterward, an extra encoder is used to further model visual features, whose output will be aligned to the textual representation at the document level by a contrastive learning module. finally, we employ a co-attention to integrate the visual and text features at the sentence level to obtain the final fused representation, which is then input to the decoder to generate the impressions. experimental results on two benchmark datasets, mimic-cxr and openi, demonstrate the effectiveness of our proposed model, which achieves better performance than most existing studies. furthermore, analysis of impression length shows that our proposed multimodal model is better at long impression generation, where our model obtains significant improvements when the impression is longer than 20. ## method we follow existing studies on report generation @xcite and impression generation @xcite @xcite and utilize the standard sequence-to-sequence paradigm for this task. in doing so, we regard patch features extracted from radiology image x i as one of the source inputs. in addition, the other input is the findings sequence x @xmath0, s 2 , • • • , s m , where m is the number of sentence and can be defined as: ## analyses 5.1 impression length to test the effect of the length of impressions in aig, we categorize the generated impressions on the mimic-cxr test set into several groups according to the length of reference impression, with the r-1 scores shown in figure 3 . note that the average impression length for mimic-cxr is 17. we can observe that these models tend to have worse performance with increasing impression length, especially in the last group, where all obtain the worst r-1 scores. our proposed model achieves more promising results in most groups, except the first group where the base-findings achieves the best results, which illustrates that our model is better at generating longer impressions. the main reason is that short impressions are usually normal observations without complicated abnormalities so that findings are enough to describe such information, and images may lead to some redundant noise due to their being too detailed. in contrast, for the long impression, detailed information can complement textual features to help the model accurately grasp complex observations. ## conclusion this paper proposes an anatomy-enhanced multimodal summarization framework to integrate radiology images and text for facilitating impression generation. in detail, for radiology images, we use a visual extractor to extract detailed visual features. for radiology findings, we first plan anatomical prompts into each sentence by keywords and rules and then apply a pre-trained encoder to distillate features from modified findings. afterward, we employ a contrastive learning module to align the visual and textual features at the document level and use a co-attention to fuse these two features at the sentence level, which are then input to the decoder to improve impression generation. furthermore, experimental results on two benchmark datasets illustrate the effectiveness of our model, especially for long impression generation, where our model achieves significant improvements.