Boosting Radiology Report Generation by Infusing Comparison Prior (2023)

Abstract: Recent transformer-based models have made significant strides in generating radiology reports from chest X-ray images. However, a prominent challenge remains; these models often lack prior knowledge, resulting in the generation of synthetic reports that mistakenly reference non-existent prior exams. This discrepancy can be attributed to a knowledge gap between radiologists and the generation models. While radiologists possess patient-specific prior information, the models solely receive X-ray images at a specific time point. To tackle this issue, we propose a novel approach that leverages a rule-based labeler to extract comparison prior information from radiology reports. This extracted comparison prior is then seamlessly integrated into state-of-the-art transformer-based models, enabling them to produce more realistic and comprehensive reports. Our method is evaluated on English report datasets, such as IU X-ray and MIMIC-CXR. The results demonstrate that our approach surpasses baseline models in terms of natural language generation metrics. Notably, our model generates reports that are free from false references to non-existent prior exams, setting it apart from previous models. By addressing this limitation, our approach represents a significant step towards bridging the gap between radiologists and generation models in the domain of medical report generation.

## introduction the analysis of radiology images and the subsequent writing of medical reports are crucial tasks performed during the diagnostic process @xcite . however, producing a radiology report is a labor-intensive and timeconsuming task for radiologists, requiring years of training to accurately identify and describe specific abnormalities in medical images @xcite . inspired by the success of image captioning models in deep learning, numerous studies have emerged proposing various models for automated radiology report generation, specifically focusing on chest x-ray images @xcite @xcite @xcite . the automated generation of reports holds the potential to alleviate the high workload of radiologists and expedite the diagnostic process by providing preliminary reports that include useful keywords or observations @xcite . despite the relative success of recent approaches in generating radiology reports from chest x-ray images @xcite @xcite @xcite , a crucial challenge remains unaddressed in these studies: the need to provide models with appropriate prior knowledge, akin to what is available to radiologists. specifically, radiologists are equipped with information about the existence of previous reports and x-ray images, enabling them to compare current exams with past ones, and assess the patient's progress, deterioration, or improvement @xcite . these medical reports often incorporate specific words or phrases for comparison, such as "compared to the previous exam," "in the interval," "referring to the prior x-ray," and so on. in this paper, we refer to these words or phrases as prior expressions, which are also present in general medical datasets such as mimic-cxr @xcite and iu x-ray @xcite , widely utilized for training and evaluating report generation tasks. the inclusion of prior expressions in medical reports is vital for accurate reporting. however, models trained on medical report datasets often generate reports with inappropriate or misused prior expressions, leading to relatively lower performance metrics. the challenge lies in effectively incorporating and utilizing prior expressions within the model's generation process, a crucial aspect yet to be fully ground truth r2gen @xcite m 2 tr @xcite heart size is normal. aorta is tor- both lungs are clear and expanded. heart and mediastinum normal. stable cardiomediastinal silhouette. no focal pulmonary pleural effusion or pneumothorax. no acute bony abnormality. the heart is normal size. the mediastinum is unremarkable. there is no pleural or focal airspace disease. mild chronic degenerative changes are present in the spine. in these falsely referenced reports, which include prior expressions, tend to yield lower evaluation metrics. in figure 1 , we utilize a rule-based labeler (explained in section 3.1) to classify synthetic reports into two categories: negative and positive. the negative class represents reports without prior expressions, while the positive class comprises reports with prior expressions. we then plot the distribution of bleu-4 @xcite scores for each class, along with their respective mean and standard deviation. the results clearly demonstrate that the positive reports achieve lower scores compared to the negative reports. this observation suggests that the transformer-based models do not effectively leverage the generated prior expressions, underscoring the significance of properly incorporating prior information during the training process. in our paper, we address this knowledge gap between generation models and radiologists by infusing prior information into existing models. our aim is to reduce the disparity and enable the improved models to produce more informative and practical reports. since existing datasets such as iu x-ray and mimic-cxr do not contain prior information (previous x-ray images), we adopt a data-driven approach by consulting experienced radiologists. inspired by the chexpert labeler @xcite , we develop a rule-based labeler that extracts prior information by identifying specific patterns and keywords in radiology reports. notably, the rule-based labeler focuses on comparison phrases that indicate whether a medical report corresponds to the first or subsequent exams for each patient. the main contributions of our paper are as follows: ## related work initial research @xcite in radiology report generation employed a basic encoder-decoder architecture, where an encoder extracted key features from medical images and converted them into a latent vector, and a decoder generated the target text from the latent vector. typically, @xcite was used as the encoder, and lstm @xcite ) was chosen as the decoder. subsequently, visual attention mechanisms were introduced to highlight specific image features and generate more interpretable reports @xcite @xcite @xcite . recent studies @xcite @xcite have explored more advanced architectures using transformers to produce more comprehensive and consistent medical reports. alternatively, generating medical reports can be approached as a retrieval task, as similar sentences and a specific writing format are often repeated in most reports. reusing diagnostic text from visually similar x-ray images may result in more consistent and accurate reports compared to generating an entire report from scratch. @xcite demonstrated the superiority of retrieval-based models, outperforming many encoder-decoder-based models. more recently, @xcite introduced a retrieval-based model called cxr-repair, which incorporated contrastive language image pretraining (clip) @xcite to calculate the similarity between text and image embeddings. cxr-repair generates predictions by selecting the most aligned report from a large report corpus given a specific x-ray image. they achieved sota performance on their newly developed metrics. previous approaches have primarily focused on enhancing model performance through advancements in model architecture, while paying relatively less attention to the distinctive characteristics of radiology reports, particularly those involving comparisons. however, there are a few notable exceptions, such as the work by ramesh et al. ( 2022 ), which specifically explores the impact of prior expressions in reports. the authors of that study introduced a novel dataset named mimic-pro, in which they identified and modified reports that contained hallucinated references to non-existent prior exams. the hallucinated references can be seen as a concept analogous to the prior expressions discussed in our paper. @xcite suggested a biobert-based model that paraphrased or removed sentences referring to previous reports or images, arguing that these expressions confuse the report label 1. cardiomegaly is noted and is stable compared to prior examination from xxxx. 1 2. ill-defined opacity is again noted in the region of the lingula. ## method as observed in ## analysis in this section, we compare the ground truths, synthetic reports created by our proposed model, and two previously published models, r2gen and m 2 tr, to assess the effectiveness of our model in generating concise and accurate reports without irrelevant or false priors. similarly, the last two rows of overall, the synthetic reports generated by our proposed model are more concise and accurate compared to those generated by r2gen and m 2 tr, as evidenced by the higher performance in nlg metrics. our model achieves this by avoiding irrelevant or false prior expressions through a rule-based labeler and generating reports that contain only relevant and accurate information. these succinct and precise reports generated by our model will effectively assist radiologists in their practice. ## limitations and ethical considerations our proposed method has certain limitations and ethical considerations that merit discussion. the effectiveness of our approach heavily relies on the rule-based labeler. however, it is important to acknowledge that the labeler may not capture unseen patterns or variations, potentially limiting improvements in various evaluation metrics. moreover, we were unable to conduct a comprehensive human evaluation of the rule-based labeler in this study due to resource constraints. therefore, future work should include a detailed evaluation to assess its performance and address any potential limitations. collaboration with three radiologists at kyoto university is a critical aspect of our work. the regular expressions designed in the rule-based labelers were validated through mutual confirmation by computer scientists and radiologists. however, it is essential to note that the radiologists involved in the collaboration primarily work in a japanese hospital setting. this may introduce potential biases or patterns that are specific to the local context. therefore, it is necessary to cross-check the performance of the rule-based labeler with radiologists from different regions and healthcare systems to ensure broader applicability and minimize any potential bias. regarding the datasets used in our study, we exclusively utilized publicly available datasets that are properly anonymized and de-identified, addressing privacy concerns. however, it is crucial to emphasize that if datasets containing comparison exams become available in the future, additional precautions must be taken to ensure that no personally identifiable information is inadvertently disclosed or used in a manner that could identify individual patients. by acknowledging these limitations and ethical considerations, we aim to encourage future research and discussions in the field, driving advancements in radiology report generation while prioritizing patient privacy, accuracy, and fairness. these considerations will contribute to the development of robust and ethically sound approaches in radiology report generation. ## conclusion in this study, we present a novel approach to generate medical reports from chest x-ray images, aiming to bridge the gap between radiologists' knowledge and the lack of prior information in generation models. to achieve this, we developed a rule-based labeler capable of extracting comparison priors from radiology reports in the iu x-ray and mimic-cxr datasets. these priors were subsequently integrated into state-of-the-art models for conditional report generation, allowing our approach to emulate the realistic diagnostic process of radiologists who possess prior information about patients. our experimental results demonstrate the superiority of our method over previous state-of-the-art models, as indicated by improved performance in terms of nlg metrics and a significant reduction in the occurrence of falsely referred prior exams. through our analysis, we show that the incorporation of comparison priors leads to the generation of more accurate and concise reports, thereby holding great potential to enhance the quality and efficiency of medical report generation for chest x-ray images. ultimately, this advancement benefits healthcare professionals and patients by providing more reliable and informative reports. furthermore, our work highlights the future potential of generating medical reports in an end-to-end fashion if a dataset containing all previous exams becomes available. the ability to leverage comprehensive prior information would further amplify the accuracy and effectiveness of medical report generation, paving the way for improved healthcare outcomes. the cardiomediastinal silhouette is normal in size and contour. low lung volumes without focal pneumothorax or large pleural effusion. normal xxxx. the trachea is midline. the cardiomediastinal silhouette is normal and unchanged compared to prior examination. densities overlying the heart xxxx are xxxx coronary artery stents. there are small round calcific densities in the bilateral lobes which are unchanged from prior exam and xxxx represent sequelae from old granulomatous disease. the cardiomediastinal silhouette is within normal limits for appearance. the thoracic aorta is tortuous. no focal areas of pulmonary consolidation. no pneumothorax. no large pleural effusion. mild degenerative changes and osteopenia of the thoracic spine. cardiomediastinal silhouette is within normal limits. no focal consolidation. there is right lower lobe scarring. no pneumothorax or large pleural effusion. granulomas present. no acute bony abnormalities. the trachea is midline. the cardiomediastinal silhouette is normal. the superior thoracic spine is again unchanged from prior. lucent pulmonary parenchyma is consistent appearance with emphysema and appears unchanged from prior xxxx. no evidence of pneumothorax. no focal airspace disease or pleural effusion. vague density in the medial right lung apex most xxxx is. the cardiomediastinal silhouette is normal in size and contour. no focal pneumothorax or large pleural effusion. negative for acute bone abnormality. ground truth m 2 tr (cornia et al., 2020) m 2 tr w/ prior (ours) no change lung xxxx. xxxx opacities are present in the right lower lobe. no focal infiltrates. heart and mediastinum are unremarkable. aorta normal. stable cardiomediastinal silhouette with normal heart mediastinal calcifications suggest a previous granulomatous process. apical irregularities also present on the previous exam suggestive of scarring. no focal alveolar no definite pleural effusion seen. no typical findings of pulmonary edema. no pneumothorax. the heart is normal in size and contour. there is no mediastinal widening. the lungs are hyperexpanded. scattered granuloma. no focal airspace disease. no large pleural effusion or pneumothorax. the xxxx are intact. the trachea is midline. negative for pleural or focal airspace consolidation. the heart size is normal. the heart is top normal in size. the mediastinum is unremarkable. the lungs are hypoinflated but grossly clear. significant degenerative changes of the xxxx are again noted bilaterally. the heart is normal in size. the mediastinum is unremarkable. the lungs are clear.