ICON : Improving Inter-Report Consistency in Radiology Report Generation via Lesion-aware Mixup Augmentation (2024)

Abstract: Previous research on radiology report generation has made significant progress in terms of increasing the clinical accuracy of generated reports. In this paper, we emphasize another crucial quality that it should possess, i.e., inter-report consistency, which refers to the capability of generating consistent reports for semantically equivalent radiographs. This quality is even of greater significance than the overall report accuracy in terms of ensuring the system’s credibility, as a system prone to providing conflicting results would severely erode users’ trust. Regrettably, existing approaches struggle to maintain inter-report consistency, exhibiting biases towards common patterns and susceptibility to lesion variants. To address this issue, we propose ICON, which improves the inter-report consistency of radiology report generation. Aiming to enhance the system’s ability to capture similarities in semantically equivalent lesions, our approach first involves extracting lesions from input images and examining their characteristics. Then, we introduce a lesion-aware mixup technique to ensure that the representations of the semantically equivalent lesions align with the same attributes, achieved through a linear combination during the training phase. Extensive experiments on three publicly available chest X-ray datasets verify the effectiveness of our approach, both in terms of improving the consistency and accuracy of the generated reports.

## introduction being part of the diagnostic process, radiology report generation @xcite @xcite has garnered significant attention within the research community, due to consistent ✓ inaccurate ✗ equal figure 1 : given two semantically equivalent cases (i.e., case a and case b), an example to illustrate the difference between three radiology report generation systems: a consistent and accurate system (i.e., system α) and a consistently inaccurate system (i.e., system β), and an inconsistent system (i.e., system γ). however, constructing a credible report generation system goes beyond the overall accuracy. there is another crucial quality for report generation systems that has been largely overlooked in the existing literature of medical report generation, which is, inter-report consistency @xcite . to illustrate the disparity between accuracy and inter-report consistency, we exemplify two semantically equivalent cases as shown in figure 1 , where they share similar observations and reports. specifically, system α demonstrates the ability to maintain both inter-report consistency and factual accuracy for two similar cases (i.e., "small bilateral pleural effusions" for positive pleural effu-sion), whereas other systems (i.e., β and γ) fail to meet these criteria. these systems might have overfitted to ordinary cases and could be vulnerable to noise or attack. in terms of enhancing the system's credibility, inter-report consistency might even hold greater significance than the overall accuracy, since a system prone to providing conflicting results would severely undermine users' trust @xcite . regrettably, existing report generation systems struggle to maintain this important quality. they tend to exhibit biases towards common patterns, primarily describing normal observations and are susceptible to lesion variants and context noise @xcite @xcite . we argue that this is largely due to their limited capability of capturing shared attributes of similar patterns, which arises from the data scarcity of distributed lesions and their semantically equivalent variants, rendering it challenging for neural models to accurately locate and describe abnormalities. in this paper, we propose icon, which aims to improves inter-report consistency of radiology report generation. our proposed method involves first extracting lesions from given input images, followed by examining the attributes of these lesions. subsequently, both the radiographs and their associated attributes are utilized as inputs for report generation. to further enhance the inter-report consistency, we introduce a lesion-aware mixup technique by learning from linearly combined lesions and synthesized attributes that belong to the same observation. in summary, the contributions of this paper are as follows: ## related works radiology report generation @xcite @xcite has gained much attention. prior research has either devised various memory mechanisms to record key information @xcite @xcite @xcite or proposed different learning methods to enhance performance @xcite to bridge the modality gap. mixup is also closely related to this research @xcite , and this method has been extensively adopted in nlp research @xcite @xcite . although consistency has been studied in many domains @xcite @xcite , it remains unexplored in medical report generation. ## conclusion and future works in this paper, we propose icon, comprising three components to improve both accuracy and interreport consistency. icon first extracts lesions and then matches fine-grained attributes for report generation. a lesion-aware mixup method is devised for attribute alignment. experimental results on three datasets demonstrate the effectiveness of icon. in the future, we plan to explore incorporating large language models (llms) into our framework, given their advanced capabilities in planning and generation, to further enhance the performance of radiology report generation. leveraging the strengths of llms could provide more refined signals to enhance the performance of icon.