JPG - Jointly Learn to Align: Automated Disease Prediction and Radiology Report Generation (2022)

Abstract: Automated radiology report generation aims to generate paragraphs that describe fine-grained visual differences among cases, especially those between the normal and the diseased. Existing methods seldom consider the cross-modal alignment between textual and visual features and tend to ignore disease tags as an auxiliary for report generation. To bridge the gap between textual and visual information, in this study, we propose a “Jointly learning framework for automated disease Prediction and radiology report Generation (JPG)” to improve the quality of reports through the interaction between the main task (report generation) and two auxiliary tasks (feature alignment and disease prediction). The feature alignment and disease prediction help the model learn text-correlated visual features and record diseases as keywords so that it can output high-quality reports. Besides, the improved reports in turn provide additional harder samples for feature alignment and disease prediction to learn more precise visual and textual representations and improve prediction accuracy. All components are jointly trained in a manner that helps improve them iteratively and progressively. Experimental results demonstrate the effectiveness of JPG on the most commonly used IU X-RAY dataset, showing its superior performance over multiple state-of-the-art image captioning and medical report generation methods with regard to BLEU, METEOR, and ROUGE metrics.

## introduction writing radiology reports and predicting disease labels are two essential procedures in clinical practice. however, manually creating them by radiologists is laborious and time-consuming @xcite . therefore, automated radiology report generation and disease prediction, which aim to generate formal-format descriptive texts (fig. 1 findings) and clinical conclusive terminologies (fig. 1 mesh), have received increasing attention recently @xcite ## methodology figure 2 exhibits an overview of jpg, which consists of three chief components: (a) shared subspace representation learning, (b) context-aware disease prediction, and (c) radiology report generation. hereafter, we will give formal notations of variables and task definitions concerning jpg, and introduce each component subsequently in detail. ## conclusion we addressed several fundamental issues concerning clinical disease prediction and radiology report generation in an overall framework, where contextaware disease terminologies act as complementary textual features coupled with visual features of images to guide and facilitate the report generation process. meanwhile, these explicit clues of lesion location effectively prevent the report generation model from generating factual erroneous texts. the proposed shared subspace provides an interaction platform for different representations extracted by image and word embeddings to overcome the misalignment of information across modalities. empirical results acquired with the most widely used dataset, including those of ablation studies, demonstrate the effectiveness of the proposed jpg, which achieves the state-of-the-art performance.