Progressive Transformer-Based Generation of Radiology Reports (2021)

Abstract: Inspired by Curriculum Learning, we propose a consecutive (i.e., image-to-text-to-text) generation framework where we divide the problem of radiology report generation into two steps. Contrary to generating the full radiology report from the image at once, the model generates global concepts from the image in the first step and then reforms them into finer and coherent texts using transformer-based architecture. We follow the transformer-based sequence-to-sequence paradigm at each step. We improve upon the state-of-the-art on two benchmark datasets.

## introduction the analysis of x-rays in medical practice is the most common and important task for radiologists. with years of training, these experts learn to recognize particular features in the image that are later translated to a written report in a clinically appropriate manner. this is a labor intensive and time consuming task, especially difficult for young trainees. with increasing demand on imaging examinations, the burden on radiologists has increased over time, requiring the addition of the technologies to improve their workflow. previous research on radiology report generation has mostly focused on image-to-text generation tasks. @xcite introduced a coattention mechanism to generate full paragraphs. @xcite explored report generation through transformers. more recently, @xcite used a preconstructed graph embedding module on multiple disease findings to assist the generation of reports. finally, @xcite proposed to generate radiology reports via memory-driven transformer and showed that their proposed approach outperforms previous models with respect to both language generation metrics and clinical evaluation. these systems have significant potential in many clinical settings, including improvement in workflow in radiology, clinical decision support, and large-scale screening using x-ray images. in this work, we focus on generating reports from chest x-ray images innovating with a double staged transformer based architecture. our contributions in this paper can be summarized as follows: ## method an essential challenge in the radiology report generation is modeling the clinical coherence across the entire report. contrary to generating the full radiology report from the image at once, we propose a consecutive (i.e., image-to-text-to-text) generation framework (inspired by curriculum learning @xcite and the work of @xcite ). as shown in figure 1 , we divide the problem of radiology report generation into two steps. in the first step, the model generates global concepts from the image and then reforms them into finer and coherent text using a transformer architecture. each step follows the transformer based sequence-to-sequence paradigm. model architecture instead of generating the full report from an input radiology image, we frame the generation process such as: @xmath0, where @xmath1, x 2 , ..., x s }, x @xmath2. x is a radiology image and x s is a sequence of patch features extracted from visual extractor and d is the size of the feature vectors. @xmath3, c 2 , ..., c t }, c @xmath4, and @xmath5, y 2 , ..., y t }, y @xmath6, are the generated tokens at intermediate and final steps, respectively. t and t are the length of generated tokens and v, v are the vocabulary of all possible tokens at each step. our framework can be partitioned into three major components such as: 1) a visual backbone 2) an intermediate encoderdecoder as a visual language model (vilm) and 3) a final encoder-decoder as a language model (lm). visual backbone given a set of radiology images (i), the visual backbone extracts the visual features x and results in the source sequence {x 1 , x 2 , ..., x s } for the subsequent visual language model. the visual backbone can be formulated based on pre-trained convolutional neural networks (cnn), e,g., densenet @xcite , vgg @xcite or resnet @xcite . we find densenet to be more effective in our generation task and therefore use it as our based visual feature extractor. ## results and discussion effect of progressive generation to show the effectiveness of our model, we conduct experiments with baseline models, including our proposed model (i.e., m 2 tr. progressive ) as reported in the progressive generation strategy. however the precision of the progressive model is lower than the baselines. we observe that the progressive generation produces long reports mostly by adding the abnormality mentions in negation mode (e.g., no evidence of pneumonia, there is no pneumothorax ), therefore it increases the number of false positives (fps) in the ce metrics. in analysis as a qualitative analysis to explain the effectiveness of our progressive model, we examine some of the generated reports with their references from the mimic-cxr test dataset (see figure 2 in the appendix). we show the text alignments between the reference text and generated one with the same colors. it can be seen in the top two examples the progressive model is able to provide reports aligned with the reference texts where the baseline model fails to cover them, e.g., post median sternotomy, and mitral valve replacement, the mediastinal contours, enlargement of the cardiac silhouette, bilateral pleural effusions and compressive atelectasis in the top two examples are not generated by m 2 tr.. although our model shows improvements in the nlg and ce metrics evaluation, it still fails to generate clinically coherent and error-free reports. for example, in the third example of figure 2 , the mild pulmonary edema is incorrect since the no new parenchymal opacities in the reference implies negative pulmonary edema. furthermore, the sentence left plueral effusion in the last example is not consistent with the previous text bilateral pleural effusion. additionally, the examples in figure 2 contain a comparison of study against to the previous study such as as compared to the previous ... and in comparison with the study ... in the generated reports. this is a little surprising since the model does not have any clue about the previous report of a patient in its design. it can be attributed to the fact that these template sentences are more frequent in the training set. the examples also show that the progressive model generates a more comprehensive report compare to the baseline.it includes occasionally the extra mentions of medical terms compared to the reference text (e.g., there is no focal consolidation and no evidence of pneumonia in examples 1 and 3, respectively), which result in false-positive mention of observations in the chexpert labeler of the ce metrics. ## conclusion we propose to produce radiology report via a simple but effective progressive text generation model by incorporating high-level concepts into the generation process.