Learning to Generate Clinically Coherent Chest X -Ray Reports (2020)

Abstract: Automated radiology report generation has the potential to reduce the time clinicians spend manually reviewing radiographs and streamline clinical care. However, past work has shown that typical abstractive methods tend to produce fluent, but clinically incorrect radiology reports. In this work, we develop a radiology report generation model utilizing the transformer architecture that produces superior reports as measured by both standard language generation and clinical coherence metrics compared to competitive baselines. We then develop a method to differentiably extract clinical information from generated reports and utilize this differentiability to fine-tune our model to produce more clinically coherent reports.

## introduction medical imaging (e.g. chest x-ray) is widely used in medicine for diagnostic purposes. however, current clinical practice requires a radiologist with specialized training to manually evaluate x-rays and note their findings in a radiology report. this manual evaluation is time-consuming and providing an automated solution for this task would help streamline the clinical workflow and improve the quality of care. image captioning has gained a large amount of attention following the curation of the coco dataset @xcite and the initial image captioning work conducted on it @xcite @xcite . although image captioning is a widely studied task, radiology report generation offers unique challenges that precludes the direct adaptation of many image captioning models for the task. for example, much of the recent image captioning research has followed the work of @xcite and utilized pre-trained objection detection models @xcite to extract image features @xcite @xcite . while this works well for generaldomain datasets, analagous pre-trained models are not available in the clinical domain. radiology reports are also typically longer and more complex than the captions available in standard image captioning datasets such as coco. evaluation of medical report generation is also difficult because the language generation metrics typically used to evaluate image captioning systems can not directly evaluate the descriptive accuracy of generated reports which is of critical importance in the medical domain. there is already a body of past work that has focused on medical report generation, the most related being that of @xcite and @xcite who both developed fully abstractive techniques for report generation. @xcite benchmarked a number of simple baselines on the mimic-cxr dataset @xcite , the largest publicly available dataset of paired chest xrays and radiology reports. they observed that typical abstractive methods often produce fluent, but clinically incoherent reports that fail to correctly convey essential information (e.g. the presence of a medical condition). @xcite attempted to directly addressed this problem by using self-critical sequence training (scst) @xcite to optimize the clinical accuracy of their generated reports. although their use of scst did increase the precision of their model, it also greatly decreased recall and ultimately reduced the f1 score of their model foot_1 . we also focus on improving the clinical coherence of generated reports in this work. @xcite also utilized recurrent architecture for report generation despite the recent success of the transformer architecture @xcite , 2017 ) with image-and video-captioning tasks outside of the medical domain @xcite @xcite . we also explore whether the transformer architecture is more effective than representative recurrent models for radiology report generation. our primary contributions can be summarized as follows: (1) we apply the transformer model to radiology report generation and demonstrate that it is superior to competitive baselines as measured by both language generation metrics and the clinical coherence of the generated reports. (2) we develop a procedure to differentiably extract clinical information from our generated reports and leverage this differentiability to train our report generation model to produce more clinically coherent reports. ## dataset the mimic-cxr dataset contains 227,835 imaging reports with 377,110 total images conducted at the beth israel deaconess medical center emergency department for 65,379 patients @xcite . the imaging studies are accompanied with free-text radiology reports that record the observations of a practicing radiologist during routine clinical care. radiology reports are semi-structured documents composed of a number of possible sections such as patient history, findings, and impressions. we follow the precedent set by previous work and focus on generating the findings section because it represents the most direct transcription of the imaging study @xcite . we thus constrain our dataset to radiology reports that contain the findings sections and then divide the remaining data into training, validation, and testing sets following a 70%/10%/20% split. we divide the data on the patient id rather than on specific radiology reports to avoid leaking data from subjects with multiple radiology exams conducted. ## methods we develop an end-to-end report generation framework that consists of two stages. the first stage consists of a report generation model that is trained using a standard language generation objective. for the second stage, we differentiably sample a report from our model and extract the clinical observations from that report. this allows us to introduce an additional learning objective based on the agreement between the observations from the generated and ground truth reports. we utilize this additional objective to fine-tune our model to produce more clinically coherent reports. an overview of this framework is provided in figure 1 . we refer the reader to the supplemental materials for further implementation details. ## results we report the cider @xcite , me-teor @xcite , rouge-l @xcite , and bleu @xcite of our report generation models. while these metrics all measure the language similarity between the the generated and ground truth reports, they can not directly evaluate how effectively the models are producing clinically correct reports. to address this shortcoming, we also report clinical coherence metrics that compare the chexpert extracted labels for the generated and ground truth reports. for this, we report the macro-and micro-averaged precision, recall, and f1 score for positive annotations. we refer the reader to the supplementary materials to view the results for the individual observations. all models in this work are decoded using beam search with a beam size of 4. @xcite benchmarked a number of report generation techniques on the mimic-cxr dataset and found a simple 1-nearest neighbor baseline to be surprisingly effective, particularly with respect to its clinical coherence. as such, we compare against a 1-nn baseline where we retrieve the report from our training set whose desnenet-induced image features have the highest cosine similarity with the test query image. we also compare against two competitive recurrent image captioning models: show, attend, and tell developed by @xcite and adaptive attention developed by @xcite . ## limitations and future work past work on radiology report generation has leveraged the semi-standardized nature of radiology reports to develop extractive or template-based methods @xcite @xcite . other work has combined templatebased methods with abstractive methods to utilize the advantages of both methods @xcite . in this work we focused on developing abstractive techniques as was done by past work on the mimic-cxr dataset @xcite . however, in the future we intend to combine the abstractive methods developed in this work with retrieval methods to further improve upon our framework. @xcite developed a model that jointly extracted conditions from chest x-rays and generated radiology reports. in this work we only focus on report generation, but augmenting our framework with an explicit image classification objective is a potential direction for future work. while the methods developed in this work lead to significantly improved performance compared to competitive baselines, the clinical coherence of our model is still insufficient for clinical practice. more work must be conducted in the future to continue improving the clinical coherence of automated report generation to enable adoption of such methods. ## conclusion in this work we develop a radiology report generation model utilizing the transformer architecture and demonstrate that it is both more fluent and clinically coherent than competitive baselines. we also develop a procedure to differentiably extract clinical information from generated reports and utilize this differentiability to further fine-tune our model for clinical coherence. our proposed architecture and fine-tuning procedure improve the micro-averaged and macro-averaged f1 of our best baselines by 6.4 and 2.2 points respectively while achieving superior fluency as measured by all of our computed nlg metrics.