Automated Generation of Accurate & Fluent Medical X -ray Reports (2021)

Abstract: Our paper aims to automate the generation of medical reports from chest X-ray image inputs, a critical yet time-consuming task for radiologists. Existing medical report generation efforts emphasize producing human-readable reports, yet the generated text may not be well aligned to the clinical facts. Our generated medical reports, on the other hand, are fluent and, more importantly, clinically accurate. This is achieved by our fully differentiable and end-to-end paradigm that contains three complementary modules: taking the chest X-ray images and clinical history document of patients as inputs, our classification module produces an internal checklist of disease-related topics, referred to as enriched disease embedding; the embedding representation is then passed to our transformer-based generator, to produce the medical report; meanwhile, our generator also creates a weighted embedding representation, which is fed to our interpreter to ensure consistency with respect to disease-related topics. Empirical evaluations demonstrate very promising results achieved by our approach on commonly-used metrics concerning language fluency and clinical accuracy. Moreover, noticeable performance gains are consistently observed when additional input information is available, such as the clinical document and extra scans from different views.

## introduction medical reports are the primary medium, through which physicians communicate findings and diagnoses from the medical scans of patients. the process is usually laborious, where typing out a medical report takes on average five to ten minutes @xcite ; it could also be error-prone. this has led to a surging need for automated generation of medical reports, to assist radiologists and physicians in making rapid and meaningful diagnoses. its potential efficiency and benefits could be enormous, especially during critical situations such as covid or a similar pandemic. clearly a successful medical report generation process is expected to possess two key properties: 1) clinical accuracy, to properly and correctly describe the disease and related symptoms; 2) language fluency, to produce realistic and human-readable text. fueled by recent progresses in the closely related computer vision problem of image-based captioning @xcite , there have been a number of research efforts in medical report generation in recent years @xcite @xcite @xcite @xcite @xcite . these methods often perform reasonably well in addressing the language fluency aspect; on the other hand, as is also evidenced in our empirical evaluation, their results are notably less satisfactory in terms of clinical accuracy. this we attribute to two reasons: one is closely tied to the textual characteristic of medical reports, which typically consists of many long sentences describing various disease related symptoms and related topics in precise and domain-specific terms. this clearly sets the medical report generation task apart from a typical image-to-text problem such as image-based captioning; another reason is related to the lack of full use of rich contextual information that encodes prior knowledge. these information include for example the patient's clinical document describing key clinical history and indication from doctors, and multiple scans from distinct 3d views -information that are typically existed in abundance in practical scenarios, as e.g. in the standard x-ray benchmarks of @xcite and @xcite . the aforementioned observations motivate us to propose a categorize-generate-interpret framework that places specific emphasis on clinical accuracy figure 1 : our approach consists of three modules: a classifier that reads chest x-ray images and clinical history to produce an internal checklist of disease-related topics, a transformer-based generator to generate fluent text, and an interpreter to examine and fine-tune the generated text to be consistent with the disease-related topics. while maintaining adequate language fluency of the generated reports. it consists of a classifier module reads chest x-ray images (e.g., either single-view or multi-view images) and related documents to detect diseases and output enriched disease embedding, a transformer-based medical report generator, and a differentiable interpreter to evaluate and finetune the generated reports for factual correctness. the main contributions are two-fold: ## our approach our framework consists of a classification module, a generation module, and an interpretation module, as illustrated in fig. 1 . the classification module reads multiple chest x-ray images and extracts the global visual feature representation via a multiview image encoder. they are then disentangled into multiple low-dimensional visual embedding. meanwhile, the text encoder reads clinical documents, including, e.g., doctor indication, and summarizes the content into text-summarized embedding. the visual and text-summarized embeddings are entangled via an "add & layernorm" operation to form contextualized embedding in terms of disease-related topics. the generation module takes our enriched disease embedding as initial input and generates text word-by-word, as shown in fig. 2 . finally, the generated text is fed to the interpretation module for fine-tuning to align to the checklist of disease-related topics from the classification module. in what follows, we are to elaborate on these three modules in detail. figure 2 : an exemplar illustration of our approach in action. specifically, the enriched disease embedding produced from the classification module are fed into the generation module as initial inputs. then, at each time step, the hidden state h i is obtained to predict the next output word. finally, the interpretation module takes as input all predicted outputs ŵ to produce a checklist of disease-related topics, which are to be gauged with the same topics output from the classification module for consistency verification. ## limitations and future work our work has several limitations that future works can take into consideration for further improvement. firstly, our model does not explicitly consider disease orientation or direction (e.g., left or right, top or bottom). for example, future works can include visual-semantic embedding (direction/orientation/location) to learn and localize diseases during generating medical reports. secondly, our work does not support time-series relationships between different studies of a patient. this information is vital to analyze existing diseases by comparing their size or structure to determine if a disease is getting worse or not. if these limitations can be addressed, the medical report system can be much more reliable for real-world applications. noticeably, we observe some hallucination facts (false positives) where some diseases are mistakenly described as positive in the medical reports. for example, some images with "pneumonia" are wrongly described as "pulmonary edema". in fact, human radiologists often mistakenly classify some diseases @xcite . for example, @xcite shows that human radiologists or physicians can accurately detect normal lung x-ray images almost all of the time; but, for abnormal lung x-ray images, the correctness of diagnosis drops to only 50% @xcite . for this reason, it is challenging to generate accurate medical reports even for experienced radiologists. in the future, we will expand our work to related medical applications such as retinal and brain medical report generation on x-ray/mri/ct scans. we believe that our model can be generalized to a wide range of medical report generation problems where common symptoms or disease labels and medical reports are available in most medical scan datasets. moreover, extending the current work to incorporate tabular data inputs could be another exciting direction because some clinical information is in the form of tabular structure such as patient's age, heart pressure, or temperature @xcite . in some cases, physicians must include this information in medical reports, which cannot be inferred from only reading medical scans. ## conclusion this paper introduces a novel three-module approach for generating medical reports from xray scans. superior performance of our approach over state-of-the-art methods has been empirically demonstrated on widely-used benchmarks with a range of evaluation metrics. our approach is also flexible and can work with additional input information, where consistent performance gains are observed. for future work, we plan to apply our approach to related medical report generation tasks that go beyond x-rays.