Generating Radiology Reports via Memory-driven Transformer (2020)

Abstract: Medical imaging is frequently used in clinical practice and trials for diagnosis and treatment. Writing imaging reports is time-consuming and can be error-prone for inexperienced radiologists. Therefore, automatically generating radiology reports is highly desired to lighten the workload of radiologists and accordingly promote clinical automation, which is an essential task to apply artificial intelligence to the medical domain. In this paper, we propose to generate radiology reports with memory-driven Transformer, where a relational memory is designed to record key information of the generation process and a memory-driven conditional layer normalization is applied to incorporating the memory into the decoder of Transformer. Experimental results on two prevailing radiology report datasets, IU X-Ray and MIMIC-CXR, show that our proposed approach outperforms previous models with respect to both language generation metrics and clinical evaluations. Particularly, this is the first work reporting the generation results on MIMIC-CXR to the best of our knowledge. Further analyses also demonstrate that our approach is able to generate long reports with necessary medical terms as well as meaningful image-text attention mappings.

## introduction radiology report generation, which aims to automatically generate a free-text description for a clinical radiograph (e.g., chest x-ray), has emerged as a prominent attractive research direction in both artificial intelligence and clinical medicine. it can greatly expedite the automation of workflows and improve the quality and standardization of health care. recently, there are many methods proposed in this area @xcite @xcite @xcite . practically, a significant challenge of radiology report generation is that radiology reports are long narratives consisting of multiple sentences. as illustrated by figure 1 , a radiology report generally consists of a section of findings which describes medical observations, including both normal and abnormal features, as well as an impression or concluding remark summarizing the most prominent observations. therefore, applying conventional image captioning approaches @xcite ) may be insufficient for radiology report generation, as such approaches are designed to briefly describe visual scenes with short sentences. the ability to provide accurate clinical descriptions for a radiograph is of the highest priority, which places a higher demand on the generation process. nevertheless, despite the difficulties posed by these evident length and accuracy requirements, radiology reports do have their own distinctive characteristics. an important feature to note is their highly patternized nature, as illustrated by the sample report described above (figure 1 ). on the basis of this patternization, many approaches have been proposed to address the challenges of radiology report generation. for example, @xcite found that a simple retrieval-based method could achieve a comparative performance for this task. @xcite combined retrieval-based and generation-based methods with manually extracted templates. although promising results may be obtained by the retrieval-based approaches, they are still limited in the preparation of large databases, or the explicit construction of template lists to determine the patterns embedded in various reports. in this paper, we propose to generate radiology reports via memory-driven transformer. in detail, a relational memory (rm) is proposed to record the information from previous generation processes and a novel memory-driven conditional layer normalization (mcln) is designed to incorporate the relational memory into transformer @xcite . as a result, similar patterns in different medical reports can be implicitly modeled and memorized during the generation process, which thereby can facilitate the decoding of transformer and is capable of generating long reports with informative content. experimental results on two benchmark datasets confirm the validity and effectiveness of our approach, where transformer with rm and mcln achieves the state-of-the-art performance on all datasets. to summarize, the contributions of this paper are four-fold: ## the proposed method generating radiology reports is essentially an image-to-text generation task, for which there exist several solutions @xcite @xcite . we follow the standard sequence-to-sequence paradigm for this task. in doing so, we treat the input from a radiology image as the source sequence @xmath0, x 2 , ..., x s }, x @xmath1, where x s are patch features extracted from visual extractors and d the size of the feature vector. the corresponding report is the target sequence @xmath2, y 2 , ..., y t }, y @xmath3, where y t are the generated tokens, t the length of generated tokens and v the vocabulary of all possible tokens. an overview of our proposed model is shown in figure 2 , where the details are illustrated in following subsections. ## related work the most popular related task to ours is image captioning @xcite @xcite , which aims to describe images with sentences. different from them, radiology report generation requires much longer generated outputs, and possesses other features such as patterns, so that this task has its own characteristics requiring particular solutions. for example, @xcite proposed a co-attention extra knowledge (e.g., pre-trained embeddings @xcite @xcite and pretrained models @xcite ) can provide useful information and thus enhance model performance for many nlp tasks @xcite . specifically, memory and memory-augmented neural networks @xcite @xcite are another line of related research, which can be traced back to @xcite , which proposed memory networks to leverage extra information for question answering; then @xcite improved it with an end-to-end design to ensure the model being trained with less supervision. particularly for transformer, there are also memory-based methods proposed. for example, @xcite proposed to solve the under-fitting problem of transformer by introducing a product-key layer that is similar to a memory module. @xcite proposed memo, an adaptive memory to reason over long-distance texts. compared to these studies, the approach proposed in this paper focuses on leveraging memory for decoding rather than encoding, and presents a relational memory to learn from previous generation processes as well as patterns for long text generation. to the best of our knowledge, this is the first study incorporating memory for decoding with transformer and applied for a particular task, which may provide a reference for studies in the line of this research. ## conclusion in this paper, we propose to generate radiology reports with memory-driven transformer, where a relational memory is used to record the information from previous generation processes and a novel layer normalization mechanism is designed to incorporate the memory into transformer. experimental results on two benchmark datasets illustrate the effectiveness of the memory by either concatenating it with the output or integrating it with different layers of the decoder by mcln, which obtains the state-of-the-art performance. further analyses investigate how memory size affects model performance and show that our model is able to generate long reports with necessary medical terms and meaningful image-text attention mappings.