Writing by Memorizing: Hierarchical Retrieval-based Medical Report Generation (2021)

Abstract: Medical report generation is one of the most challenging tasks in medical image analysis. Although existing approaches have achieved promising results, they either require a predefined template database in order to retrieve sentences or ignore the hierarchical nature of medical report generation. To address these issues, we propose MedWriter that incorporates a novel hierarchical retrieval mechanism to automatically extract both report and sentence-level templates for clinically accurate report generation. MedWriter first employs the Visual-Language Retrieval (VLR) module to retrieve the most relevant reports for the given images. To guarantee the logical coherence between generated sentences, the Language-Language Retrieval (LLR) module is introduced to retrieve relevant sentences based on the previous generated description. At last, a language decoder fuses image features and features from retrieved reports and sentences to generate meaningful medical reports. We verified the effectiveness of our model by automatic evaluation and human evaluation on two datasets, i.e., Open-I and MIMIC-CXR.

## introduction medical report generation is the task of generating reports based on medical images, such as radiology and pathology images. given that this task is timeconsuming and cumbersome, researchers endeavor to relieve the burden of physicians by automatically generating the findings and descriptions from medical images with machine learning techniques. existing studies can be roughly divided into two categories, i.e., generation-based and retrievalbased approaches. generation-based methods, including lrcn @xcite , coatt @xcite retrieval et al., 2018), and mvh+attl @xcite , focus on generating image captions with a encoderdecoder model that leverage image features. however, they are unable to produce linguistically diverse descriptions and depict rare but prominent medical findings. on the other hand, retrievalbased methods such as hrgr-agent @xcite and kepp @xcite , pay attention to memorizing templates to generate standardized reports from a predefined retrieval database. however, the quality of generated reports significantly depends on the manually curated template database. besides, they only use sentence-level templates for the generation but ignore to learn the report-level templates, which prevent them from generating more accurate reports. to address the aforementioned issues, we propose a new framework called medwriter as shown in figure 1 . medwriter introduces a novel hierarchical retrieval mechanism working with a hierarchical language decoder to automatically learn the dynamic report and sentence templates from the data for generating accurate and professional medical reports. medwriter is inspired by the process of how physicians write medical reports in real life. they keep report templates in mind and then generate reports for new images by using the key information that they find in the medical images to update the templates sentence by sentence. in particular, we use three modules to mimic this process. first, medwriter generates reportlevel templates from the visual-language retrieval (vlr) module using the visual features as the queries. to generate accurate reports, medwriter also predicts disease labels based on the visual features and extracts medical keywords from the retrieved reports. we propose a multiquery attention mechanism to learn the reportlevel template representations. second, to make the generated reports more coherent and fluent, we propose a language-language retrieval (llr) module, which aims to learn sentence-level templates for the next sentence generation by analyzing between-sentence correlation in the retrieved reports. finally, a hierarchical language decoder is adopted to generate the full report using visual features, report-level and sentence-level template representations. the designed two-level retrieval mechanism for memorization is helpful in generating accurate and diverse medical reports. to sum up, our contributions are: ## related work generation-based report generation visual captioning is the process of generating a textual de-scription given an image or a video. the dominant neural network architecture of the captioning task is based on the encoder-decoder framework @xcite @xcite , with attention mechanism @xcite @xcite @xcite . as a sub-task in the medical domain, early studies directly apply state-of-theart encoder-decoder models as cnn-rnn @xcite , lrcn @xcite and adaatt @xcite to medical report generation task. to further improve long text generation with domain-specific knowledge, later generationbased methods introduce hierarchical lstm with co-attention @xcite or use the medical concept features @xcite to attentively guide the report generation. on the other hand, the concept of reinforcement learning @xcite is utilized to ensure the generated radiology reports correctly describe the clinical findings. to avoid generating clinically non-informative reports, external domain knowledge like knowledge graphs @xcite and anchor words @xcite are utilized to promote the medical values of diagnostic reports. clara @xcite also provides an interactive solution that integrates the doctors' judgment into the generation process. ## method as shown in figure 2 , we propose a new framework called medwriter, which consists of three modules. the visual-language retrieval (vlr) module works on the report level and uses visual features to find the most relevant template reports based on a multi-view image query. the language-language retrieval (llr) module report-level retrieval spatial attention multi-query attention visual feature extraction the 1 !" sentence generation using report-level retrieval sentence lstm word lstm 1 !" sentence retrieved reports disease classification disease labels medical keywords sentence-level retrieval the 𝑡 "# sentence generation using sentence-level retrieval 𝑡 -1 "# generated sentence sentence lstm word lstm 𝑡 "# sentence retrieved sentences visual feature report template sentence template disease representation image-report matching 𝒗 𝒓 ! visual -language retrieval vlr module hierarchical language decoder 𝒉 @xmath0 sentence-sentence matching disease labels medical keywords multi-query attention 𝒗 ! 𝒓 ! 𝒖 "%$ ! 𝒉 " sentence hidden state language -language retrieval llr module works on the sentence level and retrieves a series of candidates that are most likely to be the next sentence from the retrieval pool given the generated language context. finally, medwriter generates accurate, diverse, and disease-specified medical reports by a hierarchical language decoder that fuses the visual, linguistics and pathological information obtained by vlr and llr modules. to improve the effectiveness and efficiency of retrieval, we first pretrain vlr and llr modules to build up a retrieval pool for medical report generation as follows. ## conclusions automatically generating accurate reports from medical images is a key challenge in medical image analysis. in this paper, we propose a novel model named medwriter to solve this problem based on hierarchical retrieval techniques. in particular, medwriter consists of three main modules, which are the visual-language retrieval (vlr) module, the language-language retrieval (llr) module, and the hierarchical language decoder. these three modules tightly work with each other to automatically generate medical reports. experimental results on two datasets demonstrate the effectiveness of the proposed medwriter. besides, qualitative studies show that medwriter is able to generate meaningful and realistic medical reports.