Normal-Abnormal Decoupling Memory for Medical Report Generation (2023)

Abstract: The automatic generation of medical reports plays a crucial role in clinical automation. In contrast to natural images, radiological images exhibit a high degree of similarity, while medical data are prone to data bias and complex noise, posing challenges for existing methods in capturing nuanced visual information. To address these challenges, we introduce a novel normal-abnormal semantic decoupling network that utilizes abnormal pattern memory. Different from directly optimizing the network using medical reports, we optimize visual extraction through the extraction of abnormal semantics from the reports. Moreover, we independently learn normal semantics based on abnormal semantics, ensuring that the optimization of the visual network remains unaffected by normal semantics learning. Then, we divided the words in the report into four parts: normal/abnormal sentences and normal/abnormal semantics, optimizing the network with distinct weights for each partition. The two semantic components, along with visual information, are seamlessly integrated to facilitate the generation of precise and coherent reports. This approach mitigates the impact of noisy normal semantics and reports. Moreover, we develop a novel encoder for abnormal pattern memory, which improves the network’s ability to detect anomalies by capturing and embedding the abnormal patterns of images in the visual encoder. This approach demonstrates excellent performance on the benchmark MIMIC-CXR, surpassing the current state-of-the-art methods.

## introduction report writing is a critical responsibility for radiologists. automatic report generation is clinically significant as it alleviates the workload of physicians. recently, substantial advancements have 1 our code is available at https://github.com/kzzjk/ nadm * corresponding authors. been achieved in this field @xcite @xcite . nonetheless, generating radiological reports remains a formidable undertaking. however, several aspects require further exploration, such as: 1) the presence of pronounced visual and textual data biases. within medical data, normal images predominate the section containing descriptions of normal utterances in the reports. 2) unlike natural images, radiological images frequently exhibit high similarity, posing challenges in extracting fine-grained visual details. the presence of noise in reports impacts network optimization. the descriptions of similar semantics may vary among different doctors. for instance, in figure 1a , "cardiac" and "mediastinal" are used, whereas in figure 1b , "cardiomediastinal" is employed. moreover, it is plausible for the syntax of the same description to exhibit significant variation. some normal descriptions may not be mentioned due to physicians' practices. for instance, in figure 1b , "consolidation" and "effusion" are referenced, whereas in figure 1a , they are disregarded. these sources of noise significantly impact network optimization, yet only limited research has been dedicated to addressing this issue. to address the aforementioned limitations, we propose a semantic decoupling network based on abnormal pattern memory for generating reports. in medical reports, the fundamental semantic structure comprises three components: anatomy, observation, and judgment. to mitigate the impact of noise in the reports, we optimize the visual extractor using abnormal semantics. this is based on the observation that abnormal semantics are relatively consistent, despite the occasional omission of normal descriptions and variations in similar semantic descriptions across reports from different doctors. we utilize radgraph @xcite to extract the semantics, focusing solely on the core semantic components while disregarding others. we employ a visual encoder based on higher-order feature interaction attention, enhancing the perception of fine-grained features through a bilinear pool @xcite , which has been shown to be effective in fine-grained classification @xcite . furthermore, drawing inspiration from discrete variational autoencoder @xcite , we store the abnormal modalities of images and incorporate higher-order feature interaction processes to augment anomaly perception. during the report generation phase, we fuse the two modalities, semantic and visual, to jointly guide the report generation process. in the training phase, we employ convolution for image encoding and grid feature generation. moreover, we utilize a variational selfencoder to capture anomaly modality memory and introduce two semantic branches atop it to predict both abnormal and normal semantics of the image during the visual attention phase. we argue that different words in the report possess varying levels of importance, with abnormal semantics being intuitively more significant than normal semantics. however, previous approaches @xcite @xcite @xcite tend to treat all report words equally. therefore, we divide the reported words into four categories: normal/abnormal sentences and normal/abnormal semantics. subsequently, we automatically learn distinct weighting parameters for optimization. in conclusion, our contributions are outlined as follows: ## related work image captioning involves generating relevant textual descriptions or topics for a given image. early approaches utilized templates or retrieval-based methods for caption generation @xcite . in recent years, encoder-decoder frameworks, primarily based on transformer architectures @xcite , have been widely employed to generate individual descriptive sentences from images @xcite @xcite . notably, anchor-frame-based methods for title generation have demonstrated accurate text description generation @xcite . however, a significant portion of the available data lacks annotations. to address this limitation, @xcite proposed the use of semantic extraction to enrich information and improve caption generation. furthermore, while most caption generation tasks focus on shorter descriptive discourse, existing approaches for very long utterances often overlook issues related to data bias @xcite . the image captioning task has a more explicit objective, but the presence of complex semantic contexts, judgmental utterances, and noise problems in medical reports introduces additional challenges. medical report generation, which falls under the category of image captioning, predominantly adopts an encoder-decoder framework. various attention mechanisms and hierarchical lstm models have been proposed to generate radiology reports @xcite @xcite . another approach involves constructing graphs based on medical knowledge and utilizing graph convolutional neural networks to enhance feature extraction @xcite . to address the issue of modal bias, @xcite introduced a set of latent variables as topics to guide sentence generation by aligning image and language patterns in the latent space. modal alignment was further improved through the introduction of a cross-modal memory network by @xcite , and later refined by @xcite with the proposition of a cross-modal prototype-driven network. additionally, @xcite presented a framework that leverages both a priori and a posteriori data to enhance generative reporting. @xcite utilized semantic extraction to improve generation, they overlooked the problems of noise and data bias prevalent in the reports. in @xcite , the data bias problem is considered, where the noise of normal statements still affects the optimization of the model despite the separate generation of normal/abnormal statements. ## the proposed method the proposed architecture for the memory of anomaly patterns is the semantic decoupling network, consisting of three critical components: a visual extractor, a semantic extractor, and a decoder, where the visual extractor consists of an image encoder and a visual encoder. the overall structure of the network is illustrated in figure 2 . the visual extractor is responsible for converting images into detailed features with the goal of detecting anomalies and generating precise visual representations. the semantic extractor is divided into two parts: abnormal semantic extractor and normal semantic extraction. the purpose of abnormal semantic extractor is to identify semantics related to aberrations from the preceding visual representation and to capture normal semantics while considering the abnormal semantics. the decoder is responsible for processing the visual and semantic characteristics, and guides the report generation by deep fusion. ## conclusions we propose a new framework for extracting normal and abnormal semantics separately. by decoupling the semantics, the noise that may exist in the normal semantics is avoided. and we propose the structure of anomaly modality enhancement to enhance the extraction of abnormal features. we classify the reported words into different parts to automatically learn the weights to measure the importance. this eventually makes better semantic and visual features jointly know to generate process accurate reports.