M ed C ycle: Unpaired Medical Report Generation via Cycle-Consistency (2024)

Abstract: Generating medical reports for X-ray images presents a significant challenge, particularly in unpaired scenarios where access to paired image-report data for training is unavailable. Previous works have typically learned a joint embedding space for images and reports, necessitating a specific labeling schema for both. We introduce an innovative approach that eliminates the need for consistent labeling schemas, thereby enhancing data accessibility and enabling the use of incompatible datasets. This approach is based on cycle-consistent mapping functions that transform image embeddings into report embeddings, coupled with report auto encoding for medical report generation. Our model and objectives consider intricate local details and the overarching semantic context within images and reports. This approach facilitates the learning of effective mapping functions, resulting in the generation of coherent reports. It outperforms state-of-the-art results in unpaired chest X-ray report generation, demonstrating improvements in both language and clinical metrics.

## introduction automating the generation of medical reports has the potential to improve the efficiency of patient information analysis and documentation, leading to better care and cost savings. consequently, many research efforts have been directed towards this aim @xcite @xcite @xcite . these works rely on labeled image-report paired datasets @xcite , which are relatively small and less accessible in comparison to datasets for natural images @xcite @xcite . privacy concerns, restricted access to high-quality data, and the complex nature of medical data analysis and labeling, demanding specialized expertise, contribute to this problem and limit the availability of such paired data. even when paired datasets exist, they may not be fully accessible to the public, leading to partially-available datasets, such as the one presented in @xcite , which consists solely of images. these limitations in obtaining paired data pose a significant challenge in this domain. utilizing unpaired data, i.e., images and reports originating from different sources, may help alleviating some of these limitations. specifically, it may resolve privacy or regulatory concerns and increase the available amount of training data. nonetheless, when attempting to learn report generation from images, the absence of image-report pairs introduces a significant challenge. the only previous work that addressed this task involved constructing a knowledge graph of the domain and utilizing a classification module for pathologies @xcite . this process requires expertise in constructing the knowledge graph and, most importantly, ensuring that the images and reports are labeled according to a consistent and shared schema. these labels typically pertain to various thoracic pathologies. our motivation is to eliminate these constraints. we propose to address the challenge by considering four perspectives, each contributing to a cohesive solution: embedding spaces, mapping between these spaces, initial cross-modality alignment, and report generation (figure 1 ). specifically, we construct two separate embedding spaces-one for the visual modality (images) and the other for the textual modality (reports)-utilizing joint global and local representations. subsequently, mapping functions learn the transformation from an image representation to a report representation and vice versa. as mapping functions should preserve the semantic meaning of the data, and image-report pairs are unavailable, training a mapping function to transform image representation to its corresponding report representation becomes challenging. to overcome this issue, we train the mapping between the two embedding spaces to preserve cycle consistency. additionally, to establish initial cross-modality relationships, we introduce a novel concept of pseudoreports, leveraging available domain information accompanying the image dataset (e.g., pathologies). we encourage the representation of an input image, after mapping to the report space, to closely align with the representation of its corresponding pseudoreport. lastly, a decoder is exclusively trained with reports, utilizing auto-encoding, to generate medical reports. overall, at inference time, given an image, we use both the learned mapping to the report domain and the knowledge learned through auto-encoding to generate a report that suits the image. our model adeptly handles a fundamental requirement of report generation-the need for details. recall that the available data is solely global, indicating the presence of pathologies and encompassing the entire image or report. alignment at this global level often results in overly generalized representations, which may be suitable for classification but fall short in capturing fine details in individual examples-details crucial for effective report generation. conversely, local representationsthose depicting image patches or individual report words-capture numerous details, providing essential information for the report generation process. however, they lack alignment. the effectiveness of our method is evident from improvements in both language and clinical met-rics. for instance, when compared to previous sota methods, our approach demonstrates a 9% enhancement in the bleu-1 score (language efficacy) and a 3% increase in f1 (clinical efficacy) on the dataset from @xcite , all while eliminating the need for specific training dataset requirements. furthermore, we illustrate how the absence of these requirements allows the utilization of other training datasets, resulting in further performance improvements. hence, our paper makes the following contributions: ## related work paired medical report generation. methods that rely on paired data have access to both images and their corresponding reports. these models typically employ an encoder-decoder architecture, where the encoder extracts visual features, commonly using a cnn, and the decoder generates text. some models utilize a hierarchical decoder comprising topic and word decoders @xcite @xcite , while others employ transformers @xcite @xcite @xcite . knowledge graphs @xcite @xcite and memory blocks @xcite @xcite are commonly used to learn and encode priori domain information. unpaired medical report generation. the only work that addresses the task of unpaired medical report generation is kgae @xcite . this for improved generalization, dropout masks a portion of the local representations. during inference, an input image is encoded (b1), followed by mapping to the report space @xcite . the transformed representation is then decoded to generate a report @xcite . ## method our goal is to create a model capable of generating medical reports for x-ray images, using two separate datasets-one for reports and the other for images. the grand challenge lies in the absence of paired image-report data during training; in other words, there is no direct correlation between a report from one dataset and an image from the other. we propose to tackle the problem from four perspectives, each addressing a different aspect of the challenge, which collectively provides a coherent solution. first, we establish cross-modality relationships through a new concept: pseudo-reports. second, encoding images and reports using similar procedures and relying on joint global and local representations will enable mapping between the embedding spaces. third, cycle-consistent mapping functions will learn how to transform an image rep-resentation into a report representation. finally, by using only reports, a decoder is trained to generate medical reports. our method, illustrated in figure 2 , incorporates these four ideas. its preprocessing procedure generates pseudo-reports. additionally, the model comprises two encoders: one dedicated to image encoding and the other to report encoding. furthermore, it integrates two mapping networks: one responsible for converting image representations into report representations (i2r), and another for the reverse transformation (r2i), thereby implementing the third idea. finally, the model incorporates a report decoder, encapsulating the fourth idea. during inference, an input image is encoded by the image encoder. the encoded image representation is then mapped to the report space by a mapping function (i2r). given this transformed representation, the decoder outputs a report that correlates to the input image. below, we use the following notations: d i is the image datasets, @xmath0. the pseudo-report of image i is ϕ(i) and its representation is z ϕ(i) . d r is the report datasets, @xmath1. ## ablation study losses. l cm plays a significant role in generating reports closely associated with the input image. the absence of l cm results in relatively poor performance, particularly in the f 1 metric, suggesting potential inaccuracies in capturing the essential information of the data -the pathologies. lastly, both l cyc and l adv contribute to further enhancing the results, as they are applied on representations of actual reports and images, rather than the pseudo-reports. global & local representations. ## conclusions this paper presents a novel approach to generate x-ray reports in an unpaired manner, eliminating the need for paired images and reports during training. our method integrates four key components: we show the effectiveness of our method on two different datasets, surpassing the performance of existing unpaired techniques for generating chest x-ray reports. for instance, when trained on the same image dataset as previous methods, our approach improves the bleu-1 score (language metric) by 4%-9%, depending on the dataset and the f1 score (clinical metric) by 3%. when trained on different datasets, which could not be utilized by other unpaired methods due to distinct labeling schemas, the results are further improved. while our method exhibits generality, future availability of sufficiently comprehensive datasets in other medical domains may broaden the scope of this work to encompass other types of medical data.