Fine-Grained Image-Text Alignment in Medical Imaging Enables Explainable Cyclic Image-Report Generation (2024)

Abstract: Fine-grained vision-language models (VLM) have been widely used for inter-modality local alignment between the predefined fixed patches and textual words. However, in medical analysis, lesions exhibit varying sizes and positions, and using fixed patches may cause incomplete representations of lesions. Moreover, these methods provide explainability by using heatmaps to show the general image areas potentially associated with texts rather than specific regions, making their explanations not explicit and specific enough. To address these issues, we propose a novel Adaptive patch-word Matching (AdaMatch) model to correlate chest X-ray (CXR) image regions with words in medical reports and apply it to CXR-report generation to provide explainability for the generation process. AdaMatch exploits the fine-grained relation between adaptive patches and words to provide explanations of specific image regions with corresponding words. To capture the abnormal regions of varying sizes and positions, we introduce an Adaptive Patch extraction (AdaPatch) module to acquire adaptive patches for these regions adaptively. Aiming to provide explicit explainability for the CXR-report generation task, we propose an AdaMatch-based bidirectional LLM for Cyclic CXR-report generation (AdaMatch-Cyclic). It employs AdaMatch to obtain the keywords for CXR images and ‘keypatches’ for medical reports as hints to guide CXR-report generation. Extensive experiments on two publicly available CXR datasets validate the effectiveness of our method and its superior performance over existing methods. Source code will be released.

## introduction inter-modality alignment, such as vision and language, has been an important task with growing interests in the field of computer vision, especially with the recent advancement in representation * xiang li and yixuan yuan are corresponding authors. learning @xcite . technologies like contrastive learning and self-supervised learning have dramatically improved state-of-the-art alignment performance. recent vision-language models (vlms) demonstrate two approaches: global contrastive alignment, which integrates images and texts at a global level @xcite @xcite @xcite , and local alignment, focusing on detailed connections between visual objects and textual words @xcite @xcite @xcite , as illustrated in fig. 1 . current vlms with local alignment either adopt the pre-trained object detector to extract regionof-interest (roi) features from images and match the corresponding object features with textual words @xcite @xcite , or align the visual token from each patch and the textual token into the same embedding space @xcite @xcite . the former highly relies on the quality of the object detector and its predefined classes, which is less generalizable to new domains. the latter family of methods learns the alignment in a more automatic and datadriven manner. however, most of these methods depend on a pre-defined patch size and positions (e.g., grids) across images. in the most challenging cases, such as the analysis of medical image, lesions can exhibit a wide range of shapes, sizes, and positions. a fixed partition of image patches can lead to incomplete or ambiguous representations of the key imaging abnormalities. therefore, it is highly desirable to adaptively exploit the finegrained relationship between image embeddings derived from a more flexible patching scheme and textual embeddings. another challenge in the current vlms lies in their explainability: it is generally difficult to delineate the image-text relationship learned by the model, especially for the current medical vlms. current solutions to provide such explanations in medical vlms leverage the attention maps from the intermediate layer to visualize the location of the abnormalities @xcite @xcite . other methods @xcite @xcite utilize network gradients such as grad-cam @xcite to generate the heatmaps according to lesion types based on ground-truth reports. however, both maps can only show the general areas potentially associated with the corresponding text data rather than pinpointing a specific region. in addition, gradient-based methods need groundtruth reports, prohibiting them from functioning correctly beyond training data. it is thus highly necessary to develop a mechanism that could provide explicit and specific explanations of input image or text during inference time. to address these two challenges above, we propose a novel adaptive patch-word matching (adamatch) model to match fine-grained image regions of various sizes and positions with textual data. adamatch introduces an image encoder with multiple adaptive patch extraction (adapatch) modules to adaptively acquire the patches associated with certain text tokens. it then performs patch-word alignment based on contrastive learning. adamatch is specifically developed in the context of aligning radiology images (chest x-ray, cxr) and their corresponding radiology reports with the capability of achieving cyclic (cxr-toreport and report-to-cxr) generation based on the learned alignment. our premise is that such a cyclic generation task would serve as the best use case and evaluation criterion for the desired fine-grained alignment. also, fine-grained cyclic generation between cxr and report will provide natural explainability for how the model aligns two modalities: for any given text token, we can visualize its matching imaging manifestation; and for any image region within a cxr image, we can tell the type of lesion or anatomical region it belongs to. to implement the cyclic cxr-report generation, we propose an adamatch-based bidirectional model (adamatch-cyclic). adamatch-cyclic employs adamatch to identify the keywords for cxr images and the 'keypatches' for medical reports to guide the generation tasks. since the potential keywords for cxr images cover a wide range and ground-truth reports cannot be used during inference, we predefine a textual codebook with the most common entities from medical reports as prior knowledge during fine-grained alignment. with the textual codebook, adamatch aligns it with the adaptive patches to obtain matched keywords to facilitate report generation. next, a vq-gan model encodes the cxr image into image tokens, and a large language model (llm) takes image tokens, the matched keywords, and the instructions as input to generate medical reports. similarly, we also build a visual codebook with the most commonly seen patches as 'keypatches', and use adamatch to obtain the matched keypatches from given text reports as hints for cxr generation. utilizing medical reports, matched keypatches, and instructions, llm generates image tokens, subsequently decoded by the vq-gan model to produce the resulting cxr image. our contributions are summarized as follows: input cxr image 𝑥 ! llm generate a chest x-ray report. instruction 𝐼 " patch-word similarity matrix 𝑠 # no significant interval change with bilateral pleural effusions with right pigtail catheter in the ...... generated report 𝑥 @xmath0 adaptive patches sampling tokenize a pigtail catheter is placed in the lower right chest. there might also be a small pneumothorax in the upper right region ...... text encoder patch emb encoder stage 1 fc encoder stage 2 offset scale adaptive patches sampling adaptive patch extraction module (adapatch) adapatch stage 3 adapatch stage 4 image encoder 𝐿 !,@xmath1,! 1. max 2. mean 2. mean 1. max patch-word similarity 𝑠 ! word-patch similarity 𝑠 @xmath2 (b) ports. moreover, we build the textual and visual codebook to utilize adamatch to extract useful keywords and keypatches for the report and cxr generation, respectively. ## methods in fig. 2 , we propose an adaptive patch-word matching (adamatch) model to associate cxr image regions with words in medical reports, and apply it to cxr-report generation to enhance explainability. given an input cxr image x i , the image encoder with several adaptive patch extraction modules (adapatch) predicts the location and scale of multiple adaptive patches and processes the adaptive patch embeddings z i . with z i and text embeddings z t extracted by text encoder for the medical report x t , we compute the similarities s i , s t between adaptive patches and text tokens to calculate the contrastive loss l i2t , l t 2i to optimize adamatch. for cxr-to-report generation, we utilize the frozen adamatch model to match a predefined textual codebook with the input cxr image x i to obtain matched keywords k t and feed a llm with k t , the instruction, and image tokens encoded by vq-gan from x i to generate a medical report x t ′ . then, the adamatch model is used to match a predefined visual codebook with the generated report x t ′ to acquire the matched keypatches k i . with instruction, x t ′ and k i , llm outputs image tokens of the generated cxr image x i ′ , which is optimized through l text and l img . ## conclusion we propose adamatch for fine-grained image-text alignment, which presents the first work to adaptively associate image patches with words and improve the explainability of cyclic cxr-report generation. it includes adapatch to acquire adaptive patches for abnormal regions and performs patchword alignment between adaptive patches with textual tokens. we implement cyclic cxr-report generation by using adamatch to provide explanations for the generation process. in addition, the finegrained cyclic generation process provides a natural explainability for the alignment between cxr images and reports. extensive experiments on two cxr datasets show the effectiveness of our method and its superiority over previous methods.