Paper: LLM-CXR: Instruction-Finetuned LLM for CXR Image Understanding and Generation (ICLR-2024)

Reviewer comment (Novelty):
The authors claim the proposed bidirectional LLM is different from previous ones, as illustrated in Figure 1. However, I found the difference between them (a) and (c) is really minor. The encoder and decoder parts in the VQGAN are indeed equivalent to the vision adapter and image generative model, as listed in (a). Therefore, it is a bit overclaimed that it is novel to introduce bidirectional reconstruction tasks (both image-to-text and text-to-image) in the pre-training.

Author response:
**W1. It is overclaimed to introduce bidirectional reconstruction tasks in pre-training as a novel approach.**\
We would like to more clearly delineate the differences between Figure 1’s (a) and (c). The image encoder and decoder in (a) have already been pre-trained with the paired vision-language dataset; and the encoder outputs pseudo-text, which the decoder in turn receives as input. In this process, the language model does not directly engage with visual information as it takes pseudo-text as input and outputs pseudo-text (this is why the LLM can remain frozen in methods like BLIP-2).

In our approach, visual information is passed directly to the LLM without passing through a vision-language pretrained network. The encoder simply performs tokenization of the image. LLM thus receives direct visual information, and vision-language alignment occurs within its own parameters on top of its pretrained language features. We believe that, as shown by our results, this is a promising avenue of multimodal LLM research that deserves attention separate from the currently dominating method of using vision adapter networks to implement multimodal LLMs.

Reviewer comment (Experiments):
The motivation for training an image-text aligned model is not clearly introduced and justified. First, it will be helpful to discuss how this model could be applied to the downstream clinical tasks. Then, the designed experiments only demonstrate the performance of the proposed method on these instruction tuning tasks, i.e., report generation based on images and VQA. It is hard to appreciate the benefit of adopting such a pre-trained and then SFTed model in practical use without demonstrating downstream applications. There are many chest X-ray benchmarks, and datasets are commonly used for the evaluation of pre-trained models, e.g., disease classifications and localizations.

Author response:
**W2. Motivation not clear. Explanation of downstream tasks?**\
Direction to AGI development will likely involve integration of a central LLM with various expert models operating as modules; and we envision that communication between them will heavily utilize natural language. From this perspective, expert models should have language-based communication skills. As for a specific downstream example task, models like LLM-CXR can be instructed to find patients with specific findings described by text. For example, if the inclusion criteria for a clinical trial calls for ‘patients with severe bilateral pleural effusions on CXR’, the model can be used as an agent to assist this process.

Reviewer comment (Evaluation):
The metrics utilized in the experiments for most tasks are not the commonly used ones, e.g., AUCROC/F1 and Jaccard similarity index for report generation, FID alone for image generation, and accuracy alone for VQA.

Author response:
**W3. Metrics used are uncommon.**\
We respectfully disagree on this point. We use AUROC/F1-scores based on label extraction to assess the accuracy of report generation because it is most commonly used in this area of research as shown in [1, 2 3]. Jaccard similarity index was also employed because AUROC/F1-scores favor detection of positive classes whereas the Jaccard similarity index treats both positive and negative classes equally; this is important as hallucinations are a well-known problem in multimodal language models, and negative classifications are of equal importance to positive classifications in analyzing medical images. Although we find them much less informative for medical reports, we have also added NLG metrics (e.g. BLEU, ROUGE, CIDEr).

For the image generation task, FID assesses quality of generated images, and AUROC/F1 using a pretrained CXR image classifier assesses whether generated images are in line with the given text inputs used to generate them. The latter evaluation is more important in our case as it assesses image-text alignment. To our best knowledge, there is no common or agreed upon way to evaluate the alignment between generated images from a model and the given input prompts used to generate them. We believe that this set of assessments, combined with the provided qualitative examples, showcases the capabilities of the models under investigation.

Reviewer comment (Experiments):
In the results of report generation, only LLM-based methods are compared. How about a dozen of those SOTA methods in chest X-ray reporting? IU X-ray is another dataset commonly used for the evaluation of report generation performance. It will be helpful to report results on that as well, mainly when used as a cross-domain evaluation dataset.

Author response:
**W4-1. Only LLM-based models are compared for report generation**\
Med-PaLM1/2 would also be a great target for comparison, but they are not publicly available yet. As per your suggestion, we include radiology report generation models that have the best reported performance as baselines for comparison. namely IFCC [3] &  R2Gen [4]. These models showed weak performance compared to LLM-based models.

***Table 1. CXR-to-report (part)***
|F1|**Micro**|**Macro**|**Weighted**|
|-|-|-|-|
|RadFM|0.370|0.204|0.341|
|LLM-CXR|0.360|0.211|0.350|
|IFCC|0.220|0.141|0.225|
|R2Gen|0.201|0.113|0.183|

**W4-2. IU X-ray dataset evaluation**\
Please refer to general response.

Reviewer comment (Experiments):
I am not sure why the upsampling is performed for the image data as described in section 3.1 since the X-ray images are much larger than this resolution. How this upsampling process will affect the results?

Author response:
**W5. Upsampling in section 3.1**\
We wanted to simulate a situation where UniXGen, which can only receive 512px images, receives 256px images (the resolution for input to LLM-CXR and XrayGPT). But we also report the outcomes for UniXGen using its native resolution 512x512 images in Table 1.

Reviewer comment (Reproducibility):
In section 2.2, the authors mentioned that the image tokens are parts of expanded embedding tables. I wonder how big K_img should be?

Author response:
**W6. How big should the image token space be?**\
$K_img$ is a value that is predefined as a design configuration for the VQ-GAN model architecture. We used the $K_img=1024$ as  used in the original VQ-GAN paper for 256px images.

Reviewer comment (Experiments):
How scalable is the tokenization and fine-tuning process for larger medical images (e.g., 3D CT/MR scans)?

Author response:
**Q1. How scalable is the tokenization and fine-tuning process for larger medical images (e.g. 3D/MR scans)?**\
Although we have not conducted experiments for 3D data, we presume that 3D scans will probably need a similar tokenization strategy adapted to be more appropriate for higher-dimensional data. We think that certain ideas from our approach can be extended to 3D images as well; for example, the TXV-loss we employed to preserve clinical information during tokenization would also be applicable for 3D medical images.

After reading your review and the RadFM paper, we have also begun work on 3D applications of this methodology. Our past experience informs us that methods using 3D volumes directly is vulnerable to the 'curse of dimensionality' (data requirements increase exponentially with increase in dimensionality); and so we are considering a "2.5D" approach in which a CT or MRI volume is treated as a sequence of 2D slices, each of which can be tokenized using methodologies developed for 2D images. While results still remain to be seen, we will, as per your suggestion, add discussions on this topic in the manuscript as it may be of interest to more readers.

Reviewer comment (Evaluation):
How interpretable are the model's decisions, especially given the clinical context where explanations for predictions are crucial?

Author response:
**Q2. How interpretable are the model's decisions?**\
We agree that interpretability is a crucial consideration in clinical models. We believe that the natural language-based outputs of LLM-CXR intrinsically offer a higher level of interpretability than traditional computer vision models that output classification results. LLM-CXR is also able to generate images given some input text like ‘severe left-sided pneumonia’; this offers a window into the inner workings of the model as we can ask it to show images that correspond to a certain concept described using text.

As an additional note, compared to approaches in which the LLM receives images in the form of pseudo-text produced by an adapter network (e.g. image encoding networks), we believe that our method of directly processing images and treating them equal to natural language is more promising for interpretability as if the semantics of the image are delivered as pseudo-text, interpretation of the LLM’s understanding of images themselves is bound to be limited.

As for interpretability, we are also developing methods to offer more insight into model outputs such as visualizing the parts of the image that are most influential in generating certain natural language phrases (mimicking how radiologists often mark the important parts of a scan with arrows or circles). Since LLM-CXR effectively translates information between images and text within a single neural network (the LLM) and not in disparate adapter modules, we believe that making this translation process explicit will be most feasible for our model particularly. As you mentioned, because evidence-based care and clear rationales for decision-making are crucial in healthcare, we see this as important follow-up work.

Reviewer comment (Experiments):
The comparison with other related methods is missed. https://github.com/chaoyi-wu/RadFM

Author response:
**Q3. Comparison with RadFM**\
Thank you for bringing RadFM model to our attention. This work shares many of our motivations, and we were excited to see that they have open-sourced their work. We include RadFM in our assessments for CXR-to-report generation and CXR-VQA:

***Table 1. CXR-to-report generation (part)***
|**F1**$\uparrow$|**Micro**|**Macro**|**Weighted**|
|---|---|---|---|
|RadFM|**0.370**|0.204|0.341|
|XrayGPT|0.326|0.209|0.330|
|LLM-CXR|0.350|**0.211**|**0.350**|

***Table 3. CXR-VQA (part)***

|**Accuracy**$\uparrow$|**All**|**Presence**|**Location**|**Size,severity,type**|
|---|---|---|---|---|
|XrayGPT|25.2%|27.4%|21.9%|20.3%|
|RadFM|32.7%|34.5%|31.3%|20.8%|
|**LLM-CXR**|**44.8%**|**41.3%**|**50.0%**|**62.5%**|

Note that RadFM is based on the LLaMa-13B model and LLM-CXR is based on the dolly-3B model, so there is about a 4-fold difference in parameter count; but LLM-CXR's performance is comparable if not better.

Reviewer comment (Experiments):
**Potential for Catastrophic Forgetting**: As with any endeavor to expand a pretrained model's capabilities, there's the inherent risk of the model losing or diminishing its foundational skills—known as catastrophic forgetting. While the paper's intent is to preserve and build upon the LLM's language capabilities, it doesn't extensively address measures taken to prevent this potential degradation.

Author response:
**A.** Our main goal was to leverage the pre-existing language capabilities of a large language model to tune it into an (instructionable in natural language) expert model that can describe images using text, generate images given an input text, and perform visual question-answering tasks. Because all three tasks and natural language instruction following ability require an understanding of language, we build on top of an LLM; as we simply sought to leverage the language capabilities of the LLM, it was not our goal to maintain all of the previous text-based knowledge contained in the model.

Our model is intended to be a form of expert module for future AGI that communicates in natural language, and therefore we believe that natural language skills, excluding task-related language skills, only require a basic ability to follow natural language instructions.  While it would of course be beneficial to mitigate forgetting as much as possible in a model with a given number of parameters, we view this as future work that requires its own set of innovations.

Reviewer comment (Experiments):
**Adapter Network Dismissal**: The paper critiques the prevalent "adapter network" approach but doesn't provide a comprehensive empirical comparison. A deeper, side-by-side evaluation highlighting performance, adaptability, and computational efficiency would offer a clearer picture of why their "instruction-finetuning" method is superior or preferable.

Author response:
**A.** We do not mean to dismiss approaches using adapter networks for vision-language integration. Most concurrent vision-language research works use these adapter networks (e.g. XrayGPT and ELIXR) for integration of visual information to LLMs. On the other hand, there has been a surprising lack of investigation into other novel avenues (such as our method of utilizing instruction-finetuning for vision-language alignment). Moreover, the reviewer is kindly reminded that the adapter networks cannot generate images without connecting to an image generation model such as Stable Diffusion.

In the medical domain especially, we believe that our method of instruction-tuning to achieve alignment of image-text information may hold more promise as pixel-level detailed visual information must be adequately described in the text. This is in contrast to natural images where, as shown in works such as BLIP-2, the use of adapter networks to pass information to an LLM may be enough to yield sufficiently performant models as broader features confer the semantic meaning. This was an important motivation behind our work, and we wanted to convey this need for investigation of alternative approaches to image-text alignment rather than dismissing the adapter network approaches as a whole. 

Our evaluation experiments comparing results for LLM-CXR to those of XrayGPT and ELIXR showcase the viability of our approach against these adapter network approaches, as they both utilize larger models but show similar or lower performance in image understanding.

Reviewer comment (Evaluation):
**VQ-GAN Limitations**: Using VQ-GAN for image tokenization introduces its own set of challenges. VQ-GANs, while powerful, can sometimes produce artifacts or representations that aren't entirely faithful to the original image. The paper doesn't discuss how it mitigates or addresses these potential shortcomings, leaving room for questioning the quality or accuracy of the generated content.

Author response:
**A.** As pointed out, VQ-GAN has limitations in reconstruction due to its characteristics (e.g. vector quantization). There are limitations in terms of information preservation, so VQ-GAN uses LPIPS-like loss to improve the preservation of perceptual information. An important informational aspect of CXRs is the presence or absence of certain lesions. Therefore, we introduced a clinical information-preserving loss into VQ-GAN to preserve lesion information as much as possible in the VQ-GAN structure.

Considering that our ultimate goal is not to ensure that LLM accurately restores a given image but to understand images and generate them based on text, it was more in line with our objective to aim for latent tokens that contain semantic information rather than details for exact reconstruction. In other words, in our work, we used VQ-GAN's reconstruction loss as a means of encoding and decoding an image for textual understanding rather than for complete reconstruction.

As a side note, we would like to point out that an adapter network with Stable Diffusion image generator cannot reconstruct original input images as closely as our VQ-based method.

Reviewer comment (Evaluation):
**Lack of Diverse Evaluation**: The paper's evaluation, though rigorous, might benefit from a more diverse set of metrics. For instance, qualitative evaluations or user studies involving medical professionals could provide insights into the model's practical utility. Comparisons with human diagnostic capabilities might also offer a benchmark for the model's proficiency.

Author response:
**A.** The comprehensive evaluation by the end-users (ie, clinicians) would be the eventual target, but owing to time constraints, we cannot conduct these experiments yet. While we will try to conduct such experiments, we believe that the utilized metrics can guide further model development.

Reviewer comment (Reproducibility):
**RoentGen AUC and FID numbers**: The authors need to address the discrepancy between the AUC and FID numbers provided in their paper and those in the RoentGen paper. It should be clarified whether these numbers were reproduced by the authors or cited from the RoentGen study, and if there are differences, the paper should explain the reasons behind these.

Author response:
**A.** The Difference is most likely because we used a different test set. To compare the different models from different works, we used a single test set (that is presumably different from the test set used in the RoentGen paper) across all experiments. However, we did use the model weights and inference code provided by the authors.

Reviewer comment (Writing):
Grammar throughout the text requires attention for better readability, as exemplified by the construction of the second sentence in section 2.2.

Author response:
**A.** We have fixed and improved the writing to be clearer to the reader.

Reviewer comment (Novelty):
The paper's premise relies on the fine-tuning of LLM across multiple instruction tuning tasks. This approach isn't particularly novel, as numerous large-scale medical models adopt a similar strategy but with a broader functional range. Both the innovation in method and the paper's applicative contribution seem to be lacking.

Author response:
**W1/MethodologyConcerns-Q1.  The approach of multiple instruction-tuning tasks is not particularly novel. How does our methodology differentiate itself from other existing large-scale models that employ a similar strategy?**

**A.** The main distinction of our work from other work using instruction-tuning to build medical LLMs is that we use instruction-tuning to teach an LLM to both understand and generate images. Most instruction-tuning methodologies are geared towards changing a next word-predicting model to an instruction-following model using instruction-answer pairs (e.g. GPT3.5, PaLM) rather than promoting the integration of visual information to language-based models. There is previous work on instruction-tuning with visual information (e.g. LLaVA, InstructBLIP) which are most similar to our work; but these works employ adapter networks CLIP and BLIP respectively for image-text alignment and employ instruction-tuning techniques simply as a means of passing visual information to an LLM. We on the other hand aimed to use instruction-tuning itself as a method of aligning images and text. You may have noticed that no separate image-text aligning techniques (e.g. CLIP, BLIP, ALIGN) are used at all. In more detail, image encoders in approaches such as LLaVA and InstructBLIP have already been pre-trained with the paired vision-language dataset, and the encoder already outputs pseudo-text; the decoder in turn receives pseudo-text as input. In this process, the language model does not directly deal with images as it takes pseudo-text as input and outputs pseudo-text. However, in our approach visual information is passed directly to the LLM, and the LLM itself learns visual features and vision-language alignment on top of its previous knowledge of the language.

Another distinction is that our instruction-tuning method also enables the generation of images. This confers a few benefits: 1) we can ask the model to directly visualize a certain concept encapsulated in text (e.g., pneumonia); 2) this bidirectionality in generation turns out to be vital for aligning images and text without any adapter networks; 3) the visual instruction tuning method of LLaVA can teach the model image features but only the features that are close to the semantic meaning of the texts and not the fine-grain visual information in images that are not completely described by text.

We hope that the distinction of our work from previous ones is now clearer.

Reviewer comment (Experiments):
The LLM-CXR, when compared with the open-sourced versions of XrayGPT and UniXGen, appears to be unfair. It's noted that these models aren't trained on a consistent instruction tuning dataset. It's recommended that a uniform dataset is utilized for training before making such comparisons. Furthermore, there's an evident absence of comparison with existing LLM+instruction tuning models, such as Med-PaLM and Med-PaLM 2.

Author response:
**W2/ComparativeAnalysis-Q1. Comparison with XrayGPT and UniXGen is not fair because they are not trained on the same instruction-tuning dataset.**

**A.** XrayGPT and UniXGen are trained on the same dataset as ours (MIMIC-CXR). However, we have used the dataset to train in an instruction-tuning manner. 

Regarding your comment on fair comparison, we respectfully disagree with the reviewer since we started our research from the viewpoint that using an enriched instruction-tuning set itself is a distinction from previous works such as XrayGPT and UniXGen.

Reviewer comment (Experiments):
The paper's comparative methods across various understanding and generative tasks aren't comprehensive. For instance, in the realm of report generation, several contemporary methodologies exist, such as "Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report Generation, CVPR 2023."

Author response:
**W3/Comparative-AnalysisQ2/3. Evaluation is not comprehensive. Compare with contemporary radiology report generation methodologies or Med-PaLM1/2.**

**A.** The motivation for our work was to teach a large language model to understand and generate chest X-ray images so that it can serve as a more general-purpose interface for medical images than report generation models. This is why we felt most meaningful comparisons were against similar large language models that can understand CXR images (XrayGPT, ELIXR). 

As per your suggestion, we have included radiology report generation models with best reported performance as additional baselines for comparison. The results confirm that our models are superior or comparable to the dedicated contemporary report generation models (Table 1, 2 in the revised manuscript).

***Table 1. CXR-to-report (part)***
|F1|**Micro**|**Macro**|**Weighted**|
|-|-|-|-|
|RadFM|0.370|0.204|0.341|
|LLM-CXR|0.360|0.211|0.350|
|IFCC|0.220|0.141|0.225|
|R2Gen|0.201|0.113|0.183|

As you mentioned, Med-PaLM1/2 would also be a great target for comparison, but they are not publicly available yet. Also, Mingjie Li *et al.* did not release pretrained checkpoints, and the training code does not run despite our efforts to debug the authors’ code.

Reviewer comment (Evaluation):
The evaluation metrics used for report generation seem lacking in depth. For instance, within the paper (table 1), only the AUROC scores across six categories are reported, while prior research typically reports the average Precision/Recall/F1-score across all 14 categories.

Author response:
**W4. Evaluation metrics should also be more comprehensive e.g. results for all 14 lesions.**

**A.** Upon reviewing the comments, we agree that a more comprehensive evaluation with all 14 lesions is appropriate and provides updated results using the 14 lesions above in the revised manuscript. This method of evaluation does not affect the overall results.

Reviewer comment (Writing):
The paper's writing style isn't fluid and contains multiple errors, hindering smooth reading and comprehension. For instance, repetitive phrases like "… generate these VQAs as shown as shown in …" and grammatical mistakes like "During the fine-tuning process …" detract from the paper's clarity.

Author response:
**W5. Writing needs to be improved.**

**A.** We have fixed and improved the writing to be clearer to the reader.

Reviewer comment (Novelty):
How does the instruction-finetuning method differentiate itself from existing large-scale medical models that employ a similar strategy?

Author response:
**W1/MethodologyConcerns-Q1.  The approach of multiple instruction-tuning tasks is not particularly novel. How does our methodology differentiate itself from other existing large-scale models that employ a similar strategy?**

**A.** The main distinction of our work from other work using instruction-tuning to build medical LLMs is that we use instruction-tuning to teach an LLM to both understand and generate images. Most instruction-tuning methodologies are geared towards changing a next word-predicting model to an instruction-following model using instruction-answer pairs (e.g. GPT3.5, PaLM) rather than promoting the integration of visual information to language-based models. There is previous work on instruction-tuning with visual information (e.g. LLaVA, InstructBLIP) which are most similar to our work; but these works employ adapter networks CLIP and BLIP respectively for image-text alignment and employ instruction-tuning techniques simply as a means of passing visual information to an LLM. We on the other hand aimed to use instruction-tuning itself as a method of aligning images and text. You may have noticed that no separate image-text aligning techniques (e.g. CLIP, BLIP, ALIGN) are used at all. In more detail, image encoders in approaches such as LLaVA and InstructBLIP have already been pre-trained with the paired vision-language dataset, and the encoder already outputs pseudo-text; the decoder in turn receives pseudo-text as input. In this process, the language model does not directly deal with images as it takes pseudo-text as input and outputs pseudo-text. However, in our approach visual information is passed directly to the LLM, and the LLM itself learns visual features and vision-language alignment on top of its previous knowledge of the language.

Another distinction is that our instruction-tuning method also enables the generation of images. This confers a few benefits: 1) we can ask the model to directly visualize a certain concept encapsulated in text (e.g., pneumonia); 2) this bidirectionality in generation turns out to be vital for aligning images and text without any adapter networks; 3) the visual instruction tuning method of LLaVA can teach the model image features but only the features that are close to the semantic meaning of the texts and not the fine-grain visual information in images that are not completely described by text.

We hope that the distinction of our work from previous ones is now clearer.

Reviewer comment (Experiments):
Is the comparison between LLM-CXR and the open-sourced versions of XrayGPT and UniXGen made on a consistent dataset?

Author response:
**W2/ComparativeAnalysis-Q1. Comparison with XrayGPT and UniXGen is not fair because they are not trained on the same instruction-tuning dataset.**

**A.** XrayGPT and UniXGen are trained on the same dataset as ours (MIMIC-CXR). However, we have used the dataset to train in an instruction-tuning manner. 

Regarding your comment on fair comparison, we respectfully disagree with the reviewer since we started our research from the viewpoint that using an enriched instruction-tuning set itself is a distinction from previous works such as XrayGPT and UniXGen.

Reviewer comment (Experiments):
Why hasn't the paper compared its methodology with existing LLM+instruction tuning models, such as Med-PaLM and Med-PaLM 2?

Author response:
**W3/Comparative-AnalysisQ2/3. Evaluation is not comprehensive. Compare with contemporary radiology report generation methodologies or Med-PaLM1/2.**

**A.** The motivation for our work was to teach a large language model to understand and generate chest X-ray images so that it can serve as a more general-purpose interface for medical images than report generation models. This is why we felt most meaningful comparisons were against similar large language models that can understand CXR images (XrayGPT, ELIXR). 

As per your suggestion, we have included radiology report generation models with best reported performance as additional baselines for comparison. The results confirm that our models are superior or comparable to the dedicated contemporary report generation models (Table 1, 2 in the revised manuscript).

***Table 1. CXR-to-report (part)***
|F1|**Micro**|**Macro**|**Weighted**|
|-|-|-|-|
|RadFM|0.370|0.204|0.341|
|LLM-CXR|0.360|0.211|0.350|
|IFCC|0.220|0.141|0.225|
|R2Gen|0.201|0.113|0.183|

As you mentioned, Med-PaLM1/2 would also be a great target for comparison, but they are not publicly available yet. Also, Mingjie Li *et al.* did not release pretrained checkpoints, and the training code does not run despite our efforts to debug the authors’ code.

Reviewer comment (Experiments):
Are there reasons for not exploring contemporary methodologies in report generation, like "Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report Generation, CVPR 2023"?

Author response:
**W3/Comparative-AnalysisQ2/3. Evaluation is not comprehensive. Compare with contemporary radiology report generation methodologies or Med-PaLM1/2.**

**A.** The motivation for our work was to teach a large language model to understand and generate chest X-ray images so that it can serve as a more general-purpose interface for medical images than report generation models. This is why we felt most meaningful comparisons were against similar large language models that can understand CXR images (XrayGPT, ELIXR). 

As per your suggestion, we have included radiology report generation models with best reported performance as additional baselines for comparison. The results confirm that our models are superior or comparable to the dedicated contemporary report generation models (Table 1, 2 in the revised manuscript).

***Table 1. CXR-to-report (part)***
|F1|**Micro**|**Macro**|**Weighted**|
|-|-|-|-|
|RadFM|0.370|0.204|0.341|
|LLM-CXR|0.360|0.211|0.350|
|IFCC|0.220|0.141|0.225|
|R2Gen|0.201|0.113|0.183|

As you mentioned, Med-PaLM1/2 would also be a great target for comparison, but they are not publicly available yet. Also, Mingjie Li *et al.* did not release pretrained checkpoints, and the training code does not run despite our efforts to debug the authors’ code.

Reviewer comment (Evaluation):
Why were the AUROC scores for report generation only presented for six categories instead of the standard 14 categories?

Author response:
**W4. Evaluation metrics should also be more comprehensive e.g. results for all 14 lesions.**

**A.** Upon reviewing the comments, we agree that a more comprehensive evaluation with all 14 lesions is appropriate and provides updated results using the 14 lesions above in the revised manuscript. This method of evaluation does not affect the overall results.

Reviewer comment (Experiments):
The experiments in this paper were conducted using only one dataset. As the author mentioned, the quality of the dataset used was limited, necessitating adjustments in the training strategy.

Author response:
**W1. Experiments are done on just one dataset.**

**A.** Please refer to the general response for added evaluation results on an out-of-domain dataset (IU chest X-ray).

Reviewer comment (Evaluation):
The paper lacks visualization of the alignment of images and text in the feature space. Particularly when using VQA for data augmentation, the questions posed by GPT contain rich prior information, focusing the text more on the key lesion areas in the images. I believe that appropriate visualization is necessary to demonstrate the effectiveness of incorporating VQA.

Author response:
**W2. Visualization of alignment in feature space.**

**A.** While it is true that a direct visualization image and text features in the embedding space would provide the most obvious evidence for image-text alignment, we found a dearth of research on the creation and interpretation of embedding spaces in GPT-based multimodal models. In fact, a recent paper [1] showed that the integration of visual and textual information in text-pretrained multimodal transformer models occurs in the neurons themselves rather than in the feature space. We think that more work on the interpretability of feature spaces of multimodal LLMs is necessary before we are able to apply such techniques to gauge image-text understanding in models. Thus instead, we demonstrate achievement of image-text alignment in the evaluation experiments and ablation studies: training with the VQA task leads to improvements in both the image-based text generation task and the text-based image generation task even though the VQA task does not directly employ these tasks, serving as evidence of improved image-text understanding in the model.

Reviewer comment (Writing):
This paper demonstrates the potential application of LLMs in the medical field, but the content of the study extends beyond generating diagnostic reports from radiological images. I hope the author could further clarify in the introduction whether there are specific application scenarios for this research.

Author response:
**Q1. Clarification of specific application scenarios.**

**A.** We believe that a model such as ours can be a versatile AI assistant for interacting with large volumes of medical data. For instance, it can be used to find CXR images with specific findings described using natural language from a large dataset of CXRs. This is different from using a traditional image classification network as those networks can only distinguish between predefined labels. This is also different from radiology report generation models that simply take in an image and produce a text output as models that can understand language can provide answers to more contextualized queries, which is more similar to the human radiologist’s workflow (radiologists do not look at the images alone; they understand the clinical context of the patient and the study to produce reports based on them). With confirmation of sufficient accuracy, a model like this can potentially be used to select patients for a clinical trial based on a set inclusion/exclusion criteria or be used for routine, relatively simple clinical tasks such as clearing the absence of certain lesions prior to a procedure.

Reviewer comment (Experiments):
In '2.1 CLINICAL INFORMATION-PRESERVING CXR TOKENIZATION', the author mentions '...causes loss of clinically important information such as characteristics of microscopic lesions...'. However, in this study, generating image tokens is key to aligning images with text during LLM fine-tuning. Is the mere use of L2 reconstruction loss sufficient to effectively reduce the loss of clinical information? Could there be an enhancement of features in the design of the module network structure? I think this is a very important challenge in this paper, and I hope the difference in experimental results before and after the introduction of L2 reconstruction loss can be explained.

Author response:
**Q2. L2 reconstruction loss may not be enough to reduce the loss of clinical information.**

**A.** The preservation of information in the process of tokenizing images into a token space shared by both text and image is definitely an important, major challenge that has to be overcome to develop an accurate model using our methodology. As you have hinted, our employment of the L2 reconstruction loss, while helpful, does not address this problem entirely and there is room for improvement; however, we would like to delve further into why our VQ-GAN training method is able to take a solid step towards that direction:

The original VQ-GAN improves upon the VQ-VAE’s bottleneck architecture and a quantized latent space with LPIPS and adversarial losses. These enhancements improved reconstruction quality, especially with regards to reducing blurry outputs characteristic of models trained with pixel space L1 loss. However, because LPIPS is a loss based on a network trained on natural images (namely Inception-V3), we hypothesized that it is not adequate for extracting features of CXR images and could be improved with a loss tailored for CXR images. Hence, the TXV-loss we use is a variant of the LPIPS loss but calculated using the features of a neural network trained on CXR images (torchxrayvision library). As our ablation study shows that the use of the TXV-loss does increase the performance of the final model, we believe that our hypothesis has at least some merit.