Paper: MedJourney: Counterfactual Medical Image Generation by Instruction-Learning from Multimodal Patient Journeys (ICLR-2024)

Reviewer comment (Evaluation):
The accuracy of descriptions generated by GPT-4 lacks evaluation, which directly affects whether the generated images are correct or not.

Author response:
Although GPT-4 has no specialized training in biomedicine, many prior studies have demonstrated its impressive capabilities in understanding and processing biomedical text. E.g., book by Peter Lee et al. on: “The AI Revolution in Medicine: GPT-4 and Beyond.” offers insights into this aspect.

Additionally, one of the coauthors, a board-certified practicing radiologist at a top US medical school, has played a crucial role in assessing the utility of GPT-4 in our context. The progression description generated by GPT-4 appears reasonable upon preliminary examination. We will add more systematic assessments in the final version. 

To illustrate, here are some examples reviewed by the radiologist, comparing GPT-4 vs orginal text: 
|Impression| GPT-4 |
|-----| -----|
“Resolving right middle lobe pneumonia. A followup chest radiograph in 4 weeks is recommended. If the right middle lobe opacity fails to completely resolve by that time, a chest CT should be performed at that time to exclude an endobronchial lesion. New small right pleural effusion.” |”Resolving right middle lobe pneumonia. New small right pleural effusion.”|
“Subtle right lower lobe opacity may represent early pneumonia. These findings were discussed with Dr. _ by Dr. _ at 2:30 p.m.” |"Subtle right lower lobe opacity may represent early pneumonia."

These examples demonstrate GPT-4's ability to succinctly summarize key radiological findings, which aligns well with clinical needs and workflow efficiency. This aspect will be further elaborated in the final version of our manuscript.

Likewise, in addition to the quantitative evaluation, we have conducted several review sessions with the aforementioned radiologist co-author, who expressed satisfaction with the image generation quality. We will mention this in the final version. Thanks much for your suggestion.

Reviewer comment (Writing):
Why use the word `counterfactual` instead of direct `image editing`? The latter is more intuitive and understandable. Since reference images already exist in this case, such image generation does not appear counterfactual.

Author response:
We agree that the underlying technique might share a lot of similarity, but the real-world semantics are quite different. In biomedicine, a clinician won’t be thinking about this generation problem as editing an image. Instead, their interest lies in counterfactual reasoning,  i.e., projecting what if the patient’s condition has progressed in certain ways. Many prior works have studied the “Counterfactual” factor for medical images, e.g. Cohen et, al. “Gifsplanation via Latent Shift: A Simple Autoencoder Approach to Counterfactual Generation for Chest X-rays”; Sumedha Singla, et, al. “Explaining the Black-box Smoothly- A Counterfactual Approach”. In this context, counterfactual just means answering what if questions, which could be with or without the prior image. However, in the case of progression modeling, the patient already has some prior image taken. So it is natural to condition on the prior image, rather than generating a newmage from scratch without any constraints. This distinction is pivotal in our research, as it aligns with the clinical workflow and decision-making processes, emphasizing the practical applicability in real-world medical settings. We will mention this in the final version.

Reviewer comment (Evaluation):
In Table 4, MedJourney has more accurate segmentation results than Reference Image; does this mean that segmentation results are difficult to evaluate the quality of generated images?

Author response:
Thanks for the insightful comment. Here, we are comparing the similarity of a generated image with the prior image. The reference image may have lower DICE score due to variations in the creation process for the consecutive images, which are taken at different times, potentially by different clinicians with different machines. This variation is a typical characteristic in clinical imaging and does not imply inferiority of the reference image.

By contrast, the extremely low scores for baseline systems indicate that they are completely off base with respect to the prior image. 

We will clarify this in the final version.

Reviewer comment (Experiments):
**Comparison to more recent works that closely follow the instruction and reference image**: While MedJourney has shown promise in counterfactual medical image generation, it lacks a direct comparison with more recent works, such as "Imagic: Text-Based Real Image Editing with Diffusion Models." Imagic, designed to closely follow instructions and reference images, is potentially better suited for the tasks that MedJourney addresses with medical images. A side-by-side comparison or integration of insights from Imagic could potentially strengthen the robustness and applicability of MedJourney.

Author response:
Thanks for this suggestion. Imagic requires per-example inversion and fine-tuning. Namely, for each input image and target description, it requires a fine-tuning process to obtain an optimal text embedding close to the target description, fine-tune the diffusion model conditioned on the optimal text embedding (rather than noise in the original diffusion model), and then apply the newly fine-tuned diffusion model to an interpolation of original & target text embeddings to generate the new image. This is very cumbersome and not very generalizable. 

By contrast, our method builds on the more recent InstructPix2Pix method, which learns a single model that can generate new image in a single forward pass, without requiring elaborate text embedding and diffusion learning for each test example as in Imagic.

We will add discussion to explain why our method is superior to the Imagic approach.

Reviewer comment (Evaluation):
**Need for reader study in addition to quantitative metrics for qualitative evaluation of different methods**: While the authors have introduced a comprehensive suite of quantitative tests, the paper could benefit from a reader study that provides qualitative insights. Direct feedback from radiologists or medical experts on the interpretability, authenticity, and clinical utility of the counterfactual images would enhance the evaluation depth and offer a more holistic understanding of MedJourney's real-world applicability.

Author response:
One of the coauthors is a board-certified practicing radiology at a top US medical school. In addition to the quantitative evaluation, we have conducted several review sessions with this radiologist co-author, who expressed satisfaction with the generation quality. We will mention this in the final version. Thanks much for your suggestion.

Reviewer comment (Experiments):
**How well does it apply to other medical image datasets?** While MedJourney's performance on the MIMIC-CXR dataset is commendable, its generalizability across diverse medical imaging modalities remains uncertain. Demonstrating its efficacy on a broader range of datasets, such as MRI, CT scans, or ultrasounds, would substantiate its versatility. Furthermore, insights into how the method adapts and performs across different diseases, anatomical regions, or imaging techniques would solidify its position as a universally applicable counterfactual medical image generation tool.

Author response:
Thanks much for this suggestion. While we tried to break new ground in counterfactual medical image generation, this is just the first baby step towards a universally applicable tool. We plan to explore exactly along the directions you have outlined in future work.

Reviewer comment (Experiments):
**Demographic diversity is crucial for medical datasets**: Medical datasets must accurately represent the diverse patient populations they serve to ensure that AI models generalize well across varied demographic groups. The authors acknowledge in the appendix that the dataset has an overrepresentation of the white elderly population. Such biases can inadvertently lead to models that perform sub-optimally or even erroneously for underrepresented groups, which can have significant clinical implications. Highlighting this limitation in the main body of the paper, especially in sections like 'Limitations' or 'Discussion,' would underscore the importance of demographic diversity. It would also serve as a call to action for researchers to actively pursue more balanced datasets, ensuring equitable healthcare outcomes for all patients regardless of their background.

Author response:
Thanks much for this insightful suggestions. We will move our discussion about demographic diversity to the main body. 
Further, we have conducted additional evaluations focusing specifically on racial diversity. Our analysis compared image generations for the dominant demographic group (white) and an underrepresented group (black). Despite the training data itself being skewed in terms of racial representation (**17%** black vs **78%** white), our results are promising. MedJourney successfully retained the racial features of the prior image with high fidelity, scoring **0.92** for the underrepresented group and **0.98** for the dominant group (we used the SOTA image race classifier, as mentioned in sec 4.3, as evaluator and AUC as metric). This indicates our MedJourney has a decent property of preserving demographic diversity,  and further suggests that it may contribute to mitigating racial representation disparities through data augmentation for underrepresented groups. 
While our results are promising, they also emphasize the crucial need for balanced datasets in medical research. We will include this insight in the final version.

Reviewer comment (Evaluation):
**Add FID as an additional metric**: Please add FID as an additional qualitative metric to evaluate different methods. See UnixGen and RoentGen paper for details on the FID metric.

Author response:
Thanks for the suggestion. We have incorporated this metric into our analysis and will present it in the final version. The FID results are as follows: 
| Method| FID|
|-|-| 
| Stable Diffusion| 291.13
|InstructPix2Pix| 180.72
|RoentGen| 42.61
|MedJourney| **29.68**

We believe these results further demonstrate the effectiveness of MedJouney.

Reviewer comment (Evaluation):
The paper does not indicate whether the disease progression generated by GPT-4 has been compared or verified against radiologists' opinions. Such a comparison is crucial for establishing the reliability of the generated progression description.

Author response:
**Validity of disease progression**: Thanks for the suggestion. One of the coauthors is a board-certified practicing radiology at a top US medical school. The progression description generated by GPT-4 appears reasonable upon preliminary examination. We will add more systematic assessments in the final version. In general, although GPT-4 has no specialized training in biomedicine, many prior studies have demonstrated its amazing capabilities in understanding and processing biomedical text. E.g., see the book by Peter Lee et al. on: “The AI Revolution in Medicine: GPT-4 and Beyond.”

Reviewer comment (Evaluation):
There seems to be no mention of sanity checks for the progression generated between two images. Considering GPT-4's capability to generate progression even for unrelated images of different patients, it is vital to understand how the paper ensures the accuracy and relevance of these progressions.

Author response:
**Lack of sanity check**: In addition to the quantitative evaluation, we have conducted several review sessions with the aforementioned radiologist co-author, who expressed satisfaction with the generation quality. We will mention this in the final version. Thanks much for your suggestion.

Reviewer comment (Evaluation):
The advantages of using GPT-4 over the impression section in reports are minimal (as evident in Table 2). Clarification on how the impression section was used and whether GPT-4 is actually essential for this study would be beneficial, given the marginal gains across metrics.

Author response:
**Comparison between Impression and GPT-4**: The impression section is taken from the gold report describing the target image. The small difference between using impression vs GPT-4 generated progression may stem from the characteristics of this dataset, where in many cases the first image is normal, and thus the pathological description of the follow-up image essentially describes progression. Additionally, some impression would include comparison against the prior image, also similar to what GPT-4 would generate as progression. As in 5.3 “impression v.s. GPT-4 generation”, GPT-4 generated is usually more succinct (averaging 146 characters v.s. 179) and more naturally phrased. For example,  
|Impression| GPT-4 |  
|-----| -----|  
“Resolving right middle lobe pneumonia. A followup chest radiograph in 4 weeks is recommended. If the right middle lobe opacity fails to completely resolve by that time, a chest CT should be performed at that time to exclude an endobronchial lesion. New small right pleural effusion.” |”Resolving right middle lobe pneumonia. New small right pleural effusion.”|  
“Subtle right lower lobe opacity may represent early pneumonia. These findings were discussed with Dr. _ by Dr. _ at 2:30 p.m.” |"Subtle right lower lobe opacity may represent early pneumonia."

Reviewer comment (Novelty):
The substantial improvements seem to stem from image registration, primarily leveraging the SimpleITK toolkit (Beare et al. 2018). This raises concerns about the originality of the MedJourney method's contributions. An expanded discussion on how image registration was adapted for the proposed method would provide a clearer understanding of its impact.

Author response:
Note that image registration is only a preprocessing step that helps align the two consecutive images for a given patient in the training data. This is a standard technique to reduce immaterial variations of images taken at different times by different technicians, e.g., translational shift or rotation. Image registration alone can’t perform counterfactual image generation and is thus irrelevant with respect to the novelty of MedJourney. We will clarify this in the final version.

A significant challenge in our research was the limited dataset size (approximately 10,000 triplets), posing the risk of learning spurious correlations due to minor, non-essential variations in images. By implementing image registration, we effectively minimized these immaterial variations, enabling the model to focus on material changes indicative of disease progression.

This image registration procedure is used merely for preprocessing training data. This procedure is illustrated in Appendix Figure 8, and to enhance understanding, we will move it to the main part in the final version.  

This innovative use of image registration in our specific learning pipeline is, to our knowledge, unprecedented in both generic and medical image generation research. We want to reiterate that, image registration cannot generate counterfactual image on its own, its use in MedJourney represents an inventive adaptation of existing techniques, augmenting rather than diminishing the method's originality.

Reviewer comment (Experiments):
The necessity of the prompt “a photo of chest x-ray” is not justified. Including results without this prompt might offer more insight into its impact.

Author response:
# Re prompt requirement:

Thanks for pointing out. This prompt “a photo of chest x-ray” facilitates to generate better results in our exploration. As we observed in our experiments, without the prompt, the baseline Stable Diffusion (SD) model tends to produce arbitrary results, getting lost from the descriptions of disease progression. That being said, we introduced this prompt to elicit the maximal capacity of SD for conducting the fair comparison as possible as we can. We will include quantitative results in the final version when this prompt is not used for the justification.

Reviewer comment (Novelty):
There is a missing citation for GPT-4 in the related work section on page 3.

Author response:
# Re issue on citations and figures:

Thank you very much for all the good catches. We will fix them in the final version.

Reviewer comment (Presentation):
Figure 3 could be better placed in the experiments section, as it is not referenced beforehand.

Author response:
# Re issue on citations and figures:

Thank you very much for all the good catches. We will fix them in the final version.

Reviewer comment (Presentation):
Figure 1 is not referenced in the text.

Author response:
# Re issue on citations and figures:

Thank you very much for all the good catches. We will fix them in the final version.

Reviewer comment (Presentation):
The paper uses "Figure" and "Fig." interchangeably.

Author response:
# Re issue on citations and figures:

Thank you very much for all the good catches. We will fix them in the final version.