Paper: Enhancing Small Medical Learners with Privacy-preserving Contextual Prompting (ICLR-2024)

Reviewer comment (Novelty):
Privacy Protection as an Innovation Point. Regarding the extraction of key words for privacy protection, the paper uses a medical NER model proposed by Neumann et al in 2019. We suggest further improvement of this model, for example, considering age as a crucial keyword for certain diseases and extracting it as necessary to better enrich the innovative aspects of the paper.

Author response:
In our paper, we utilize NER to extract keywords automatically, and we mention in the introduction that keywords can also be extracted by other methods, e.g., a manually created dictionary based on domain expertise. In practice, we can also use rules to post-process the extracted keywords or  keep clinicians in the loop to add or delete medical terms, thereby improving the quality of the medical keywords.

We adopt NER here because it is the most convenient way to extract relatively high-quality medical information from raw data. This supports our main idea that by utilizing LLMs as a database, we can generate medical knowledge for auxiliary SLMs in medical decision-making while reducing privacy concerns. We will further clarify the reason to utilize NER in our revision to improve clarity.

Reviewer comment (Novelty):
The overall innovation of the methodology needs improvement, as the majority of the content relies on existing methods, such as the medical NER (Named Entity Recognition) model.

Author response:
We appreciate the opportunity to further elucidate the innovative aspects of our work. Our paper introduces a novel approach that simplifies the integration of medical knowledge into SLMs by leveraging LLMs as a knowledge base. Unlike prior research that often relies on complex training algorithms and the intricate process of constructing medical knowledge graphs[1-3], our method streamlines this integration, which is substantiated by the performance improvements showcased in Table 1 in the paper.

Moreover, our paper begins a crucial discussion on privacy issues when using LLMs in healthcare. This is a significant issue that hasn't been given enough focus in past studies [4-7]. By bringing this up, we aim to encourage more careful and ethical use of LLMs where patient privacy is a concern.

Reviewer comment (Reproducibility):
As this research utilized a named entity recognition model to extract keywords, it is possible that the NER model can extract privacy information such as patient names. Is there any filtering or postprocessing step to avoid that? In addition, it is not guaranteed that NER system will never extract sensitive patient information; for example, if the NER system incorrectly extracts a patient's address as a symptom, then the address may be leaked to LLM. Although it is very rare, it is still necessary to comment on this.

Author response:
**1. Concerns on privacy preserving in practical usage.**
The data we utilized in experiments have already undergone post-processing; however, even well-processed data cannot be directly shared with third parties in a real-hospital setting. Here, we adopt NER methods directly, solely for automation, to show that LLM can be utilized as a medical database to query knowledge under privacy-restricted scenarios. Practically, we can leverage de-identification models and rules to remove personal information and then extract medical keywords to query third-party LLMs for auxiliary knowledge generation. In this paper, we take an initial step to discuss the significant privacy-preserving situations in the medical domain and demonstrate the promising results of utilizing LLM to improve SLM while mitigating privacy concerns.

Reviewer comment (Experiments):
As the LLM already provides a preliminary decision, I am curious about the performance if we only feed the preliminary decision from LLM to SLM. It is worth knowing which part of the LLM-generated information improves the SLM most.

Author response:
**2.Question about what SLM learns for decision making.**

We feed preliminary decisions (PD) as context into SLM with backbone BioLinkBert-Base on three datasets. Three separate runs for each setting are conducted and the average results along with the standard deviation are reported. The results are shown in the Table below.


|            | MEDQA         | HeadQA        | MEDMCQA       |
|------------|---------------|---------------|---------------|
| SLM w PD   | 47.21 ± 0.31  | 53.64 ± 1.09  | 45.42 ± 0.17  |
| FTC        | 50.17 ± 0.42  | 61.35 ± 0.16  | 49.20 ± 0.45  |

FTC, which integrates extensive medical knowledge into the decision-making, shows a consistent and significant improvement over the SLM that only uses PD for context. These findings underscore the valuable contribution of leveraging comprehensive medical knowledge, provided by LLM, in enhancing the medical decision-making capabilities.

Reviewer comment (Novelty):
The related work section need to discuss more LLM application in the clinical area, especially the knowledge-enhanced LLM in clinical settings. For example, paper "Qualifying Chinese Medical Licensing Examination with Knowledge Enhanced Generative Pre-training Model." also utilized external knowledge for clinical questions.

Author response:
**3. Suggestion about related work in LLM application in the clinical domain.**

Thanks for the suggestion in the related work. We will add the suggested work into the related work section in the revision.

Reviewer comment (Reproducibility):
By adding the LLM-generated content, will the new concatenated input be too long and out of the word window in SLM? How do you deal with the long content problem?

Author response:
**4. Question abut address long medical context generated by LLM.**

We utilize the Fusion-in-Decoder [1] approach in our general domain experiments. This strategy is also effective for encoding long contexts. It works by dividing the input into smaller passages, encoding each one separately, and then combining the encoded representations for decision-making.

[1] Izacard et al. ( 2020) Leveraging passage retrieval with generative models for open domain question answering

Reviewer comment (Writing):
(Clarity) There is no specific definition of the private information. From Figure 1, it seems that privacy definition is restricted to private identifiable information (PII). The authors should clarify the scope of privacy risks. Importantly, the proposed method cannot address general private information leakage that is considered by strict formulations like differential privacy.

Author response:
**1. Concern  on privacy definition.**
Privacy is a multifaceted concept with varying definitions that depend on the context and use cases. Our definition of privacy differs from that of differential privacy, which offers a theoretical guarantee but may not always be practically applicable. In our setting, privacy issues do not arise during the training process but instead when calling third-party APIs with tokens to query medical knowledge. Specifically, we define the privacy budget as the ratio of the number of words from the original question that are provided to the third party for prompting the language model, as detailed in Section 5. Intuitively, the less information provided to the third party, the better the privacy condition we achieve; if no words are sent, there is no privacy risk. 

Furthermore, Figure 1 is for demonstration purposes only and is not used to define privacy in terms of personally identifiable information (PII). In practice, we can employ additional de-identification methods to remove PII from medical terms in the preprocessing or postprocessing stages. In this paper, our primary contribution is to demonstrate the effectiveness of using LLM as a database for generating medical knowledge to enhance SLMs' performance. We also initiate the discussion of LLM utilization under privacy-restricted conditions, which previous works [1-4] have overlooked. We will revise our paper to include a clear definition of privacy in our context, thereby improving its clarity.

Reviewer comment (Evaluation):
(Quality - Risks) The evaluation of privacy is not strict. It is possible that the keyword extraction includes private identifiable information (PII), for instance, names and dates as shown in Figure 1. There is no theoretical guarantee for privacy protection or empirical evaluation of the leakage rates of such PII.

Author response:
In practice, we can employ additional de-identification models and rules to in the preprocessing or postprocessing stages to avoid the PII information leakage. Additionally, we can involve a human review to double-check the data before it is transmitted to an API for practical applications.

The concept of 'privacy-preserving' is crucial because actual hospital environments prohibit sharing even anonymized data with external entities [1]. Our privacy budget is designed in accordance with this policy, determining the proportion of words from the initial query that prompting medical knowledge from the third party LLMs.


[1] Johnson et al. (2016). MIMIC-III Clinical Database.

Reviewer comment (Evaluation):
(Quality - Metric) The evaluation of privacy is not strict. The authors used the privacy budget for quantifying privacy risks: the ratio of the number of words provided to the LLM to the total words in the original question. However, I doubt if the metric can imply some privacy risks. There essentially lacks an intuitive explanation of the relationship between the privacy budget and privacy risks.

Author response:
**1. Concern  on privacy definition.**
Privacy is a multifaceted concept with varying definitions that depend on the context and use cases. Our definition of privacy differs from that of differential privacy, which offers a theoretical guarantee but may not always be practically applicable. In our setting, privacy issues do not arise during the training process but instead when calling third-party APIs with tokens to query medical knowledge. Specifically, we define the privacy budget as the ratio of the number of words from the original question that are provided to the third party for prompting the language model, as detailed in Section 5. Intuitively, the less information provided to the third party, the better the privacy condition we achieve; if no words are sent, there is no privacy risk. 

Furthermore, Figure 1 is for demonstration purposes only and is not used to define privacy in terms of personally identifiable information (PII). In practice, we can employ additional de-identification methods to remove PII from medical terms in the preprocessing or postprocessing stages. In this paper, our primary contribution is to demonstrate the effectiveness of using LLM as a database for generating medical knowledge to enhance SLMs' performance. We also initiate the discussion of LLM utilization under privacy-restricted conditions, which previous works [1-4] have overlooked. We will revise our paper to include a clear definition of privacy in our context, thereby improving its clarity.

Reviewer comment (Experiments):
(Motivation) As the authors said, SLM presents a large gap compared to LLMs and thus there is no clear motivation to use SLM for prediction. Although the authors mention that ChatGPT requires access to data, it is essentially ignored that open-source LLMs, for example, Llama, can be used. In the paper, there is no referred evidence for the large gap between open-source LLMs and ChatGPT on the concerned medical tasks. Thus, I strongly doubt if the motivation of the paper can hold.

Author response:
**2.  Concerns about motivation to utilize ChatGPT instead of open-source LLMs.**

We evaluate medical domain specific LLMs based on LLaMA [5] : AlpaCare [6], PMC_LLAMA [7], ChatDoctor [8], Medalpaca [9] and Baize-healthcare [10]  on the three QA datasets following [11]. The results are shown in the table below.

|               | MEDQA  | HeadQA | MEDMCQA |
| ------------- | ------ | ------ | ------- |
| FTC           | 53.85  | 63.17  | 52.09   |
| ChatGPT-privacy | 41.7  | 47.5   | 35.2    |
| AlpaCare      | 35.0   | 29.2   | 33.5    |
| PMC_LLAMA     | 34.2   | 28.1   | 28.6    |
| ChatDoctor    | 34.3   | 30.0   | 33.5    |
| MedaIpaca     | 38.4   | 30.3   | 31.3    |
| Baize-healthcare | 34.5 | 29.3   | 32.5    |


ChatGPT significantly outperforms domain-specific LLMs across three QA datasets under privacy-restricted settings, demonstrating that the medical knowledge within open-source LLMs still lags behind the GPT-series models. Additionally, our FTC, which further leverages the knowledge provided by ChatGPT, achieves better performance, indicating the effectiveness of our methods.


[1] Ho et al., (2022)  Large language models are reasoning teachers \
[2] Shridhar et al., (2022) Distilling multi-step reasoning capabilities of large language models into smaller models via semantic decompositions \
[3] Wang et al., (2022) Pinto: Faithful language reasoning using prompt-generated rationales \
[4] Li et al., (2022)  Explanations from large language models make small reasoners better \
[5] Touvron et al. (2023) LLaMA: Open and Efficient Foundation Language Models \
[6] Zhang et al.(2023) AlpaCare:Instruction-tuned Large Language Models for Medical Application \
[7] Wu et al. (2023)  PMC-LLaMA: Towards Building Open-source Language Models for Medicine \
[8] Li et al (2023) ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge \
[9] Han et al (2023) MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Data \
[10] Xu et al (2023) Baize: An Open-Source Chat Model with Parameter-Efficient Tuning on Self-Chat Data \
[11]  Gao et Al. (2021) A framework for few-shot language model evaluation

Reviewer comment (Experiments):
There is no clear motivation to see SLM for prediction. Although the authors mention that ChatGPT requires access to data, it is essentially ignored that open-source LLMs, for example, Llama, can be used. Is there any evidence for the large gap between open-source LLMs and ChatGPT on the concerned medical tasks?

Author response:
**2.  Concerns about motivation to utilize ChatGPT instead of open-source LLMs.**

We evaluate medical domain specific LLMs based on LLaMA [5] : AlpaCare [6], PMC_LLAMA [7], ChatDoctor [8], Medalpaca [9] and Baize-healthcare [10]  on the three QA datasets following [11]. The results are shown in the table below.

|               | MEDQA  | HeadQA | MEDMCQA |
| ------------- | ------ | ------ | ------- |
| FTC           | 53.85  | 63.17  | 52.09   |
| ChatGPT-privacy | 41.7  | 47.5   | 35.2    |
| AlpaCare      | 35.0   | 29.2   | 33.5    |
| PMC_LLAMA     | 34.2   | 28.1   | 28.6    |
| ChatDoctor    | 34.3   | 30.0   | 33.5    |
| MedaIpaca     | 38.4   | 30.3   | 31.3    |
| Baize-healthcare | 34.5 | 29.3   | 32.5    |


ChatGPT significantly outperforms domain-specific LLMs across three QA datasets under privacy-restricted settings, demonstrating that the medical knowledge within open-source LLMs still lags behind the GPT-series models. Additionally, our FTC, which further leverages the knowledge provided by ChatGPT, achieves better performance, indicating the effectiveness of our methods.


[1] Ho et al., (2022)  Large language models are reasoning teachers \
[2] Shridhar et al., (2022) Distilling multi-step reasoning capabilities of large language models into smaller models via semantic decompositions \
[3] Wang et al., (2022) Pinto: Faithful language reasoning using prompt-generated rationales \
[4] Li et al., (2022)  Explanations from large language models make small reasoners better \
[5] Touvron et al. (2023) LLaMA: Open and Efficient Foundation Language Models \
[6] Zhang et al.(2023) AlpaCare:Instruction-tuned Large Language Models for Medical Application \
[7] Wu et al. (2023)  PMC-LLaMA: Towards Building Open-source Language Models for Medicine \
[8] Li et al (2023) ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge \
[9] Han et al (2023) MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Data \
[10] Xu et al (2023) Baize: An Open-Source Chat Model with Parameter-Efficient Tuning on Self-Chat Data \
[11]  Gao et Al. (2021) A framework for few-shot language model evaluation

Reviewer comment (Novelty):
The contribution of this paper to the algorithm and the significance of the clinical problems it addresses seem not to be very high.

Author response:
The reviewer asserts that there seems to be room for improvement in our algorithm, but it is not clear to us from which perspective we should improve. In addition, we wish to highlight that the contributions of a paper can be evaluated from various perspectives, not limited to algorithms alone.

The primary contribution of our paper is the proposal of a simple yet effective method to utilize LLM as a medical knowledge database, which enhances the decision-making abilities of downstream SLMs. Unlike previous works in medical QA that utilize more complex algorithms and extensive engineering to integrate medical knowledge graphs or databases [1-3], our approach simplifies this by foregoing the complex engineering involved in building knowledge graphs and training algorithms such as GNNs. It has been empirically validated to outperform these methods, as demonstrated in Table 1 in the paper.

Furthermore, our work pioneers the discussion on privacy concerns within the utilization of LLMs in the medical field, a critical issue that has been overlooked in previous research. This innovation opens up new pathways for the deployment of LLMs in sensitive domains.

We also note that the reviewer has acknowledged our contributions to the medical domain with comprehensive experiments and our efforts to mitigate privacy concerns in the review of the strengths section. In this case, we kindly request the reviewer reevaluates our paper based on the clear contribution.