Paper: BayesPrompt: Prompting Large-Scale Pre-Trained Language Models on Few-shot Inference via Debiased Domain Abstraction (ICLR-2024)

Reviewer comment (Experiments):
The PLMs utilized in the evaluation are not clear. Using various PLMs may be better to show the generality of the proposed method.

Author response:
**A:** Thanks for the review. The PLM used in the evaluation is **RoBERTa-large**, which is mentioned in Appendix C.2, Page 19, of the revised manuscript. Theoretically, the proposed method can generally apply to all problems that involve “**sub-knowledge domains**”, such as text classification, entity recognition, entity disambiguation, etc.

To further demonstrate the generality of the proposed BayesPrompt, we conducted text classification tests on a specific automotive industry dataset using another pre-trained language model, i.e., **BERT**, with a parameter count of 110 million. The obtained classification accuracy was 74.0%. We also employed TextRCNN to train the classification model with a parameter count of 15 million. The resulting classification accuracy reached 95.3%, significantly outperforming the pre-trained language model BERT, which is due to the over-multitudinous conceptual knowledge learned by BERT. However, when using BayesPrompt, it achieves the best classification accuracy of 97.4%. Note that the method does not require training (fine-tuning) the network of BERT. Therefore, the proposed method can be generalized into various tasks with various PLMs and obtain performance improvements to baselines.

Reviewer comment (Theory):
Why can GMM approximate the target domain? What are its benefits than a learnable generator (VAE or GAN)?

Author response:
**A:** Thanks for the review. GMM is a probabilistic model, which assumes that all data points are generated from a mixture of Gaussian distributions with unknown parameters. Based on the fact that the Gaussian mixture model can theoretically fit any probability density distribution and the fact that due to the limited sample size in the target downstream domain, the factual distribution of the target downstream domain **does NOT** fit the conventional Gaussian distribution well, directly adopting the Gaussian distribution-based approach, e.g., conventional VAE, may learn biased distribution of the target downstream domain, such that we use the Gaussian mixture distribution to approximate the debiased factual distribution of the downstream domain within the knowledge space of PLMs.. In contrast, VAE and GAN, as learnable generators, can also approximate the target domain, but the inherent defects of VAE and GAN degenerate their performance of approximating the distribution of the target domain with insufficient samples, and we elaborate on the reasons as follows.

VAE conventionally holds the prior distribution as the **typical Gaussian distribution** and then learns the latent representation of data by maximizing the KL divergence of latent variables of the posterior distribution and the prior distribution. The inherent mechanism presents that conventional VAE can only perform limited distortion on the prior distribution, i.e., the typical Gaussian distribution, and cannot well fit the complex distribution, e.g., the distribution of the target domain with **limited sample size**.

GAN learns to generate data by training two neural networks in an adversarial manner **without any priori**. One network is the generator, responsible for generating new data, while the other is the discriminator, tasked with distinguishing between real and generated data. Due to the lack of priori, GAN requires **sufficient data** to approximate the distribution of the target domain. However, in few-shot scenarios, the limited samples are far from completely supporting the sufficient training of GAN.

Compared to learnable generators, using GMM to approximate the target domain provides the following advantages:

1.**Strong interpretability**. GMM is a probabilistic model, and its parameters directly correspond to the mean, covariance, and mixture coefficients of the data. This makes the results of GMM more easily interpretable and understandable.

2.**High computational and sample efficiency**. Compared to VAE and GAN, the training and inference of GMM are generally more efficient, and GMM requires relatively fewer samples for optimization. GMM is a classical statistical model based on maximum likelihood estimation, while GAN and VAE typically involve more complex optimization processes.

3.**Stable training**. Compared to GAN, GMM tends to be more stable during the training process. GAN training may be affected by issues such as mode collapse, while GMM is less prone to these problems.

Reviewer comment (Reproducibility):
Is it required to train a specific GMM for each input sentence (X, Y)?

Author response:
**A:** Thanks for the review. In our approach, we do not train a specific GMM for each input sentence (X, Y). Instead, we train a GMM on the downstream task dataset and use the obtained Gaussian mixture distribution to approximate the debiased factual distribution of the target domain. Subsequently, prompts with domain discriminative information are generated by sampling from the approximated distribution. During each prediction, the prompt containing domain discriminative information needs to be added to the input of the downstream task to provide de-ambiguous guidance for PLMs. As a result, it increases the time complexity. In experiments, several optimization strategies can be considered to reduce the time complexity, such as sampling or subsampling datasets, employing feature selection or dimensionality reduction techniques to decrease the input feature dimensions, or implementing a training strategy with a warm-up. 

Additionally, the time complexity of BayesPrompt is only slightly higher than the baseline. For instance, in the 1-shot setting of the SemEval dataset, the training time cost of each epoch at the main baseline is 16 seconds, while the training time cost of each epoch at BayesPrompt is 27 seconds. Nevertheless, the main baseline achieves an F1 score of 28.6%, whereas BayesPrompt achieves an F1 score of 35.1%. Compared to the improvement of BayesPrompt on the F1 score, the time complexity brought by Bayesprompt is acceptable.

Reviewer comment (Presentation):
The paper is essentially a shortened version of a much longer manuscript, where the authors are constantly cutting the content short and referring the reader to different sections of the appendix (appendix is referred to 11 times throughout the paper!). So, the main body of the paper is not really self-contained and heavily relies on the appendix. By the same token, the main algorithm of the paper had to be pushed to the Appendix, which could be a natural choice in the main text to clarify the end-to-end procedure.

Author response:
**A:** Thank you for your review and suggestions. Due to the page limitation, we have tried to organize our original manuscript, but there are still some critical parts of our paper that need to be improved, such that we have revised the paper to make the main body more self-contained and then submitted the revised manuscript for now.

Specifically, in the revised version, we have relocated the main algorithm from the Appendix to the main body of the paper to enhance overall clarity, which is demonstrated in Section 4, Page 6, of the revised manuscript. Additionally, a wealth of the referred contents in Appendix are the extended descriptions and extended experiments of the main paper, which are unnecessary for understanding our works, such that we have removed certain references of the Appendix in the main content of our paper to make the manuscript more understandable, e.g., “**Appendix C.3** provides further empirical evidence.” in Introduction, Page 1, of the original manuscript, “Refer to **Appendix B** for the procedure of BayesPrompt” in Section 4, Page 6, of the original manuscript, “The statistical details are provided in **Appendix D.1**.” and “Refer to **Appendix D.2** for more implementation details.” and “and please refer to **Appendix C.2** for details” in Section 6, Page 8, of the original manuscript, and “Refer to **Appendix C.5** for further analysis of the empirical results.” in Section 6, Page 9, of the original manuscript. Concretely, there are only a few necessary references of the Appendix that are reserved in the main paper, and the main paper of the revised manuscript is well self-contained for now.

Reviewer comment (Evaluation):
The impact of the proposed approach is rather marginal when compared to the closest competitors (say RetrievalRE), especially on standard RE performance in Table 3, while at the same it comes at the cost of extra training complexity. Any reason behind this?

Author response:
**A:** Thanks for the review. In this paper, we choose KnowPrompt as our primary baseline. RetrievalRE is the follow-up work of KnowPrompt, such that RetrievalRE achieves superior performance. However, whether in the few-shot or standard setting, on average, BayesPrompt consistently outperforms both KnowPrompt and RetrievalRE, confirming the effectiveness of this approach. Specifically, in few-shot scenarios, the proposed BayesPrompt widely improves the benchmark methods, including the primary baseline (KnowPrompt) and RetrievalRE, e.g., BayesPrompt beats KnowPrompt by 3.24% and beats RetrievalRE by 1.29% on average. In standard scenarios, the improvements of benchmark methods to their baseline are consistently limited, e.g., RetrievalRE beats KnowPrompt by 0.2%, such that the improvement of BayesPrompt to RetrievalRE is relatively acceptable. Note that, the primary baseline of BayesPrompt is KnowPrompt, and BayesPrompt beats KnowPrompt by 0.4%, which is relatively significant. We also perform the significance test, i.e., t-test, with the primary baseline, and observe that the P values are consistently lower than 0.05, e.g., 0.045 on the SemEval dataset, indicating that the improvement of BayesPrompt is significant. Concretely, for the reason behind such a deviation existing in the few-shot scenarios and standard scenarios, we further derive a conclusive analysis as follows:

As we discussed in the Abstract and Introduction sections on Page 1 and Page 2, of the submitted manuscript, the limited and discrete semantic information contained in the training samples from downstream domains can barely support the conventional trainable prompts to acquire sufficient supervision, such that the guidance of the generated prompts is trivial to PLMs. Especially, such a challenge further exacerbates the performance of PLMs in few-shot scenarios. With this motivation, the proposed BayesPrompt aims to learn a prompt that contains relatively sufficient discriminative knowledge for the target downstream domain, which is achieved by proposing the debiased domain abstraction and then generating the prompt. Thus, the empirical observation, i.e., the performance improvement derived by introducing BayesPrompt on few-shot scenarios is more significant than that on standard scenarios, jointly proves the motivation and the effectiveness of the proposed BayesPrompt.

From the perspective of time complexity, the time complexity introduced by BayesPrompt primarily stems from the necessity of adding prompts containing domain discriminative information for downstream tasks during each prediction, providing de-ambiguous guidance for PLMs. In experiments, several optimization strategies can be considered to reduce the time complexity, such as sampling or subsampling datasets, employing feature selection or dimensionality reduction techniques to decrease the input feature dimensions, or implementing a training strategy with a warm-up. Additionally, the time complexity of BayesPrompt is only slightly higher than the baseline. For instance, in the 1-shot setting of the SemEval dataset, the training time cost of each epoch for the main baseline is 16 seconds, while the training time cost of each epoch for BayesPrompt is 27 seconds. Nevertheless, the main baseline achieves an F1 score of 28.6%, whereas BayesPrompt achieves an F1 score of 35.1%. Compared to the improvement of BayesPrompt on the F1 score, the time complexity brought by Bayesprompt is acceptable.

Reviewer comment (Evaluation):
As can be seen from Tables 1 and 3, the proposed BayesPrompt presents a completely different improvement. Can the authors provide a detailed explanation?

Author response:
**A:** Thanks for the review. By comparing Table 2 and Table 3, it can be observed that the performance improvement of BayesPrompt in few-shot scenarios widely surpasses its performance improvement in the standard scenario, i.e., with the full dataset. This observation precisely validates our motivation and the effectiveness of the method. 

As we discussed in the Abstract and Introduction sections on Page 1 and Page 2, of the submitted manuscript, the limited and discrete semantic information contained in the training samples from downstream domains can barely support the conventional trainable prompts to acquire sufficient supervision, such that the guidance of the generated prompts is trivial to PLMs. Especially, such a challenge further exacerbates the performance of PLMs in **few-shot** scenarios. With this motivation, the proposed BayesPrompt aims to learn a prompt that contains relatively sufficient discriminative knowledge for the target downstream domain, which is achieved by proposing the debiased domain abstraction and then generating the prompt. Thus, the empirical observation, i.e., the performance improvement derived by introducing BayesPrompt on few-shot scenarios is more significant than that on standard scenarios, jointly proves the motivation and the effectiveness of the proposed BayesPrompt.

Specifically, in few-shot scenarios, the limited amount of data may hinder PLMs from fully learning the distribution and features of the downstream task data, leading to relatively poorer performance. However, when utilizing prompts obtained from BayesPrompt, which contain domain discriminative information, to guide PLMs in locating knowledge domains relevant to the downstream domain, the deficiencies caused by the limited data are noticeably mitigated.

In the standard scenario, where the amount of downstream task data is relatively sufficient, PLMs can better grasp the features and distribution of the downstream task data by learning from a more comprehensive dataset. Consequently, the performance improvement brought by BayesPrompt may not be as pronounced in this context.

Concretely, as our intuition and motivation in the Abstract and Introduction, the proposed BayesPrompt mainly aims to improve the inference performance of PLMs in few-shot scenarios. While the results in standard scenarios further demonstrate the generalizability of our exploration and the effectiveness of BayesPrompt.

Reviewer comment (Evaluation):
Please provide more discussion about the ablation results at Figure 4(c).

Author response:
**A:** Thanks for the review. The role of the type prompt word is to inject knowledge related to entity information into the prompt. Figure 4(c) illustrates the impact of the type prompt word on task performance in the full dataset. It can be seen that the performance improvement brought by the type prompt word in the full dataset is significantly weaker than its performance enhancement in the few-shot scenario (Figure 4(b)). We attribute this difference to the fact that, in the full dataset, the relatively abundant downstream task data provides a rich knowledge background, resulting in a substantial reduction in the deviation between the knowledge domain located by PLMs and the real downstream knowledge domain. This, in turn, weakens the de-biasing effect of the type prompt word on the few-shot dataset. Therefore, the performance improvement brought by the type prompt word is not significant in the full dataset.

Reviewer comment (Experiments):
BayesPrompt's training complexity is higher than its baseline, is there any potential for optimization?

Author response:
**A:** Thanks for the review. The time complexity introduced by BayesPrompt primarily arises from the necessity of adding prompts containing domain discriminative information for downstream tasks during each prediction to provide de-ambiguous guidance for PLMs. In experiments, several optimization strategies can be considered to reduce the time complexity, such as sampling or subsampling datasets, employing feature selection or dimensionality reduction techniques to decrease the input feature dimensions, or implementing a training strategy with a warm-up. We will further explore the optimization strategy for BayesPrompt in the following works.

Additionally, the time complexity of BayesPrompt is only slightly higher than the baseline. For instance, in the 1-shot setting of the SemEval dataset, the training time cost of each epoch at the main baseline is 16 seconds, while the training time cost of each epoch at BayesPrompt is 27 seconds. Nevertheless, the main baseline achieves an F1 score of 28.6%, whereas BayesPrompt achieves an F1 score of 35.1%. Compared to the improvement of BayesPrompt on the F1 score, the time complexity brought by Bayesprompt is acceptable.

Reviewer comment (Presentation):
It is unclear what the author refers to for "unabridged Domain" and "partial domain" in Figure 2.

Author response:
**Q1: What does the author refer to for “unabridged domain” and “partial domain”, in Figure 2?**  

**A:** “**Unabridged**” means “**complete**” and “**partial**” means “**incomplete**”. In Figure 2, the “**unabridged domain**” represents the complete knowledge domain corresponding to the knowledge required for the downstream task, approximately covering all the semantic knowledge needed in the downstream domain. “**Partial domain**” represents the partial knowledge domain corresponding to the knowledge required for the downstream task, which only covers a part of the knowledge required for the downstream domain. Thanks for this careful review, and to improve the understandability of our work, we have changed “**unabridged**” into “**complete**”, e.g., “**unabridged domain**” is replaced with “**complete domain**”, and “**unabridged knowledge**” is replaced with “**complete knowledge**”, and we further change “**abridged**” and “**partial**” into “**incomplete**”, e.g., “**abridged domain**” and “**partial domain**” are replaced with “**incomplete domain**”, and “**abridged knowledge**” and “**partial knowledge**” are replaced with “**incomplete knowledge**”, in the submitted revised manuscript.

Reviewer comment (Theory):
It is unclear why the proposed approach can better approximate the downstream task distribution — is it by injecting label-related information? What is the bias referred to here? Is there any process to reduce the bias? The author may refer to the bias as "irrelevant pretraining knowledge" that is confounding for the downstream tasks, but it is not very clear why introducing "Gaussian mixture distribution" can help solve the problem — is it for sampling and easily injecting label-related knowledge?

Author response:
**Q4: What is the bias referred here? Is there any produces to reduce the bias? The author may refer the bias as “irrelevant pretraining knowledge” that is confounding for the downstream tasks ?**  

**A:** Thanks for the review. The “**bias**” refers to that, without given appropriate prompts, PLMs exhibit a bias in locating the knowledge domain relevant to downstream tasks, deviating from the actual knowledge domain corresponding to the downstream tasks. This bias introduces irrelevant pretraining knowledge, that is confounding for the inference of PLMs on the downstream tasks. As the discussion in A to Q3 and Q5, we attempt to leverage the Gaussian mixture distribution as the known distributions to approximate the debiased factual distributions of the target downstream domains, thereby generating the prompts to mitigate the bias of PLMs.

Reviewer comment (Experiments):
By injecting label-dependent knowledge, the PLM may learn a PLM distribution that is useful for the downstream task, which makes sense, but is it unfair, as BayesPrompt already uses label information while other methods do not?

Author response:
**Q6: By injecting label dependent knowledge, the PLM may learn a PLM distribution that is useful for the downstream task, which makes sense. but is it unfair, as BayesPrompt already uses label information but other methods don't?**  

**A:** Thanks for the review. The benchmark prompt learning methods, e.g., PTR, KnowPrompt, and RetrievalRE, consistently follow the same benchmark, that is, using the available labeled training set of the target downstream task, which presents that our approach follows the benchmark experimental setting and avoids the unfair comparisons. Specifically, all the methods, compared with BayesPrompt, use the labeled training set of the target downstream dataset to tune the prompt generation network. Moreover, PTR applies logic rules to encode prior knowledge about tasks and classes into prompt tuning, which requires the label information. KnowPrompt injects latent knowledge contained in labels into prompt construction and synergistically optimizes their representation with structured constraints. RetrievalRE is the follow-up work of KnowPrompt, and it constructs an open-book datastore for retrieval regarding prompt-based instance representations and corresponding relation labels as memorized key-value pairs. Additionally, BayesPrompt and the associated benchmark methods, e.g., PTR, KnowPrompt, and RetrievalRE, **do NOT** require extra data besides the available information on the target downstream dataset, which is detailed in Table 3, Page 9, of the submitted revised manuscript. Concretely, the proposed BayesPrompt follows the benchmark experimental setting to use the available labeled training set of downstream tasks and avoids unfair comparisons.

Reviewer comment (Reproducibility):
In the Method section, it is unclear how label prompt word lp and type prompt word tp fit in equation (6). The author is also requested to clarify the training procedure.

Author response:
**Q8: how does label prompt word lp and type prompt word tp fit in eq(6)? Can the author also bring some clarify to training?**  

**A:** Thanks for the review. A typical prompt consists of a template and a set of label words. The label prompt word $l_p$ and type prompt word $t_p$ are used to inject relational semantic knowledge and entity type knowledge into the prompt to predict the “**[Mask]**”, corresponding to computing $\mathcal{P}\left ( \left [MASK  \right ]=M\left (y \right ) \mid T\left ( x \right )   \right )$ in Equation (6).  

During the training process, the label prompt word $l_p$ is utilized to initialize the relation label corresponding to the “**[Mask]**”, while the type prompt word $t_p$ is employed to inject entity knowledge into the template $T\left ( x \right )$. By computing $\mathcal{P}\left ( \left [MASK  \right ]=M\left (y \right ) \mid T\left ( x \right )   \right )$ in Equation (6), we can predict the relation label corresponding to the “**[Mask]**”. To fully associate the initialized label prompt word $l_p$ and type prompt word $t_p$ with the surrounding context, we conduct further optimization of their representations using a loss function. The loss function is computed as the cross-entropy between $y$ and $\mathcal{P}\left ( y\mid x \right )$, as shown in Equation (6). For the details of training, we provide a pseudo-code in Algorithm 1, Page 6, of the submitted revised manuscript, and the implementation details of BayesPrompt are provided in Appendix C, Page 18, of the submitted revised manuscript.

Reviewer comment (Experiments):
Why does the approach focus on relation extraction tasks (used in method section)? How about other tasks? Is this method currently specific for relation extraction tasks?

Author response:
**Q9: Why does the approach focus on relation extraction tasks (used in method section)? how about other tasks? Is this method currently specific for relation extraction tasks?**  

**A:** Thanks for the review. Relation extraction aims to extract structured knowledge from unstructured text and plays a critical role in information extraction and knowledge base construction. Therefore, we choose the relation extraction task to verify the effectiveness of the proposed method, but this does not mean that our method is limited to relation extraction tasks. Theoretically, our method applies to all problems that involve “**sub-knowledge domains**”, such as text classification, entity recognition, entity disambiguation, etc.  

Thus, we further conducted **text classification** experiments on a specific automotive industry dataset using a pre-trained language model, BERT, with a parameter count of 110 million. The obtained classification accuracy was 74.0%. We also employed TextRCNN to train the classification model with a parameter count of 15 million. The resulting classification accuracy reached 95.3%, significantly outperforming the pre-trained language model BERT, which is due to the over-multitudinous conceptual knowledge learned by BERT. However, when using BayesPrompt, it achieves the best classification accuracy of 97.4%. Note that our method does not require training (fine-tuning) the network of BERT. Therefore, our method can be generalized into various tasks and obtain performance improvements to baselines. The corresponding experiments can be found in Table 5, Page 16, Appendix B.3, of the submitted manuscript.