Paper: Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineering (ICLR-2024)

Reviewer comment (Novelty):
"Survival of the Most Influential Prompts: Efficient Black-Box Prompt Search via Clustering and Pruning" has been accepted by EMNLP-2023. Please cite it correctly.

Author response:
We thank the reviewer for pointing them out. We have corrected them in the updated manuscript.

Reviewer comment (Presentation):
Why is the related work in section 5? Introducing it in section 2 will make readers understand the background better.

Author response:
We appreciate the reviewer’s comments on the structure of the presentation. Section 2 serves as a unified overview of the most relevant existing calibration methods while addressing two important research questions behind their design principles. Therefore, we found that having the related work with the current analysis section together may be chunky and could potentially dilute the attention of readers. Hence, we decided to move the related work to section 5 and present the main body of our work, section 3, to readers as early as possible for a better flow of content.

Reviewer comment (Presentation):
Unify the usage of abbreviated words. Tab. -> Table.

Author response:
We thank the reviewer for pointing them out. We have corrected them in the updated manuscript.

Reviewer comment (Experiments):
When introducing BCL, the author takes some examples to make readers understand the $\gamma$. I suggest authors provide more experiments on how different $\gamma$s affect the results.

Author response:
We thank the reviewer for suggesting more experiments. We have now conducted additional experiments to illustrate the impact of the strength term on BCL and have included more plots in Figure 11, Appendix D, in the updated manuscript. To better understand the $\gamma$ term, We kindly refer the reviewer to Table 1 and Figure 2, which visualize the decision boundary for BC. The strength term is controlling the strength of the shift (BC) operation for the decision boundary, such that $p_0 = p_1 + \gamma \alpha$. When $\gamma =1$, the estimation of $\alpha$ by zero-shot BC serves as a strong but not optimal prior for shifting the decision boundary. Therefore, by leveraging labelled data to learn the strength term, we can further refine the decision boundary by learning $\gamma$.

Reviewer comment (Evaluation):
As shown in Figure 3, the improvement from BC to BCL seems marginal.

Author response:
We acknowledge that having labelled data brings marginal improvement to BC for certain tasks, which has also been reflected in Figure 5, Page 6, where the gain of performance between the BC (purple line) and BCL (green line) is observable but not significant. However, this empirical evidence exactly indicates the strength point of BC, such that BC at $\gamma=1$ already serves as a good prior for estimating the contextual bias from the target (test) statistics only. It further consolidates the effectiveness of BC even in the zero-shot setups. By incorporating an additional strength hyperparameter, we are enabled to estimate the source prior to source statistics in labelled data. This BCL design further refines the robustness of BC under covariate shift and shows a much more enhanced calibration performance in a range of tasks (SST-2, MRPC, QQP, CB) than BC.

Reviewer comment (Experiments):
My main concern goes to the $\textit{strength}$ parameter in BCL; the current results show $\gamma = 1$ seems to give strong performance (though not optimal) across tasks on CB and SST-2. Does this generalize to other tasks as well? It would be good to provide the optimal $\gamma$ for each task; maybe include it in Table 2? Besides, would it be more reasonable to sample \gamma in [0, 5]?

Author response:
We thank the reviewer for expressing interest in the strength parameter in BCL and the generalization of performance in terms of the strength term. We recall that when strength is at 1, Equation 5 becomes the zero-shot BC, and BC shows state-of-the-art performance across many tasks over ICL and calibration baselines in Table 2. To better understand the $\gamma$ term, We kindly refer the reviewer to Figure 2 that visualizes the decision boundary for BC. The strength term is controlling the strength of the shift (BC) operation for the decision boundary, such that $p_0 = p_1 + \gamma \alpha$. When $\gamma =1$, the estimation of contextual prior, $\alpha$, by zero-shot BC serves as a strong but not always optimal prior for shifting the decision boundary, which is due to the dependency on test-time statistics. Therefore, by leveraging labelled data to learn the strength term as a source prior, we can further refine the decision boundary while mitigating calibration errors.

We thank the reviewer for suggesting to sample $\gamma$ in [0, 5]. By having the $\gamma$ in the negative range, we can explore what the decision boundary looks like after shifting it in the opposite direction. We then retain both directions for a sufficient range to sample the optimal source prior. To provide a better intuition of $\gamma$ across other tasks, we have included more plots in Figure 11, Appendix E in the updated manuscript, where optimal $\gamma$ for the test set can be read from each task plot. BC ($\gamma=1$) demonstrates strong performance across tasks, where further slight gains can be achieved by BCL when we incorporate the source prior via labelled data. We welcome the reviewer for further discussion.

Reviewer comment (Experiments):
There are some statements that are not well presented or supported. In Sec. 4.3, the authors claimed BC retains the performance even when using emoji pairs as verbalizers. Is there an example or result related to this specific experiment?

Author response:
We thank the reviewers for expressing their interest in individual test performance for each verbalizer choice in Figure 8. We kindly refer the reviewer to Table 10, which lists the detailed verbalizer choices and descriptions for each emoji pair. We agree that it will be interesting to interpret individual experimental results corresponding to each design choice. Therefore, we have updated our manuscript and included each test sample result before and after calibration in a new Table 7, Page 17. Empirically, we observe that BC can recover the performance of LLM for emoji pairs with opposed but context-relevant semantics. We hope this will address your remaining concerns.

Reviewer comment (Writing):
In e.q.(3), how to obtain the contextual prior $\hat{\mathbf{p}}$?

Author response:
In Equation 3, the contextual prior $\hat{\mathbf{p}}$ is obtained via Equation 2, which is equivalent by obtaining the estimated mean of log probability for each class. In the updated manuscript, we have updated the notation in Equation 2 to $\mathbf{p}(y|C)_{j}$ better to emphasize the link between Equations 2 & 3.

Reviewer comment (Experiments):
How would BC work on the task other than classification?

Author response:
We thank the reviewer for discussing this appealing research direction that is worthy of being explored by the research community. We suspect that the contextual bias may also exist in short-form generation tasks. Motivated by [1], one possible solution for generative tasks is to calibrate over the logits of the first output token since the following tokens are likely deterministic based on the first token. We will endeavor to explore generative calibration. We thank the reviewer for this insight, and we include this idea in the future work section in Appendix in the updated manuscript. 

[1] Guo, C., Pleiss, G., Sun, Y. and Weinberger, K.Q. On calibration of modern neural networks. ICML 2017.

Reviewer comment (Experiments):
It seems the calibration perform is not stable across different model architectures in 1-shot setting on some tasks. E.g., MNLI, 75.12/60.02/81.34 for PaLM-2-S/PaLM-2-M/PaLM-2-L. I understand that PaLM-2-S/L are based on 5 runs and PaLM-2-M result is from a single run. Could you help to understand, what causes the high perform variance, I may miss something here.

Author response:
We acknowledge that different sizes of models preserve distinctive robustness features for different tasks, and it’s occasionally possible that a larger model underperforms a smaller model for specific tasks (this has also been reflected in, for example, the PaLM 2 technical report [2], Table 2) due to multiple possible factors such as pretraining strategies, type and size of input data used in training and instruction finetuning, and/or slight differences in templates, but in general, PaLM 2-M still shows a stronger 1-shot ICL performance than PaLM 2-S across the set of evaluation tasks, where BC also shows consistent performance improvement across all model scales. 

[2] Anil, R., Dai, A.M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z. and Chu, E. Palm 2 technical report. arXiv preprint arXiv:2305.10403.

Reviewer comment (Writing):
The term calibration can be confusing, maybe using bias is better. At least should add a footnote to the Introduction. Another widely used notion of calibration is from the perspective of uncertainty, explainability, and reliability.

Author response:
We appreciate the reviewer’s suggestion for clarifying the notion of calibration. In our updated manuscript, we have included a clarification on the definition of calibration as a footnote in the introduction. We agree with the reviewer that the term “calibration” is usually aligned with the trustworthiness, reliability, and interpretability of neural networks [2]. In the context of our work, we follow the notion of calibration defined in [3] as a technique for mitigating the label shift and contextual bias in LLMs prediction.

[2] Guo, C., Pleiss, G., Sun, Y. and Weinberger, K.Q. On calibration of modern neural networks. ICML 2017.

[3] Zhao, Z., Wallace, E., Feng, S., Klein, D. and Singh, S. Calibrate before use: Improving few-shot performance of language models. ICML 2021.

Reviewer comment (Reproducibility):
Please include model sizes and details in the paper, as the performance of ICL and prompting is quite irreproducible and context-dependent.

Author response:
We thank the reviewer for being aware of the nature of the context-dependency of prompting. We acknowledge the sensitivity of LLM prediction to the context and the prompt template. Therefore, we included detailed templates in Tables 8, 9, 10, & 11 for reproducibility and conducted all main experiments with 5 random seeds to reveal the variance for each study. Regarding the details of the model size, we included the PaLM 2-S, M, L variants, and we kindly refer the reviewer to [4] that lists the details for each variant, including experiments on PaLM 2 with comparable model sizes as PaLM, and PaLM 2-L shows competitive performance to GPT-4 [5]. We then follow the notion of model variants in [4] and hope this has addressed the reviewer’s concern.

[4] Anil, R., Dai, A.M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z. and Chu, E. Palm 2 technical report. arXiv preprint arXiv:2305.10403.

[5] OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.

Reviewer comment (Presentation):
Please adjust the margins of subfigures in fig. 2, it's overlapped.

Author response:
We appreciate the reviewer's suggestions on the presentation. We have adjusted the margins and provided improved visualization in the updated manuscript.

Reviewer comment (Presentation):
The liteature review of Test time tuning is a bit unnecessary as ICL itself is not a tuning method.

Author response:
In Appendix A, we included a discussion of test-time adaptation (TTA), where we explicitly focus on the previous literature using test-time batch statistics for BN without tuning. This line of research has shed light on the theoretical foundation for batch calibration, where contextual bias can also be framed as a form of covariate label shift [6, 7]. Batch calibration takes inspiration from test-time batch adaptation in the novel context of LLMs, and we would be grateful if researchers in the TTA community could be inspired to deliver advanced research ideas on top of batch calibration.

[6] Nado, Z., Padhy, S., Sculley, D., D'Amour, A., Lakshminarayanan, B. and Snoek, J., 2020. Evaluating prediction-time batch normalization for robustness under covariate shift. arXiv preprint arXiv:2006.10963.

[7] Sun, Q., Murphy, K., Ebrahimi, S. and D'Amour, A., 2022. Beyond invariance: Test-time label-shift adaptation for distributions with "spurious" correlations. arXiv preprint arXiv:2211.15646.

Reviewer comment (Evaluation):
In fig.5, why does a strength greater than 1 consistently decrease accuracy?

Author response:
We would like to recall that when strength is at 1, Equation 5 becomes the zero-shot BC. We then refer the reviewer to Figure 2, which visualizes the decision boundary for BC. BC is equivalent to shifting the original ICL decision boundary $p_0 = p_1 + \alpha$ by the amount of $\alpha$, where $\alpha$ is the contextual prior estimated by marginalizing the output scores. Following that, the strength term that can be learned from labelled data is controlling the strength of this shift operation, such that $p_0 = p_1 + \gamma \alpha$. As visualized in Figure 2, an optimal linear decision boundary that separates the data points should always exist. When $\gamma =1$, the estimation of $\alpha$ by zero-shot BC serves as a strong but not optimal prior for shifting the decision boundary. Therefore, by leveraging labelled data to learn the strength term, we can further refine the decision boundary by learning $\gamma$. Consequently, having an inappropriate strength term can shift the decision boundary to become an undesirable separation plane, which results in a degradation of performance.

Reviewer comment (Experiments):
How significant is the bias/miscalibration of the ICL performance? that say is the problem still relevant for models with greater scales? Should consider 65B llama2, GPT-3.5, GPT-4 etc as the method does not require tuning.

Author response:
LLMs have been shown to be very sensitive to the context and biased prediction across all scales. This phenomenon was first revealed by [2] on GPT-2 and three sizes of GPT-3. [1, 8] further observe the same problem of biased prediction in GPT-J and Bloom family of models. Since PaLM 2-L [4] has achieved results that are competitive with GPT-4, we follow this line of research work and observe the same pattern in three variants of PaLM 2 models, where BC brings a significant margin of performance improvement across all scales. We also extended the current study to a new modality, calibrating vision-language models, and we will endeavor to include other models as part of future work. We have incorporated this discussion in the revised version of the paper. 

[8] Fei, Y., Hou, Y., Chen, Z. and Bosselut, A. Mitigating Label Biases for In-context Learning. ACL 2023.

Reviewer comment (Theory):
The proposed method is not thoroughly justified.

Author response:
We’d like to refer the reviewer to Section 2, Pages 2-5, where we have conducted a comprehensive analysis of existing calibration methods through their decision boundaries. We identified the current deficiency of existing methods with individual case studies, and the design of each aspect of BC has been strongly justified with empirical evidence in Section 2.3 and motivated in Section 3. In particular, we addressed two important research questions for calibration methods with an interpretation of decision boundaries and model biases, as illustrated in Figures 2, 3, & 10, delivering novel guidelines for opting for a linear decision boundary and content-based calibration. **We’d be grateful if the reviewer could point out what precisely, in their opinion, is lacking justification, and we are happy to provide further clarifications.**

Reviewer comment (Writing):
The writings could be improved in terms of the analysis of existing methods and the proposed method.

Author response:
We thank the reviewer for their suggestion. We have revisited the analysis section and cleaned up any language problems. We’d be grateful if the reviewer could let us know of any further concerns, and we are happy to provide further clarifications.

Reviewer comment (Theory):
The authors argued that linear decision boundaries produced by calibration methods can be more robust and generalizable across tasks. This argument is not well supported theoretically and empirically.

Author response:
We stated in Section 2.3, as the alternative design choice to a linear decision boundary, the non-linear decision boundary by GMM tends to be susceptible to overfitting and vulnerable to instability, which, empirically, was further evidenced by the extensive results in Table 2 and discussed in Section 4.2 in the paper, where the non-linear decision boundary by PC exhibits the worst stability with the largest standard deviation even across the number of shots. 

Theoretically, we have derived the non-linear boundary equation for GMM in Section 2.3, which was known to be unstable with outlier data points, and the robustness of GMM has been theoretically analyzed in previous work [1, 2]. Therefore, we opt for a linear decision boundary for BC due to its robustness, and we provide consistent and generalizable empirical evidence across tasks and modalities. We also conducted extensive robustness analysis to further validate the empirical robustness for the linear decision boundary design in Section 4.3, Page 8. We are open to any further discussion. 

[1] Diakonikolas, I., Kane, D. M., & Stewart, A, 2017. Statistical query lower bounds for robust estimation of high-dimensional gaussians and gaussian mixtures. FOCS 2017. IEEE.

[2] Diakonikolas, I., Hopkins, S.B., Kane, D. and Karmalkar, S., 2020. Robustly learning any clusterable mixture of gaussians. arXiv preprint arXiv:2005.06417.

Reviewer comment (Theory):
The advantage of Batch Calibration over existing methods is not thoroughly justified. Could you provide more intuitive descriptions and theoretical analysis for it?

Author response:
We thank the reviewer for expressing their interest in a more intuitive description of BC. In contrast with existing methods that rely on content-free tokens and fail on multi-sentence classification tasks, BC serves as a content-based method using test-time statistics, thereby being more robust against distribution shifts and more generalizable across tasks. Intuitively,  in Section 2.3, we discussed that the calibration can be framed as an unsupervised decision boundary learning problem. Figures 2 & 10 then provide a visualization of the decision boundary for BC and existing methods. To constitute a better decision boundary, we recall from Section 3 that the most critical component for calibration is to accurately estimate the contextual bias term $p(y|C)$ that shifts the original ICL decision boundary. This serves as a prior that can be approximately estimated by marginalizing the output likelihood over all samples within the batch in Equation 2. This forms the mathematical foundation of BC, and we welcome the reviewer for further discussion.

Reviewer comment (Theory):
The derivations of Table 1 could be provided in more details in the main contents or supplementary. Currently, it is not easy to understand.

Author response:
We appreciate the reviewer for pointing this out, and we have made updates in the caption of Table 1 to provide a supplementary explanation for each cell, including a reference to Section 2.3 that presents detailed derivations of decision boundaries for each method.