Paper: Calibrated on Average, but not Within Each Slice: Few-shot Calibration for All Slices of a Distribution (ICLR-2024)

Reviewer comment (Experiments):
The lack of prior works for comparison makes less clear the effectiveness of the recalibrator.

Author response:
**The lack of prior works for comparison makes less clear the effectiveness of the recalibrator.**  
We thank the reviewer for this suggestion. We will include more discussions and comparisons with prior works in the next version of the paper.

Reviewer comment (Reproducibility):
The reviewer would like some clarification with regard to the specifics of the recalibrator.

Author response:
**"Does the recalibrator learn just a temperature scaling term?"**  
No. From a high level, our recalibrator learns to predict the precision curve for a specific domain of questions (e.g., math questions, or history questions). We then use the precision curve to recalibrate the model’s raw confidence and choose when to abstain.  

**What does the learned recalibrator parameters look like?**  
We finetune a LLaMA-7B model to serve as the recalibrator even for larger LMs. So, the recalibrator model parameters are the post-finetuning weights of LLaMA-7B.

Reviewer comment (Experiments):
In particular, what does the learned recalibrator parameters look like?

Author response:
**What does the learned recalibrator parameters look like?**  
We finetune a LLaMA-7B model to serve as the recalibrator even for larger LMs. So, the recalibrator model parameters are the post-finetuning weights of LLaMA-7B.

Reviewer comment (Novelty):
Does the recalibrator learn just a temperature scaling term?

Author response:
**"Does the recalibrator learn just a temperature scaling term?"**  
No. From a high level, our recalibrator learns to predict the precision curve for a specific domain of questions (e.g., math questions, or history questions). We then use the precision curve to recalibrate the model’s raw confidence and choose when to abstain.

Reviewer comment (Evaluation):
The proposed method is simple to follow. What is the limitation of this method? It is important to see the boundaries of method use.

Author response:
Like many few-shot learning methods, our method can struggle when given target domains that are outside of the distribution seen during training. Our recalibrator is trained via meta-learning framing to generalize to different domains of MMLU questions (e.g., from math questions to history questions). Given this training data, our recalibrator could fail to generalize to more distant distributions (e.g., beyond academic question such as cooking questions).

Reviewer comment (Reproducibility):
While the first 2 sections are written well, I found section 3 lacking some details and precision. Could you have a more elaborate section 3 with the following: a. Provide technical details of what is meant by a slice and how they are constructed for training $f_\theta$. b. Provide an algorithm block listing all the steps involved in training (starting from how the data slices are constructed) to outputting the recalibrated scores. c. Could you give a technical definition of the precision curve, it is there but in loose words. How is this curve estimated for each slice? I do get it roughly from the paper but would be better if a precise mathematical expression/definition is provided. d. In 3.2 what is the loss computed on? It only has $\theta$ (parameters) but data is missing.

Author response:
**Rewrite Section 3 (Method section) to be more elaborate.**
We agree that the method section lacks clarity and will rewrite the section with greater technical detail about both the learning settings and our methods, following LrPN’s suggestions.

Reviewer comment (Theory):
It seems the method uses some set of precision curves $p_i$ (corresponding to some slices/domains) and tries to predict the precision curve for some given slice. So if those curves $p_i$ are not good, the method would not be able to correct them, since it is just using some unlabeled data points to identify the domain or the relevant $p_i$ to combine in some way.

Author response:
**So if those precision curves are not good, the method would not be able to correct them**
We wish to clarify a misunderstanding: the goal of our method is not to improve a model’s precision curve, but rather to predict it, and use the predicted curve to choose when to abstain: From a high-level, models with “bad” precision curves, meaning that they are low accuracy, should abstain more, and models with “good” precision curves should abstain less. Intuitively, we predict which regime we are in and abstain appropriately.

Reviewer comment (Experiments):
Could you provide some simulations (on synthetic data) demonstrating the precision curves for slices and how the proposed method actually does recalibration.

Author response:
**"Could you provide some simulations (on synthetic data) demonstrating the precision curves for slices and how the proposed method actually does recalibration."**
We will rewrite the method section to clarify this and specifically provide details of the recalibration procedure in the next version of the paper.

Reviewer comment (Novelty):
The observation that classifiers can be "calibrated on average but not within a slice" is not novel. It is well-understood in the community that calibrating for a given target could lead to miscalibration for targets at finer granularity. This occurs even within the same domain when looking at different classes, leading to unexpected behavior like that demonstrated in Sec 2. For example, see https://arxiv.org/abs/2107.08353. A standard solution is to recalibrate for the finer-granularity targets using a small supervised set adapted to that target, as the paper does.

Author response:
The novelty of our approach is that we meta-learn the recalibration in a few-shot manner. In the referenced standard solution, the small supervised set still requires a non-trivial number of examples with ground-truth labels in the target domain. In contrast, **our approach removes the need for supervised labels entirely** and can recalibrate with **as few as 5 unsupervised examples**. Specifically, each training example in our approach consists of 5 to 20 unlabeled examples and a set of labeled examples that are used to draw the precision curve.  Then, our approach can accurately predict the precision curve and consequently recalibrate predictions for a new domain, given only 5 to 20 unsupervised examples.

Reviewer comment (Evaluation):
Although the change in metric to precision is somewhat well-motivated, it introduces ambiguity since the improved performance could result from the method being tailored to that metric. Because the paper is written as a calibration paper rather than a paper on improving precision@k, standard metrics like accuracy and classwise-ECE (in addition to conf-ECE) should be reported to allow fair comparison.

Author response:
We thank the reviewer for this suggestion, we will incorporate NLL, Brier score, and accuracy in the future version of the paper.  
Since language models typically output only one answer, users mainly care about calibration over the highest probability answer, i.e., top-label calibration. Hence, we focus on improving top-label calibration rather than classwise calibration, such as classwise-ECE.

Reviewer comment (Experiments):
Benchmarking against previous methods is severely lacking. The only calibration method compared against is temperature scaling (TS), which is far from state-of-the-art. Many calibration methods exist, and the authors should conduct their own literature survey beyond Guo et al. TS is known to perform poorly once moving beyond conf-ECE.

Author response:
We acknowledge that the submission misses important related work, which we will include in the next version of the paper.

Reviewer comment (Experiments):
The final method used is unclear in terms of whether it is supervised or not. Based on Sec 3.2, it appears that access to ground-truth labels is required to compute the true precision curve. However, in Sec 4.3 under "Empirical," the paper states that access to labeled queries gives an unfair advantage. The TS fewshot baseline also uses labels.

Author response:
- **What supervision do we use in our method?**  
During training, we leverage both labeled and unlabeled examples: we learn a precision curve from a set of labeled examples, and then learn to predict it from a small number of unlabeled examples. This enables us to predict the precision curve in a new domain at inference time from only a small number of unlabeled examples, and no labeled examples.

Reviewer comment (Experiments):
It is unclear how TS is trained. Is a separate TS temperature learned for every slice, or are all categories merged to learn one temperature? If it is the latter, the underperformance is unsurprising. Regardless, TS is not a strong baseline.

Author response:
- **How is temperature scaling trained?**  
Our temperature scaling baseline TS (fewshot) learns a separate TS temperature for every slice. TS (all domains) learns a temperature with all domains merged.

Reviewer comment (Writing):
The goal of Section 5.1 is not understandable. The meaning of "We first experiment with measuring the success rate of selecting a confidence threshold that achieves a given target precision on the slice’s ground-truth precision curve" should be paraphrased for clarity.

Author response:
- **What is the goal of section 5.1?**  
Our high-level goal is to produce language models that can choose to abstain when asked about information they don’t know, hence avoiding “hallucination” or misinformation. We achieve this in Section 5.1 by finding a threshold of the model’s raw confidence such that the model answers the question if the predicted raw confidence exceeds the threshold and abstains otherwise. We choose the confidence threshold that achieves a certain precision target, while maintaining as high recall as possible.

Reviewer comment (Evaluation):
Other forms of calibration error should be considered and reported, such as classwise-ECE, NLL, etc.

Author response:
We thank the reviewer for this suggestion, we will incorporate NLL, Brier score, and accuracy in the future version of the paper.  
Since language models typically output only one answer, users mainly care about calibration over the highest probability answer, i.e., top-label calibration. Hence, we focus on improving top-label calibration rather than classwise calibration, such as classwise-ECE.