Paper: LitCab: Lightweight Language Model Calibration over Short- and Long-form Responses (ICLR-2024)

Reviewer comment (Experiments):
There are still some efficient fine-tuning methods (e.g. LoRA) that can leverage additional training data, but the paper lacks results compared with such methods.

Author response:
In Sections 6.3 and 6.4 (highlighted in green), we introduced and compared LitCab with Label Smoothing, where the LM is fine-tuned using LoRA. Results in Table 1 indicate that it does not yield satisfactory calibration performance, possibly due to the small size of the training data leading to model overfitting. In contrast, LitCab can leverage small training sets for improved calibration, suggesting that LitCab achieves superior data efficiency compared to label smoothing.

Reviewer comment (Novelty):
The proposed method seems to have minimal differences compared to the original LM, and the experimental results also indicate that the original LM performs not very badly. Therefore, what are the main advantages of the proposed method? How does it improve LLMs qualitatively or quantitatively?

Author response:
In response to the observation of the baseline's relatively low ECE, which falls within the range of approximately 0.1 to 0.3, it is important to note that the typical ECE range is not confined to 0 to 1, as randomness in model confidence and correctness alone can yield an ECE of 0.25. However, our proposed method, LitCab, has demonstrated a significant and noteworthy impact on calibration. Table 1, highlighted in green, serves as compelling evidence, showcasing LitCab's effectiveness in achieving a remarkable 25% decrease in the average expected calibration error—from 0.118 to 0.089. This substantial reduction underscores the significant improvement LitCab brings to calibration, emphasizing its practical importance and contribution to the field.

Reviewer comment (Evaluation):
Why do you use different thresholds (e.g., cov@50, cov@90, cov@60) for different datasets?

Author response:
This is because language models demonstrate varied performance across different datasets. Cov@p, as defined, assesses the model's recall by identifying the largest percentage denoted as 'c,' for which the most confident c percent of predictions exhibit accuracy surpassing the threshold 'p.' It is crucial to note that a lower or higher value of 'p' can result in meaningless comparisons; for example, all models will achieve 100% when p is set to 0 and 0% when p is set to 1.0.

To illustrate, consider SciQ, which is relatively easy for LMs. If we set a lower value of 'p' in cov@p (e.g., 50), it results in all 100% for all models in cov@50. This poses a challenge in utilizing this metric for comparisons. Therefore, in alignment with the approach outlined by Tian et al. (2023) [3], we opt for a relatively high value of 'p' to facilitate more meaningful and informative comparisons.

It is important to emphasize that we do not solely rely on this metric (due to its aforementioned limitations), we also report acc@q, ECE, and Brier score, to include a comprehensive set of metrics for evaluating the calibration performance of models.

Reviewer comment (Experiments):
The CaT benchmark has two major shortcomings. Firstly, the motivation for creating CaT is to be able to evaluate calibration for long-form responses. However, CaT consists of five datasets with short-form responses and only one with long-form responses. There are a couple of datasets that could have been included, such as ELI5, WebGPT, ASQA, and QAMPARI (which is not passage-level but contains multiple claims).

Author response:
## 2. There is only 1 dataset with long-form responses. More datasets, such as ELI5, WebGPT, ASQA, and QAMPARI should be considered. 

Thank you for your suggestion. We followed your advice and conducted the experiment on QAMPAR because there are several issues when experimenting with other datasets. Regarding ELI5, collecting support documents for correctness evaluation is extremely time-consuming, and unfortunately, we won't be able to do that due to the limited rebuttal time. However, we will definitely consider experimenting with ELI5  in the future. As for WebGPT, there is currently no available and consistent source for checking the correctness of model responses, given the open-ended nature of the questions. While quoted sentences are provided, they are not sufficient for verifying the knowledge in various model responses. ASQA primarily inspects the model's capability on ambiguous questions, which includes the uncertainty of questions, and it's not within our scope. This leaves QAMPARI, where the correctness of model responses can be verified by comparing them with provided references. LitCab demonstrates its superiority in this task, based on the results below
|   Task  | Metric | Original LM | Label Smoothing | Temp. Scaling | LitCab    | LitCab w/ Temp.Scaling |
|:-------:|--------|-------------|-----------------|---------------|-----------|------------------------|
| QAMPARI | acc@50 | 0.193       | 0.180           | 0.188         | **0.207** | **0.207**              |
|         | cov@30 | **0.260**   | 0.156           | 0.239         | 0.257     | 0.257                  |
|         | ECE    | 0.290       | 0.213           | 0.303         | **0.096** | 0.104                  |
|         | Brier  | 0.228       | 0.208           | 0.273         | **0.142** | 0.157                  |

Results of LitCab and baselines on CaT

Furthermore, to evaluate LitCan on more tasks, we introduce a new task for long-form generation—WikiGen, where the LM is tasked to generate Wiki-style descriptions for entities, with an average generation length of 104, sourced from the fact verification dataset FEVER [1]. Compared with BioGen which is limited to biographies, WikiGen covers a wider spectrum of subjects beyond individuals, including events, concepts, and objects, resulting in more diverse topics. LitCab again obtains the best calibration performance, as evidenced by the results presented in the following table:

| Task    | Metric | Original LM | Label Smoothing | Temp. Scaling | LitCab    | LitCab w/ Temp.Scaling |
|---------|--------|-------------|-----------------|---------------|-----------|------------------------|
| WikiGen | acc@50 | 0.876       | 0.860           | 0.876         | **0.872** | **0.872**              |
|         | cov@80 | 0.745       | 0.733           | 0.745         | **0.756** | **0.756**              |
|         | ECE    | 0.045       | 0.075           | 0.049         | **0.037** | 0.065                  |
|         | Brier  | 0.172       | 0.187           | 0.173         | **0.171** | 0.174                  |

Results of LitCab and baselines on CaT

We also evaluate seven LLMs using both QAMPARI and WikiGen:

| Task    | Metric | GPT-J (6B) | LLaMA-7B  | LLaMA-13B | LLaMA-30B | Llama2-7B | Llama2-13B | Vicuna-13B |
|---------|--------|------------|-----------|-----------|-----------|-----------|------------|------------|
| QAMPARI | acc@50 | 0.115      | 0.229     | 0.293     | 0.267     | 0.193     | 0.253      | **0.305**  |
|         | cov@30 | 0.000      | 0.387     | **0.580** | 0.501     | 0.260     | 0.464      | 0.501      |
|         | ECE    | 0.372      | **0.257** | 0.291     | 0.268     | 0.290     | 0.323      | 0.268      |
|         | Brier  | 0.306      | **0.228** | 0.250     | 0.230     | **0.228** | 0.265      | 0.230      |

| Task    | Metric | GPT-J (6B) | LLaMA-7B | LLaMA-13B | LLaMA-30B | Llama2-7B | Llama2-13B | Vicuna-13B |
|---------|--------|------------|----------|-----------|-----------|-----------|------------|------------|
| WikiGen | acc@50 | 0.395      | 0.667    | 0.773     | 0.750     | 0.876     | **0.900**  | 0.822      |
|         | cov@80 | 0.001      | 0.220    | 0.450     | 0.354     | 0.745     | **0.914**  | 0.675      |
|         | ECE    | 0.102      | 0.102    | 0.124     | 0.165     | **0.045** | 0.048      | 0.168      |
|         | Brier  | 0.220      | 0.239    | 0.226     | 0.252     | 0.172     | **0.164**  | 0.227      |

Different calibration metrics of popular LMs computed over CaT


**Reference**

[1] James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a Large-scale Dataset for Fact Extraction and VERification. NAACL 2018.

Reviewer comment (Experiments):
I think it would help the paper if there were extrinsic evaluations to support Litcab's usage in a real-world task setup. Self-consistency is now commonly used because the original paper showed promising results on arithmetic and commonsense reasoning benchmarks. How would the use of Litcab fare on those datasets?

Author response:
## 4. How would the use of Litcab fare on arithmetic and commonsense reasoning benchmarks? 

In this paper, we focus on evaluating the calibration of LMs on outputs of varied lengths. Therefore, we design LitCab in the context of generation tasks and conduct experiments related to such tasks.

Following your suggestion, we further conduct experiments on the CommonsenseQA task under the multiple-choice setting. We compare the results of the original LM, self-consistency and LitCab. LitCab demonstrates superior calibration performance compared to both the original language model and self-consistency, as evidenced by its lower ECE and Brier score, as outlined below:

| Task                            | Metric | Original LM | Self-consistency | LitCab |
|---------------------------------|--------|-------------|------------------|--------|
| CommonsenseQA (multiple-choice) | acc@50 | 0.237       | 0.243            | 0.235  |
|                                 | cov@60 | 0.006       | 0.011            | 0.011  |
|                                 | ECE    | 0.039       | 0.196            | 0.034  |
|                                 | Brier  | 0.166       | 0.209            | 0.165  |

Reviewer comment (Experiments):
The paper mentions that Litcab has the "advantage of requiring a reduced number of data samples for training, leading to enhanced computational and data efficiency". There is no evidence shown in the paper that supports this statement.

Author response:
## 5. There is no evidence shown in the paper that supports "advantage of requiring a reduced number of data samples for training, leading to enhanced computational and data efficiency".

We apologize for our imprecise claim. We have rephrased the corresponding description in the revision: LitCab achieves superior computational efficiency with fewer tunable parameters, compared with the fine-tuning-based baselines.

Reviewer comment (Experiments):
It is not clear why adding a linear layer on the last hidden state is the best option. Other options include LoRA or prefix tunning, which also only requires a fraction of parameters to tune. More evaluation results should be presented to justify the model choice of LITCAB.

Author response:
## 2. It’s not clear why adding a linear layer on the last-layer hidden states is the best option.

This is because we aim to prioritize computational efficiency in the design of the lightweight calibration mechanism. Fine-tuning the LM through LoRA involves gradient back-propagation through all layers of LM. In contrast, training LitCab only necessitates running the LM's inference process to acquire the last-layer hidden states and logits. The single linear layer can then be trained based on these gathered features. 

In our pilot study, we experimented with both the second-to-last layer and the last layer's hidden states and found that using the last layer performed better. This is likely since because the last layer is closest to the logits and encodes more information about the future token. Exploring the optimal layer or combinations of the layers is a topic we intend to delve into in our future work.

In Section 6.3 and 6.4 (highlighted in green), we compare LitCab with Label Smoothing fine-tuning with LoRA. Results in Table 1 indicate that Label Smoothing does yield satisfactory calibration performance, possibly due to the small size of the training data leading to model overfitting. In contrast, LitCab can leverage small training sets for improved calibration, suggesting that LitCab achieves superior data efficiency compared to label smoothing.

Reviewer comment (Experiments):
How did you select/generate incorrect answers for the QA tasks?

Author response:
## 1. It’s not clear how to select the incorrect answers.

For phrase- and sentence-level tasks where references are available, we use the references directly as positive samples. We then repeatedly sample model generations, treating incorrect ones, evaluated by GPT4, as negative samples. For the paragraph-level tasks, we directly instruct each LM to generate biographies or descriptions for the given people’s names or entities. Subsequently, we categorize the correct claims as positive samples and the incorrect ones as negative samples. We have included this description in the final paragraph of section 4 in our revised version–highlighted in blue.

Reviewer comment (Theory):
The confidence of a sentence is the geometric mean of all its tokens. However, different tokens contribute differently to semantics. For instance, think about "The 2023 Oscar best actress is Michelle Yeoh" as the answer to "Who wins the 2023 Oscar best actress?". Only the last two tokens decide the correctness of the sentence. How did you handle the different importance of tokens?

Author response:
## 3. How to handle the different importance of tokens?

We answer your comment in two points. Firstly, this aggregation method has been studied and shown as effective in approximating the sentence-level probability by Kuhn et al., 2023 [1], and Si et al., 2023 [2], we aggregate token probabilities to calculate the sentence probability in our work.

Secondly, we argue that there is no straightforward definition of what constitutes an important token, as **the token importance heavily relies on the context**, such as the question, and it could easily change if any tokens related to the question are altered. In reference to your example, the model may generate an answer like "The 2022 Oscar best actress is Michelle Yeoh." If we only focus on the individual’s name, the incorrect generation of "2022" might be easily overlooked.

In light of these considerations, we chose to treat all tokens equally for the sake of simplicity and generalizability to text of varied lengths in this work. We are also open to and appreciate any suggestions from this perspective.

**Reference**

[1] Kuhn, Lorenz, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. ICLR 2023.

[2] Si, Chenglei, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan Boyd-Graber, and Lijuan Wang. Prompting gpt-3 to be reliable. ICLR 2023.