Paper: Investigating Uncertainty Calibration of Aligned Language Models under the Multiple-Choice Setting (ICLR-2024)

Reviewer comment (Theory):
The framing of sections 4.2 and 4.3 seems wrong. While it's true that we'd want "aligned" models to be as or, better, more calibrated than the raw language models, that doesn't necessarily mean we'd want the answer uncertainty to remain the same. The text claims that aligned LMs should change the format uncertainty and preserve the answer uncertainty of pre-trained LMs, but this is questionable because the point of "aligning" models in this setting is to improve accuracy. It is difficult to imagine improving accuracy while also preserving the approximate answer uncertainties of the original model, as that could result in unnecessarily brittle models with tiny margins of error. Models can have the same calibration error but vastly different answer uncertainties, meaning high accuracy and good calibration could still be achieved while tuning answer uncertainties.

Author response:
**A:** Thanks for pointing this out! We agree with your statement that changing the answer uncertainty does not necessarily cause overconfidence and miscalibration. Indeed, keeping the answer uncertainty of the pre-trained LMs is not desired during alignment since updating it could also boost the accuracy. Hence, the claim (such as "Ideally, ...") in our initial submission is inaccurate and inappropriate, and we removed them. 

Here, we want to make some clarifications about the somewhat convoluted phenomenon when aligning pre-trained LMs. The key point here is changing the answer uncertainty _in some human preference dialog data_ during alignment would _cause overconfidence in MCQs like MMLU_, which is usually beyond the scope of and not present in the dataset of alignment. This problem is similar to the in-distribution (ID) and out-of-distribution (OOD) settings, where optimizing the answer uncertainty on the ID dataset (in this case, the human preference data) would cause overconfident on the OOD dataset (in this case, the MCQs that is not presented in the alignment data). Tuning the answer uncertainty would result in good accuracy and calibration in the ID dataset, but the accuracy of the OOD dataset is basically unchanged, and the confidence is affected by the alignment process. This is especially true for larger LMs (e.g., Llama-2-70B in our experiments), where the alignment process does not improve the accuracy of MMLU but only increases confidence.

In Section 4.2 of our revised version, we demonstrate that the common alignment processes update the LMs' answer \& format uncertainty simultaneously and cause overconfidence in MCQs, which is undesired since the alignment process does not involve much domain knowledge required for performing the MCQs in our evaluation. In Section 4.3, we add new synthetic schemes using pair-wise preference data with DPO and show that we could increase the accuracy of the synthetic task (illustrated in Table 1 in Appendix C.4) while preserving the calibration and changing format preference on MMLU.

Reviewer comment (Theory):
Essentially the same questionable claim about answer uncertainty is repeated in Proposition 2, which states that common alignment processes like SFT and LPF alter both answer uncertainty and format uncertainty, with the changed answer uncertainty causing miscalibration. This reasoning is problematic because enforcing preservation of answer uncertainty may not be desirable or necessary, and could conflict with accuracy improvements.

Author response:
**A:** Thanks for pointing this out! We agree with your statement that changing the answer uncertainty does not necessarily cause overconfidence and miscalibration. Indeed, keeping the answer uncertainty of the pre-trained LMs is not desired during alignment since updating it could also boost the accuracy. Hence, the claim (such as "Ideally, ...") in our initial submission is inaccurate and inappropriate, and we removed them. 

Here, we want to make some clarifications about the somewhat convoluted phenomenon when aligning pre-trained LMs. The key point here is changing the answer uncertainty _in some human preference dialog data_ during alignment would _cause overconfidence in MCQs like MMLU_, which is usually beyond the scope of and not present in the dataset of alignment. This problem is similar to the in-distribution (ID) and out-of-distribution (OOD) settings, where optimizing the answer uncertainty on the ID dataset (in this case, the human preference data) would cause overconfident on the OOD dataset (in this case, the MCQs that is not presented in the alignment data). Tuning the answer uncertainty would result in good accuracy and calibration in the ID dataset, but the accuracy of the OOD dataset is basically unchanged, and the confidence is affected by the alignment process. This is especially true for larger LMs (e.g., Llama-2-70B in our experiments), where the alignment process does not improve the accuracy of MMLU but only increases confidence.

In Section 4.2 of our revised version, we demonstrate that the common alignment processes update the LMs' answer \& format uncertainty simultaneously and cause overconfidence in MCQs, which is undesired since the alignment process does not involve much domain knowledge required for performing the MCQs in our evaluation. In Section 4.3, we add new synthetic schemes using pair-wise preference data with DPO and show that we could increase the accuracy of the synthetic task (illustrated in Table 1 in Appendix C.4) while preserving the calibration and changing format preference on MMLU.

Reviewer comment (Evaluation):
The issue is sidestepped in Figure 6 because accuracies remain roughly constant, which makes it unclear why the intervention is even being performed if the goal is to improve these metrics.

Author response:
**A:** 
Thank you for asking this. We add a reference in Section 3.1 to the detailed evaluation protocol in Appendix B.1. The accuracy, ECE, and confidence are calculated based on the target generation position for a choice letter (no matter which choice format, i.e., `A` or `(A)`, it is). So, these three metrics are computed based on **one** position that is supposed to be a choice letter.

As for the probability of the format identifier, for an autoregressive LM, we could get all conditional probabilities $p(x_i|x_{<i})$ for all $x_i, i \in [2, L+1]$ within a single run of inference on the input, where $L$ is the length of the input text. Hence, we could get the probability of the format identifier without letting the LMs generate it by themselves.

Reviewer comment (Writing):
The notation in equation 6 isn't great and would be better if it incorporated the T.

Author response:
**A**: Thank you for your carefulness! We have fixed it.

Reviewer comment (Evaluation):
How does the proposed temperature scaling technique (Eq. 6) affect MC accuracy? Figure 7 only shows calibration error, but it is possible this intervention could undo some of the accuracy gains from the alignment process.

Author response:
**A**: Temperature scaling preserves the accuracy since the argmax of the scaled distribution is still the same. The cost of TS is that it requires an additional calibration set for optimizing the temperature, which may not be available for all cases.

Reviewer comment (Evaluation):
Regarding Figure 6, isn’t it expected that finetuning on a binary classification task would wreck calibration on MMLU (a 4-class task) by de-weighting choices C and D? If so, the rise in ECE could be due to this change rather than solely increased confidence.

Author response:
**A**: That's a great critical point. In response to that, we changed the dataset in Section 4.3 to a synthetic four-class MCQ dataset, where the LM is asked to pick the word that appeared in the question from four given options.

Reviewer comment (Experiments):
Regarding Figure 6, it would be interesting to see results for Llama-1+Format, ICL.

Author response:
**A**: We present the results here (we use the setup in our revised version, which is conducted with Llama-1 7B):



| Model     | Acc. | ECE  | Avg.Conf. | Prob. of `(` |
| ----------- | ---- | ---- | --------- | -------------------------- |
| Llama-1 ZSL           |   31.74   |  3.71    |     31.16      |          36.34                  |
| Llama-1 ICL            |   30.82   |  7.69    |    34.89       |         99.33                   |
| Llama-1 + Format, ZSL         |  32.52    |    2.10  |      32.92     |         85.11                   |
| Llama-1 + Format, ICL | 31.03 | 7.23 |   36.16        |         99.91                   |


The result shows that the performance of pre-trained Llama-1 and Llama-1 + Format has similar performance under ICL.

Reviewer comment (Writing):
The paper contains many unclear sentences and claims.

Author response:
**A:** Sorry for the confusion. We want to highlight the fact that pre-trained LMs need ICL to be well-calibrated compared to ZSL. We revise the sentence to  `Pre-trained LMs achieve good calibration by ICL`.

Reviewer comment (Writing):
In Section 3.3, the sentence `Pre-trained LMs are calibrated in-context learners` is confusing, and simply saying `Pre-trained LMs calibrate well in ICL setting` would be enough.

Author response:
**A:** We change the sentence to `The probability of format identifier indicates the format preference of LMs in MCQs.` and we rewrite the corresponding paragraph. The main point we want to convey is the difference between the choice format `A` and `(A)`, where the choice letter in the former case is responsible for representing both the answer and the format of MCQs, whereas in the latter case, the format preference is separate from the choice letters.

Reviewer comment (Writing):
In Section 3.3, the sentence `Format identifier decomposes format preference in MCQs` and the whole paragraph are incomprehensible even after multiple readings, and the paragraph needs to be rewritten for better clarity.

Author response:
**A:** We change the sentence to `The probability of format identifier indicates the format preference of LMs in MCQs.` and we rewrite the corresponding paragraph. The main point we want to convey is the difference between the choice format `A` and `(A)`, where the choice letter in the former case is responsible for representing both the answer and the format of MCQs, whereas in the latter case, the format preference is separate from the choice letters.

Reviewer comment (Theory):
Proposition 1 is unclear, as a proposition is expected to be derived from rigorous mathematical proof, whereas the current content is only an observation from experiments. It is also unclear why the first half of the sentence in Proposition 1 implies the second sentence. The claim that \( p_{\theta}^{PT}(y_c |x, F_{MC}) \approx p_{\theta}^{PT}(y_c |x, F_{MC}, S_{K}) \) is not justified, and it is unclear why the demonstrations do not affect the answer uncertainty in LMs.

Author response:
**A:** Thank you for pointing it out. We admit that using propositions here is inappropriate, and we reframe this part to improve clarity. The revised version replaces the proposition with an assumption for pre-trained LMs that comes from the empirical observations that they have well-calibrated answer uncertainty $p_{\theta}^{\rm{PT}}(y_c|x, F_{\rm{MC}})$. And we intuitively and briefly discuss how ICL estimates $p_{\theta}^{\rm{PT}}(y_c|x, F_{\rm{MC}})$ with $p_{\theta}^{\rm{PT}}(y_c|x, S_K)$.

To achieve this, ICL should (1) provide the signal for the format $F_{MC}$, i.e., $p_{\theta}^{\rm{PT}}(F_{\rm{MC}}|x, S_K) = 1$; (2) have minimal impact on the answer uncertainty of the test example, i.e., $p_{\theta}^{\rm{PT}}(y_c|x, F_{\rm{MC}}) \approx p_{\theta}^{\rm{PT}}(y_c|x, S_K, F_{\rm{MC}})$. A rigorous demonstration of (2) would require introducing a number of premises [1]. Here, we just utilize the intuition that when the LMs comprehend the MCQ task, their decision to the test example will not be influenced by the ICL examples since they are independent of each other. 

[1] [An Explanation of In-context Learning as Implicit Bayesian Inference](https://arxiv.org/abs/2111.02080) (Xie, Sang Michael, et al., ICLR 2022).

Reviewer comment (Theory):
Proposition 2 has the same problem as Proposition 1 in terms of lack of clarity. It is unclear why changing the answer uncertainty would cause miscalibration. SFT affects both confidence and accuracy, so altering answer uncertainty does not necessarily mean the model will be miscalibrated. Furthermore, the claim `optimizing LMs with maximum likelihood in SFT encourages two uncertainties to be eliminated toward the SFT’s data distribution, and the impact on answer uncertainty will lead to aligned LMs’ overconfidence` is unclear—both in the phrase `eliminated toward the SFT's data distribution` and in the reasoning why SFT would cause overconfidence in answer uncertainty. The text seems to suggest that SFT fine-tuned models will be miscalibrated without sufficient justification.

Author response:
**A:** We admit that Proposition 2 could not provide a solid claim for explaining the source of the miscalibration of aligned LMs, and we remove that. The key point here is changing the answer uncertainty in some human preference dialog data during alignment would cause overconfidence in MCQs like MMLU, which is usually beyond the scope of and not present in the dataset of alignment. This is analogous to the problem of in-distribution (ID) and out-of-distribution (OOD), where optimizing the answer uncertainty on the ID dataset (in this case, the alignment dataset) will cause overconfidence on the OOD dataset (in this case, the general MCQs). Hence, for the ID dataset, the changed answer uncertainty and increased accuracy may result in a calibrated model, while for the OOD dataset, the accuracy remains unchanged, but the answer uncertainty is altered toward the direction of overconfidence. That's basically what happens when aligning the pre-trained LM and could not be demonstrated in the way in the original Proposition 2. 

In the revised version, we demonstrate this phenomenon based on the existing empirical results and support the claim that `altered answer uncertainty during alignment will cause overconfidence in MCQs` with our synthetic alignment schemes in Section 4.3.

Reviewer comment (Reproducibility):
Many experimental settings are underspecified and hard to understand.

Author response:
**A:** Thank you for asking this. In the initial submission, we put the details about the selection of ICL samples and prompt sensitivity analysis in the Appendix without referring to them in the main text. In our revised version, we add the reference to these details in Section 3.1. For a short overview, we perform all evaluations with three sets of randomly selected in-context examples, and the conclusion is still valid. We also find that larger pre-trained LMs are more robust to the changes of ICL examples.

Reviewer comment (Experiments):
It is unspecified how ICL samples are selected, and it is unclear whether the paper's observations hold when varying ICL demonstrations.

Author response:
**A:** Thank you for asking this. In the initial submission, we put the details about the selection of ICL samples and prompt sensitivity analysis in the Appendix without referring to them in the main text. In our revised version, we add the reference to these details in Section 3.1. For a short overview, we perform all evaluations with three sets of randomly selected in-context examples, and the conclusion is still valid. We also find that larger pre-trained LMs are more robust to the changes of ICL examples.

Reviewer comment (Experiments):
The SFT and LPF datasets used in Figure 5 are not specified, making it unclear if the SFT dataset has a fixed or preferred format.

Author response:
**A**: We add a reference to the detailed information of these LMs in Appendix B.2 in our revised manuscript. The LMs in Fig.5 adopt common preference data similar to the aligned LMs in Section 3 (e.g., dialogs) for alignment. We use the official checkpoints to evaluate them on MCQs.

Reviewer comment (Presentation):
The figures are very difficult to read when printed on A4 paper.

Author response:
**A:** Sorry for the inconvenience. We improved the readability of all figures in our manuscript.

Reviewer comment (Presentation):
The dashed lines in Figure 4 look very similar to non-dashed lines, making them hard to distinguish.

Author response:
**A:** Sorry for the inconvenience. We improved the readability of all figures in our manuscript.

Reviewer comment (Presentation):
The words (or symbols) above the bars in Figure 7 are completely unreadable.

Author response:
**A:** Sorry for the inconvenience. We improved the readability of all figures in our manuscript.

Reviewer comment (Experiments):
The method proposed in Section 4.3, which uses only the format for SFT, is highly limited to the MCQ setting and is not straightforward to adapt to other settings.

Author response:
**A:** Thank you for asking this!
We would like to clarify that the alignment schemes in Section 4.3 do not serve as novel practical solutions. Instead, by optimizing the answer and format uncertainty separately, their role is to help understand the source of miscalibration during alignment and to support that `altered answer uncertainty during alignment will cause overconfidence in MCQs`. However, our initial submission does not convey this point well, and we rewrite this section thoroughly.

Besides, in our revised version, we add three pairwise preference learning schemes with DPO to make it closer to realistic settings and show that format learning with DPO could add new knowledge to the LLM (illustrated in Table 1 in Appendix C.4 due to space limitation) while preserving the calibration and changing format preference on MMLU.

Reviewer comment (Theory):
The method in Section 4.3 does not allow adding new knowledge to the LLM during SFT, making it impractical.

Author response:
**A:** Thank you for asking this!
We would like to clarify that the alignment schemes in Section 4.3 do not serve as novel practical solutions. Instead, by optimizing the answer and format uncertainty separately, their role is to help understand the source of miscalibration during alignment and to support that `altered answer uncertainty during alignment will cause overconfidence in MCQs`. However, our initial submission does not convey this point well, and we rewrite this section thoroughly.

Besides, in our revised version, we add three pairwise preference learning schemes with DPO to make it closer to realistic settings and show that format learning with DPO could add new knowledge to the LLM (illustrated in Table 1 in Appendix C.4 due to space limitation) while preserving the calibration and changing format preference on MMLU.

Reviewer comment (Experiments):
Have you considered instructing the LLMs to answer in parentheses format to eliminate format uncertainty instead of using ICL?

Author response:
**A:** That's a great question! We tried this by adding the sentence like `Begin your response by selecting one option from [(A),(B),(C),(D)]` in the instruction, and we have the following ZSL results on MMLU (we mark the setting with such sentence as + Hint):

For choice format A:

| Model |  Sum of Prob. for [A, B, C, D]   |
| ----- | ---- |
| Llama-1 30B  | 75.54 |
| Llama-1 30B + Hint  | 70.85 | 
| Vicuna-v1.3 33B  | 54.30 |
| Vicuna-v1.3 33B + Hint| 90.14  | 


For choice format (A):
| Model |  Prob. of `(`   |
| ----- | ---- |
| Llama-1 30B  | 34.81 |
| Llama-1 30B + Hint  | 42.01 | 
| Vicuna-v1.3 33B  | 46.50 |
| Vicuna-v1.3 33B + Hint  | 57.88 | 

While the hints could partly eliminate the format uncertainty of LMs, it is not as effective as in-context demonstrations for the LMs we evaluate.

Reviewer comment (Experiments):
The paper is mainly focusing on MCQ and while they argue that many tasks can be framed as MCQ tasks, many other tasks can not. Limiting the scope of the study to MCQ makes it unclear whether these findings will generalize to free-form generation, which I believe is more aligned with how LLMs are deployed in practice.

Author response:
**A:** Indeed, our main concern is the calibration of LMs in MCQs, and we do not expect our findings to be naturally extended to free-form generation. We focus on understanding and mitigating the degradation of calibration from pre-trained to aligned LMs with logit-based UQ that originate from the evaluation of MCQs, where the LM's confidence is well-defined [1, 2]. For free-form generation, the pressing issue is how to quantify its confidence rationally, and logit-based UQ may not be a good solution. Hence, our work is somewhat orthogonal to the work related to UQ for free-form generation.

Reviewer comment (Novelty):
I have to say that most of the findings in the paper are expected and some of them have already been established in the literature. For example, we know that alignment hurts calibration and we should expect ICL to mitigate issues with answer format uncertainty. I do not understand the motivation behind formalizing some of the paper findings as *propositions*. I do not think that adds anything to the paper. I would expect the paper to briefly discuss these expected findings and then move on to attacking the issue.

Author response:
**A:** While some of our work's findings and results may align with expectations and intuitions, we believe that we still need to present them rigorously and comprehensively. Besides, the focus of our study is markedly different from existing work. Here is a brief overview of previous work related to logit-based UQ with MCQs in the context of pre-trained and aligned LMs:

* [1]: Demonstrate the calibrated performance of the pre-trained LMs in MCQs and show that larger LMs with ICL are well-calibrated, with ECE and the reliability diagram as metrics. Show that the aligned LMs suffer from miscalibration and propose the TS with a constant temperature method.

* [2]: Show that aligning GPT-4 with PPO degrades the ECE on a subset of MMLU with a single pair of reliability diagrams.

* [3]: Present a comprehensive benchmark for all available LMs and compare different schemes (e.g., using the likelihood of sentence or choice letters) for logit-based calibration evaluation.

It follows that even with the simple MCQ setting, there are still questions to be answered, such as **how _aligning LMs with human-preference data_ leads them to be _overconfident in MCQs_**, which is not quite straightforward to answer. 

Our study addresses this question by specifically pairing pre-trained LMs with their aligned counterparts for comparative evaluation. We report additional metrics previously unexplored in depth by other research, employ a framework for decomposing answer and format uncertainty in LMs, study different alignment stages, and design synthetic align schemes to investigate the different roles of the alignment process, which helps enhance the understanding of the source of miscalibration in aligned LMs.

Besides, our work shows that ICL could only mitigate the miscalibration of pre-trained LMs caused by _underconfidence_ while having a marginal effect on aligned LMs' miscalibration due to _overconfidence_, which suggests the alignment process impairs the answer uncertainty of the pre-trained LMs. Again, we believe that this kind of difference between pre-trained and aligned LMs is not obvious and needs to be carefully discussed across model sizes, alignment algorithms, and tasks.


[1] [Language Models (Mostly) Know What They Know](https://arxiv.org/abs/2207.05221) (Kadavath, Saurav, et al., 2023).

[2] [GPT-4 Technical Report](https://arxiv.org/abs/2303.08774) (OpenAI, 2023).

[3] [Holistic Evaluation of Language Models](https://arxiv.org/abs/2211.09110) (Liang, Percy, et al., TMLR 2023).

Reviewer comment (Experiments):
The calibration-preserving training approach in section 4.3 is not novel; It is expected that training on the format only will likely reduce the overconfidence issue. In addition, there are two issues with the experimental setup there. First, the QQP dataset is not a pairwise preference ranking dataset, and therefore it's not entirely similar to the pairwise datasets used in realistic RLHF settings. Second, the experiments were run with LoRA, again limiting the generalizability of these results to the true RLHF setting.

Author response:
**A:** 
Thank you for your feedback! We admit that the propositions are unnecessary and may cause confusion, and we accept your suggestion to remove Proposition 4.2 and re-organize the Proposition 4.1 part to an empirically established assumption with some intuitive discussions. We still preserve the formalization of the decomposition for a concrete demonstration of "answer" and "format" and enrich the experiment part with new results in Section 4, where we think careful discussion of the findings is necessary.

Reviewer comment (Experiments):
The proposed temperature scaling approach is hard to apply in practice if we do not have access to the standard LM logits (the authors also point to this limitation) and is twice as computationally demanding as other approaches, while not being significantly better than simply using a constant temperature.

Author response:
**A:**
We would like to clarify that the alignment schemes do not serve as novel practical solutions. Instead, the role of them is to understand the source of miscalibration during alignment. However, our initial submission does not convey this point well, and we made substantial revisions to Section 4.3:

(1) We modify this section's title to clarify its purpose.

(2) We change the dataset to a synthetic MCQ task (see next answer).

(3) We add new alignment schemes with DPO to better reflect the LPF case and support the idea that _the altered answer uncertainty during alignment causes overconfidence in aligned LMs_.

Reviewer comment (Experiments):
In section 5.3, you write that you use the last prediction pair of each prompt with your proposed TS method. Does that mean your approach effectively uses less data than the baselines? Have you tried using all pairs with your approach?

Author response:
**A:** Thank you for pointing this out! We address the two points below:

(1) The goal of the experiments in Section 4.3 is to understand the role of different components (i.e., optimizing answer and format uncertainty) of the alignment process through a simple and controllable setting. We hope a dataset that (a) could be formulated to MCQs to ease the decomposition of answer & format uncertainty; (b) should be simple enough that it does not require some complex background knowledge; \(c\) the pre-trained LMs could not handle it perfectly. Hence, the QQP, which is a simple paraphrase detection task, meets the above principles.

Nevertheless, as pointed out by other reviewers, there is a gap between the binary-class QQP at training and, equivalently, four-class MMLU at evaluation so QQP is not a proper choice. Therefore, we choose to use a synthetic four-class MCQ dataset that meets the above principles. Besides, although we are not using a real RLHF dataset, we add three new alignment schemes based on preference learning using DPO, which is closer to the real RLHF setting compared to previous SFT schemes.

(2) As mentioned in the paper, aligning a large LM with a single and small dataset will cause overfitting and hurt the general performance of other tasks, such as MMLU. Meanwhile, we do not expect these simple alignment schemes to be directly generalized to realistic settings, but rather that they will support our claims and provide some insights.
Therefore, we think optimizing the synthetic dataset with LoRA is enough to demonstrate the concept of this section, though there is indeed a gap between realistic alignment processes.

Reviewer comment (Experiments):
Why did you pick the Quora-Question Pairs dataset? Why not a dataset designed for RLHF?

Author response:
**A**: We have already admitted that (1) is the main limitation of our proposed method, as you mentioned. (2) is a tradeoff between better sample efficiency and computations. Although it's twice more computationally demanding, it could work well with few-shot examples, so we think the cost is acceptable. As for (3), we would like to address that _there is no principal way to determine this universal temperature_ and can only make a guess. So, there's a fair chance of making a bad guess to undermine the calibration further, as demonstrated in the case of HellaSWAG in Fig.8 of our revised manuscript.

Reviewer comment (Experiments):
They restrict their study to multiple-choice questions (MCQ). While understandable due tractability issues for open-ended generation, their identified problems and proposed solutions for calibration issues are only really applicable to MCQ settings and aren’t really generalizable to open-ended generation (what is “format uncertainty” in this case). This is especially problematic since the “alignment process” is normally applied through open-ended generation tasks. While I understand the difficulty in the open-ended analysis, some discussion on how the methods could generalize this setting (for example, the proposed temperature scaling should in principle work for open-ended generation as well?) would help to ease these concerns.

Author response:
**A:** Thank you for pointing this out. First, for the decomposition of answer \& format uncertainty and performing controlled optimization for them, we think this idea has the potential to be generalized in the open-ended generation. Although, as you mentioned, the quantification of format uncertainty, in this case, is not as straightforward as it is in MCQs. Nevertheless, we could still try to optimize the format preference only during general alignment processes. 

For example, in LPF, we could select the pair of $\boldsymbol{y}_w$ and $\boldsymbol{y}_l$ that share the same semantic meaning while structured with a different format. In our revised version, we design a new synthetic scheme following this idea by letting $\boldsymbol{y}_w$ = `(A)` and $\boldsymbol{y}_l$ = `It's (A)` and fine-tune the LM with DPO. The result aligned LM exhibits similar behavior to the original Format scheme using SFT, i.e., preserving the pre-trained LM's calibration on MMLU. Though such an alignment scheme still performs on MCQs, it's closer to the alignment objective on open-ended generation compared to only increasing the likelihood of the format identifier in the original Format scheme.

Second, generalizing the proposed TS technique to open-ended generation scenario depends on whether the premises of the method are met. The key fact to making this approach work is that the pre-trained LMs are well-calibrated while the corresponding aligned LMs are not, which is well-established in MCQs. This would be true for short-form answers where our method could be naturally applied. However, to the best of our knowledge, for open-ended generation, the likelihood of a long answer may not be a proper indicator of LMs' confidence [1], and other forms of confidence quantification, such as linguistic confidence, do not show a clear superiority of pre-trained LMs over aligned LMs [2]. Hence, our proposed TS method may best fit the scenario similar to MCQs where logit-based UQ works well.

[1] [Holistic Evaluation of Language Models](https://arxiv.org/abs/2211.09110) (Liang, Percy, et al., TMLR 2023) (The [ablation study of calibration in MCQs](https://crfm.stanford.edu/helm/latest/?group=ablation_multiple_choice)).

[2] [Llamas Know What GPTs Don't Show: Surrogate Models for Confidence Estimation.](https://arxiv.org/abs/2311.08877) (Shrivastava, V., Liang, P., & Kumar, A, 2023) (Table 6 in Appendix A.6.).

Reviewer comment (Experiments):
Their proposed temperature-scaling with pretrained-LMs distribution requires access to the non-aligned pretrained LLM and a few examples *for every* new task that we might want to run our aligned model with. This might not always be feasible, particularly as closed-access LLMs proliferate and since one of the goals of alignment is making them work well in the zero-shot scenario. Is there some way to generalize this such that we only really need to use the pretrained model once (for example, before the release of the aligned model)?

Author response:
**A:** We admit that our proposed TS method has the limitations you mentioned, and we also addressed them in the manuscript. In the context of post-hoc calibration, generalizing our task-wise approach to a model-wise one is not trivial since the degree of overconfidence is not quite the same on different tasks. Achieving this goal may need to include extra and potentially heavier procedures, and we would like to explore possibilities in future works.