Paper: Automatic Calibration and Error Correction for Generative Large Language Models via Pareto Optimal Self-Supervision (ICLR-2024)

Reviewer comment (Theory):
My biggest concern about this paper is I think it's over-claimed. This paper frames the learning objective to be a Pareto loss scalarizer, and also requires the Pareto scalarizer to be a convex function, which is too strong. In this case, the Pareto optimal is only a single point in the whole function space, which seems not satisfying the realistic scenario.

Author response:
$\textbf{Regarding Pareto optimal Solution:}$ 
- Based on our claim in Theorem 1, the harmonizer model $h^*$ identified via our approach approximates one of the possibly many Pareto optimal solutions. Indeed, different scenarios may favor different solutions on the Pareto frontier. We achieve this diversity through the selection of different Pareto scalarizers $G$, as outlined in our paper. Specifically, we explored three distinct Pareto scalarizers and the non-Pareto Chebyshev scalarizer for comparative analysis. Our empirical results, as shown in Table 2, suggest that the quadratic scalarizer is particularly effective when paired with the BERT harmonizer model. This scalarizer emphasizes challenging examples where the LLM and weak sources exhibit significant discrepancies, as briefly discussed following Definition 2.

- We understand your concern regarding the need for reweighting among the LLM and weak sources. To address this, we provided two alternative strategies in Appendix B, designed to automatically adjust weights based on different scenarios, such as high dependency and significant differences in source accuracy. We hope these provide a good variety of Pareto optimality for different scenarios, addressing your concern.

Reviewer comment (Theory):
A more natural way should be giving a different weight to different $l$, and then finding the Pareto optimal under specific weight. Thus, we have the Pareto frontier. According to different scenario, we may choose different Pareto optimal on the frontier.

Author response:
$\textbf{Regarding Pareto optimal Solution:}$ 
- Based on our claim in Theorem 1, the harmonizer model $h^*$ identified via our approach approximates one of the possibly many Pareto optimal solutions. Indeed, different scenarios may favor different solutions on the Pareto frontier. We achieve this diversity through the selection of different Pareto scalarizers $G$, as outlined in our paper. Specifically, we explored three distinct Pareto scalarizers and the non-Pareto Chebyshev scalarizer for comparative analysis. Our empirical results, as shown in Table 2, suggest that the quadratic scalarizer is particularly effective when paired with the BERT harmonizer model. This scalarizer emphasizes challenging examples where the LLM and weak sources exhibit significant discrepancies, as briefly discussed following Definition 2.

- We understand your concern regarding the need for reweighting among the LLM and weak sources. To address this, we provided two alternative strategies in Appendix B, designed to automatically adjust weights based on different scenarios, such as high dependency and significant differences in source accuracy. We hope these provide a good variety of Pareto optimality for different scenarios, addressing your concern.

Reviewer comment (Novelty):
If in this paper's setting, then it seems the pipeline is just like first it learns some score functions using additional supervision signals. Then it uses these scoring function to enhance prompt, and finally it gets better responses based on better prompts, which seems no novelty to me.

Author response:
$\textbf{On the novelty of using scoring function:}$
- We respectfully express our disagreement on "Finding scoring function from external weak sources and use them for calibration and correction has no novelty". Any calibration method, including early ones like Platt scaling, involves some form of scoring function. The challenge lies in deriving a robust scoring function with limited resources. Our approach uniquely tackles this by using weak supervision sources in the absence of ground truth labels, ensuring independence from the LLM and mitigating bias. We believe this is a reasonable approach to provide information that is independent from the LLM itself, which is essential in as a guardrail.

- Our work's novelty is threefold:
 1. We are pioneers in using weak supervision sources for LLM calibration and correction.
 2. To our knowledge, this is arguably the first attempt to estimate confidence levels using only weak sources, a concept previously unexplored. We are excited to report our results that may inspire future work. 
 3. Our methodology for aggregating weak sources is innovative and distinct from prior works focused on label weighting.

Reviewer comment (Theory):
In Theorem 1, if G is always a convex function, then Theorem 1 always holds. I don't see any importance from Theorem 1. Does Theorem 1 still hold when G is a non-convex function?

Author response:
$\textbf{Responses to Specific Questions:}$ 
- Question 1. Theorem 1 tells us the criteria for the function $G$ that are beyond mere convexity; it must also be strictly increasing element-wise (as per Definition 2). An illustrative counterexample is the Chebyshev/max scalarizer. Though convex, it fails to meet the strictly increasing condition, hence does not provide the guarantees outlined in Theorem 1. Our experiments support this, showing its relative underperformance compared to other Pareto scalarizers.

Reviewer comment (Reproducibility):
In Table 1 and Figure 1, what scalarizer is used?

Author response:
$\textbf{Responses to Specific Questions:}$ 
-  Question 3: We utilized the quadratic scalarizer in our experiments, as detailed in the 'Harmonizer Training' section of Section 4.

Reviewer comment (Writing):
This paper has major issues with clarity. You have a lot of typos / odd grammatical issues and sentence structures that make it hard to understand. E.g., "Unlike task-specific models, there lack an effective method..." in the abstract. Even beyond these, there is some major missing information throughout in important sections. For example, on the top of page 4 (from the end of Proposition 1 to the start of Proposition 2), you have a paragraph that contains a lot of statements illustrating potential problems and arguing that certain things are therefore needed with little to no explanation of these key ideas and why we should believe they are true. E.g., why does Proposition 1 show a path towards calibrating an LLM via distillation? How does this show the need of external signals? etc. In addition, some mathematical statements are not sufficiently explicitly stated to be correct as written, which need to be corrected for sure.

Author response:
Dear Reviewer,

Thank you for your thorough review and constructive feedback on our manuscript. We sincerely appreciate your acknowledgement of our contribution. The suggestions you made to improve our paper presentation are highly valuable, and we have made significant revisions to address your concerns, particularly regarding clarity and the specificity of our theoretical claims. Here is a summary of the key revisions and responses to your questions:

1. $\textbf{Clarity and Writing Quality:}$ We have carefully revised the manuscript to eliminate typos and grammatical issues. We have also restructured sentences for better clarity and coherence, particularly in sections where the methodology and experimental details are explained. This includes clarifying the role and the limitations of Propositions 1 and 2, and providing a more explicit explanation of our approach in the end of Section 3.3. Although there are many changes in the paper, for your time and convenience we want to highlight one example of modification for your reference. This is the part after Proposition 1 (distillation from imperfect source promotes calibration), and before Proposition 2 (noisy weak signals helps accurate LLM):

"(After Proposition 1) ... Muller et al. (2019) empirically demonstrated
that label smoothing improves or at least doesn’t harm model calibration compared to standard
supervised learning. Therefore, Proposition 1 suggests that access to ground truth labels is not
necessary for obtaining well-calibrated outputs. If LLM errors were uniformly “random”, we could
directly use it as a teacher model to distill a small network on the specific task for calibration.

However, LLM errors are rarely “random”. Deterministic functions typically err on specific types of
inputs. Distilling a model solely from the LLM could lead to overfitting, thus unable to signal errors
in the teacher LLM. Our solution is to integrate other weak supervision functions as external signals
independent of the LLM. As each supervision source makes different types of errors on different
examples, their ensemble approximates a “random” scenario... (Before Proposition 2)"

Reviewer comment (Theory):
Your proposition 2 is not sufficiently well specified to be true. You need to expand on what "weak independent signal ensemble" means to eliminate the case that the weak ensemble contains no new information relative to your LLM itself (and therefore cannot contribute meaningful improvements over the LLM). Also your notation for the function $\psi$ is inconsistent, as $\psi$ is listed as a function of both the functions $\Lambda$ and $w$ as well as of their outputs. And you need to clarify why your $w$ is a continuous estimator, not a categorical, given you work in the categorical space in other settings.

Author response:
2. $\textbf{Theoretical Clarification:}$ Regarding Proposition 2, we have expanded the explanation of "weak independent signal ensemble" to clarify that these signals provide information independent of the LLM. We have also revised the notation for the function "$\psi(\Lambda, w)$" to "$\psi(\Lambda(x), w(x))$" to avoid confusion. Additionally, we now explicitly state why the weak supervision signal ensemble $\omega(x)$ is modeled with a continuous random variable in Proposition 2.

Reviewer comment (Theory):
The inference you make in the second sentence of 3.3 is invalid given that you don't know that these "weakest supervision sources such as reg-ex or knowledge base" are actually independent of the LLM. I know you go back and state this in the next sentence or two, but if you're going to acknowledge it there, just be specific and clear from the outset.

Author response:
3. $\textbf{Section 3.3 Clarification}$ We now explicitly states in this section that our main theoretical results takes a leap from the inspirational propositions. We also added a brief summary for the theoretical part at the end of Section 3.3 to discuss the connections between the theoretical results and their limitations. Here is the content for your reference:

"Let’s make a brief theoretical summary here. If the LLM makes uniformly “random” errors, simply distilling a smaller network would give a calibration model. Since this is not usually the case, we incorporate multiple independent supervision functions alongside the LLM, fitting a harmonizer model to all simultaneously to prevent overfitting to the LLM. Theoretically, we approximate a harmonizer model that is Pareto optimal. However, due to the complexity of modeling the errors made by the LLM and supervision functions, we cannot offer further theoretical guarantees at this point. We empirically demonstrate the calibration capability of the harmonizer model in Section 4."

Reviewer comment (Theory):
Isn't your "pareto optimization scalarizer" functionally reducing all your weak supervision signals to a single signal, thereby suffering the same concerns that you state the existing works will suffer by combining loss terms via a weighted sum?

Author response:
4. $\textbf{Pareto Optimization Scalarizer:}$ We have further clarified the concept of the Pareto optimization scalarizer in our framework. We clarified that while it does aggregate multiple supervision signals into a single harmonized output, this process is fundamentally different from a simple weighted sum because of Jensen's inequality. One example is the quadratic scalarizer, which automatically emphasizes on the harder examples with larger overall loss. The trade-offs between different supervision sources is handled in the Pareto optimal manner, thereby avoiding the pitfalls of traditional weighted sum approaches.

Reviewer comment (Reproducibility):
Do you iterate repeatedly in Algorithm 2 until your POLAR score is below $\delta$? Or do you just do the process once.

Author response:
5. $\textbf{Dynamic Prompting Iteration in Algorithm 2:}$ The POLAR-assisted dynamic prompting process in Algorithm 2 is typically executed once per instance. If we take the threshold $\delta = 0.5$, after one iteration of correction it must be below $\delta$ already. However, one can re-fit the harmonizer using the updated LLM response and do it again. In our experiments, we didn't observe additional significant improvement on performance, likely due to the information limit.

Reviewer comment (Writing):
I want to see in the revision a significant improvement in writing quality (elimination of typos, grammatical errors, etc.), clarification of all sections of the text—especially with respect to methodology and details behind the experiments—and formal, correct, and complete statements behind the mathematical claims made in the work.

Author response:
Dear Reviewer,

Thank you for your thorough review and constructive feedback on our manuscript. We sincerely appreciate your acknowledgement of our contribution. The suggestions you made to improve our paper presentation are highly valuable, and we have made significant revisions to address your concerns, particularly regarding clarity and the specificity of our theoretical claims. Here is a summary of the key revisions and responses to your questions:

1. $\textbf{Clarity and Writing Quality:}$ We have carefully revised the manuscript to eliminate typos and grammatical issues. We have also restructured sentences for better clarity and coherence, particularly in sections where the methodology and experimental details are explained. This includes clarifying the role and the limitations of Propositions 1 and 2, and providing a more explicit explanation of our approach in the end of Section 3.3. Although there are many changes in the paper, for your time and convenience we want to highlight one example of modification for your reference. This is the part after Proposition 1 (distillation from imperfect source promotes calibration), and before Proposition 2 (noisy weak signals helps accurate LLM):

"(After Proposition 1) ... Muller et al. (2019) empirically demonstrated
that label smoothing improves or at least doesn’t harm model calibration compared to standard
supervised learning. Therefore, Proposition 1 suggests that access to ground truth labels is not
necessary for obtaining well-calibrated outputs. If LLM errors were uniformly “random”, we could
directly use it as a teacher model to distill a small network on the specific task for calibration.

However, LLM errors are rarely “random”. Deterministic functions typically err on specific types of
inputs. Distilling a model solely from the LLM could lead to overfitting, thus unable to signal errors
in the teacher LLM. Our solution is to integrate other weak supervision functions as external signals
independent of the LLM. As each supervision source makes different types of errors on different
examples, their ensemble approximates a “random” scenario... (Before Proposition 2)"

7. $\textbf{Experimental Details:}$ The details behind the experiments have been expanded for greater clarity. This includes more information on the datasets used, the specifics of the implementation of the harmonizer, and the rationale behind the choice of our evaluation metrics. Explanations of the benchmark comparison methods are also expanded.

Thank you again for your time and valuable comments, which did put our work and paper into a better state! We believe these revisions have significantly strengthened the manuscript, addressing the key concerns you raised. Hope that the improvements in clarity, presentation, citation, and extra experiments adequately address your feedback and make the paper a strong candidate for acceptance.

Thank you once again for your valuable feedback, and we look forward to your further thoughts!

Best regards,
Authors

Reviewer comment (Novelty):
Weighting dilemma is not something new in weak supervision. A major goal is rather than using one weight per supervision function across the entire dataset, we'd like to have input-specific weight parameters (such as https://arxiv.org/abs/2107.02233).

Author response:
- In response to your suggestion, we have included the End-to-end weak supervision paper in our citations. This paper presents an nice framework in weak supervision, assigning instance-specific weights in an end-to-end manner.

- We conducted benchmark experiments using the WeaSEL model and have included these results in our comparison Table 1. While WeaSEL shows commendable performance on specific tasks and LLMs (e.g., calibrating GPT-3.5 on the CDR relation extraction), it does not consistently deliver across all scenarios. Notably, it performs optimally with GPT-3.5 and in situations with a limited number of supervision functions. Compared to the Snorkel model, WeaSEL exhibits superior performance in most cases, likely due to its greater expressiveness and instance-specific weighting. For your convenience, we provide the extended comparison table in the next comment.

Reviewer comment (Theory):
Limitations of theoretical results are not discussed well. For instance in proposition 2, the existence of a function doesn't imply that this function class is how $\Lambda(x), w(x)$ is aggregated.

Author response:
- We have revised the theoretical section for enhanced clarity, adding detailed explanations and discussions about the implications and limitations of our propositions and theorems. We acknowledge that our propositions are more inspirational in nature and may contain unrealistic assumptions (e.g., non-random LLM errors, independence of weak supervision sources, and an insufficient number of weak sources for CLT approximation). For main results, we provide theoretical guarantee with Theorem 1 free from these assumptions, and we demonstrate the empirical effectiveness of our approach.

- A new paragraph at the end of Section 3.3 now provides a summary and discusses the connection between the theoretical results and their limitations. Here is the added content for you reference:

"Let’s make a brief theoretical summary here. If the LLM makes uniformly “random” errors, simply distilling a smaller network would give a calibration model. Since this is not usually the case, we incorporate multiple independent supervision functions alongside the LLM, fitting a harmonizer model to all simultaneously to prevent overfitting to the LLM. Theoretically, we approximate a harmonizer model that is Pareto optimal. However, due to the complexity of modeling the errors made by the LLM and supervision functions, we cannot offer further theoretical guarantees at this point. We empirically demonstrate the calibration capability of the harmonizer model in Section 4."

Reviewer comment (Experiments):
See my question about comparing to additional weak supervision methods.

Author response:
- In response to your suggestion, we have included the End-to-end weak supervision paper in our citations. This paper presents an nice framework in weak supervision, assigning instance-specific weights in an end-to-end manner.

- We conducted benchmark experiments using the WeaSEL model and have included these results in our comparison Table 1. While WeaSEL shows commendable performance on specific tasks and LLMs (e.g., calibrating GPT-3.5 on the CDR relation extraction), it does not consistently deliver across all scenarios. Notably, it performs optimally with GPT-3.5 and in situations with a limited number of supervision functions. Compared to the Snorkel model, WeaSEL exhibits superior performance in most cases, likely due to its greater expressiveness and instance-specific weighting. For your convenience, we provide the extended comparison table in the next comment.

Reviewer comment (Theory):
The proposed framework is restricted to tasks where the desired output space $\mathcal{Y}$ is finite, which can be impractical for real-world applications when there is no definitive answer in a finite set, e.g., recommending websites for learning languages.

Author response:
**Applicability to Real-World Tasks with Non-Finite Output Spaces:** We agree with you and recognize the limitation of our framework being applicable primarily to tasks with finite output spaces. As identifying and correcting LLM error is a challenging problem and an emerging field, we chose to be focus on and tackle a smaller problem. What we hope from our paper is not only the introduction of the proposed method, but also theoretical and empirical results that can inspire future work. In that sense, targeting a smaller problem helps us dig deeper.

Reviewer comment (Novelty):
I am sorry, but I don't think the two intuitions behind how calibration can be achieved (mentioned by the authors in the introduction) are easy to understand, perhaps due to my lack of expertise. They look like two conclusions that other works have confirmed. Are there any references?

Author response:
**Clarity on Theoretical Intuitions:** We have revised the introduction to provide clearer explanations of the intuitions behind our approach. The two intuitions are proved in Proposition 1 and 2 respectively. In short, if the LLM error is uniformly random, the distilled model from it serves as a good calibration model. Introducing multiple external supervision sources diversifies the output, avoiding the distilled model being completely bias by the LLM and not able to signal its error.

Reviewer comment (Theory):
I do not see a natural connection between **Proposition 1**, **Proposition 2** and Pareto optimal learning. According to the **Proposition 1**, fitting a model $h(x)$ to $Λ(x)$ is equivalent to training on ground truth labels with label smoothing. However, I do not find that the author mentioned using the "label smoothing" technique while optimizing the harmonizer network $h(x)$. Also, I'm not sure how this is solved by using Pareto optimal learning.

Author response:
**Connection Between Propositions and Pareto Optimal Learning:**

To clarify the relationship between Propositions 1 and 2 and Pareto optimal learning, we have added a detailed explanation in Section 3.2. We also added brief summary of the connections between the theoretical results, as well as their limitations. The content is below for your reference:

"Let’s make a brief theoretical summary here. If the LLM makes uniformly “random” errors, simply distilling a smaller network would give a calibration model. Since this is not usually the case, we incorporate multiple independent supervision functions alongside the LLM, fitting a harmonizer model to all simultaneously to prevent overfitting to the LLM. Theoretically, we approximate a harmonizer model that is Pareto optimal. However, due to the complexity of modeling the errors made by the LLM and supervision functions, we cannot offer further theoretical guarantees at this point. We empirically demonstrate the calibration capability of the harmonizer model in Section 4."

**Label Smoothing:** In Proposition 1, label smoothing is implicitly achieved by fitting to the imperfect LLM output. This technique is not adopted for training, but as a theoretical concept to justify our approach. We have made the explanation for this clearer in the revision to avoid confusion.

Reviewer comment (Writing):
As a major part contributing to the multi-objective optimization, the supervision functions $\lambda_j$ do not seem to be clearly presented.

Author response:
**Presentation of Supervision Functions:** We have revised the manuscript to provide a more detailed explanation of the supervision functions. In short, they are any programmatic supervision sources, such as keywords, regex, knowledge base, that outputs noisy heuristic labels on the input. We explained the choice of the supervision function in the experimental part, and also discussed their contribution to the overall calibration performance.

Reviewer comment (Theory):
In the **Definition 2** (Pareto scalarizer), what is the difference between $G(\mathcal{l}_0, \cdots, \mathcal{l}^{'}_j, \cdots, \mathcal{l}_m)$ and $G(\mathcal{l}_0, \cdots, \mathcal{l}_j, \cdots, \mathcal{l}_m)$, as far as I understand, they have the same $m$ objectives?

Author response:
**Objectives in Pareto Scalarizer:** The difference between $G(\ell_0, \cdots, \ell_j', \cdots, \ell_m)$ and $G(\ell_0, \cdots, \ell_j, \cdots, \ell_m)$ is $\ell_j'$ and $\ell_j$. The requirement for $G$ is, for any $\ell_j' < \ell_j$, there must be $G(\ell_0, \cdots, \ell_j', \cdots, \ell_m) < G(\ell_0, \cdots, \ell_j, \cdots, \ell_m)$, i.e. strictly increasing (but not necessarily continuous).

Reviewer comment (Theory):
So is the Pareto scalarizer actually equivalent to the uniform weighting scheme?

Author response:
**Clarification on Pareto Scalarizer and Weighting Schemes:** The Pareto scalarizer in our experiment is a nonlinear function aggregating the losses, thus different from a uniform weighting scheme. When performing SGD types of optimization, Jensen's inequality makes the nonlinearity take effect. For example, the quadratic scalarizer, which is the one we experimented to be most effective, automatically emphasize on the hard examples where the LLM and the other sources have larger disagreement. In our revision, we highlight its unique role in balancing multiple objectives in a harmonized manner.

Reviewer comment (Evaluation):
How are the two metrics based on POLAR score, i.e., expected calibration error (ECE) and $R^2$ calculated?

Author response:
**Computation of POLAR Score Metrics:** 
- ECE: Split the examples into 10 bins according to their POLAR score, e.g. 0-10%, 10-20%, etc. For the examples in each bin, calculate the real LLM error rate using the ground truth labels. Compare the difference between the POLAR score and the true LLM error rate, and take expectation over all the examples.
- $R^2$: Ranks the LLM responses into POLAR score bins each with 100 examples, and plot the average POLAR score and true error rate for each bin. The R2 is calculated from the (POLAR - error rate) scatter plot.

Reviewer comment (Evaluation):
For every baseline method, is the error rate calculated by comparing the estimated class probabilities with ground truth labels?

Author response:
**Error Rate Calculation in Baseline Methods:** Yes, the error rate is indeed calculated by comparing the estimated class probabilities from the baseline model with ground truth labels.

Reviewer comment (Evaluation):
From Figure 3(a), it seems that we can always use the dynamic self-supervision prompting strategy to achieve a decent error rate?

Author response:
**Dynamic Self-Supervision Prompting Strategy:** Yes, as long as the supervision functions are independent from the LLM (thus provide extra information), we can always improve the error rate for the high POLAR score examples. Once the error got corrected, because the POLAR score already dropped below the threshold, re-prompting doesn't further improve the results. This is consistent with the information limit available from the supervision functions.