Paper: Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR-2024)

Reviewer comment (Experiments):
The supervised model is trained on the in-domain corpus and then used to enhance the inference of LLMs. Have you tried fine-tuning the LLMs with the in-domain corpus? Although it has a much higher cost, I wonder whether there is a much larger performance improvement after SFT.

Author response:
We appreciate your suggestion regarding instruction-tuning the LLMs with the in-domain corpus. Existing literature indicates that while tuned LLMs demonstrate high performance on many reasoning datasets, they still face challenges with out-of-distribution tests. This insight, derived from recent studies such as Liu et al. (2023), informed our decision to focus on enhancing LLM inference with supervised models trained on in-domain data, a less explored approach with potential benefits in handling OOD scenarios.  

[1] Liu, Hanmeng, et al. "Evaluating the logical reasoning ability of chatgpt and gpt-4." arXiv preprint arXiv:2304.03439 (2023).

Reviewer comment (Presentation):
I cannot capture the motivation of presenting Section 4.2. It seems that this section is not related to your core idea of the paper.

Author response:
Thank you for highlighting the need for clarity regarding Section 4.2. The purpose of this section is to investigate the limitations of in-context learning, as observed in Table 2, where LLMs underperform in this setting compared to our SuperContext approach with zero-shot demonstrations. We designed two interpretation tasks in Section 4.2 to delve into why in-context learning might be less effective, thereby providing a deeper understanding of the dynamics between LLMs and supervised learning models. This analysis is crucial for comprehending the full scope of our proposed methodology.

Reviewer comment (Experiments):
Apart from appending the model prediction to the prompt, you also include the model confidence score to the prompt. How much does the framework benefit from this design? If removing the model confidence score, how does the performance change?

Author response:
| **Model (ChatGPT)** | **SST-2** | **MNLI** | **QNLI** | **RTE** | **MRPC** | **QQP** | **STS-B** | **CoLA** | **Avg** |
|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------|
SuperContext (w/o confidence) | 94.84 | 77.21 | 82.66 | 78.45 | 63.60 | 78.08 | **80.74** | 40.29 | 78.43
SuperContext (zero-shot) | **95.19** | **87.24** | **82.91** | **78.71** | **63.87** | **78.65** | 78.75 | **41.47** | **80.05**

| **Model (LLaMA2-7B)** | **SST-2** | **MNLI** | **QNLI** | **RTE** | **MRPC** | **QQP** | **STS-B** | **CoLA** | **Avg** |
|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------|
SuperContext (w/o confidence 16-shot) | 94.29 | 76.68 | 82.66 | 78.46 | 43.41 | **78.17** | **80.74** | 40.26 | 75.68
SuperContext (16-shot) | **94.95** | **87.14** | **82.17** | **79.07** | **54.63** | 77.18 | **80.74** | **45.47** | **79.08**

Your inquiry into the role of model confidence in our framework is useful. We have incorporated ablation studies in Table 2 to address this. The results show that including the confidence score of small models significantly boosts the performance of LLMs in various NLU tasks. For instance, the performance of ChatGPT improves from an average of 78.43 to 80.05 when confidence scores are included. Similarly, LLaMA2-7B's performance increases from 75.68 to 79.08, demonstrating the effectiveness of integrating model confidence in our approach.

Reviewer comment (Evaluation):
How do you explain the extremely low scores in Table 3, i.e., 5.32, 6.08 and 3.72?

Author response:
The low scores in Table 3 primarily result from the models' challenges in generating correctly formatted responses and handling no-answer scenarios. We observed that the original Llama2-7B-chat model struggled with format adherence and hallucinations, especially in answering no-answer questions. This is reflected in the notably low score of 3.72. In other words, it cannot output ``I don't know`` when the question is unanswerable. However, when applying in-context learning with a mix of no-answer and has-answer instances, we noticed an improvement in handling no-answer questions, though this came at the cost of reduced accuracy in has-answer questions.  These observations and their implications are further discussed in the revised manuscript following Table 3.

Reviewer comment (Presentation):
Illustration of the proposed method in Figure 1 is not clear. It is unclear how the supervised knowledge participate in the whole procedure, while the capture states that it plays a core role.

Author response:
Thank you for your feedback. We acknowledge the need for clearer representation of supervised knowledge in our methodology. Accordingly, we have revised the caption of Figure 1 to better illustrate how supervised knowledge plays a pivotal role in our approach, clarifying its integration and significance in the overall process.

Reviewer comment (Writing):
The explanation of proposed method in section 2.2 is not clear enough or may have grammar error:  
a. "Specifically, our receipt r_i": receipt from what? in form of what?  
b. "learning from which ... and which ... is important": who learn what? how does it learn?  
c. "as typical data points are OOD for the model": for the discriminative model or for the LLM?

Author response:
Thank you for highlighting these ambiguities. We have made the following revisions for clarity:

Question A: The term 'receipt r_i' refers to the output and confidence score generated by Supervised Language Models (SLMs), in form of the prompt.
Question B: We have elaborated on how Large Language Models (LLMs) are trained to learn from in-context examples, providing detailed descriptions of the prompt design and learning process.
Question C: The term 'Out-Of-Distribution (OOD)' applies to both discriminative models and LLMs in our context. The training set and in-context examples are both drawn from the in-domain dataset, while the training set is used to fine-tune the SLM and in-context examples are used as the prompt, which we now explain more comprehensively in Section 2.2.

Reviewer comment (Evaluation):
Table 2 has two lines for the same baseline "ELECTRA-large" with different scores and no explanation for such difference. Specifically, the scores of ELECTRA-large in those two lines have small divergence except that it got 63.60 on MRPC in the first line but 37.59 in the second line. This raises serious questions about the reliability of the experimental results.

Author response:
We understand the importance of clarity in our experimental results. The discrepancy in Table 2 for ELECTRA-large arises due to differing test data sizes for ChatGPT and LLaMA2, as elaborated in Section 3.1. We have included a note in Table 2 explaining this variance: for LLaMA2-7B-chat, we used 37,438 instances, whereas, for ChatGPT, the number was 43,728. This clarification should address concerns regarding the reliability of our findings.

Reviewer comment (Evaluation):
The improvement of proposed method in managing OOD data is not obvious or even negative. For example, equipped with both ELECTRA-large and LLMs, the scores of proposed method only surpass ELECTRA-large within 1 point on most datasets in Table 2, and even become less than ELECTRA-large on some of the datasets. Nevertheless, there is no analysis for the casualty in evaluation, e.g., the variance of those scores.

Author response:
Your point regarding the evaluation of OOD data management is well-taken. In response, we have added detailed variance data for three separate runs in the revised manuscript. This additional information demonstrates that our method, compared to ELECTRA-large, consistently exhibits lower variance, indicating a more stable performance. These findings provide a deeper understanding of our method's efficacy in handling OOD data, supporting the reliability of our results.

| **Model (ChatGPT)** | **SST-2** | **MNLI** | **QNLI** | **RTE** | **MRPC** | **QQP** | **STS-B** | **CoLA** | **Avg** |
|----------|------------|------------|------------|------------|------------|------------|------------|------------|------------|
ELECTRA-Large | 94.84±1.4e-5 | **87.30±1.3e-4** | 82.66±1.0e-4 | 78.45±6.6e-3 | 63.60±1.0e-2 | 78.08±1.4e-4 | **80.74±3.6e-4** | 40.29±6.5e-4 | 79.86
SuperContext (zero-shot) | **95.19±9.6e-6** | 87.24±1.2e-4 | **82.91±5.0e-5** | **78.71±6.1e-3** | **63.87±1.1e-2** | **78.65±1.0e-4** | 78.75±4.3e-4 | **41.47±3.5e-4** | **80.05**

| **Model (LLaMA2-7B)** | **SST-2** | **MNLI** | **QNLI** | **RTE** | **MRPC** | **QQP** | **STS-B** | **CoLA** | **Avg** |
|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------|
ELECTRA-Large | **95.42±5.0e-5** | **87.29±1.3e-4** | **82.69±1.1e-4** | 78.84±6.6e-3 | 37.59±2.5e-2 | **77.18±5.8e-4** | **80.74±3.6e-4** | **45.73±2.6e-4** | 76.84
SuperContext (16-shot) | 94.95±4.5e-5 | 87.14±1.0e-4 | 82.17±1.4e-4 | **79.07±7.0e-3** | **54.63±2.3e-2** | **77.18±5.8e-4** | **80.74±3.6e-4** | 45.47±3.3e-4 | **79.08**

Reviewer comment (Novelty):
I disagree with the statement in the introduction that says, "However, since our goal is to allow reliable task adaptation rather than knowledge acquisition, the consulting agent becomes SLMs rather than search engines." It seems that the approach in this paper is quite similar to what Hugging Face's [1] GPT does, with the only difference being that it uses its own trained API instead of Hugging Face's SOTA small model API.

Author response:
We appreciate your observation. Our work, indeed, shares a conceptual similarity with Hugging Face's GPT in leveraging Language Model (LM) architectures. However, the key distinction lies in our approach's application and analysis under out-of-distribution (OOD) conditions, a less explored area in the existing literature. Our method uniquely combines Supervised Language Models (SLMs) and Large Language Models (LLMs) to assess and correct potential inaccuracies in SLMs' outputs. We've designed specialized tasks for interpretation and calibration laws to delve into this interaction, emphasizing the novelty of our approach. These distinctions and our experimental framework are now more clearly articulated in the revised sections of our manuscript, as shown in Related Work.

Reviewer comment (Novelty):
The paper lacks significant novelty, essentially enhancing the output of small models with larger models. However, it extensively validates the effectiveness of this approach (even though Hugging Face's GPT has also done similar validations).

Author response:
Thank you for your feedback. While recognizing HuggingGPT's contributions, our study pivots on a critical gap: the systematic evaluation of Large Language Models (LLMs) in Out-Of-Distribution (OOD) scenarios. In addition, a significant novelty of our work is investigating whether Supervised Language Models (SLMs) can mitigate LLMs' hallucination tendencies, an aspect scarcely addressed in current research. This inquiry not only extends the current understanding of model interactions but also proposes a novel perspective in improving LLM reliability.

Reviewer comment (Experiments):
The paper intentionally incorporates confidence when using the output of small models, but it lacks a detailed ablation study on the role of confidence. I am particularly interested in understanding the significance of confidence in this context.

Author response:
| **Model (ChatGPT)** | **SST-2** | **MNLI** | **QNLI** | **RTE** | **MRPC** | **QQP** | **STS-B** | **CoLA** | **Avg** |
|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------|
SuperContext (w/o confidence) | 94.84 | 77.21 | 82.66 | 78.45 | 63.60 | 78.08 | **80.74** | 40.29 | 78.43
SuperContext (zero-shot) | **95.19** | **87.24** | **82.91** | **78.71** | **63.87** | **78.65** | 78.75 | **41.47** | **80.05**

| **Model (LLaMA2-7B)** | **SST-2** | **MNLI** | **QNLI** | **RTE** | **MRPC** | **QQP** | **STS-B** | **CoLA** | **Avg** |
|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------|
SuperContext (w/o confidence 16-shot) | 94.29 | 76.68 | 82.66 | 78.46 | 43.41 | **78.17** | **80.74** | 40.26 | 75.68
SuperContext (16-shot) | **94.95** | **87.14** | **82.17** | **79.07** | **54.63** | 77.18 | **80.74** | **45.47** | **79.08**

We're grateful for your interest in the role of model confidence. To address your query, we've included Table 2 in our revised manuscript, presenting a detailed ablation study. The data demonstrate that incorporating the confidence metric from small models significantly enhances LLM performance across various Natural Language Understanding (NLU) tasks. For instance, the average performance of ChatGPT improved from 78.43 to 80.05, and for LLaMA2-7B, it rose from 75.68 to 79.08. 

These findings underscore the importance of confidence as a valuable component in our approach, offering a more nuanced understanding of model interactions.