Original: P ROTOTYPICAL C ALIBRATION FOR F EW - SHOT L EARNING OF L ANGUAGE M ODELS
Revised: P ROTOTYPICAL C ALIBRATION FOR F EW - SHOT L EARNING OF L ANGUAGE M ODELS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In-context learning of GPT-like models has been recognized as fragile across different hand-crafted templates, and demonstration permutations.
Revised: In-context learning of GPT-like models has been recognized as fragile across different hand-crafted templates, and demonstration permutations.

Original: In this work, we propose prototypical calibration to adaptively learn a more robust decision boundary for zero- and few-shot classification, instead of greedy decoding.
Revised: In this work, we propose prototypical calibration to adaptively learn a more robust decision boundary for zero- and few-shot classification, instead of greedy decoding.

Original: Concretely, our method first adopts Gaussian mixture distribution to estimate the prototypical clusters for all categories.
Revised: Concretely, our method first adopts Gaussian mixture distribution to estimate the prototypical clusters for all categories.

Original: Then we assign each cluster to the corresponding label by solving a weighted bipartite matching problem.
Revised: Then we assign each cluster to the corresponding label by solving a weighted bipartite matching problem.

Original: Given an example, its prediction is calibrated by the likelihood of prototypical clusters.
Revised: Given an example, its prediction is calibrated by the likelihood of prototypical clusters.

Original: Experimental results show that prototypical calibration yields a substantial improvement on a diverse set of tasks.
Revised: Experimental results show that prototypical calibration yields a substantial improvement on a diverse set of tasks.

Original: Extensive analysis across different scales also indicates that our method calibrates the decision boundary as expected, greatly improving the robustness of
Revised: Extensive analysis across different scales also indicates that our method calibrates the decision boundary as expected, greatly improving the robustness of

Original: GPT to templates, permutations, and class imbalance.
Revised: GPT to templates, permutations, and class imbalance.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Large-scale language models (LMs) have shown strong generalization ability on a wide range of downstream tasks (Devlin et al., 2018;
Revised: Large-scale language models (LMs) have shown strong generalization ability on a wide range of downstream tasks (Devlin et al., 2018;

Original: Radford et al., 2019;
Revised: Radford et al., 2019;

Original: Yang et al., 2019; Lewis et al., 2019;
Revised: Yang et al., 2019; Lewis et al., 2019;

Original: Brown et al., 2020; Dong et al., 2019; Bao et al., 2020).
Revised: Brown et al., 2020; Dong et al., 2019; Bao et al., 2020).

Original: Fine-tuning has been the common strategy to transfer the extensive knowledge to downstream tasks for a long time.
Revised: Fine-tuning has been the common strategy to transfer the extensive knowledge to downstream tasks for a long time.

Original: However, fine-tuning such large LMs suffers from over-parameterization issue under few-shot settings.
Revised: However, fine-tuning such large LMs suffers from over-parameterization issue under few-shot settings.

Original: Brown et al.
Revised: Brown et al.

Original: (2020) propose the concept of in-context learning with GPT, which enables LMs to quickly adapt to a new task by conditioning on hand-crafted prompts as shown in Figure 1.
Revised: (2020) propose the concept of in-context learning with GPT, which enables LMs to quickly adapt to a new task by conditioning on hand-crafted prompts as shown in Figure 1.

Original: The prompts consist of task-specific templates and several input-label pairs (demonstrations).
Revised: The prompts consist of task-specific templates and several input-label pairs (demonstrations).

Original: In-context learning is surprising as GPT can perform various tasks without any parameter updating.
Revised: In-context learning is surprising as GPT can perform various tasks without any parameter updating.

Original: It has been noticed that the predictions of GPT conditioned on prompts tend to bias toward some specific answers and can be highly volatile across different templates, demonstrations, and their permutations (Lu et al., 2021; Jiang et al., 2020).
Revised: It has been noticed that the predictions of GPT conditioned on prompts tend to bias toward some specific answers and can be highly volatile across different templates, demonstrations, and their permutations (Lu et al., 2021; Jiang et al., 2020).

Original: Zhao et al.
Revised: Zhao et al.

Original: (2021) propose to calibrate the model prediction by the content-free output to mitigate this problem.
Revised: (2021) propose to calibrate the model prediction by the content-free output to mitigate this problem.

Original: Rubin et al.
Revised: Rubin et al.

Original: (2021) and Lu et al.
Revised: (2021) and Lu et al.

Original: (2021) focus on the training examples retrieval and optimal ordering selection respectively to produce more performant prompts than random sampling.
Revised: (2021) focus on the training examples retrieval and optimal ordering selection respectively to produce more performant prompts than random sampling.

Original: However, they did not explain why the in-context learning performance is fragile across different scenarios.
Revised: However, they did not explain why the in-context learning performance is fragile across different scenarios.

Original: In this paper, we analyze the intrinsic reason for the instability of few-shot learning with GPT.
Revised: In this paper, we analyze the intrinsic reason for the instability of few-shot learning with GPT.

Original: We observe significant distinctions among the prediction distributions of GPT under different prompts.
Revised: We observe significant distinctions among the prediction distributions of GPT under different prompts.

Original: As shown in Figure 2, the conventional decision boundary of GPT (i.e., naively uses the output with the largest probability as the predicted label) often fails to discriminate the predictions.
Revised: As shown in Figure 2, the conventional decision boundary of GPT (i.e., naively uses the output with the largest probability as the predicted label) often fails to discriminate the predictions.

Original: We argue that the predictions can be more discriminative when provided a calibrated decision boundary.
Revised: We argue that the predictions can be more discriminative when provided a calibrated decision boundary.

Original: Specifically, we term the model outputs of examples whose ground-truth are the same category as prototypical cluster and adopt Gaussian Mixture Model (GMM) to estimate the distributions of them for all categories.
Revised: Specifically, we term the model outputs of examples whose ground-truth are the same category as prototypical cluster and adopt Gaussian Mixture Model (GMM) to estimate the distributions of them for all categories.

Original: The decision boundaries of the prototypical clusters are adaptively learned, which is called prototypical calibration (P RO C A ).
Revised: The decision boundaries of the prototypical clusters are adaptively learned, which is called prototypical calibration (P RO C A ).

Original: Then the prototypical clusters are assigned to the corresponding labels through a weighted bipartite matching.
Revised: Then the prototypical clusters are assigned to the corresponding labels through a weighted bipartite matching.

Original: We also propose to improve estimations according to cluster-label assignment.
Revised: We also propose to improve estimations according to cluster-label assignment.

Original: Finally, the predictions of test examples are more precise owing to the calibrated decision boundary (as shown in Figure 2).
Revised: Finally, the predictions of test examples are more precise owing to the calibrated decision boundary (as shown in Figure 2).

Original: Experimental results show that we achieve on average 13% absolute improvement for different sizes of GPT models across nine text classification datasets.
Revised: Experimental results show that we achieve on average 13% absolute improvement for different sizes of GPT models across nine text classification datasets.

Original: We demonstrate that P RO C A is effective across various templates and different demonstration permutations.
Revised: We demonstrate that P RO C A is effective across various templates and different demonstration permutations.

Original: To summarize, our key contributions are as follows:
Revised: To summarize, our key contributions are as follows:

Original: • We find that the decision boundary plays a critical role in few-shot evaluation.
Revised: • We find that the decision boundary plays a critical role in few-shot evaluation.

Original: Moreover, performant decision boundaries are inconsistent across language models and prompts.
Revised: Moreover, performant decision boundaries are inconsistent across language models and prompts.

Original: • We propose prototypical calibration to adaptively learn a better decision boundary for few-shot classification of language models.
Revised: • We propose prototypical calibration to adaptively learn a better decision boundary for few-shot classification of language models.

Original: • Experiments show that P RO C A achieves a 13% absolute improvement over the conventional approach on a wide range of text classification tasks.
Revised: • Experiments show that P RO C A achieves a 13% absolute improvement over the conventional approach on a wide range of text classification tasks.

Original:  D ECISION B OUNDARY OF F EW -S HOT L EARNING WITH GPT
Revised:  D ECISION B OUNDARY OF F EW -S HOT L EARNING WITH GPT

Original: A decision boundary refers to an explicit prediction criterion in the output space for a given classification problem.
Revised: A decision boundary refers to an explicit prediction criterion in the output space for a given classification problem.

Original: As shown in Figure 2, two dashed lines represent two different decision boundaries, which classifies examples into negative and positive categories.
Revised: As shown in Figure 2, two dashed lines represent two different decision boundaries, which classifies examples into negative and positive categories.

Original: In this section, we explore the effect of decision boundary on few-shot learning.
Revised: In this section, we explore the effect of decision boundary on few-shot learning.

Original: We demonstrate that optimal decision boundaries are inconsistent under different LMs and prompts.
Revised: We demonstrate that optimal decision boundaries are inconsistent under different LMs and prompts.

Original: Decision boundary greatly influences the few-shot performance.
Revised: Decision boundary greatly influences the few-shot performance.

Original: We evaluate the performance of different models and prompts using different decision boundaries.
Revised: We evaluate the performance of different models and prompts using different decision boundaries.

Original: Results are shown in Figure 3.
Revised: Results are shown in Figure 3.

Original: The red rectangle indicates the conventional decision boundary used by GPT, which naively decodes the label with larger prediction probability.
Revised: The red rectangle indicates the conventional decision boundary used by GPT, which naively decodes the label with larger prediction probability.

Original: We observe that shifting the decision boundary can cause wild fluctuations of few-shot accuracy, from near state-of-the-art to random guessing.
Revised: We observe that shifting the decision boundary can cause wild fluctuations of few-shot accuracy, from near state-of-the-art to random guessing.

Original: For each prompt, there is an exclusive region where the decision boundary is relatively robust.
Revised: For each prompt, there is an exclusive region where the decision boundary is relatively robust.

Original: The model exhibits poor performance when the decision boundary is far from the robust region.
Revised: The model exhibits poor performance when the decision boundary is far from the robust region.

Original: Performant decision boundaries are not transferable across LMs or prompts.
Revised: Performant decision boundaries are not transferable across LMs or prompts.

Original: Figure 3 demonstrates that all prompts exhibit strong performance if the decision boundary locates in the robust region.
Revised: Figure 3 demonstrates that all prompts exhibit strong performance if the decision boundary locates in the robust region.

Original: However, different prompts and models lead to different degrees of deviation between the optimal decision boundary and the conventional one.
Revised: However, different prompts and models lead to different degrees of deviation between the optimal decision boundary and the conventional one.

Original: It suggests that performant decision boundaries are inconsistent across models or prompts.
Revised: It suggests that performant decision boundaries are inconsistent across models or prompts.

Original: Based on the above analysis, we argue that all prompts can achieve better performance when the decision boundary is calibrated into the robust region.
Revised: Based on the above analysis, we argue that all prompts can achieve better performance when the decision boundary is calibrated into the robust region.

Original:  P ROTOTYPICAL C ALIBRATION
Revised:  P ROTOTYPICAL C ALIBRATION

Original: We have illustrated that the conventional decision boundary generally deviates from the robust region, which renders in-context learning fragile.
Revised: We have illustrated that the conventional decision boundary generally deviates from the robust region, which renders in-context learning fragile.

Original: In this section, we present prototypical calibration (P RO C A ) to adaptively learn a better decision boundary.
Revised: In this section, we present prototypical calibration (P RO C A ) to adaptively learn a better decision boundary.

Original: 3. P ROTOTYPICAL C LUSTER E STIMATION
Revised: 3. P ROTOTYPICAL C LUSTER E STIMATION

Original: Considering an N -way few-shot classification task, let X denote the N -dimensional model outputs.
Revised: Considering an N -way few-shot classification task, let X denote the N -dimensional model outputs.

Original: For examples whose ground-truth is the n -th category, the model outputs compose a prototypical cluster.
Revised: For examples whose ground-truth is the n -th category, the model outputs compose a prototypical cluster.

Original: For instance, the red and blue areas in Figure 2 refer to two prototypical clusters respectively.
Revised: For instance, the red and blue areas in Figure 2 refer to two prototypical clusters respectively.

Original: We assume that each prototypical cluster follows a Gaussian distribution:
Revised: We assume that each prototypical cluster follows a Gaussian distribution:

Original: [Equation]
Revised: [Equation]

Original: where µ n  and Σ n  are the mean vector, and covariance matrix of the distribution, respectively.
Revised: where µ n µ n µ n and Σ n Σ n Σ n are the mean vector, and covariance matrix of the distribution, respectively.

Original: Next, we estimate N prototypical clusters for N categories with Gaussian mixture model (GMM):
Revised: Next, we estimate N prototypical clusters for N categories with Gaussian mixture model (GMM):

Original: [Equation]
Revised: [Equation]

Original: where α n is the mixing coefficient for the n -th distribution.
Revised: where α n is the mixing coefficient for the n -th distribution.

Original: In our work, we formulate the model prediction x = [ x 1 , x 2 , ..., x N ] as follows:
Revised: In our work, we formulate the model prediction x = [ x 1 , x 2 , ..., x N ] as follows:

Original: [Equation]
Revised: [Equation]

Original: where o n and o i are the logits predicted by GPT, corresponding to label token n and label token i respectively.
Revised: where o n and o i are the logits predicted by GPT, corresponding to label token n and label token i respectively.

Original: Intuitively, x n represents the log probability of the n -th category.
Revised: Intuitively, x n represents the log probability of the n -th category.

Original: After clarifying the GMM definition under few-shot learning, we utilize a small-scale unlabeled in-domain dataset, named as estimate set ( D esti ), to estimate the parameters { α n , µ n , Σ n  } Nn =1 by the Expectation-Maximization (EM) algorithm
Revised: After clarifying the GMM definition under few-shot learning, we utilize a small-scale unlabeled in-domain dataset, named as estimate set ( D esti ), to estimate the parameters { α n ,µ n µ n µ n , Σ n Σ n Σ n } Nn =1 by the Expectation-Maximization (EM) algorithm

Original: (Moon, 1996).
Revised: (Moon, 1996).

Original: Notice that the estimate set does not contain any human annotation.
Revised: Notice that the estimate set does not contain any human annotation.

Original: Specifically, EM is an iterative method to find the optimal estimation of GMM’s parameters by maximizing the likelihood (cid:81)x ∈ D esti
Revised: Specifically, EM is an iterative method to find the optimal estimation of GMM’s parameters by maximizing the likelihood (cid:81)

Original: P GMM ( x ) .
Revised: [Equation]

Original: 3. C LUSTER -L ABEL A SSIGNMENT
Revised: 3. C LUSTER -L ABEL A SSIGNMENT

Original: Then we assign the estimated prototypical clusters to the target labels.
Revised: Then we assign the estimated prototypical clusters to the target labels.

Original: Concretely, for an estimation e = { ( α n , µ n , Σ n  ) } Nn =1 ,the estimated parameter µ n,l indicates the belongingness of cluster n  to label l .
Revised: Concretely, for an estimation e = { ( α n ,µ n µ n µ n , Σ n Σ n Σ n ) } Nn =1 , µ n,l is the l -th element of µ n and indicates how much the n -th cluster of e belongs to label l .

Original: Therefore, we propose a cluster-label assignment score CLA( · ) , which represents the overall belongingness of a cluster-label assignment.
Revised: Therefore, we propose a cluster-label assignment score CLA( · ) , which represents the overall belongingness of a cluster-label assignment.

Original: Let the tuple k =
Revised: Let the tuple k =

Original: ( k 1 , k 2 , · · · , k N ) denote a cluster-label assignment, where k is a permutation of {1, 2, ..., N}.
Revised: ( k 1 , k 2 , · · · , k N ) denote a cluster-label assignment, where k is a permutation of {1, 2, ..., N}.

Original: It means that the n -th cluster is assigned to the label k n .
Revised: It means that the n -th cluster is assigned to the label k n .

Original: The assignment score CLA( · ) is defined as:
Revised: The assignment score CLA( · ) is defined as:

Original: [Equation]
Revised: [Equation]

Original: where µ n,k n indicates how much the n -th cluster of e belongs to label k n .

Original: Then it is transformed to a weighted bipartite matching problem between N clusters and N labels.
Revised: Then it is transformed to a weighted bipartite matching problem between N clusters and N labels.

Original: The optimal assignment is obtained by maximizing CLA( e, k ) :
Revised: The optimal assignment is obtained by maximizing CLA( e, k ) :

Original: [Equation]
Revised: [Equation]

Original: where K indicates the set of all assignment permutations.
Revised: where K indicates the set of all assignment permutations.

Original: In the worst case, this process requires N !
Revised: In the worst case, this process requires N !

Original: attempts to find the optimal assignment, which is time-consuming when N is large, thus we adopt
Revised: attempts to find the optimal assignment, which is time-consuming when N is large, thus we adopt

Original: Kuhn-Munkres algorithm (Kuhn, 1955) to accelerate it.
Revised: Kuhn-Munkres algorithm (Kuhn, 1955) to accelerate it.

Original: 3. E STIMATION S ELECTION B ASED ON C LUSTER -L ABEL A SSIGNMENT
Revised: 3. E STIMATION S ELECTION B ASED ON C LUSTER -L ABEL A SSIGNMENT

Original: The EM algorithm is empirically sensitive to the different initializations of GMM parameters.
Revised: The EM algorithm is empirically sensitive to the different initializations of GMM parameters.

Original: So we repeat the estimation multiple times with different random seeds.
Revised: So we repeat the estimation multiple times with different random seeds.

Original: Then we define a metric to evaluate how good these estimations are and select the best estimation.
Revised: Then we define a metric to evaluate how good these estimations are and select the best estimation.

Original: As CLA( e, k ∗ ) reflects the overall label belongingness of the optimal assignment of an estimation e , thus it can be used to evaluate estimations.
Revised: As CLA( e, k ∗ ) reflects the overall label belongingness of the optimal assignment of an estimation e , thus it can be used to evaluate estimations.

Original: Formally, we select the estimation e ∗ according to the assignment score of k ∗ as follows:
Revised: Formally, we select the estimation e ∗ according to the assignment score of k ∗ as follows:

Original: [Equation]
Revised: [Equation]

Original: where E is the set of estimations obtained by different initializations of GMM parameters.
Revised: where E is the set of estimations obtained by different initializations of GMM parameters.

Original: 3. I NFERENCE
Revised: 3. I NFERENCE

Original: After selecting the desired estimation e ∗ , we use GMM to make predictions instead of the conventional approach used in GPT (Brown et al., 2020).
Revised: After selecting the desired estimation e ∗ , we use GMM to make predictions instead of the conventional approach used in GPT (Brown et al., 2020).

Original: Due to the class-distribution discrepancy between the estimate set and the test set, we discard the mixing coefficient α n of each sub-distribution during inference.
Revised: Due to the class-distribution discrepancy between the estimate set and the test set, we discard the mixing coefficient α n of each sub-distribution during inference.

Original: For a test example, the LM prediction is x .
Revised: For a test example, the LM prediction is x .

Original: It will be assigned to the most likely cluster:
Revised: It will be assigned to the most likely cluster:

Original: [Equation]
Revised: [Equation]

Original: Finally, the predicted label is k ∗ ˜ n ( e ∗ ) , where the cluster-label assignment k ∗ ( e ∗ ) is obtained according to Equation (5).
Revised: Finally, the predicted label is k ∗ ˜ n ( e ∗ ) , where the cluster-label assignment k ∗ ( e ∗ ) is obtained according to Equation (5).

Original:  E XPERIMENTS 4. E XPERIMENTAL S ETUP
Revised:  E XPERIMENTS 4. E XPERIMENTAL S ETUP

Original: We evaluate five models from GPT-family including GPT-2-large (Radford et al., 2019) with 0.8B parameters, GPT-2-XL (Radford et al., 2019) with 1.5B parameters, GPT-neo (Black et al., 2021) with
Revised: We evaluate five models from GPT-family including GPT-2-large (Radford et al., 2019) with 0.8B parameters, GPT-2-XL (Radford et al., 2019) with 1.5B parameters, GPT-neo (Black et al., 2021) with

Original: As for the estimate set, it can be constructed by generating from LMs (Lu et al., 2021;
Revised: As for the estimate set, it can be constructed by generating from LMs (Lu et al., 2021;

Original: Wang et al., 2021; Meng et al., 2022; Ye et al., 2022) or sampling a light sub set of training examples but without golden labels.
Revised: Wang et al., 2021; Meng et al., 2022; Ye et al., 2022) or sampling a light sub set of training examples but without golden labels.

Original: For simplicity, we choose the later way to construct the estimate set and we further compare their differences in Section 4.5.
Revised: For simplicity, we choose the later way to construct the estimate set and we further compare their differences in Section 4.5.

Original: Moreover, the estimate set size is proportional to the number of classes of the task.
Revised: Moreover, the estimate set size is proportional to the number of classes of the task.

Original: For more details, please refer to Table 7 in Appendix.
Revised: For more details, please refer to Table 7 in Appendix.

Original: We use the k -means algorithm to initialize GMM parameters to accelerate the convergence.
Revised: We use the k -means algorithm to initialize GMM parameters to accelerate the convergence.

Original: The maximum iterations and the convergence threshold for each EM process are set to 100 and 1erespectively.
Revised: The maximum iterations and the convergence threshold for each EM process are set to 100 and 1erespectively.

Original: Moreover, we repeat the estimation multiple times with different random initializations to avoid getting stuck in local optima.
Revised: Moreover, we repeat the estimation multiple times with different random initializations to avoid getting stuck in local optima.

Original: It is worth noting that multiple repetitions bring little additional time consumption compared to the inference of GPT, we thus simply set it to 100 for all tasks.
Revised: It is worth noting that multiple repetitions bring little additional time consumption compared to the inference of GPT, we thus simply set it to 100 for all tasks.

Original: 4. E VALUATION P ROTOCOL
Revised: 4. E VALUATION P ROTOCOL

Original: We evaluate the proposed method on nine widely-used text-classification datasets including SST2 (Socher et al., 2013),
Revised: We evaluate the proposed method on nine widely-used text-classification datasets including SST2 (Socher et al., 2013),

Original: SST-5 (Socher et al., 2013), Subj (Pang & Lee, 2004), MR (Pang & Lee, 2005), AP (Zhang et al., 2015), DBPedia (Zhang et al., 2015), AGNews (Zhang et al., 2015),
Revised: SST-5 (Socher et al., 2013), Subj (Pang & Lee, 2004), MR (Pang & Lee, 2005), AP (Zhang et al., 2015), DBPedia (Zhang et al., 2015), AGNews (Zhang et al., 2015),

Original: RTE (Dagan et al., 2005), and TREC (Voorhees & Tice, 2000).
Revised: RTE (Dagan et al., 2005), and TREC (Voorhees & Tice, 2000).

Original: SST-2, SST-5, MR and AP are sentiment classification tasks.
Revised: SST-2, SST-5, MR and AP are sentiment classification tasks.

Original: RTE is a textual entailment recognition task and TREC is a text retrieval question classification task.
Revised: RTE is a textual entailment recognition task and TREC is a text retrieval question classification task.

Original: Subj and AGNews are subjectivity and topic classification tasks respectively, and DBPeida is an ontology classification task.
Revised: Subj and AGNews are subjectivity and topic classification tasks respectively, and DBPeida is an ontology classification task.

Original: We use the full validation set for evaluation except for AGNews, DBPedia and AP, for which we randomly sample 2000 test examples.
Revised: We use the full validation set for evaluation except for AGNews, DBPedia and AP, for which we randomly sample 2000 test examples.

Original: We compare P RO C A with the conventional approach used by GPT (Brown et al., 2020) and contextual calibration (Zhao et al., 2021).
Revised: We compare P RO C A with the conventional approach used by GPT (Brown et al., 2020) and contextual calibration (Zhao et al., 2021).

Original: Experiments are conducted under 0-shot, 1-shot, 4-shot and 8-shot scenarios.
Revised: Experiments are conducted under 0-shot, 1-shot, 4-shot and 8-shot scenarios.

Original: We fix the template format for each dataset (details of templates are shown in Table 6) and use the randomly sampled training examples as demonstrations.
Revised: We fix the template format for each dataset (details of templates are shown in Table 6) and use the randomly sampled training examples as demonstrations.

Original: We compute the average accuracy on validation set over five random seeds for each setting except for Bloom using 2 seeds.
Revised: We compute the average accuracy on validation set over five random seeds for each setting except for Bloom using 2 seeds.

Original: We conduct evaluation on 8 Tesla A100 GPUs for Bloom and Tesla V100 GPUs for other models.
Revised: We conduct evaluation on 8 Tesla A100 GPUs for Bloom and Tesla V100 GPUs for other models.

Original: 4. M AIN R ESULTS
Revised: 4. M AIN R ESULTS

Original: We report the mean and standard deviation of accuracy across five different random seeds for GPT-2XL, GPT-J and Bloom in Table 1.
Revised: We report the mean and standard deviation of accuracy across five different random seeds for GPT-2XL, GPT-J and Bloom in Table 1.

Original: The results of GPT-2-Large and GPT-neo are shown in Table 4 of
Revised: The results of GPT-2-Large and GPT-neo are shown in Table 9 of

Original: Appendix.
Revised: Appendix.

Original: From Table 1 and Table 4, we observe that P RO C A achieves, on average, a 13% absolute improvement compared to the conventional approach and a 6% absolute improvement over contextual calibration.
Revised: From Table 1 and Table 9, we observe that P RO C A achieves, on average, a 13% absolute improvement compared to the conventional approach and a 6% absolute improvement over contextual calibration.

Original: In some cases, the absolute improvement can be up to 40% and 20% respectively, like
Revised: In some cases, the absolute improvement can be up to 40% and 20% respectively, like

Original: GPT-J 0-shot on DBpedia and GPT-2-XL 8-shot on AGNews.
Revised: GPT-J 0-shot on DBpedia and GPT-2-XL 8-shot on AGNews.

Original: Results show that P RO C A maintains high effectiveness across different model sizes and few-shot scenarios, indicating its strong generalization ability.
Revised: Results show that P RO C A maintains high effectiveness across different model sizes and few-shot scenarios, indicating its strong generalization ability.

Original: Moreover, compared to the conventional approach, P RO C A achieves considerable improvements with lower variance across different prompts in most cases, which suggests that P RO C A can effectively calibrate the decision boundary for various prompts (as illustrated in Figure 2).
Revised: Moreover, compared to the conventional approach, P RO C A achieves considerable improvements with lower variance across different prompts in most cases, which suggests that P RO C A can effectively calibrate the decision boundary for various prompts (as illustrated in Figure 2).

Original: It also reflects that our estimation strategy is reliable and insensitive to different estimate sets, because of the low variance of P RO C A ’s zero-shot performance.
Revised: It also reflects that our estimation strategy is reliable and insensitive to different estimate sets, because of the low variance of P RO C A ’s zero-shot performance.

Original: We observe that the performance gain on Bloom is smaller than that on relatively small models.
Revised: We observe that the performance gain on Bloom is smaller than that on relatively small models.

Original: It also shows that Bloom’s performance variation under different prompts is also much smaller than other models.
Revised: It also shows that Bloom’s performance variation under different prompts is also much smaller than other models.

Original: It suggests that huge LMs have less suffering on the decision boundary deviation problem and are more robust to different prompts.
Revised: It suggests that huge LMs have less suffering on the decision boundary deviation problem and are more robust to different prompts.

Original: In addition, P RO C A seems invalid for GPT-2-XL on RTE.
Revised: In addition, P RO C A seems invalid for GPT-2-XL on RTE.

Original: We identify the reason is that the entailment recognition task is too challenging for relatively small models like GPT-2-XL and the output of LM on such challenging tasks is no more discriminative (same for GPT-2-Large, as shown in Table 4 in Appendix).
Revised: We identify the reason is that the entailment recognition task is too challenging for relatively small models like GPT-2-XL and the output of LM on such challenging tasks is no more discriminative (same for GPT-2-Large, as shown in Table 9 in Appendix).

Original: 4. E FFECTIVENESS A NALYSIS
Revised: 4. E FFECTIVENESS A NALYSIS

Original: We conduct more experiments to verify the effectiveness of P RO C A .
Revised: We conduct more experiments to verify the effectiveness of P RO C A .

Original: The experimental results are the average accuracy of GPT-2-XL conditioned on 5 different 4-shot prompts unless otherwise specified.
Revised: The experimental results are the average accuracy of GPT-2-XL conditioned on 5 different 4-shot prompts unless otherwise specified.

Original: [Table]
Revised: [Table]

Original: P RO C A is robust under demonstration perturbations.
Revised: P RO C A is robust under demonstration perturbations.

Original: Previous works (Zhao et al., 2021; Lu et al., 2021) have noticed that the order of training examples has significant effects on the performance of few-shot demonstrations.
Revised: Previous works (Zhao et al., 2021; Lu et al., 2021) have noticed that the order of training examples has significant effects on the performance of few-shot demonstrations.

Original: In this part, we evaluate our prototypical calibration conditioned on nine 8-shot prompts with different class proportions for SST-2, and show the accuracy of twelve randomly sampled orderings for each proportion in Figure 6.
Revised: In this part, we evaluate our prototypical calibration conditioned on nine 8-shot prompts with different class proportions for SST-2, and show the accuracy of twelve randomly sampled orderings for each proportion in Figure 6.

Original: We find that the original GPT-2-XL with some specific compositions and permutations achieves high performance while some others are close to random guesses similar to the findings of Lu et al.
Revised: We find that the original GPT-2-XL with some specific compositions and permutations achieves high performance while some others are close to random guesses similar to the findings of Lu et al.

Original: Contextual calibration can improve the performance in most cases but is still sensitive to the orderings.
Revised: Contextual calibration can improve the performance in most cases but is still sensitive to the orderings.

Original: However, P RO C A is significantly superior to the others and keeps an extremely low variance across different permutations, indicating the non-sensitiveness to the class proportion and permutation.
Revised: However, P RO C A is significantly superior to the others and keeps an extremely low variance across different permutations, indicating the non-sensitiveness to the class proportion and permutation.

Original: It is also shown that although the class-balanced prompts tend to have higher performance, there are some exceptions that the prompt with all negative samples is the most performant one for both the conventional approach and contextual calibration.
Revised: It is also shown that although the class-balanced prompts tend to have higher performance, there are some exceptions that the prompt with all negative samples is the most performant one for both the conventional approach and contextual calibration.

Original: We think that it is GPT-2-XL’s intrinsic bias to the positive class that leads to the counter-intuitive results.
Revised: We think that it is GPT-2-XL’s intrinsic bias to the positive class that leads to the counter-intuitive results.

Original: P RO C A is robust to class imbalance.
Revised: P RO C A is robust to class imbalance.

Original: Due to the unavailability of the labels of the estimate examples, P RO C A may suffer the problem of class imbalance.
Revised: Due to the unavailability of the labels of the estimate examples, P RO C A may suffer the problem of class imbalance.

Original: We construct nine estimate sets with different imbalance levels for SST-2 and Subj by controlling the proportion of positive examples in the sampled set.
Revised: We construct nine estimate sets with different imbalance levels for SST-2 and Subj by controlling the proportion of positive examples in the sampled set.

Original: Then we evaluate P RO C A and contextual calibration on them.
Revised: Then we evaluate P RO C A and contextual calibration on them.

Original: The experimental results in Figure 5 show that the estimate set’s class imbalance level affects the performance of
Revised: The experimental results in Figure 5 show that the estimate set’s class imbalance level affects the performance of

Original: P RO C A to some extent and the class-balanced estimate set can lead to higher accuracy.
Revised: P RO C A to some extent and the class-balanced estimate set can lead to higher accuracy.

Original: As described in Section 3.1, standard GMM estimates the weight of each cluster, which reflects the proportion of different classes in the estimate set.
Revised: As described in Section 3.1, standard GMM estimates the weight of each cluster, which reflects the proportion of different classes in the estimate set.

Original: Owing to our "weights-cutting" operation, the problem of class imbalance has much less negative impact on P RO C A , which even with an extremely classimbalanced estimate set surpasses contextual calibration on both SST-2 and Subj.
Revised: Owing to our "weights-cutting" operation, the problem of class imbalance has much less negative impact on P RO C A , which even with an extremely classimbalanced estimate set surpasses contextual calibration on both SST-2 and Subj.

Original: Besides, the absolute advancement for the class-balanced estimate set can reach 20% and 15% respectively.
Revised: Besides, the absolute advancement for the class-balanced estimate set can reach 20% and 15% respectively.

Original: [Table]
Revised: [Table]

Original: 4. A BLATION S TUDIES
Revised: 4. A BLATION S TUDIES

Original: Comparison between different estimate set construction methods.
Revised: Comparison between different estimate set construction methods.

Original: There are two ways to construct the estimate set.
Revised: There are two ways to construct the estimate set.

Original: One is using light unlabeled examples from the training set, which is simple and convenient.
Revised: One is using light unlabeled examples from the training set, which is simple and convenient.

Original: The other is utilizing the generation ability of LMs to construct unlabeled dataset (Lu et al., 2021;
Revised: The other is utilizing the generation ability of LMs to construct unlabeled dataset (Lu et al., 2021;

Original: Wang et al., 2021;
Revised: Wang et al., 2021;

Original: Meng et al., 2022; Ye et al., 2022).
Revised: Meng et al., 2022; Ye et al., 2022).

Original: For the generation method, we follow Lu et al.
Revised: For the generation method, we follow Lu et al.

Original: (2021) to generate diverse estimate examples based on the various permutations of demonstrations.
Revised: (2021) to generate diverse estimate examples based on the various permutations of demonstrations.

Original: Specifically, we only use two labeled examples per category as demonstrations for generation except for DBPedia(one labeled example per category) and all these labeled examples are not involved in the evaluation.
Revised: Specifically, we only use two labeled examples per category as demonstrations for generation except for DBPedia(one labeled example per category) and all these labeled examples are not involved in the evaluation.

Original: The size of generated estimate set maintains consistent with the estimate set sampled from the training set and the LM used for generation is the same as that in the evaluation.
Revised: The size of generated estimate set maintains consistent with the estimate set sampled from the training set and the LM used for generation is the same as that in the evaluation.

Original: The 4-shot and 8-shot experimental results are shown in Table 2 and the 0-shot and 1-shot results are shown in Table 5 of Appendix.
Revised: The 4-shot and 8-shot experimental results are shown in Table 2 and the 0-shot and 1-shot results are shown in Table 5 of Appendix.

Original: We observe that P RO C A greatly outperforms the original LM whether using unlabeled data generated by LM or randomly sampled from the training set, which validates the effectiveness of P RO C A under a restricted setting.
Revised: We observe that P RO C A greatly outperforms the original LM whether using unlabeled data generated by LM or randomly sampled from the training set, which validates the effectiveness of P RO C A under a restricted setting.

Original: It also shows that P RO C A -t performs slightly better than P RO C A -g
Revised: It also shows that P RO C A -t performs slightly better than P RO C A -g

Original: and we speculate that it is due to the lower quality of the unlabeled data generated by LM.
Revised: and we speculate that it is due to the lower quality of the unlabeled data generated by LM.

Original: A relatively small-scale estimate set is sufficient for
Revised: A relatively small-scale estimate set is sufficient for

Original: P RO C A .
Revised: P RO C A .

Original: In Figure 7, we evaluate P RO C A with ten different estimate set sizes across five datasets.
Revised: In Figure 7, we evaluate P RO C A with ten different estimate set sizes across five datasets.

Original: We report the average accuracy over five randomly sampled estimate sets, conditioning on the same 4-shot prompt in each setting.
Revised: We report the average accuracy over five randomly sampled estimate sets, conditioning on the same 4-shot prompt in each setting.

Original: We observe that increasing the scale of the estimate set within a certain small range can greatly improve the classification accuracy and reduce the variance.
Revised: We observe that increasing the scale of the estimate set within a certain small range can greatly improve the classification accuracy and reduce the variance.

Original: However, a larger estimate set can hardly bring further improvement, which indicates that a small estimate set can support P RO C A to be optimal.
Revised: However, a larger estimate set can hardly bring further improvement, which indicates that a small estimate set can support P RO C A to be optimal.

Original: It also shows that P RO C A has acceptable performances with just several estimate examples on
Revised: It also shows that P RO C A has acceptable performances with just several estimate examples on

Original: SST-2, MR, and Subj, which even surpasses both the conventional approach and contextual calibration.
Revised: SST-2, MR, and Subj, which even surpasses both the conventional approach and contextual calibration.

Original: Estimation selection according to assignment
Revised: Estimation selection according to assignment

Original: score is useful to P RO C A .
Revised: score is useful to P RO C A .

Original: The standard estimation of GMM aims to maximize the likelihood of all observations and select the optimal estimated parameters among multiple repetitions.
Revised: The standard estimation of GMM aims to maximize the likelihood of all observations and select the optimal estimated parameters among multiple repetitions.

Original: We argue that the estimation with maximum likelihood is not consistently beneficial to P RO C A especially in multi-classes tasks, because there is no supervision to force the predictions to be assigned to their inclined classes during the estimation procedure.
Revised: We argue that the estimation with maximum likelihood is not consistently beneficial to P RO C A especially in multi-classes tasks, because there is no supervision to force the predictions to be assigned to their inclined classes during the estimation procedure.

Original: We propose to determine estimation according to the assignment score (as described in Section 3.3), which can theoretically select the most performant estimation for P RO C A .
Revised: We propose to determine estimation according to the assignment score (as described in Section 3.3), which can theoretically select the most performant estimation for P RO C A .

Original: We compare P RO C A with the two strategies for GPT-2-XL and GPT-J respectively.
Revised: We compare P RO C A with the two strategies for GPT-2-XL and GPT-J respectively.

Original: The experimental results are shown in Table 3.
Revised: The experimental results are shown in Table 3.

Original: It indicates that our determination strategy can achieve more stable improvements on AGNnews and DBPedia regardless of model size.
Revised: It indicates that our determination strategy can achieve more stable improvements on AGNnews and DBPedia regardless of model size.

Original: [Table]
Revised: [Table]

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Understanding In-context Learning with Language Models.

Original: Xie et al.

Original: (2022) provide a theoretical perspective to understand in-context learning and cast it as implicit Bayesian inference, where LM can infer latent concepts across demonstrations.

Original: Shin et al.

Original: (2022) observe that corpora sources play a crucial rule in the emergence of in-context learning.

Original: Razeghi et al.

Original: (2022) also show that low-order co-occurrence statistics in pretraining corpora can significantly impact the few-shot performance.

Original: Other works analyze the role of natural language instructions and demonstrations in prompts, which are considered crucial for the success of in-context learning.

Original: Liu et al.

Original: (2021) construct more semantically-similar demonstrations during evaluation.

Original: Min et al.

Original: (2022) demonstrate that such success is mainly attributed into three factors, i.e., label space, input distribution and the prompt format.

Original: Instability of Few-shot Learning with Language Models.
Revised: Instability of Few-shot Learning with Language Models.

Original: It has been recognized that the few-shot performance of language models is unstable under different in-context scenarios.
Revised: It has been recognized that the few-shot performance of language models is unstable under different in-context scenarios.

Original: Language models are prone to predict some specific labels due to the intrinsic bias or demonstration permutations (Zhao et al., 2021;
Revised: Language models are prone to predict some specific labels due to the intrinsic bias or demonstration permutations (Zhao et al., 2021;

Original: Lu et al., 2021).
Revised: Lu et al., 2021).

Original: Lu et al.
Revised: Lu et al.

Original: (2021) demonstrate LM’s sensitiveness to the order of few-shot demonstrations, and introduced an Entropy-based metric to select the most performant prompts.
Revised: (2021) demonstrate LM’s sensitiveness to the order of few-shot demonstrations, and introduced an Entropy-based metric to select the most performant prompts.

Original: Zhao et al.
Revised: Zhao et al.

Original: (2021) attribute the instability to three biases of prompts, including majority bias, recency bias and common token bias, and proposed a contextual calibration approach.
Revised: (2021) attribute the instability to three biases of prompts, including majority bias, recency bias and common token bias, and proposed a contextual calibration approach.

Original: However, the selected content free test inputs can not precisely reflect the bias of models and lead to the problem of over-correction or under-correction.
Revised: However, the selected content free test inputs can not precisely reflect the bias of models and lead to the problem of over-correction or under-correction.

Original: On the contrary, we adaptively provide the classification criterion according to the text inputs’ overall prediction distribution, and completely calibrate the bias introduced by models and prompts.
Revised: On the contrary, we adaptively provide the classification criterion according to the text inputs’ overall prediction distribution, and completely calibrate the bias introduced by models and prompts.

Original: C ONCLUSION AND L IMITATION
Revised: Improving and Understanding In-context Learning with Language Models. Due to the instability, prior efforts propose various methods to improve the in-context learning performance. Holtzman et al. (2021) explores the surface form competition problem in zero-shot models and proposes Domain

Revised: Conditional Pointwise Mutual Information to reweigh the answer scores.

Revised: Min et al.

Revised: (2021) introduces a noisy channel approach that computes the conditional probability of the input given the label, which boosts improvements with lower variance and higher worst-case accuracy.

Revised: Moreover, Liu et al.

Revised: (2021) focus on prompt engineering to construct more semantically-similar demonstrations.

Revised: To the best of our knowledge, we are the first to study the intrinsic reason for the instability of in-context learning from the perspective of decision boundary and propose prototypical calibration to improve it.

Revised: Another line of work aim to understand how in-context learning works through casting it as implicit

Revised: Bayesian inference (Xie et al., 2022), analyzing corpora sources and statistics (Shin et al., 2022;

Revised: Razeghi et al., 2022) and learning which aspects of demonstrations contribute most to the downstream performance (Min et al., 2022).

Revised:  C ONCLUSION AND L IMITATION

Original: To our analysis, decision boundary is of critical importance to the performance of few-shot demonstrations and the traditional decision boundary leads to the fragility of prompting LMs.
Revised: To our analysis, decision boundary is of critical importance to the performance of few-shot demonstrations and the traditional decision boundary leads to the fragility of prompting LMs.

Original: We propose prototypical calibration to adaptively learn a more robust decision boundary.
Revised: We propose prototypical calibration to adaptively learn a more robust decision boundary.

Original: Experiments show that the calibrated decision boundary is effective across various prompt templates, class proportions and permutations.
Revised: Experiments show that the calibrated decision boundary is effective across various prompt templates, class proportions and permutations.

Original: We achieve on average a 13% absolute improvement across different sizes of pretrained language models on nine popular text classification tasks.
Revised: We achieve on average a 13% absolute improvement across different sizes of pretrained language models on nine popular text classification tasks.

Original: A limitation of our method is that it is not applicable for tasks whose label space is open-ended since a fixed label space is necessary for estimating prototypical clusters.
Revised: A limitation of our method is that it is not applicable for tasks whose label space is open-ended since a fixed label space is necessary for estimating prototypical clusters.

Original: Furthermore, our method is designed for in-context learning on individual downstream tasks, it fails to calibrate the inherent bias of language models like gender and occupation bias.
Revised: Furthermore, our method is designed for in-context learning on individual downstream tasks, it fails to calibrate the inherent bias of language models like gender and occupation bias.

Original: For future work, we would like to extend our method to the tasks with open-ended answer space, such as generative question-answering and text summarization tasks.
Revised: For future work, we would like to extend our method to the tasks with open-ended answer space, such as generative question-answering and text summarization tasks.

Original:  R EPRODUCIBILITY
Revised:  R EPRODUCIBILITY

Original: We developed our code based on the open-source work Conctextul Calibration 1 and the code was attached in the supplementary material.
Revised: We developed our code based on the open-source work Conctextul Calibration 1 and the code was attached in the supplementary material.

Original: Moreover, we used the same hyperparameters for all of our experiments, which we specify in the section 4.1 and
Revised: Moreover, we used the same hyperparameters for all of our experiments, which we specify in the section 4.1 and

Original: Appendix A
Revised: Appendix A

Original: A E XPERIMENTAL D ETAILS AND A DDITIONAL R ESULTS
Revised: A E XPERIMENTAL D ETAILS AND A DDITIONAL R ESULTS

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: B C OMPARISONS AMONG USING LOGITS , PROBABILITY , AND LOG - PROBABILITY FOR P RO C A .
Revised: B C OMPARISONS AMONG USING LOGITS , PROBABILITY , AND LOG - PROBABILITY FOR P RO C A .

Original: We evaluate the performance of P RO C A based on three commonly used model output formalizations including logits, probability, and log-probability on SST-2, AGNews, and AP for GPT-2-XL.
Revised: We evaluate the performance of P RO C A based on three commonly used model output formalizations including logits, probability, and log-probability on SST-2, AGNews, and AP for GPT-2-XL.

Original: The experimental results are plotted in Figure 8.
Revised: The experimental results are plotted in Figure 8.

Original: We find that P RO C A with log-probability consistently outperforms that with probability, especially for AGNews.
Revised: We find that P RO C A with log-probability consistently outperforms that with probability, especially for AGNews.

Original: We identify that the log operation makes the predictions more fitted to Gaussian distribution and more discrepant.
Revised: We identify that the log operation makes the predictions more fitted to Gaussian distribution and more discrepant.

Original: Although P RO C A with logits seems to have superior performance when few-shot demonstrations are provided, it degrades severely at zero-shot setting.
Revised: Although P RO C A with logits seems to have superior performance when few-shot demonstrations are provided, it degrades severely at zero-shot setting.

Original: It suggests that output embedding normalization over label space is necessary to P RO C A , because the output is more likely biased to the tokens outside the label space when no prompts are accessed, which is consistent with the conclusion of Min et al.
Revised: It suggests that output embedding normalization over label space is necessary to P RO C A , because the output is more likely biased to the tokens outside the label space when no prompts are accessed, which is consistent with the conclusion of Min et al.

Original: [Table]
Revised: [Table]

Original: Question:
Revised: Question:

Original: Is the sentiment of the above review
Revised: Is the sentiment of the above review

Revised: [Table]

Revised: [Table]

Revised: C P RO C A FOR B IDIRECTIONAL L ANGUAGE M ODELS .

Revised: To validate the effectiveness of P RO C A for bidirectional language models

Original: [Table]
Revised: [Table]

Revised: we conduct the evaluation on RoBERTa-Large (Liu et al., 2019) with 355M parameters across the nine text classification tasks.

Revised: The templates for RoBERTa are constructed by replacing the label symbols in the templates in Tablewith [MASK] symbols.

Revised: We compare the 0- and 8-shot average performance of RoBerta-Large and

Revised: P RO C A over 5 random seeds and the detailed results are shown in Table 9.

Revised: We can see that P RO C A greatly outperforms RoBERTa-Large on all tasks except for 8-shot on SST-2.

Revised: Therefore, we suggest that P RO C A is also effective for Bidirectional Language Models.