The Unreliability of Explanations in Few-Shot In-Context Learning
Anonymous Author(s)
AffiliationAddress email
Abstract
Does prompting a large language model like GPT-3 with explanations improve incontext learning?
We study this question specifically on two NLP tasks that involvereasoning over text, namely question answering and natural language inference.
For these tasks, we find that including explanations GPT-3’s prompt and havingthe model generate them only mildly improves accuracy over standard few-shotlearning, contrary to recent results on symbolic reasoning tasks [29, 41].
Moreover,explanations generated by GPT-3 may not entail the predictions nor be factuallygrounded in the input, even on simple tasks with extractive explanations.
However, these flawed explanations can still be useful as a way to verify GPT-3’spredictions post-hoc.
Through analysis in three settings, we show that explanationsjudged as good by humans—those that are logically consistent with the input andthe prediction—usually cooccur with more accurate predictions.
Following theseobservations, we present a framework for calibrating model predictions basedon the reliability of the explanations.
We train calibrators using automaticallyextracted scores that approximately assess the reliability of explanations, whichhelps improve performance across three different datasets.
Introduction
Recent breakthroughs in pre-training have empowered large language models to learn NLP tasks fromfewer and fewer examples.
In-context learning, learning a new task from just a few training examplesin a prompt without updating the model’s parameters, has started to show promising performance forvery large language models like GPT-3 [3].
However, this learning process is still poorly understood:models are biased by the order of examples they are shown [51] and may not leverage the instructionsor even the labels of the few-shot examples in the ways we expect [28, 40].
It is difficult to investigatethese issues or explain the predictions of in-context learning when existing tools for interpretingmodel predictions have high computational cost [34] or require access to gradients [37, 39].
One appealing way to gain more insight into predictions obtained through in-context learning is tolet the language model “explain itself” [29, 41, 9, 26, 22].
In addition to input-label pairs, one canprompt the language model with explanations for input-label pairs and expect the model to generatean explanation corresponding to the prediction it gives (Figure 1).
Prompting with explanationsintroduces much richer information compared to using labels alone, which might guide the in-contextlearning process and allow the model to leverage more information about the examples.
In this work, we closely investigate the nature of the explanations that GPT-3 generates and whetherthey can really improve few-shot in-context learning, specifically for textual reasoning tasks.
Recentprior work that finds success with this approach largely targets symbolic reasoning tasks with a very
Calibrator
P r o m p t
Train Example
T e s t E x a m p l e
E x p l a n a t i o n + L a b e l
O u t p u t
The prediction is incorrect.
The explanation is not factual with respect to the context.
G P T -
[Figure]
A: First, Crestfallen's artwork is done by Yelena Yemchuk.
Second, Yelena Yemchuk is a Croatian professional photographer.
The answer is Croatian.
Crestfallen is
[Figure]
track on The Smashing Pumpkins’
[Figure]
, Adore
[Figure]
s artwork is
[Figure]
Yelena Yemchuk.
Johnny McDaid
[Figure]
Croatian professional photographer
[Figure]
Yemchuk
[Figure]
Ukrainian professional photographer.
Q
[Figure]
Crestfallen’s
[Figure]
nationality
[Figure]
different structure, such as math word problem solving [29, 41].
We show that explanations onlymildly improve performance when plugged into the prompt (Figure 1) across three different datasetsspanning QA and NLI.
Surprisingly, we find that the explanations generated by GPT-3 are unreliable , even for a very simplesynthetic dataset.
Specifically, we check the explanations along two axes: factuality , whether theexplanation is correctly grounded in the input, and consistency , whether the explanation entails thefinal prediction.
On realistic datasets, GPT-3 tends to generate consistent explanations that accountfor the predictions, but the explanations may not be factually grounded in the input context, as asshown in Figure 1.
Furthermore, our analysis suggests an unreliable explanation more likely indicatesa wrong prediction compared to a reliable explanation.
Despite GPT-3’s failures here, we can still benefit from model-generated explanations by using themfor calibration, given the connection between unreliable explanations and incorrect predictions.
Ifwe are able to automatically assess the reliability of an explanation, we can allow GPT-3 to return anull answer when its explanation is unreliable.
Unfortunately, there is no automated way to perfectlyassess the reliability, but we can extract features that approximately reflect it.
We use these features tocalibrate GPT-3’s predictions, and successfully improve the in-context learning performance acrossall the datasets.
In summary, our main findings are: 1) Simply plugging explanations into the prompt does notsignificantly boost the in-context learning performance for textual reasoning tasks.
2) GPT-3generates mostly consistent explanations, but these explanations might not be factually grounded inthe inputs.
The reliability of an explanation can serve as an indicator for the correctness of thecorresponding prediction.
4) Using features that can approximate the reliability of explanations, wesuccessfully use explanations to improve the in-context learning performance across all tasks.
Does Prompting with Explanations Improve In-Context Learning?
In this paper we specifically focus on tasks involving reasoning over natural language.
These aretasks where explanations have been traditionally studied [5, 32], but which are more complex thantasks like sentiment which are well explained by extractive rationales [49, 10].
We experiment ontwo tasks, reading comprehension question answering (QA) and natural language inference (NLI), onthree English-language datasets.
For each dataset, we create a test set with 250 examples.
2.1 Datasets
Synthetic Multi-hop QA (S YNTH )
In order to have a controlled setting where we completelyunderstand whether explanations are factual and consistent with the answer, we create a syntheticmulti-hop QA dataset.
As in Figure 2, each example in our synthetic dataset asks a bridge question(using the terminology of [45]) over a context consisting of supporting facts paired with controlleddistractors.
This dataset is carefully designed to avoid spurious correlations, giving us full understand-
S YN T
H
Context: Christopher agrees with Kevin.
Tiffany agrees with Matthew.
Mary hangs out with Danielle.
James hangs out with Thomas.
Kevin is a student.
Matthew is a plumber.
Danielle is a student.
Thomas is a plumber.
Question: Who hangs out with a student?
Answer: Mary
Explanation: Danielle is a student and Mary hangs out with Danielle.
E - S N
L I
Premise: A toddler in a green jersey is being followed by a wheelchair bound woman in a red sweater past a wooden bench.
Hypothesis: A toddler is walking near his wheelchair bound grandmother.
Label:
Neither Explanation: the woman may not be his grandmother.
ing over the correct reasoning process as well as the explanation for every example, which naturallyconsists of the two supporting sentences.
Refer to Appendix B for full details of this dataset.
Adversarial HotpotQA (A DV H OTPOT )
We also test on the English-language Adversarial HotpotQA dataset [45, 19] (license: MIT).
We use the adversarially augmented version since GPT-3achieves high performance on the distractor setting of the original dataset.
We make a challenging setof examples by balancing sets of questions on which GPT-3 makes correct and incorrect predictions.
The context of each question includes two ground truth supporting paragraphs and two adversarialparagraphs.
Full details of preprocessing the A DV H OTPOT dataset can be found in Appendix C.
For A DV H OTPOT , we manually annotated explanations for the training examples.
Figure 1 shows anexample of this explanation, highlighted in orange.
We could use the supporting sentences as theexplanations, but we found they are usually too verbose and not sufficient, e.g., with anaphors thatresolve outside of the supporting sentences.
Therefore, we manually annotate a set of explanationswhich clearly describe the reasoning path for each question.
E-SNLI
E-SNLI [5] is an English-language classification dataset (license: MIT) commonly usedto study explanations.
Shown in Figure 2, each example consists of a premise and a hypothesis, andthe task is to classify the hypothesis as entailed by, contradicted by, or neutral with respect to thepremise.
As a notable contrast to the other datasets, the explanations here are more abstract naturallanguage written by human annotators, as opposed to mostly constructed from extracted snippets ofcontext.
2.2 Baselines
We study the effectiveness of plugging in explanations by comparing the in-context learning performance of prompting with or without explanations.
Prompting without explanations resembles thestandard few-shot in-context learning approach ( Few-Shot ).
To incorporate explanations into theprompt, we consider the following two most commonly used paradigms:
Explain-then-Predict (E-P) which prepends an explanation before the label (Figure 1).
The languagemodel is expected to generate an explanation first followed by the prediction.
The prompting style ofpast work involving computational traces can be categorized into this paradigm, including Nye et al.[29] and Wei et al.
This approach is also called a pipeline model in other literature on trainingmodels using explanations [18, 44].
Predict-then-Explain (P-E) which generates the explanation after the prediction.
Unlike E-P, thepredicted explanation does not influence the predicted label, since we use greedy inference and theexplanation comes afterwards.
However, the explanations in the prompt still impact the predictions.
2.3 Setup
For few-shot learning, we use roughly the maximum allowed shots in the prompt that can fit the lengthlimit of GPT-3, which is 16 for S YNTH , 6 for A DV H OTPOT , and 32 for E-SNLI, respectively.
Because the results of in-context learning vary with the examples presented in the input prompt,for each dataset, we randomly sample 5 groups of training shots, and report the mean and standard2 This dataset is by the bAbI dataset [42].
In our preliminary experiments with some of the more complex bAbI tasks, poor performance GPT- our results on S YNTH , both with
Zhao et al.
[51] that focuses on improving performance in the 1-4-shot setting; by using more data we achieve much stronger results on our tasks.
N o n f a c t u a l
Pedro Rubens!
The individual chapters were published into 64 "tank¯obon" by Kodansha.
Y¯oko Sh¯oji (born 4 June 1950, in Mobara, Chiba) is a Japanese manga artist.
She is best known for writing "Seito Shokun!
Mulder Scully!
The individual chapters were published into 14 "tank¯obon" by Kodansha.
Seito Shokun!
The individual chapters were published into 24 "tank¯obon" by Kodansha between.
Q: How many chapters does Y¯oko Sh¯oji’s most famous manga have?
A: First, Y¯oko Sh¯oji’s most famous manga is "Seito Shokun!".
Second, "Seito Shokun!" has 64 chapters.
The answer is 64.
I n c o n s i s t e n t
Tim Minchin (December 29, 1808 July 31, 1875) was the President of the United States.
Andrew Johnson (December 29, 1808 July 31, 1875) was the President of the United States.
George Andrew Atzerodt (June 12, 1835 – July 7, 1865) was a conspirator, with John Wilkes Booth.
Jesse Andrew Williams (June 12, 1835 – July 7, 1865) was a conspirator, with John Wilkes Booth.
Q: Who was older, George Atzerodt or Andrew Johnson?
A: First, George Atzerodt was born on June 12, 1835.
Second, Andrew Johnson was born on December 29, 1808.
The answer is George Atzerodt.
deviation of the results (subscript).
All our experiments use the 175B GPT-3 [3] Instruct series API(text-davinci-001), the strongest available model at the time of our experiments.
The completionis obtained through greedy decoding (temperature set to be 0).
Our prompt formats follow those in
Brown et al.
The explanations are inserted before/after the prediction with conjunction wordslike because .
Please refer to Appendix A for full prompts.
2.4 Results
without and with explanations.
Using explanations mildly improves performance.
S YNTH
A DV H OTPOT E
[Table]
SNLI
F EW
[Table]
S HOT
[Table] [Table]
[Table] [Table]
[Table] [Table]
E-P 58.
[Table]
[Table]
[Table]
P-
[Table] [Table] [Table]
[Table]
In general, using explanations mildly improvesthe performance for the text reasoning tasks,as show in Table 1.
On the two QA tasks,
S YNTH and A DV H OTPOT , E-P improves theperformance from 54.8 to 58.5 and 56.8 to 59.4,respectively.
On E-SNLI, P-E outperforms
F EW -S HOT by 2.6, whereas E-P substantiallylags F EW -S HOT .
There is no single winner between the two paradigms of plugging in explanations; choosing the most effective way is task-specific.
Our results do not suggest immediate strong improvements from incorporating explanations, even forthe simple synthetic dataset, contradicting recent prior work.
This can be attributed to the differencebetween the tasks we study.
The tasks that receive significant benefits from using explanations in
Nye et al.
[29] and Wei et al.
[41] are all program-like (e.g., integer addition and program execution),whereas the tasks in this work emphasize textual reasoning grounded in provided inputs.
In fact, in
Wei et al.
[41] and Chowdhery et al.
[9], explanations only show mild benefit on open-domain QAtasks like StrategyQA [14] that are closer to our setting.
Can GPT-3 Generate Factual and Consistent Explanations?
Prompting GPT-3 with explanations and generating explanations does not lead to much higherperformance on our tasks.
But what about the quality of the model-generated explanations themselves?
We assess the reliability of the explanations for the three datasets, measured in terms of two aspects.
Factuality refers to whether a generated explanation is faithfully grounded in the correspondinginput context (context for QA and premise/hypothesis pair for NLI).
A factual explanation should notcontain hallucinations that contradict the context.
See Figure 3 for a nonfactual explanation.
Consistency measures if the explanation entails the prediction.
Our concept of consistency resemblesplausibility as described in the literature [18], in that we assess whether the prediction follows fromthe explanation as perceived by a human .
See Figure 3 for an inconsistent explanation.
For S YNTH , we uses rules to automatically judge whether an explanation is factual and consistent.
For A DV H OTPOT and E-SNLI, the authors manually inspected the explanations and annotated
[Table]
[Figure]
[Table]
[Figure]
them for these two characteristics (Cohen’s Kappa between these annotations: 0.894; more detailsin Appendix D).
Note for each setting, the results are based on the explanations and predictionsobtained with a single set of training shots.
We only show the results of P-E on E-SNLI, as E-P issubstantially worse here.
Results We summarize the results in Table 2.
We only report consistency on E-SNLI, as theexplanations for E-SNLI often require some external commonsense knowledge which cannot beeasily grounded in the inputs or judged as true or false (examples in Appendix F).
The results suggesta disconnect between the model predictions and the “reasoning” in explanations: even though usingexplanations improves GPT-3’s performance, the generated explanations are unreliable , even for thestraightforward synthetic setting.
Overall, GPT-3 tends to generate consistent explanations (>90%for all three datasets with the right prompt structure), but the explanations are less likely to be factual,which is concerning as they can deceive a user of the system into believing the model’s answer.
3.1 Reliability of Explanations and Prediction Accuracy
GPT-3 may hallucinate problematic explanations, but this could actually be advantageous if itgives us a way of spotting when the model’s “reasoning” has failed.
We investigate the connectionbetween the reliability of an explanation and the accuracy of a prediction, and ask whether a reliableexplanation indicates an accurate prediction.
(This resembles the linguistic calibration of Mielke et al.[27], but using a different signal for calibration.)
As shown in the right section of Table 2, accuracy and factuality/consistency are typically correlated,especially factuality.
By knowing whether an explanation is factual, we can guess the model’s prediction a high fraction of the time (Accuracy = Factuality).
A nonfactual explanation very likely (89.6%)means an incorrect prediction on the S YNTH dataset.
On A DV H OTPOT , factuality and the model’sprediction correspond 80.0% of the time, substantially surpassing the prediction accuracy itself.
Weshow fractions of correct and incorrect predictions when the explanations are factual/nonfactual andconsistent/inconsistent in Figure 4 for two of our settings.
Factual explanations are much more likelypaired with correct predictions compared to nonfactual explanations.
Consistency also connects tothe accuracy, but is an inferior indicator compared to factuality in general (Table 2).
Calibrating In-Context Learning using Explanations
From Section 3.1, we see that a human oracle assessment of the factuality of an explanation could beof substantial use for calibrating the corresponding prediction.
Can we automate this process?
We show how to achieve this goal on the perfectly controlled S YNTH dataset (Section 4.1).
Onour other two datasets, we use surface lexical matching to approximate semantic matching andgive real-valued scores approximately reflecting factuality.
Following past work on supervisedcalibration [20, 7, 47], we can learn a calibrator that tunes the probabilities of a prediction based onthe score of its explanation (Section 4.2).
We show such a calibrator can be trained with a handful of examples beyond those used for in-context learning and successfully improve the in-context learningperformance on realistic datasets.
4.1 Motivating Example: Improving S YNTH Dataset
We first show how post-hoc calibration functions in the controlled S YNTH setting, where we cansimply check the factuality of an explanation.
Since the generated explanation always followsthe format “ B is [profession] and A [verb] B. ” (example in Figure 2), we can split theexplanation into two sentences.
The explanation is factual if and only if both of the sentences exactlymatch one of the sentences in the context.
We use the assessment to improve the performance of P-E for S YNTH , where a nonfactual explanationtypically indicates an incorrect prediction.
This gives us a way to reject presumably incorrect answers.
Specifically, we iterate through the top 5 candidate answers (restricted by the API) given by GPT-3and reject any answer-explanation pair if the explanation is nonfactual until we find a factual one.
Thisprocedure dramatically improves the accuracy from 54.8% to 79.2%.
Note that this S YNTH datasetwithout any possible reasoning shortcuts is a challenging task.
For reference, neither R O BERT A [25]and D E BERT A [17] finetuned with 16 examples can achieve an accuracy surpassing 50%.
With thehelp of the explanations and the checking procedure, we can use GPT-3 to achieve strong resultsusing few-shot learning.
4.2 Learning-based Calibration Framework
Framework We now introduce the framework that can leverage the factuality assessment of anexplanation to calibrate a prediction.
Let p be the vector of predicted probabilities associated witheach class label in NLI (or the probability score of predicted answer in QA).
Let v be a scalarvalue extracted from the explanation to describe the factuality.
Then, we can adjust the probabilitiesaccordingly using a linear model:
[Equation]
where ˆ p is the tuned probabilities.
Our calibration framework is extended from classical calibration methods [30, 15, 51], which apply an affine transformation on the probabilities alone:ˆ p = softmax( W p + b ) .
In contrast, we use an additional factor v in calibration to incorporate thefactuality assessment of the explanation.
There are a small number of parameters ( W and b ) that need to be trained in such a calibrationframework.
We will rely on a few more examples in addition to the shots we use in the prompt to trainthe calibrator.
Specifically, we use the prompt examples to generate the predictions and explanationsfor these extra examples, and extract predicted probabilities, factors, and target probabilities triples toconstruct training data points used to train the calibrator.
Note this procedure requires no explanationannotations for the extra examples.
Approximating Factuality We approximate the factuality using lexical overlap between theexplanations and the inputs, which we found to work fairly well for our tasks.
A DV H OTPOT :
We use an explanation consisting of two sentences (examples in Figure 3) asan illustration.
Let E =
( E (1) , E (2) ) be the generated explanation, where E (1) and E (2) arethe two sentences, and the E ( i ) =
( e 1 , e 2 , · · · ) contain tokens e 1 , e 2 , · · · .
Similarly, let P =( P (1) , P (2) , P (3) , P (4) ) be the context paragraphs, and P ( i )
= ( p 1 , p 2 , · · · ) be the tokens.
Thefactuality estimation of one explanation sentence E ( i ) is defined as:
[Equation]
Intuitively, the factuality score for a sentence E is defined as the maximum number of overlappingtokens over all paragraphs P , normalized by the number of tokens in E .
We then define the factuality
[Table]
score for the whole explanation as: V ( E ) =
min E ∈E V ( E ) , as it requires all sentences to be factualin order to make the entire explanation factual.
E-SNLI:
On the E-SNLI dataset whose explanations do not really involve a concept of factuality, westill use an analogous score following the same principle, where we regard the premise as the context.
Let E =
( e 1 , e 2 , · · · ) be the explanation and P =
( p 1 , p 2 , · · · ) be the premise.
We simply score theexplanation by V ( E ) =
| E |∩| P | | E | .
Namely, the more an explanation overlaps with the premise, themore factual it is.
4.3 Calibrating E-SNLI
[Table]
Setup For E-SNLI, we use calibration methods to postprocess the final probabilities.
Unlikeclassical temperature scaling [30], note that themethods we use here can actually change theprediction; we will therefore evaluate on accuracy of the calibrated model.
We study the effectiveness of our explanationbased calibrator under different training datasizes varying from 32 to 128.
Recall that weonly require explanation annotations for 32 datapoints, and only need the labels for the rest totrain the calibrator.
For E-SNLI, we calibrate
P-E, which is shown to be more effective than
E-P in this setting (Section 2.4).
Baselines We provide the performance of finetuned R O BERT A [25] model as a reference,finding this to work better than DeBERTa [17].
To isolate the effectiveness of using explanations for calibration, we introduce three additional baselines using non-explanation-based calibrators.
We apply the probability-based calibrator as described in Section 4.2 on the results obtained onfew-shot learning (F EW -S HOT +P ROB C AL ) and predict-then-explain pipeline (P-E+P ROB C AL ).
Wenote that the parameters of these calibrators are trained using the addition data points, as opposedto being heuristically determined as in Zhao et al.
Furthermore, we experiment with a recently proposed supervised calibrator from Zhang et al.
[50], which uses the CLS representationsfrom an additional language model as features in the calibrator.
The probabilities are tuned usingˆ p = softmax( W [ p ; h ] + b ) , where h is the CLS representation.
Since we do not have access to theembeddings obtained by GPT-3, we use R O BERT A to extract the vectors instead.
We use such acalibrator on top of our best-performing base model, P-E, resulting P-E+Z HANG [50].
Limited by the maximum prompt length, in-context learning is not able to take as input the additionaldata used for training the calibrator.
For a fair comparison, we can allow the in-context model to usethis data by varying the prompts across test examples, dynamically choosing the prompt examples tomaximize performance.
Choosing closer data points for prompting is a common and effective way ofscaling up the training data size for in-context learning [36, 24].
Following Liu et al.
[24], we test theperformance of choosing nearest neighbors for the prompt based on CLS embedding produced by a
R O BERT A model [25], referred as F EW -S HOT (NN).
It is worth clarifying that the F EW -S HOT and
F EW -S HOT +P ROB C AL approaches use the same set of 32 training shots in the prompt for every testexample, whereas the shot sets vary from example to example in F EW -S HOT (NN).
Results We show the results in Table 3.
We use 5 different groups of training examples and reportthe mean and standard deviation across the groups.
For F EW -S HOT (NN), we only report the resultsobtained using 128 examples.
Under 128 training examples, applying a trained calibrator on top of prompting with explanation (i.e.,
P-E+E XPL C AL ) achieves the best accuracy of 68.5%, which is 12% higher than the performance ofthe vanilla uncalibrated few-shot in-context learning (F EW -S HOT ).
P-E+E XPL C AL also outperforms
[Figure]
[Table]
[Figure]
F EW -S HOT +P ROB C AL and P-E+P ROB C AL
by 5% and 3%, respectively.
Using explanations is moreeffective than using probabilities alone.
In addition, P-E+E XPL C AL also outperforms P-E+Z HANG
ET AL .
[50], whose performance is on par with P-E+P ROB C AL .
This suggests the additional CLSinformation is not very helpful in this setting.
As the data size increases from 32 to 128, the performance of the explanation-based calibrator keepsimproving notably, whereas the performance of probability-based calibrators nearly saturates at adata size of 96.
The performance of F EW -S HOT (NN) with 128 training instances only improvesthe performance by 3.3%, compared to F EW -S HOT with 32 training instances.
Choosing nearestneighbors as the shots, while being effective when having access to a large amount of data, is nothelpful in the extreme data-scarce regime.
Calibrating using explanations is an effective way of usinga few extra data points that cannot fit in the prompt, which is a pitfall of standard in-context learning.
Finally, R O BERT A finetuned using 128 shots only achieves an accuracy of 54.9%, lagging theperformance of GPT-3 based models.
The limited training data size is insufficient for finetuningsmaller language models like R O BERT A , but is sufficient for P-E+E XPL C AL to be effective.
4.4 Calibrating A DV H OTPOT
Setup For the A DV H OTPOT dataset, our calibration takes the form of tuning the confidence scoresof the predicted answers to better align them with the correctness of predictions.
These confidencescores can be used in a “selective QA” setting [20], where the model can abstain on a certain fractionof questions where it assigns low confidence to its answers.
We use the area under coverage-accuracycurve (AUC) to evaluate how well a model is calibrated as in past literature [20, 7, 50, 13, 47].
Thecurve plots the average accuracy with varying fractions (coverage) of questions being answered(examples in Figure 5).
For any given coverage, a better calibrated model should be able to identifyquestions that it performs best on, hence resulting a higher AUC.
We experiment with training data set sizes of 6, 32, and 64.
We report the results averaged from 5trials using different training sets.
For A DV H OTPOT , we calibrate E-P, which is shown to be moreeffective than P-E in this setting (Section 2.4).
Our approach is also effective for calibrating P-E;please refer to Appendix E for details.
Results We show the AUC scores in Table 4.
By leveraging explanations, E-P+E XPL C AL successfully achieves an AUC of 68.8, surpassing both F EW -S HOT by 7 points and E-P by 4 points.
We notethis is substantial improvement, given that the upperbound of AUC is constrained by the accuracy ofthe answers and cannot reach 100.
Figure 5 shows the coverage-accuracy curves of various methodsaveraged across the 5 training runs.
E-P+E XPL C AL always achieves a higher accuracy than itsuncalibrated counterpart, E-P, under a certain coverage, and the gap is especially large in the mostconfident intervals (coverage < 50%).
E-P+Z HANG ET AL .
[50] is able to calibrate the predictionson this dataset, but still lags our explanation-based calibrator, E-P+E XPL C AL .
In addition, the explanation-based calibrator can be effective with as few as 32 examples.
This isbecause there are only two parameters (the probability of predicted answer and the explanation- based factor) in the calibrator, which can be easily learned in this few-shot setting.
Comparing
E-P+E XPL C AL against F EW -S HOT (NN), using nearest neighbors in the prompt is also able toimprove the performance compared to using a fixed set of shots (F EW -S HOT ), yet our lightweightcalibrator can better utilize such a small amount of data, and learn to distinguish more accuratepredictions based on the explanations.
Related Work
Our investigation is centered around in-context learning [3], which has garnered increasing interestsince the breakthrough of various large pretrained language models.
Recent work has been devotedto studying different aspects of in-context learning, including its wayward behaviors [28, 40] andapproaches to overcome them [51], whereas our exploration focuses on using explanations.
The utility of explanations for few-shot in-context learning has also been discussed concurrently [29,41, 26, 9, 22, 43], especially in symbolic reasoning tasks.
We differ in that we study more free-formexplanations in tasks (QA and NLI, specifically) focusing on textual reasoning over provided contexts.
Furthermore, our work focuses on the nature of the explanations generated by GPT-3, which are foundto be unreliable.
Regarding our use of calibration, similar ideas of explanation-based performanceestimation have been applied to other tasks [33, 48, 47], but we rely on the free-text explanationsgenerated by the model instead of interpretations obtained through post-hoc interpretation techniques.
More broadly, how to use explanations in various forms (textual explanation, highlights, etc.) to trainbetter models is a longstanding problem [49].
Past work has built a series of pipeline models thatfirst generate the explanations and then make predictions purely based on the generated explanations[44, 52, 6].
Prior research has also explored using explanations as additional supervision to train jointmodels [16, 11, 21, 38].
Another line of work seeks to aligning the reasoning process of a trainedmodels with the explanations, which is typically done by interpreting a prediction post-hoc throughexplanation techniques and optimize the distance between the obtained explanation and ground truthexplanation [23, 35, 31, 12, 46].
These aforementioned methods all update the model parameters andtypically require a considerable amount of explanation annotations to be effective.
By contrast, oursetting treats language models as pure black boxes and only requires few-shot explanations.
Discussion & Conclusion
Caveats and Risks of Explanations from Large Language Models Our analysis suggests that
GPT-3’s internal “reasoning” does not always align with explanations that it generates, as shownby our consistency results.
More concerning, the explanations might not be factually grounded inthe provided prompt.
This shortcoming should caution against any deployment of this technologyin practice: because the explanations are grammatical English and look very convincing, they maydeceive users into believing the system’s responses even when those responses are incorrect.
Section6 of Bender et al.
[1] discusses these risks in additional detail.
The fact that language models canhallucinate explanations is also found in other work [52].
This result is unsurprising in some sense:without sufficient supervision or grounding, language models do not learn meaning as distinct fromform [2], so we should not expect their explanations to be strongly grounded.
We have shown that even explanations which don’t lead to accuracy gains can still be useful forcalibration.
However, the lexical overlap feature we use here is a weak signal of explanationcorrectness (see the example in Figure 1).
Strong enough entailment models should theoreticallybe able to perform this task and work across a range of tasks without fine-tuning.
This explanationassessing model can even be a language model itself trained for this particular propose to approachthe verification tasks for a given domain by in-context learning.
Conclusion We have explored the capabilities of GPT-3 in using explanations in in-context learningfor textual reasoning.
Through our experiments on two QA datasets and an NLI dataset, we find thatsimply including explanations in the prompt does not always improve the performance of in-contextlearning.
Our manual analysis demonstrates that GPT-3 tends to generate nonfactual explanationswhen making wrong predictions, which can be a useful leverage to assess the correctness of thepredictions.
Lastly, we showcase how to use explanations to build lightweight calibrators, whichsuccessfully improve in-context learning performance across all three datasets.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See
Section 6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] We use GPT-3 instruct series API
[Table]
(text-davinci-001).
[Table]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
[Table]
(a) If your work uses existing assets, did you cite the creators?
[Yes] See reference [19]and [5].
[Table]
(b) Did you mention the license of the assets?
[Yes] See Section 2.1.
[Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
[Table]
We included the Synthetic dataset in the supplemental material.
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[No]
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
[Table]
information or offensive content?
[No]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[-(2) P 2 = P 1 and V 
-]
[N/A]
[Table]