Original: D ISCOVERING L ATENT K NOWLEDGE IN L ANGUAGE M ODELS W ITHOUT S UPERVISION
Revised: D ISCOVERING L ATENT K NOWLEDGE IN L ANGUAGE M ODELS W ITHOUT S UPERVISION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Existing techniques for training language models can be misaligned with the truth: if we train models with imitation learning, they may reproduce errors that humans make; if we train them to generate text that humans rate highly, they may output errors that human evaluators can’t detect.
Revised: Existing techniques for training language models can be misaligned with the truth: if we train models with imitation learning, they may reproduce errors that humans make; if we train them to generate text that humans rate highly, they may output errors that human evaluators can’t detect.

Original: We propose circumventing this issue by directly finding latent knowledge inside the internal activations of a language model in a purely unsupervised way.
Revised: We propose circumventing this issue by directly finding latent knowledge inside the internal activations of a language model in a purely unsupervised way.

Original: Specifically, we introduce a method for accurately answering yes-no questions given only unlabeled model activations.
Revised: Specifically, we introduce a method for accurately answering yes-no questions given only unlabeled model activations.

Original: It works by finding a direction in activation space that satisfies logical consistency properties, such as that a statement and its negation have opposite truth values.
Revised: It works by finding a direction in activation space that satisfies logical consistency properties, such as that a statement and its negation have opposite truth values.

Original: We show that despite using no supervision and no model outputs, our method can recover diverse knowledge represented in large language models: across 6 models and 10 questionanswering datasets, it outperforms zero-shot accuracy by 4% on average.
Revised: We show that despite using no supervision and no model outputs, our method can recover diverse knowledge represented in large language models: across 6 models and 10 questionanswering datasets, it outperforms zero-shot accuracy by 4% on average.

Original: We also find that it cuts prompt sensitivity in half and continues to maintain high accuracy even when models are prompted to generate incorrect answers.
Revised: We also find that it cuts prompt sensitivity in half and continues to maintain high accuracy even when models are prompted to generate incorrect answers.

Original: Our results provide an initial step toward discovering what language models know, distinct from what they say, even when we don’t have access to explicit ground truth labels.
Revised: Our results provide an initial step toward discovering what language models know, distinct from what they say, even when we don’t have access to explicit ground truth labels.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The increasing deployment of language models in real-world applications opens up exciting possibilities, but it also raises the stakes of AI research and presents new risks (Bommasani et al., 2021;
Revised: The increasing deployment of language models in real-world applications opens up exciting possibilities, but it also raises the stakes of AI research and presents new risks (Bommasani et al., 2021;

Original: Weidinger et al., 2021; Bender et al., 2021).
Revised: Weidinger et al., 2021; Bender et al., 2021).

Original: One of these risks is that language models do not always output text that is true (Evans et al., 2021; Hendrycks et al., 2021; Kenton et al., 2021).
Revised: One of these risks is that language models do not always output text that is true (Evans et al., 2021; Hendrycks et al., 2021; Kenton et al., 2021).

Original: Common training objectives can cause models to learn internal representations related to truth, since truth is a useful feature for many tasks.
Revised: Common training objectives can cause models to learn internal representations related to truth, since truth is a useful feature for many tasks.

Original: However, these objectives can also cause language models to output text that is false, at least in some circumstances.
Revised: However, these objectives can also cause language models to output text that is false, at least in some circumstances.

Original: For example, if we train a model to imitate human-generated text, it may learn to output common misconceptions (Lin et al., 2022).
Revised: For example, if we train a model to imitate human-generated text, it may learn to output common misconceptions (Lin et al., 2022).

Original: Or if we train a chat bot to optimize a reward such as engagement, it may learn to generate text that is compelling but false (Roller et al., 2021).
Revised: Or if we train a chat bot to optimize a reward such as engagement, it may learn to generate text that is compelling but false (Roller et al., 2021).

Original: If we try to reward model outputs that look true, a model may still learn to output false text if human raters can’t evaluate the correctness of that text (Kenton et al., 2021).
Revised: If we try to reward model outputs that look true, a model may still learn to output false text if human raters can’t evaluate the correctness of that text (Kenton et al., 2021).

Original: In each case, this is an issue that stems from the misalignment between a training objective and the truth.
Revised: In each case, this is an issue that stems from the misalignment between a training objective and the truth.

Original: As models are applied to more complex domains, human supervision may become less effective at mitigating this misalignment.
Revised: As models are applied to more complex domains, human supervision may become less effective at mitigating this misalignment.

Original: Moreover, because this is a problem with the training objective rather than a model’s capabilities it likely won’t be solved by scaling up models alone.
Revised: Moreover, because this is a problem with the training objective rather than a model’s capabilities, it likely won’t be solved by scaling up models alone.

Original: We propose a different approach for addressing this misalignment: using models to answer questions in a purely unsupervised way.
Revised: We propose a different approach for addressing this misalignment: using models to answer questions in a purely unsupervised way.

Original: Intuitively, instead of trying to explicitly, externally specify truth, we search for implicit, internal “beliefs” or “knowledge” learned by a model.
Revised: Intuitively, instead of trying to explicitly, externally specify truth, we search for implicit, internal “beliefs” or “knowledge” learned by a model.

Original: We approach this problem by leveraging the fact that a model’s representation of truth must satisfy logical consistency properties, which are unlikely to be satisfied by many other features.
Revised: We approach this problem by leveraging the fact that a model’s representation of truth must satisfy logical consistency properties, which are unlikely to be satisfied by many other features.

Original: We implement this idea by introducing Contrast-Consistent Search (CCS), a method that learns a linear projection of the hidden states that is consistent across negations, as illustrated in Figure 1.
Revised: We implement this idea by introducing Contrast-Consistent Search (CCS), a method that learns a linear projection of the hidden states that is consistent across negations, as illustrated in Figure 1.

Original: We find that despite its simplicity and despite not having access to any labels or model outputs, 
Revised: We find that despite its simplicity and despite not having access to any labels or model outputs, *

Original: CCS can accurately recover knowledge from model representations: evaluated across 6 models and 10 question-answering datasets, CCS outperforms the accuracy of strong zero-shot baselines by

Original: [Figure]
Revised: [Figure]

Original: Given a set of Yes-No questions, answer each question with both "Yes" and "No"
Revised: Given a set of Yes-No questions, answer each question with both "Yes" and "No"

Original: [Figure]
Revised: [Figure]

Original: internal model activations Map each activation to a probability of being true Optimize unsupervised loss to make probabilities consistent and conﬁ
Revised: internal model activations Map each activation to a probability of being true Optimize unsupervised loss to make probabilities consistent and con

Revised: [Figure]

Original: dent
Revised: dent

Original: For each question, estimate the probability that the answer is "Yes" {{{{
Revised: For each question, estimate the probability that the answer is "

Original:  4% on average (Section 3.2.1).
Revised: [Figure] CCS can accurately recover knowledge from model representations: evaluated across 6 models and 10 question-answering datasets, CCS outperforms the accuracy of strong zero-shot baselines by 4% on average (Section 3.2.1).

Original: The resulting classifier is also less sensitive to different prompts than zero-shot, cutting the standard deviation in accuracy in half.
Revised: The resulting classifier is also less sensitive to different prompts than zero-shot, cutting the standard deviation in accuracy in half.

Original: Additionally, we try deliberately prompting models to make incorrect outputs, which should intuitively change what models say but which shouldn’t affect their latent knowledge.
Revised: Additionally, we try deliberately prompting models to make incorrect outputs, which should intuitively change what models say but which shouldn’t affect their latent knowledge.

Original: We find that this causes zero-shot accuracy to drop by up to 9.5% (Section 3.2.2), without decreasing the accuracy of CCS.
Revised: We find that this causes zero-shot accuracy to drop by up to 9.5% (Section 3.2.2) without decreasing the accuracy of CCS.

Original: We systematically analyze CCS to understand the features it discovers.
Revised: We systematically analyze CCS to understand the features it discovers.

Original: We show that it transfers across unrelated tasks, suggesting that models may have a task-agnostic representation of the truth and that CCS is able to approximately discover it (Section 3.3.1).
Revised: We show that it transfers across unrelated tasks, suggesting that models may have a task-agnostic representation of the truth and that CCS is able to approximately discover it (Section 3.3.1).

Original: Moreover, CCS sometimes works best using the hidden states in the middle layers of a network and can work even when model outputs aren’t very useful, suggesting that it can leverage different features from those used by the outputs (Section 3.3.2).
Revised: Moreover, CCS sometimes works best using the hidden states in the middle layers of a network and can work even when model outputs aren’t very informative, suggesting that it can leverage different features from those used by the outputs (Section 3.3.2).

Original: Finally, we show that representations of truth tend to be salient in models: they can often be found without much data, and they can often be found by taking the top principal component of a slightly modified representation space (Section 3.3.3).
Revised: Finally, we show that representations of truth tend to be salient in models: they can often be found without much data, and they can often be found by taking the top principal component of a slightly modified representation space (Section 3.3.3).

Original: Most existing techniques for making models truthful use human supervision to explicitly specify what is correct.
Revised: Most existing techniques for making models truthful use human supervision to explicitly specify what is correct.

Original: However, it is not feasible to provide supervision in some settings.
Revised: However, it is not feasible to provide supervision in some settings.

Original: Our work suggests that an external source of ground truth may not actually be necessary: we may instead be able to find a model’s latent representation of truth, independent of what a model says, without using any supervision in the first place.
Revised: Our work suggests that an external source of ground truth may not actually be necessary: we may instead be able to find a model’s latent representation of truth, independent of what a model says, without using any supervision in the first place.

Original:  P ROBLEM S TATEMENT AND F RAMEWORK
Revised:  P ROBLEM S TATEMENT AND F RAMEWORK

Original: In this section we describe our problem setup in more detail and introduce Contrast-Consistent Search (CCS), a method for discovering latent knowledge in language models without supervision.
Revised: In this section we describe our problem setup in more detail and introduce Contrast-Consistent Search (CCS), a method for discovering latent knowledge in language models without supervision.

Original: 2. P ROBLEM : D ISCOVERING L ATENT K NOWLEDGE
Revised: 2. P ROBLEM : D ISCOVERING L ATENT K NOWLEDGE

Original: Given a pre-trained neural language model and a set q 1 , . . .
Revised: Given a pre-trained neural language model and a set q 1 , . . .

Original: , q n of yes-no questions 1 , our goal is to answer each q i correctly.
Revised: , q n of yes-no questions 1 , our goal is to answer each q i correctly.

Original: Here, q i can be any question with a well-defined answer, including procedural questions like “Is 22+59 = 237?”, for which the answer is “No”, and factual questions like “Are cats mammals?”, for which the answer is “Yes”.
Revised: Here, q i can be any question with a well-defined answer, including procedural questions like “Is 22+59 = 237?”, for which the answer is “No”, and factual questions like “Are cats mammals?”, for which the answer is “Yes”.

Original: Importantly, we want methods that do not rely on the model generating correct outputs and  do not rely on external supervision.
Revised: Importantly, we want methods that do not rely on the model generating correct outputs and that do not rely on external supervision.

Original: We consequently do not assume access to labeled ground-truth data.

Original: Instead, we turn to the model’s  hidden representations.
Revised: Instead, we turn to the model’s unlabeled hidden representations.

Original: Specifically, let ϕ ( x ) ∈
Revised: Specifically, let ϕ ( x ) ∈

Original: R
Revised: R

Original: d denote some feature representation on a natural langauge input x , such as the hidden states of a Transformerbased language model.
Revised: d denote some feature representation on a natural langauge input x , such as the hidden states of a Transformer-based language model.

Original: Our goal is to answer the questions q 1 , . . .
Revised: Our goal is to answer the questions q 1 , . . .

Original: , q n only given access to ϕ ( · ) .
Revised: , q n only given access to ϕ ( · ) .

Original: In Section 2.2 we introduce a method for this problem that attains high accuracy (see Section 3), demonstrating that this task is tractable.
Revised: In Section 2.2 we introduce a method for this problem that attains high accuracy (Section 3), demonstrating that this task is tractable.

Original: 2. M ETHOD : C ONTRAST -C ONSISTENT S EARCH
Revised: 2. M ETHOD : C ONTRAST -C ONSISTENT S EARCH

Original: To make progress on the goal described above, we exploit the fact that truth has special structure: it satisfies consistency properties that few other features in a language model are likely to satisfy.
Revised: To make progress on the goal described above, we exploit the fact that truth has special structure: it satisfies consistency properties that few other features in a language model are likely to satisfy.

Original: Our method, Contrast-Consistent Search (CCS), leverages this idea by finding a direction in activation space that is consistent across negations.
Revised: Our method, Contrast-Consistent Search (CCS), leverages this idea by finding a direction in activation space that is consistent across negations.

Original: As we illustrate in Figure 1, CCS works by (1) answering each question
Revised: As we illustrate in Figure 1, CCS works by (1) answering each question

Original: q i as both “Yes” ( x + i ) and “No” ( x − i ), (2) computing the representations ϕ
Revised: q i as both “Yes” ( x + i ) and “No” ( x − i ), (2) computing the representations ϕ

Original: ( x + i ) and ϕ ( x − i ) of each answer, (3) mapping the answer representations to probabilities p
Revised: ( x + i ) and ϕ ( x − i ) of each answer, (3) mapping the answer representations to probabilities p

Original: + i and p − i of being true, then (4) optimizing that mapping so that the probabilities are both consistent and confident.
Revised: + i and p − i of being true, then (4) optimizing that mapping so that the probabilities are both consistent and confident.

Original: Concretely, the input to CCS is a set of Yes-No questions, q 1 , . .
Revised: Concretely, the input to CCS is a set of Yes-No questions, q 1 , . .

Original: . , q n , and access to a pretrained model’s representations, ϕ ( · ) ; the output of CCS is a lightweight probe on top of ϕ ( · ) that can answer new questions.
Revised: . , q n , and access to a pretrained model’s representations, ϕ ( · ) ; the output of CCS is a lightweight probe on top of ϕ ( · ) that can answer new questions.

Original: Here, ϕ ( · ) is fixed but should contain useful information about the answers to q 1 , . .
Revised: Here, ϕ ( · ) is fixed but should contain useful information about the answers to q 1 , . .

Original: . , q n , in the sense that if one did  have access to the ground-truth labels for q 1 , . .
Revised: . , q n , in the sense that if one did (hypothetically) have access to the ground-truth labels for q 1 , . .

Original: . , q n , one should be able to train a small supervised probe on ϕ ( · ) that attains high accuracy.
Revised: . , q n , one would be able to train a small supervised probe on ϕ ( · ) that attains high accuracy.

Original: Importantly, CCS does not modify the weights of the pretrained model and it does not use labels.
Revised: Importantly, CCS does not modify the weights of the pretrained model and it does not use labels.

Original: Constructing contrast pairs.
Revised: Constructing contrast pairs.

Original: An important property that truth satisfies is negation consistency: the answer to a clear-cut question cannot be both “Yes” and “No” at the same time, as these are negations of each other.
Revised: An important property that truth satisfies is negation consistency: the answer to a clear-cut question cannot be both “Yes” and “No” at the same time, as these are negations of each other.

Original: Probabilistically, for each question q i , the probability that the answer to q i is “Yes” should be one minus the probability that the answer to q i is “No”.
Revised: Probabilistically, for each question q i , the probability that the answer to q i is “Yes” should be one minus the probability that the answer to q i is “No”.

Original: To use this property, we begin by constructing contrast pairs: for each question q i , we answer q i both as “Yes”, resulting in the new natural language statement x + i , and as “No”, resulting in the natural language statement x − i .
Revised: To use this property, we begin by constructing contrast pairs: for each question q i , we answer q i both as “Yes”, resulting in the new natural language statement x + i , and as “No”, resulting in the natural language statement x − i .

Original: We illustrate this in Figure 1 (left).
Revised: We illustrate this in Figure 1 (left).

Original: We will then learn to classify x
Revised: We will then learn to classify x

Original: + i and x − i as true or false; if x + i is true, then the answer to q i should be “Yes”, and if x − i is true, then the answer to q i should be “No”.
Revised: + i and x − i as true or false; if x + i is true, then the answer to q i should be “Yes”, and if x − i is true, then the answer to q i should be “No”.

Original: In practice, we convert each task into a question-answering task with two possible labels, then we use task-specific zero-shot prompts to format questions and answers as strings to construct each contrast pair.
Revised: In practice, we convert each task into a question-answering task with two possible labels, then we use task-specific zero-shot prompts to format questions and answers as strings to construct each contrast pair.

Original: The opposite labels we use to construct contrast pairs can be “Yes” and “No” for a generic task, or they can be other tasks-specific labels, such as “Positive” and “Negative” in the case of sentiment classification.
Revised: The opposite labels we use to construct contrast pairs can be “Yes” and “No” for a generic task, or they can be other tasks-specific labels, such as “Positive” and “Negative” in the case of sentiment classification.

Original: We describe the exact prompts we use to for each task in Appendix B.
Revised: We describe the exact prompts we use to for each task in Appendix B.

Original: Feature extraction and normalization.
Revised: Feature extraction and normalization.

Original: Given a contrast pair ( x + i , x − i ) , CCS first computes the representations ϕ
Revised: Given a contrast pair ( x + i , x − i ) , CCS first computes the representations ϕ

Original: ( x + i ) and ϕ ( x − i )
Revised: ( x + i ) and ϕ ( x − i )

Original: using the feature extractor ϕ ( · ) .
Revised: using the feature extractor ϕ ( · ) .

Original: Intuitively, there are two salient differences between ϕ ( x + i ) and ϕ ( x − i ) : (1) x
Revised: Intuitively, there are two salient differences between ϕ ( x + i ) and ϕ ( x − i ) : (1) x

Original: + i ends with “Yes” while x − i ends with “No”, and (2) one of x + i or x − i is true while the other is false.
Revised: + i ends with “Yes” while x − i ends with “No”, and (2) one of x + i or x − i is true while the other is false.

Original: We want to find (2) rather than (1), so we first try to remove the effect of (1) by normalizing { ϕ ( x + i ) } and { ϕ ( x − i ) } independently.
Revised: We want to find (2) rather than (1), so we first try to remove the effect of (1) by normalizing { ϕ ( x + i ) } and { ϕ ( x − i ) } independently.

Original: In particular, we construct normalized representations ˜ ϕ
Revised: In particular, we construct normalized representations ˜ ϕ

Original: ( x ) as follows:
Revised: ( x ) as follows:

Original: [Equation]
Revised: [Equation]

Original: where( µ + ,σ + ) and ( µ − , σ − ) are the meansand standard deviations of { ϕ ( x + i ) } ni =1 and { ϕ ( x − i ) } ni = respectively, and where all operations are element-wise along each dimension.
Revised: where µ + , µ − ∈ R d are the means of { ϕ ( x + i ) } ni =1 and { ϕ ( x − i ) } ni =1 .

Original: This normalization ensures that { ˜ ϕ
Revised: This normalization ensures that { ˜ ϕ

Original: ( x + i ) } and { ˜ ϕ ( x − i ) } no longer form two separate clusters.
Revised: ( x + i ) } and { ˜ ϕ ( x − i ) } no longer form two separate clusters.

Revised: In practice we also normalize the scale of the features, but this isn’t essential for the method to work; see Appendix G.1 for details.

Original: Mapping activations to probabilities.
Revised: Mapping activations to probabilities.

Original: Next, we learn a probe p θ,b (˜ ϕ ) that maps a (normalized) hidden state ˜ ϕ
Revised: Next, we learn a probe p θ,b (˜ ϕ ) that maps a (normalized) hidden state ˜ ϕ

Original: ( x ) to a number between 0 and 1 representing the probability that the statement x is true.
Revised: ( x ) to a number between 0 and 1 representing the probability that the statement x is true.

Original: We use a linear projection followed by a sigmoid σ ( · ) , i.e. p θ,b ( ˜ ϕ )
Revised: We use a linear projection followed by a sigmoid σ ( · ) , i.e. p θ,b (˜ ϕ )

Original: =
Revised: =

Original: σ ( θ T ˜ ϕ
Revised: σ ( θ T ˜ ϕ

Original: + b ) , but nonlinear projections can also work.
Revised: + b ) , but nonlinear projections can also work.

Original: For simplicity, we sometimes omit the θ, b subscript in p .
Revised: For simplicity, we sometimes omit the θ, b subscript in p .

Original: Training objective.
Revised: Training objective.

Original: To find features that represent the truth, we leverage the consistency structure of truth.
Revised: To find features that represent the truth, we leverage the consistency structure of truth.

Original: First, we use the fact that a statement and its negation should have probabilities that add up to 
Revised: First, we use the fact that a statement and its negation should have probabilities that add up to 1 .

Original: This motivates the consistency loss:
Revised: This motivates the consistency loss:

Original: [Equation]
Revised: [Equation]

Original: However, this objective alone has a degenerate solution: p ( x + ) ≈ p ( x − ) ≈ 0 .
Revised: However, this objective alone has a degenerate solution: p ( x + ) = p ( x − ) = 0 .

Original: To avoid this problem, we encourage the model to also be confident with the following confidence loss:
Revised: To avoid this problem, we encourage the model to also be confident with the following confidence loss:

Original: [Equation]
Revised: [Equation]

Original: We can equivalently interpret L confidence as imposing a second consistency property on the probabilities: the law of excluded middle (every statement must be either true or false).
Revised: We can equivalently interpret L confidence as imposing a second consistency property on the probabilities: the law of excluded middle (every statement must be either true or false).

Original: The final unsupervised loss is the sum of these two losses, averaged across all contrast pairs:
Revised: The final unsupervised loss is the sum of these two losses, averaged across all contrast pairs:

Original: [Equation]
Revised: [Equation]

Original: Note that both losses are necessary; L confidence alone also has a degenerate solution.
Revised: Note that both losses are necessary; L confidence alone also has a degenerate solution.

Original: Inference.
Revised: Inference.

Original: Both p ( x + i ) and 1 − p ( x − i ) should represent the probability that the answer to q i is “Yes”.
Revised: Both p ( x + i ) and 1 − p ( x − i ) should represent the probability that the answer to q i is “Yes”.

Original: However, because we use a soft consistency constraint, these may not be exactly equal.
Revised: However, because we use a soft consistency constraint, these may not be exactly equal.

Original: To make a prediction on an example x i after training, we consequently take the average of these:
Revised: To make a prediction on an example x i after training, we consequently take the average of these:

Original: [Equation]
Revised: [Equation]

Original: We then predict that the answer to q i is “Yes” based on whether ˜ p ( q i ) is greater than 0 .
Revised: We then predict that the answer to q i is “Yes” based on whether ˜ p ( q i ) is greater than 0 .

Original: Technically, we also need to determine whether ˜ p ( q i ) > 0 .
Revised: Technically, we also need to determine whether ˜ p ( q i ) > 0 .

Original: 5 corresponds to “Yes” or “No,” as this isn’t specified by L CCS .
Revised: 5 corresponds to “Yes” or “No,” as this isn’t specified by L CCS .

Original: For simplicity in our evaluations we take the maximum accuracy over the two possible ways of labeling the predictions of a given test set.
Revised: For simplicity in our evaluations we take the maximum accuracy over the two possible ways of labeling the predictions of a given test set.

Original: However, in Appendix A we describe how one can identify the two clusters without any supervision in principle by leveraging conjunctions.
Revised: However, in Appendix A we describe how one can identify the two clusters without any supervision in principle by leveraging conjunctions.

Original:  R ESULTS 3. E XPERIMENTAL S ETUP
Revised:  R ESULTS 3. E XPERIMENTAL S ETUP

Original: Here we provide an overview of our experimental setup; see Appendix H for complete details.
Revised: Here we give an overview of our experimental setup; see Appendix G for full details.

Revised: We provide code at https://www.github.com/collin-burns/discovering_latent_knowledge .

Original: Models.
Revised: Models.

Original: We test six models: encoder-decoder models (T5 (Raffel et al., 2020), UnifiedQA (Khashabi et al., 2020), T0 (Sanh et al., 2021)), autoregressive models (GPT-J (Wang & Komatsuzaki, 2021)), and encoder-only models (RoBERTa (Liu et al., 2019), DeBERTa
Revised: We test six models: encoder-decoder models (T5 (Raffel et al., 2020), UnifiedQA (Khashabi et al., 2020), T0 (Sanh et al., 2021)), autoregressive models (GPT-J (Wang & Komatsuzaki, 2021)), and encoder-only models (RoBERTa (Liu et al., 2019), DeBERTa

Original: (He et al., 2021)).
Revised: (He et al., 2021)).

Original: Data.
Revised: Data.

Original: We test models on 10 datasets: sentiment classification (IMDB (Maas et al., 2011) and Amazon (McAuley & Leskovec, 2013)), topic classification (AG-News (Zhang et al., 2015) and DBpedia- (Lehmann et al., 2015)), NLI (RTE (Wang et al., 2018) and QNLI (Rajpurkar et al., 2016)), story completion (COPA (Roemmele et al., 2011) and Story-Cloze (Mostafazadeh et al., 2017)), question answering (BoolQ (Clark et al., 2019)), and common sense reasoning (PIQA (Bisk et al., 2020)).
Revised: We test models on 10 datasets: sentiment classification (IMDB (Maas et al., 2011) and Amazon (McAuley & Leskovec, 2013)), topic classification (AG-News (Zhang et al., 2015) and DBpedia- (Lehmann et al., 2015)), NLI (RTE (Wang et al., 2018) and QNLI (Rajpurkar et al., 2016)), story completion (COPA (Roemmele et al., 2011) and Story-Cloze (Mostafazadeh et al., 2017)), question answering (BoolQ (Clark et al., 2019)), and common sense reasoning (PIQA (Bisk et al., 2020)).

Original: We convert each dataset to a yes-no question-answering task or a binary classification task, as described in Appendix H. We balance the labels and randomly subsample 1000 examples from each dataset (except for COPA, which has only 500 examples total), then randomly split each dataset into an unsupervised training set ( 60% of the data) and test set ( 40% ).
Revised: We convert each dataset to a yes-no question-answering task or a binary classification task, as described in Appendix G. We balance the labels and randomly subsample 1000 examples from each dataset (except for COPA, which has only 500 examples total), then randomly split each dataset into an unsupervised training set ( 60% of the data) and test set ( 40% ).

Original: We subsample each dataset for computational efficiency reasons; because we aggregate over 9 prompts per dataset, 10 datasets, and 6 models, 1000 datapoints per dataset actually corresponds to approximately 180k examples in total.We results Appendix I.
Revised: We subsample each dataset for computational efficiency reasons; because we aggregate over 9 prompts per dataset, 10 datasets, and 6 models, 1000 datapoints per dataset actually corresponds to approximately 180k examples in total.

Original: Methods.
Revised: Methods.

Original: We test four main methods: zero-shot, calibrated zero-shot, Contrast-Consistent Search (CCS), and Logistic Regression (LR).
Revised: We test four main methods: zero-shot, calibrated zero-shot, Contrast-Consistent Search (CCS), and Logistic Regression (LR).

Original: Zero-shot works by predicting the answer with the highest log probability according to the language model, averaged across the tokens that make up that label.
Revised: Zero-shot works by predicting the answer with the highest log probability according to the language model, averaged across the tokens that make up that label.

Original: Calibrated zero-shot works by balancing zero-shot predictions to be 50 / 50 for each answer, as we describe in more detail below, similar to Zhao et al.
Revised: Calibrated zero-shot works by balancing zero-shot predictions to be 50 / 50 for each answer, as we describe in more detail below, similar to Zhao et al.

Original: For Logistic Regression we train on the training split for each dataset using (˜ ϕ
Revised: For Logistic Regression we train on the training split for each dataset using (˜ ϕ

Original: ( x + ) , ˜ ϕ
Revised: ( x + ) , ˜ ϕ

Original: ( x − ))
Revised: ( x − ))

Original: as the covariates, then evaluate on the corresponding test split.
Revised: as the covariates, then evaluate on the corresponding test split.

Original: We treat LR as a ceiling since it uses labeled data.
Revised: We treat LR as a ceiling since it uses labeled data.

Original: When testing CCS, we optimize it 10 times using AdamW (Loshchilov & Hutter, 2017) with learning rate 0 .
Revised: When testing CCS, we optimize it 10 times using AdamW (Loshchilov & Hutter, 2017) with learning rate 0 .

Original: 01 , then take the run with the lowest unsupervised loss.
Revised: 01 , then take the run with the lowest unsupervised loss.

Original: Unless otherwise specified, we train
Revised: Unless otherwise specified, we train

Original: CCS using all prompts for a single training set, then evaluate it on the corresponding test split.
Revised: CCS using all prompts for a single training set, then evaluate it on the corresponding test split.

Original: [Table] [Table] Zero-shot outputs sometimes suffer from miscalibration (Zhao et al., 2021), in which models are biased towards predicting specific answers.
Revised: Zero-shot baselines. Zero-shot outputs sometimes suffer from miscalibration (Zhao et al., 2021), in which models are biased towards predicting specific answers.

Original: Calibrating the outputs to be uniform over different answers can mitigate this problem.
Revised: Calibrating the outputs to be uniform [Table] mitigate this problem.

Original: We use a variant of the calibration method presented in Zhao et al.
Revised: We use a variant of the calibration method presented in Zhao et al.

Original: (2021) by balancing predictions to be 50/50 across the two output labels.
Revised: (2021) by balancing predictions to be 50/50 across the two output labels.

Original: Specifically, if l + and l − are the logits for the positive and negative label respectively, then instead of classifying an example as positive if l + > l − , we classify it as positive if l + > l − + γ , where we select the threshold γ ∈ R so that the predictions are balanced.
Revised: Specifically, if l + and l − are the logits for the positive and negative label respectively, then instead of classifying an example as positive if l + > l − , we classify it as positive if l + > l − + γ , where we select the threshold γ ∈ R so that the predictions are balanced.

Original: We find this increases accuracy by about 5% on average.
Revised: We find this increases accuracy by about 5% on average.

Original: Unless otherwise specified, we always report zero-shot accuracy after calibration.
Revised: Unless otherwise specified, we always report zero-shot accuracy after calibration.

Original: Encoder-only models (e.g. RoBERTa and DeBERTa) cannot be easily  do zero-shot classification out of the box, so to evaluate them we follow the method of Yin et al.
Revised: Encoder-only models (e.g. RoBERTa and DeBERTa) cannot be easily used to do zero-shot classification out of the box, so to evaluate them we follow the method of Yin et al.

Original: (2020): we finetune both models on an NLI dataset (MNLI, which we do not evaluate on) and treat the difference between the entailment and contradiction probabilities as the effective logit.
Revised: (2020): we finetune both models on an NLI dataset (MNLI, which we do not evaluate on) and treat the difference between the entailment and contradiction probabilities as the effective logit.

Original: This provides a strong zero-shot baseline for encoder-only models that works even for non-NLI tasks (Yin et al., 2020).
Revised: This provides a strong zero-shot baseline for encoder-only models that works even for non-NLI tasks (Yin et al., 2020).

Original: This finetuning isn’t necessary for CCS to work on encoder-only models, but we test CCS using the same MNLI-finetuned models for ease of comparison.
Revised: This finetuning isn’t necessary for CCS to work on encoder-only models (see Appendix C), but we test CCS using the same MNLI-finetuned models for ease of comparison.

Original: Hidden states.
Revised: Hidden states.

Original: We extract the hidden states corresponding to the last token in the last layer of each model, unless otherwise specified.
Revised: We extract the hidden states corresponding to the last token in the last layer of each model for simplicity, unless otherwise specified.

Original: For encoder-decoder models, we evaluate CCS on the last layer hidden states of both the encoder and decoder, and use whichever one generally achieves a lower unsupervised loss; for T0 this is the decoder hidden states, while for T5 and UnifiedQA this is the encoder hidden states.
Revised: For encoder-decoder models, we evaluate CCS on the last layer hidden states of both the encoder and decoder, and use whichever one generally achieves a lower unsupervised loss; for T0 this is the decoder hidden states, while for T5 and UnifiedQA this is the encoder hidden states.

Original: See Appendix H.1 for  implementation details.
Revised: See Appendix G.2 for further implementation details, such as tokenization.

Original: We compare the performance of CCS on the hidden states across different layers in Section 3.3.

Original: Prompts.
Revised: Prompts.

Original: To reduce prompt sensitivity,for each dataset, we use between 8 and 13 prompts  ( 9 on average), derived or slightly modified from Sanh et al.
Revised: To reduce prompt sensitivity, we use between 8 and 13 prompts for each dataset ( 9 on average), derived or slightly modified from Sanh et al.

Original: We average across all prompts when showing results.
Revised: Unless otherwise specified, we average across all prompts when showing results.

Original: To construct contrast pairs, we let x + i be the zero-shot prompt using q i and the first label (e.g. “Positive” for sentiment classification datasets) and let x −
Revised: To construct contrast pairs, we let x + i be the zero-shot prompt using q i and the first label (e.g. “Positive” for sentiment classification datasets) and let x −

Original: i be the prompt using q i and the second label (e.g. “Negative”).
Revised: i be the prompt using q i and the second label (e.g. “Negative”).

Original: We describe allof the promptswe use in Appendix G.
Revised: We describe all prompts in Appendix I.

Original: 3. E VALUATING CCS 3.2. CCS O UTPERFORMS Z ERO -S HOT
Revised: 3. E VALUATING CCS 3.2. CCS O UTPERFORMS Z ERO -S HOT

Original: We evaluate CCS on all 6 models and compute the average accuracy across all datasets and prompts.
Revised: We evaluate CCS on all 6 models and compute the average accuracy across all datasets and prompts.

Original: T0 was trained on 9 out of 10 of the datasets we evaluate on, including some of the data in our test splits, so we ignore it when averaging over models to avoid unfair comparisons.
Revised: T0 was trained on 9 out of 10 of the datasets we evaluate on, including some of the data in our test splits, so we ignore it when averaging over models to avoid unfair comparisons.

Original: We display the results in Table 1.
Revised: We display the results in Table 1.

Original: To assess prompt sensitivity, for each model and dataset we compute the standard deviation (s.d.) of accuracy across different prompts, then average the resulting standard deviations across all datasets, which we show in parentheses in Table 1.
Revised: To assess prompt sensitivity, for each model and dataset we compute the standard deviation (s.d.) of accuracy across different prompts, then average the resulting standard deviations across all datasets, which we show in parentheses in Table 1.

Original: For comparison, we also include results when training CCS on all datasets simultaneously, which we refer to as CCS (All Data).
Revised: For comparison, we also include results when training CCS on all datasets simultaneously, which we refer to as CCS (All Data).

Original: CCS attains an accuracy of 71 .
Revised: CCS attains an accuracy of 71 .

Original: 2% on average, compared to 67 .
Revised: 2% on average, compared to 67 .

Original: 2% for calibrated zero-shot.
Revised: 2% for calibrated zero-shot.

Original: It outperforms zero-shot accuracy for every model, except for RoBERTa (where it does 2% worse) and
Revised: It outperforms zero-shot accuracy for every model, except for RoBERTa (where it does 2% worse) and

Original: T0 (for which zero-shot accuracy is inflated).
Revised: T0 (for which zero-shot accuracy is inflated).

Original: Training on all datasets improves accuracy by only an insignificant amount on average ( 0 . 3% ), but with large gains for T0 in particular ( 77 . 6% → 84 .
Revised: Training on all datasets improves accuracy by only an insignificant amount on average ( 0 . 3% ), but with large gains for T0 in particular ( 77 . 6% → 84 .

Original: These results show that CCS can exceed the performance of strong baseline methods that access the model outputs, even though this wasn’t our main goal.
Revised: These results show that CCS can exceed the performance of strong baseline methods that access the model outputs, even though this wasn’t our main goal.

Original: This indicates that it is indeed possible to classify examples with high accuracy using only unlabeled representations from ϕ ( · ) .
Revised: This indicates that it is indeed possible to classify examples with high accuracy using only unlabeled model representations.

Original: 3.2. CCS I S R OBUST T O M ISLEADING P ROMPTS
Revised: 3.2. CCS I S R OBUST T O M ISLEADING P ROMPTS

Original: Recall our goal: to discover latent knowledge in a language model even when the model outputs false text.
Revised: Recall our goal: to discover latent knowledge in a language model even when the model outputs false text.

Original: In particular, language models are typically trained to imitate text whether or not it is correct, so if a model sees false text it should intuitively be more likely to predict that subsequent text will also be false.
Revised: In particular, language models are typically trained to imitate text whether or not it is correct, so if a model sees false text it should intuitively be more likely to predict that subsequent text will also be false.

Original: Based on this idea, we provide an initial proof of concept that CCS can make progress toward out goal by constructing prompts that aim to mislead the outputs of language models.
Revised: Based on this idea, we provide an initial proof of concept that CCS can make progress toward our goal by constructing prompts that aim to mislead the outputs of language models.

Original: Specifically, we add a prefix to the beginning of our zero-shot prompts that consists of questions answered incorrectly (Figure 6).
Revised: Specifically, we add a prefix to the beginning of our zero-shot prompts that consists of questions answered incorrectly (Figure 5).

Original: The hope is that such a prefix will decrease zero-shot accuracy because the model will imitate its context and answer subsequent questions incorrectly even if it internally “knows” better.
Revised: The hope is that such a prefix will decrease zero-shot accuracy because the model will imitate its context and answer subsequent questions incorrectly even if it internally “knows” better.

Original: We found that while most models are robust to this type of prefix (see Appendix B), it significantly drops calibrated zero-shot performance in UnifiedQA, decreasing accuracy from 80 .
Revised: We found that while most models are robust to this type of prefix (see Appendix B), it significantly drops calibrated zero-shot performance in UnifiedQA, decreasing accuracy from 80 .

Original: 4% to 70 .
Revised: 4% to 70 .

Original: We evaluate CCS on these examples and show the results in Figure 4.
Revised: We evaluate CCS on these examples and show the results in Figure 4 of the Appendix.

Original: We find that despite the 9 . 5% drop in zero-shot accuracy, CCS maintains high accuracy ( 82 . 1% → 83 . 8% ).
Revised: We find that despite the 9 . 5% drop in zero-shot accuracy, CCS maintains high accuracy ( 82 . 1% → 83 . 8% ).

Original: This provides evidence that our method can still work well even whenlanguage model outputs are unreliable.
Revised: This provides evidence that our method can still work well even when model outputs are unreliable.

Original: 3. A NALYZING CCS
Revised: 3. A NALYZING CCS

Original: We have shown that CCS attains strong classification performance in standard settings and when we deliberately mislead models.
Revised: We have shown that CCS attains strong classification performance in standard settings and when we deliberately mislead models.

Original: Moreover, we have described our motivation as discovering latent representations of truth in language models, but in practice CCS just finds a direction in representation space that attains high accuracy.
Revised: Moreover, we have described our motivation as discovering latent representations of truth in language models, but in practice CCS just finds a direction in representation space that attains high accuracy.

Original: This raises the question: in what sense is CCS actually finding “truth” features?
Revised: This raises the question: in what sense is CCS actually finding “truth” features?

Original: We now provide a preliminary investigation of this question.
Revised: We now provide a preliminary investigation of this question.

Original: 3.3. CCS F INDS A T ASK -A GNOSTIC R EPRESENTATION OF T RUTH
Revised: 3.3. CCS F INDS A T ASK -A GNOSTIC R EPRESENTATION OF T RUTH

Original: From the results described so far, it may be possible that the classifier we find is capturing datasetspecific properties, such as which of two labels (e.g. “Yes” vs. “No”) is more likely to be correct.
Revised: From the results described so far, it may be possible that the classifier we find is capturing datasetspecific properties, such as which of two labels (e.g. “Yes” vs. “No”) is more likely to be correct.

Original: We rule this out by showing that it generalizes across completely different tasks, including ones with different label spaces (such as from “Yes” and “No” for a generic task to “Positive” and “Negative” for sentiment classification).
Revised: We rule this out by showing that it generalizes across completely different tasks, including ones with different label spaces (such as from “Yes” and “No” for a generic task to “Positive” and “Negative” for sentiment classification).

Original: We train and test CCS on every pair of datasets, and show results for several models in Figure 2.
Revised: In particular, we train and test CCS on every pair of datasets, and show the resulting transfer for several models in Figure 2.

Original: (See Appendix F for results with other models.)
Revised: (See Appendix F for results with other models.)

Original: We find that CCS indeed transfer wells: in the majority of datasets, the transfer accuracy of CCS is competitive with training and testing on the same dataset (the last row in Figure 2, “No transfer”).
Revised: We find that CCS indeed transfer wells: in the majority of datasets, the transfer accuracy of CCS is competitive with training and testing on the same dataset (the last row in Figure 2, “No transfer”).

Original: Transfer performance can even outperform the no-transfer setting, especially when we train on simpler tasks, such as sentimentor topic classification.
Revised: Transfer performance can even outperform the no-transfer setting, especially when we train on simpler tasks, such as sentiment classification.

Original: For example, training CCS on the Amazon sentiment dataset achieves an average transfer accuracy of 71 .
Revised: For example, training CCS on the Amazon sentiment dataset achieves an average transfer accuracy of 71 .

Original: 8% , which iseven 0 .
Revised: 8% , which is 0 .

Original: 6% higher than CCS without transfer.
Revised: 6% higher than CCS without transfer.

Original: We speculate that training just on Amazon then transferring performs  well because the difference in representations between correct and incorrect answers is especially pronounced for easier tasks.
Revised: We speculate that this performs so well because the difference in representations between correct and incorrect answers is especially pronounced for easier tasks like Amazon.

Original: Additionally, transfer accuracy tends to be similar for many datasets.
Revised: Additionally, transfer accuracy tends to be similar for many datasets.

Original: For example, training on IMDB (row 1 of Figure 2) has similar accuracy as training on DBPedia (row 4).
Revised: For example, training on IMDB (row 1 of Figure 2) has similar accuracy as training on DBPedia (row 4).

Original: This provides evidence that CCS can find a functionally similar direction across many different types of training datasets.
Revised: This provides evidence that CCS can find a functionally similar direction across many different types of training datasets.

Original: Overall, these results suggest that (1) models may have a task-agnostic representation related to what is true, and that (2) CCS may approximately find this representation even without diverse data.
Revised: Overall, these results suggest that (1) models may have a task-agnostic representation related to what is true, and that (2) CCS may approximately find this representation even without diverse data.

Original: 3.3. CCS D OES N OT J UST R ECOVER M ODEL O UTPUTS
Revised: 3.3. CCS D OES N OT J UST R ECOVER M ODEL O UTPUTS

Original: One possibility is that CCS can only recover knowledge already contained in a model’s outputs.
Revised: One possibility is that CCS can only recover knowledge already contained in a model’s outputs.

Original: We have shown that CCS can outperform zero-shot accuracy (Section 3.2.1), especially when model outputs are misled (Section 3.2.2), which already provides evidence against this possibility.
Revised: We have shown that CCS can outperform zero-shot accuracy (Section 3.2.1), especially when model outputs are misled (Section 3.2.2), which already provides evidence against this possibility.

Original: We now provide additional evidence against this concern.
Revised: We now provide additional evidence against this concern.

Revised: First, if CCS were just recovering knowledge in the model outputs, using the last layer of a network (right before the outputs) should presumably outperform intermediate layers (which are more causally distant from the outputs).

Original:  similar between using this prefix with correct and incorrect answers, in agreement with the findings of Min et al.
Revised: However, for T5 and UnifiedQA, we find that using hidden states in the we found that model behavior was qualitatively similar between using this prefix with correct and incorrect answers, in agreement with the findings of Min et al.

Original: 2022b)
Revised: (2022b)

Original: that using incorrect labels in demonstrations often does not significantly degrade the  of a 
Revised: that using incorrect labels in demonstrations often does not significantly degrade the performance of a few-shot prompt.

Original: Consequently, the  may instead actually be reducing accuracy because it is 
Revised: Consequently, the prefix may instead actually be reducing accuracy because it is out-of-distribution

Original: I M D B A m a z
Revised: I M D B A m a z

Original: o n A G N e w s D B p
Revised: o n A G N e w s D B p

Original: e d i a C
Revised: e d i a C

Original: O P
Revised: O P

Original: A R T E B
Revised: A R T E B

Original: O O L Q Q N L
Revised: O O L Q Q N L

Original: I P I Q
Revised: I P I Q

Original: A S t o r y C
Revised: A S t o r y C

Original: l
Revised: l

Original: o z e
Revised: o z e

Original: I M D B
Revised: I M D B

Original: A m a z o n
Revised: A m a z o n

Original: A G N e w s
Revised: A G N e w s

Original: D B p
Revised: D B p

Original: e d i a
Revised: e d i a

Original: C O P A
Revised: C O P A

Original: R T E
Revised: R T E

Original: B O O L Q
Revised: B O O L Q

Original: Q N L
Revised: Q N L

Original: I
Revised: I

Original: P I Q A
Revised: P I Q A

Original: S t o r y C l o z e
Revised: S t o r y C l o z e

Original: N o T r a n s f e r
Revised: N o T r a n s f e r

Original: T r a i n
Revised: T r a i n

Original: [Table]
Revised: [Table]

Original: UQA
Revised: UQA

Original: I M D B A m a z
Revised: I M D B A m a z

Original: o n A G N e w s D B p
Revised: o n A G N e w s D B p

Original: e d i a C
Revised: e d i a C

Original: O P
Revised: O P

Original: A R T E B
Revised: A R T E B

Original: O O L Q Q N L
Revised: O O L Q Q N L

Original: I P I Q
Revised: I P I Q

Original: A S t o r y C
Revised: A S t o r y C

Original: l
Revised: l

Original: o z e
Revised: o z e

Original: Test
Revised: Test

Original: [Table]
Revised: [Table]

Original: T
Revised: T

Original: [Table]
Revised: [Table]

Original: n A
Revised: n A

Original: [Table]
Revised: [Table]

Original: w
Revised: w

Original: [Table]
Revised: [Table]

Original: D
Revised: D

Original: [Table]
Revised: [Table]

Original: p e d i a C
Revised: p e d i a C

Original: O P
Revised: O P

Original: A R T E B
Revised: A R T E B

Original: O O L Q Q N L
Revised: O O L Q Q N L

Original: I P I Q
Revised: I P I Q

Original: A S t o r y C
Revised: A S t o r y C

Original: l
Revised: l

Original: o z e
Revised: o z e

Original: [Table] [Table]
Revised: [Table] [Table]

Original: DeBERTaFirst, if CCS were just recovering knowledge in the model outputs, using the last layer of a network (right before the outputs) should presumably outperform intermediate layers (which are more causally distant from the outputs). However, for T5 and UnifiedQA, we find that using hidden states in the      middle of the network outperform hidden states at the end of the network when using CCS (see
Revised: DeBERTa      middle of the network outperform hidden states at the end of the network when using CCS (see

Original: Figure 9).
Revised: Figure 10).

Original: This is especially true for UnifiedQA on misleading prefixes; we find that using the encoder hidden states is robust to misleading prefixes (Section 3.2.2), but that accuracy using the decoder hidden states drops from 81 .
Revised: This is especially true for UnifiedQA on misleading prefixes; we find that using the encoder hidden states is robust to misleading prefixes (Section 3.2.2), but that accuracy using the decoder hidden states drops from 81 .

Original: 0% to 73 .
Revised: 0% to 73 .

Original: 5% , a similar amount to zero-shot accuracy.
Revised: 5% , a similar amount to zero-shot accuracy.

Original: This suggests that compared to the later layers of a model, intermediate layers are more robust and less correlated with the model outputs, and that CCS can take advantage of this.
Revised: This suggests that compared to the later layers of a model, intermediate layers are more robust and less correlated with the model outputs, and that CCS can take advantage of this.

Original: Finally, if CCS were just recovering knowledge in the model outputs, we would only expect it to work in cases where model outputs are informative.
Revised: Finally, if CCS were just recovering knowledge in the model outputs, we would only expect it to work in cases where model outputs are informative.

Original: However, we show in Appendix C that CCS still works with masked language models when their outputs are uninformative: when we don’t [MASK] any input tokens, and when we prompt models so that the labels used to construct contrast pairs appear in the middle of a prompt rather than at the end.
Revised: However, we show in Appendix C that CCS still works with masked language models when their outputs are uninformative: when we don’t [MASK] any input tokens, and when we prompt models so that the labels used to construct contrast pairs appear in the middle of a prompt rather than at the end.

Original: These results show that CCS can  recover latent knowledge in a model that is distinct from—and more useful than—what the model outputs.
Revised: These results show that CCS can sometimes recover latent knowledge in a model that is distinct from—and more useful than—what the model outputs

Original:  3.3. T RUTH IS A S ALIENT F EATURE
Revised: . 3.3. T RUTH IS A S ALIENT F EATURE

Original: From the results we have presented so far, it is possible that the direction learned by CCS is difficult to find and requires using a large amount of unsupervised data.
Revised: From the results we have presented so far, it is possible that the direction learned by CCS is difficult to find and requires using a large amount of unsupervised data.

Original: We provide evidence against this possibility by showing that finding such a direction can both (1) often be done with only a small amount of data, and can also (2) often be done by essentially taking the top principal component of a slightly modified representation space.
Revised: We provide evidence against this possibility by showing that finding such a direction can both (1) often be done with only a small amount of data, and can also (2) often be done by essentially taking the top principal component of a slightly modified representation space.

Original: CCS doesn’t require much data.
Revised: CCS doesn’t require much data.

Original: We  evaluate how  CCS performs with different amounts of data.
Revised: We now evaluate how well CCS performs with different amounts of data.

Original: Whereas before we trained CCS using the full training set and all prompts, here we use limited data and a single prompt.
Revised: Whereas before we trained CCS using the full training set and all prompts, here we use limited data and a single prompt.

Original: Specifically, we train using only k unlabeled contrast pairs, using the single prompt for each model and dataset that achieves the highest zero-shot accuracy.
Revised: Specifically, we train using only k unlabeled contrast pairs, using the single prompt for each model and dataset that achieves the highest zero-shot accuracy.

Original: We still test on all prompts for each dataset.
Revised: We still test on all prompts for each dataset.

Original: We resample k points 32 times for each of k = 1 , 2 , 4 , · · · , and take the average accuracy across those 32 samples.
Revised: We resample k points 32 times for each of k = 1 , 2 , 4 , · · · , and take the average accuracy across those 32 samples.

Original: Finally, we plot the average such accuracy across all dataset and prompts for several models in Figure 3.
Revised: Finally, we plot the average such accuracy across all datasets and prompts for several models in Figure 3.

Original: We find that while CCS benefits from more data, it can often do well with very limited data.
Revised: We find that while CCS benefits from more data, it can often do well with very limited data.

Original: In fact, it can sometimes even do well with only a single contrast pair, though we find high variance across individual datasets; see Appendix D for more details.
Revised: In fact, it can sometimes even do well with only a single contrast pair, though we find high variance across individual datasets; see Appendix D for more details.

Original: This suggests that the strong performance of
Revised: This suggests that the strong performance of

Original: CCS does not primarily come from using a large amount of unsupervised data, and indicates that the direction learned by CCS may be relatively easy to find.
Revised: CCS does not primarily come from using a large amount of unsupervised data, and indicates that the direction learned by CCS may be relatively easy to find.

Original: Contrastive Representation Clustering.
Revised: Contrastive Representation Clustering.

Original: We now show that directions correlated with the truth may be “salient” in a different way: by showing that we can also find such directions using either [Figure] [Figure]
Revised: We now show that directions correlated with the truth may be “salient” in a different way: by showing that we can also find such directions using either (1) PCA or (2) clustering.

Original: While it benefits from more examples, for several models CCS can work well with limited data.

Original: (1) PCA or (2) clustering.

Original: Specifically, suppose we construct contrast pairs ( x + i , x − i ) as before.
Revised: Specifically, suppose we construct contrast pairs ( x + i , x − i ) as before.

Original: Intuitively, these two examples are qualitatively almost identical except that one is true and the other is false, so the main difference between the representations ˜ ϕ
Revised: Intuitively, these two examples are qualitatively almost identical except that one is true and the other 512 #Samples [Figure] [Figure]

Revised: DeBERTa is false, so the main difference between the representations ˜ ϕ

Original: ( x + i ) and ˜ ϕ
Revised: ( x + i ) and ˜ ϕ

Original: ( x − i ) should relate to truth.
Revised: ( x − i ) should relate to truth.

Original: Consequently, we can take the differences in (normalized) hidden states, { ˜ ϕ
Revised: Consequently, we can take the differences in (normalized) hidden states, { ˜ ϕ

Original: ( x + i ) − ˜ ϕ
Revised: ( x + i ) − ˜ ϕ

Original: ( x − i ) } ni =1 , and cluster them.
Revised: ( x − i ) } ni =1 , and cluster them.

Original: We call this method Contrastive Representation Clustering (CRC).
Revised: We call this method Contrastive Representation Clustering (CRC).

Original: Clustering can be achieved by taking the top principal component (TPC) and thresholding at 0 , or by doing a “bimodal salience search” (BSS) to find a direction that looks bimodal; see Appendix H.3 for further details.
Revised: Clustering can be achieved by taking the top principal component (TPC) and thresholding at 0 , or by doing a “bimodal salience search” (BSS) to find a direction that looks bimodal; see Appendix G.4 for further details.

Original: We compare CCS and the two versions of Contrastive Representation Clustering in Table 2, using the same setting as in Table 1.
Revised: We compare CCS and these two variants of Contrastive Representation Clustering in Table 2, using the same setting as in Table 1.

Original: While CCS performs best, all methods attain high accuracy and are competitive with zero-shot performance.
Revised: While CCS performs best, all methods attain high accuracy and are competitive with zero-shot performance.

Original: This indicates both that (1) representations of truth often lie in a high-variance direction in the contrastive representation space { ˜ ϕ
Revised: This indicates both that (1) representations of truth often lie in a high-variance direction in the contrastive representation space { ˜ ϕ

Original: ( x + i ) − ˜ ϕ
Revised: ( x + i ) − ˜ ϕ

Original: ( x − i ) } ni =1 , and also that (2) true and false examples are often well-clustered in this same contrastive space.
Revised: ( x − i ) } ni =1 , and also that (2) true and false examples are often well-clustered in this same contrastive space.

Original: This strengthens the idea that representations of truth may be salient features inside models that are relatively easy to find.
Revised: This strengthens the idea that representations of truth may be salient features inside models that are relatively easy to find.

Original: This may help explain why CCS can perform well without using any supervision, and how it can do so even with only a limited amount of unlabeled data.
Revised: This may help explain why CCS can perform well without using any supervision, and how it can do so even with only a limited amount of unlabeled data.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Zero-Shot Prompting.
Revised: Zero-Shot Prompting.

Original: Since the release of GPT-3 (Brown et al., 2020), one of the main paradigms for eliciting what models know has been zero-shot prompting (Liu et al., 2022; Beltagy et al., 2022).
Revised: Since the release of GPT-3 (Brown et al., 2020), one of the main paradigms for eliciting what models know has been zero-shot prompting (Liu et al., 2022; Beltagy et al., 2022).

Original: Zero-shot exploits how language models are trained to predict diverse data from the internet, which incidentally includes tasks such as question-answering.
Revised: Zero-shot exploits how language models are trained to predict diverse data from the internet, which incidentally includes tasks such as question-answering.

Original: If prompted appropriately, this can be used to solve various useful tasks with reasonable performance (Brown et al., 2020).
Revised: If prompted appropriately, this can be used to solve various useful tasks with reasonable performance (Brown et al., 2020).

Original: However,since these models are trained to imitate human-generated data, this bounds the quality of their outputs.
Revised: However, these models are trained to imitate human-generated data, which bounds the quality of their outputs.

Original: Many methods improve upon vanilla zero-shot prompting (Liu et al., 2022;
Revised: Many methods improve upon vanilla zero-shot prompting (Liu et al., 2022;

Original: Zhao et al., 2021;
Revised: Zhao et al., 2021;

Original: Lu et al., 2022; Wei et al., 2022b; Min et al., 2022a).
Revised: Lu et al., 2022; Wei et al., 2022b; Min et al., 2022a).

Original: While our goal is not to improve zero-shot performance, some of the ideas underlying these methods are similar to CCS.
Revised: While our goal is not to improve zero-shot performance, some of the ideas underlying these methods are similar to CCS.

Original: Particularly relevant are methods that also leverage unsupervised consistency properties, such as Jung et al.
Revised: Particularly relevant are methods that also leverage unsupervised consistency properties, such as Jung et al.

Original: ; Zhou et al.
Revised: ; Zhou et al.

Original: However, these methods still bootstrap from language model outputs trained via imitation learning, which limits their applicability to our main goals.
Revised: However, these methods still bootstrap from language model outputs trained via imitation learning, which limits their applicability to our main goals

Original: [Table]
Revised: [Table]

Original: To illustrate this, imagine we train reinforcement learning agents to play a game such as Diplomacy (Gray et al., 2021), in which players have incentives to lie to each other.
Revised: To illustrate this, imagine we train reinforcement learning agents to play a game such as Diplomacy (FAIR et al., 2022), in which players have incentives to lie to each other.

Original: Then those agents may learn to lie in a way that is difficult to detect, but they may still internally represent whether they are lying.
Revised: Then those agents may learn to lie in a way that is difficult to detect, but they may still internally represent whether they are lying.

Original: Because their outputs would be deliberately misleading, standard zero-shot methods may be very unreliable.
Revised: Because their outputs would be deliberately misleading, standard zero-shot methods may be very unreliable.

Original: In contrast, techniques like CCS may still be able to detect whether those models are lying by finding representations of truth in their activations that contradict their outputs.
Revised: In contrast, techniques like CCS may still be able to detect whether those models are lying by finding representations of truth in their activations that contradict their outputs.

Original: Truthfulness.
Revised: Truthfulness.

Original: There has been increasing interest in making language models truthful (Evans et al., 2021; Lin et al., 2022).
Revised: There has been increasing interest in making language models truthful (Evans et al., 2021; Lin et al., 2022).

Original: One aspect of truthfulness that has received substantial attention is factuality (Thorne et al., 2018; Maynez et al., 2020).
Revised: One aspect of truthfulness that has received substantial attention is factuality (Thorne et al., 2018; Maynez et al., 2020).

Original: For instance, many techniques aim to improve the factuality of models by augmenting them with retrieval methods (Nakano et al., 2021;
Revised: For instance, many techniques aim to improve the factuality of models by augmenting them with retrieval methods (Nakano et al., 2021;

Original: Menick et al., 2022), which allows them to cite their sources.
Revised: Menick et al., 2022), which allows them to cite their sources.

Original: In contrast, we focus on truthfulness more generally, which also includes procedural knowledge such as reasoning or  inference tasks.
Revised: In contrast, we focus on truthfulness more generally, which also includes procedural knowledge such as reasoning or natural language inference tasks.

Original: An approach to making language models truthful in this more general setting is to finetune them using either human demonstrations (Khashabi et al., 2020; Sanh et al., 2021;
Revised: An approach to making language models truthful in this more general setting is to finetune them using either human demonstrations (Khashabi et al., 2020; Sanh et al., 2021;

Original: Zhong et al., 2021;
Revised: Zhong et al., 2021;

Original: Wei et al., 2022a) or reinforcement learning from human feedback (Christiano et al., 2017; Stiennon et al., 2020; Askell et al., 2021; Bai et al., 2022;
Revised: Wei et al., 2022a) or reinforcement learning from human feedback (Christiano et al., 2017; Stiennon et al., 2020; Askell et al., 2021; Bai et al., 2022;

Original: Ouyang et al., 2022).
Revised: Ouyang et al., 2022).

Original: These techniques have been widely successful at improving performance, but unlike our method they rely on being able to provide ground truth labels, which can be intractable in many settings.
Revised: These techniques have been widely successful at improving performance, but unlike our method they rely on being able to provide ground truth labels, which can be intractable in many settings.

Original: Some work has aimed to go beyond the direct supervision humans can provide by augmenting supervision with AI systems (Christiano et al., 2018; Irving et al., 2018; Leike et al., 2018; Perez et al., 2022).
Revised: Some work has aimed to go beyond the direct supervision humans can provide by augmenting supervision with AI systems (Christiano et al., 2018; Irving et al., 2018; Leike et al., 2018; Perez et al., 2022).

Original: This may expand the range of applications we can supervise, but many of these proposals remain theoretical, and it is unclear just how far these techniques can generalize.
Revised: This may expand the range of applications we can supervise, but many of these proposals remain theoretical, and it is unclear just how far these techniques can generalize.

Original: Christiano et al.
Revised: Christiano et al.

Original: (2022) reframes this issue by posing the problem of Eliciting Latent Knowledge (ELK).
Revised: (2022) reframes this issue by posing the problem of Eliciting Latent Knowledge (ELK).

Original: Like our problem statement, ELK is about eliciting knowledge from models even in cases where humans cannot evaluate that knowledge.
Revised: Like our problem statement, ELK is about eliciting knowledge from models even in cases where humans cannot evaluate that knowledge.

Original: However, ELK frames this as a worst-case theoretical problem, while we frame this as an empirical problem that we can make progress on using current models.
Revised: However, ELK frames this as a worst-case theoretical problem, while we frame this as an empirical problem that we can make progress on using current models.

Original:  D ISCUSSION 5. L IMITATIONS AND F UTURE W ORK
Revised:  D ISCUSSION 5. L IMITATIONS AND F UTURE W ORK

Original: Our work has a number of limitations.
Revised: Our work has a number of limitations.

Original: First, CCS relies on the existence of a direction in activation space that separates true and false inputs well, in the sense that a supervised probe on the activations would be able to attain high accuracy (if it hypothetically had access to the ground truth labels).
Revised: First, CCS relies on the existence of a direction in activation space that separates true and false inputs well, in the sense that a supervised probe on the activations would be able to attain high accuracy (if it hypothetically had access to the ground truth labels).

Original: This requires that a model is both capable of evaluating the truth of a given input, and also that the model actively evaluates the truth of that input.
Revised: This requires that a model is both capable of evaluating the truth of a given input, and also that the model actively evaluates the truth of that input.

Original: It is not clear when these conditions holds precisely.
Revised: It is not clear when these conditions hold precisely.

Original: Second, we did not evaluate our method on setups involving active “lying” or “deception” (Kenton et al., 2021; Evans et al., 2021) by models, as we aren’t aware of existing evaluation setups for this setting.
Revised: Second, we did not evaluate our method on setups involving active “lying” or “deception” (Kenton et al., 2021; Evans et al., 2021) by models, as we aren’t aware of existing evaluation setups for this setting.

Original: If future work develops such a setup, a good stress test would be to apply CCS to do “lie detection” in that setting – though this mayalso require slight tweaks to the method.
Revised: If future work develops such a setup, a good stress test would be to apply CCS to do “lie detection” in that setting. This may require modifications or extensions to the method, such as more explicitly ensuring that it recovers the truth of an input rather than what the model says.

Original: There are also various straightforward improvements to our method that one could explore.
Revised: There are also various straightforward improvements to our method that one could explore.

Original: This includes adding additional consistency constraints,  calibrating its probabilities, generalizing it beyond the yes-no question-answering setting, generalizing it to cases where answers aren’t clear-cut, and closing the remaining gap betweenthe performance of CCS and the logistic regression ceiling.
Revised: This includes adding additional consistency constraints, improving its reliability, calibrating its probabilities, generalizing it beyond the yes-no question-answering setting, generalizing it to cases where answers aren’t clear-cut, and closing the remaining gap between CCS and the logistic regression ceiling.

Original: 5. C ONCLUSION
Revised: 5. C ONCLUSION

Original: As language models become more capable, they will be increasingly used as components in larger
Revised: As language models become more capable, they will be increasingly used as components in larger

Original: AI systems trained with reinforcement learning.
Revised: AI systems trained with reinforcement learning.

Original: As this occurs, falsehoods arising from misaligned training objectives may become more common, severe, and difficult to detect.
Revised: As this occurs, falsehoods arising from misaligned training objectives may become more common, severe, and difficult to detect.

Original: In principle, models may even develop instrumental incentives to lie: for example, if a human evaluator would disapprove of bad model behavior, models may learn to lie about their behavior to achieve higher reward.
Revised: In principle, models may even develop instrumental incentives to lie: for example, if a human evaluator would disapprove of bad model behavior, models may learn to lie about their behavior to achieve higher reward.

Original: If so, those models would be optimizing against human evaluators, making it precarious to rely on those evaluators for assessing the truth of what models say.
Revised: If so, those models would be optimizing against human evaluators, making it precarious to rely on those evaluators for assessing the truth of what models say.

Original: Because unsupervised methods for eliciting answers circumvent this issue, they may still work even in this scenario.
Revised: Because unsupervised methods for eliciting answers circumvent this issue, they may still work even in this scenario.

Original: We show that it is possible to make progress on such methods today; the empirical success of our approach suggests that unsupervised methods are both a tractable and underexplored research direction. 
Revised: We show that it is possible to make progress on such methods today; the empirical success of our approach suggests that unsupervised methods are both a tractable and underexplored research direction. A CKNOWLEDGEMENTS

Revised: We are very grateful to Jared Kaplan for helpful experiment suggestions and resources early on in the project.

Revised: We thank Beth Barnes and Paul Christiano for valuable discussions regarding the longer-term impacts of this work.

Revised: We are also grateful to Jessy Lin, Alex Pan, Ruiqi Zhong, Yaodong Yu, the anonymous reviewers, and several others for useful feedback on earlier versions of this paper.

Revised: CB is supported by an Open Philanthropy AI Fellowship

Original: A I DENTIFYING C LUSTERS
Revised: A I DENTIFYING C LUSTERS

Original: CCS predicts yes-no answers to a set of unlabeled questions q 1 , . .
Revised: CCS predicts yes-no answers to a set of unlabeled questions q 1 , . .

Original: . , q n .
Revised: . , q n .

Original: However, it does not identify which prediction label corresponds to true and which corresponds to false.
Revised: However, it does not identify which prediction label corresponds to true and which corresponds to false.

Original: We now describe how one can do so in principle, as long as a model is also able to take conjunctions.
Revised: We now describe how one can do so in principle, as long as a model is also able to take conjunctions.

Original: Suppose we run our method and find, for example, that x and x ′ end up in different clusters.
Revised: Suppose we run our method and find, for example, that x and x ′ end up in different clusters.

Original: Then we know that these two statements have opposite truth values.
Revised: Then we know that these two statements have opposite truth values.

Original: As a result, if we take the conjunction of these two statements, x ∧ x ′ then it should be false, and if we take the disjunction, x ∨ x ′ , then it should be true.
Revised: As a result, if we take the conjunction of these two statements, x ∧ x ′ then it should be false, and if we take the disjunction, x ∨ x ′ , then it should be true.

Original: This allows us to identify which cluster is true and which is false in a completely unsupervised way.[Table]
Revised: This allows us to identify which cluster is true and which is false in a completely unsupervised way.

Original: B M ISLEADING P REFIX D ETAILS B. M ISLEADING P REFIX
Revised: B M ISLEADING P REFIX D ETAILS B. M ISLEADING P REFIX

Original: In Section 3.2, we mentioned that model outputs can be biased by using the misleading few-shot prefix shown in Figure 6.
Revised: In Section 3.2, we mentioned that model outputs can be biased by using the misleading few-shot prefix shown in Figure 5.

Original: We show the effect of the misleading prefix for all models in Table 3, and we illustrate the drop in accuracy for each dataset and prompt in Figure 4.[Figure]
Revised: We show the effect of the misleading prefix for all models in Table 3, and we illustrate the drop in accuracy for each dataset and prompt in Figure 4.

Original: B.1. E VALUATING T HE E FFECT OF T HE M ISLEADING P REFIX
Revised: B.1. E VALUATING T HE E FFECT OF T HE M ISLEADING P REFIX

Original: The hope is that the misleading prefix causes models to imitate the sorts of incorrect answers in the prefix.
Revised: The hope is that the misleading prefix causes models to imitate the sorts of incorrect answers in the prefix.

Original: However, we found that results are qualitatively similar with correct answers, making the actual interpretation of the effect of this prefix ambiguous.
Revised: However, we found that results are qualitatively similar with correct answers, making the actual interpretation of the effect of this prefix ambiguous.

Original: Nevertheless, to better understand what this prefix is doing, we visualize the top 100 tokens for both this misleading prefix and no prefix on average across all examples for UnifiedQA on the IMDB sentiment dataset, and show the results in Figure 5.
Revised: Nevertheless, to better understand what this prefix is doing, we visualize the top 100 tokens for both this misleading prefix and no prefix on average across all examples for UnifiedQA on the IMDB sentiment dataset, and show the results in Figure 6.

Original: Without this prefix, the top two tokens are the actual labels (‘positive’ and ‘negative’).
Revised: Without this prefix, the top two tokens are the actual labels (‘positive’ and ‘negative’).

Original: In contrast, with the prefix, these tokens have lower probabilities relative to other tokens, with the highest probability tokens instead being irrelevant to the true answer.
Revised: In contrast, with the prefix, these tokens have lower probabilities relative to other tokens, with the highest probability tokens instead being irrelevant to the true answer.

Original: For example, since the prefix includes several answers with numbers, many of the top tokens are also numbers.
Revised: For example, since the prefix includes several answers with numbers, many of the top tokens are also numbers.

Original: This provides some evidence that the prefix is actually causing the model to output false text by imitating its context in a meaningful sense. 
Revised: This provides some evidence that the prefix is actually causing the model to output false text by imitating its context in a meaningful sense. [Figure] Method Prefix

Original: C M ASKED L ANGUAGE M ODELING R ESULTS
Revised: RoBERTa DeBERTa GPT-J T

Revised: UQA

Revised: T

Revised: [Table]

Revised: Mean

Revised: [Table]

Revised: Calibrated

Revised: [Table]

Revised: shot

Revised: [Table]

Revised: C M ASKED L ANGUAGE M ODELING R ESULTS

Original: We now provide an initial demonstration that CCS can work well even when a model’s outputs are not very useful.
Revised: We now provide an initial demonstration that CCS can work well even when a model’s outputs are not very useful.

Original: First, we evaluate our method on a model trained exclusively with the masked language modeling (MLM) objective: DeBERTa-v2 (He et al., 2021) without NLI finetuning (unlike in Section 3, where we finetuned DeBERTa on an NLI task to use it in the zero-shot setting).
Revised: First, we evaluate our method on a model trained exclusively with the masked language modeling (MLM) objective: DeBERTa-v2 (He et al., 2021) without NLI finetuning (unlike in Section 3, where we finetuned DeBERTa on an NLI task to use it in the zero-shot setting).

Original: The outputs of DeBERTa are unlikely to be meaningful if we provide a raw input without using any [MASK] tokens 4 .
Revised: The outputs of DeBERTa are unlikely to be meaningful if we provide a raw input without using any [MASK] tokens 3 .

Original: If CCS only works when the model outputs are useful, then it should perform poorly in this setting.
Revised: If CCS only works when the model outputs are useful, then it should perform poorly in this setting.

Original: We also consider one other way that model outputs can be uninformative.
Revised: We also consider one other way that model outputs can be uninformative.

Original: Given a sequence of tokens as input, DeBERTa has a different set of output logits for each input token.
Revised: Given a sequence of tokens as input, DeBERTa has a different set of output logits for each input token.

Original: For a given contrast example (a question and a candidate answer), the most informative output should intuitively be the logits that predict the answer.
Revised: For a given contrast example (a question and a candidate answer), the most informative output should intuitively be the logits that predict the answer.

Original: For instance, if the contrast example x + is “Is 2+2=4?
Revised: For instance, if the contrast example x + is “Is 2+2=4?

Original: Yes”, then the most informative output should be the logits corresponding to candidate answer “Yes”.
Revised: Yes”, then the most informative output should be the logits corresponding to candidate answer “Yes”.

Original: Consequently, if we instead format each example so that the label (e.g. “Yes” or “No”) appears in the middle of the prompt, then the output corresponding to the final token should not be very informative.
Revised: Consequently, if we instead format each example so that the label (e.g. “Yes” or “No”) appears in the middle of the prompt, then the output corresponding to the final token should not be very informative.

Original: If CCS only works well when the model outputs are useful, then it should also perform poorly in this setting (as long as CCS uses the last-token hidden states as usual).
Revised: If CCS only works well when the model outputs are useful, then it should also perform poorly in this setting (as long as CCS uses the last-token hidden states as usual).

Original: To show that CCS can work even when model outputs aren’t informative, based on the above discussion we now do an initial test of CCS when we simultaneously (1) use DeBERTa-v2 (MLMpretrained only), and (2) format inputs so that the label is in the middle of the prompt rather than at the end.
Revised: To show that CCS can work even when model outputs aren’t informative, based on the above discussion we now do an initial test of CCS when we simultaneously (1) use DeBERTa-v2 (MLMpretrained only), and (2) format inputs so that the label is in the middle of the prompt rather than at the end.

Original: In particular, we apply CCS on the Amazon dataset, where we randomly samplenew (unlabeled) points and use a 60/40 train-test split as before.
Revised: In particular, we apply CCS on the Amazon dataset, where we randomly samplenew (unlabeled) points and use a 60/40 train-test split as before.

Original: As usual we continue to use the last-token hidden states for CCS.
Revised: As usual we continue to use the last-token hidden states for CCS.

Original: As usual, we use the default Huggingface tokenizer and we do not [MASK] any input tokens.
Revised: As usual, we use the default Huggingface tokenizer and we do not [MASK] any input tokens.

Original: We use the following custom prompt to test (2):
Revised: We use the following custom prompt to test (2):

Original: The following movie review expresses a [label] sentiment:\n[text]
Revised: The following movie review expresses a [label] sentiment:\n text]

Original: Even though it is unclear why the model outputs should be useful with this setup, we find that
Revised: Even though it is unclear why the model outputs should be useful with this setup, we find that

Original: CCS can indeed still perform well, attaining an accuracy of 93 .
Revised: CCS can indeed still perform well, attaining an accuracy of 93 .

Original: For comparison, this is nearly identical to the approximately 94% accuracy of CCS when evaluated on the Amazon dataset using
Revised: For comparison, this is nearly identical to the approximately 94% accuracy of CCS when evaluated on the Amazon dataset using

Original: NLI-finetuned DeBERTa and the original prompts (see Figure 11).
Revised: NLI-finetuned DeBERTa and the original prompts (see Figure 9).

Original: We argued above that the model outputs should not be very useful in this setting.
Revised: We argued above that the model outputs should not be very useful in this setting.

Original: We now verify that this is the case.
Revised: We now verify that this is the case.

Original: To do so, we evaluate a modified version of zero-shot prompting adapted to work for masked language models when we use no [MASK] tokens in the input.
Revised: To do so, we evaluate a modified version of zero-shot prompting adapted to work for masked language models when we use no [MASK] tokens in the input.

Original: Specifically, on an input x + or x − , we (1) take the logit vector corresponding to the last input token, (2) select the corresponding logit for “positive” and the corresponding logit for “negative” (the two candidate labels), (3) take the difference between these, resulting in l + pos − l + neg for x + and l − pos − l − neg for x − , respectively, then (4) take the difference between these, ( l + pos − l + neg ) − ( l − pos − l − neg ) , to form the effective logit.
Revised: Specifically, on an input x + or x − , we (1) take the logit vector corresponding to the last input token, (2) select the corresponding logit for “positive” and the corresponding logit for “negative” (the two candidate labels), (3) take the difference between these, resulting in l + pos − l + neg for x + and l − pos − l − neg for x − , respectively, then (4) take the difference between these, ( l + pos − l + neg ) − ( l − pos − l − neg ) , to form the effective logit.

Original: Intuitively, this final expression should be large if the example is positive, and small if the example is negative.
Revised: Intuitively, this final expression should be large if the example is positive, and small if the example is negative.

Original: As before, we calibrate this zero-shot method so that its predictions are balanced.
Revised: As before, we calibrate this zero-shot method so that its predictions are balanced.

Original: We evaluate this method and find that it recovers non-trivial knowledge in the model outputs, but that it still performs substantially worse than CCS: calibrated zero-shot accuracy in this setting is 71 . 6% , compared to almost 94% for CCS.
Revised: We evaluate this method and find that it recovers non-trivial knowledge in the model outputs, but that it still performs substantially worse than CCS: calibrated zero-shot accuracy in this setting is 71 . 6% , compared to almost 94% for CCS.

Original: This suggests that model outputs in this setting indeed aren’t very useful, especially relative to CCS.
Revised: This suggests that model outputs in this setting indeed aren’t very useful, especially relative to CCS.

Original: Overall, our results in this section show that CCS can still work in at least some settings where model outputs don’t seem to be very useful.
Revised: Overall, our results in this section show that CCS can still work in at least some settings where model outputs don’t seem to be very useful.

Original: This provides additional evidence that out approach does not simply recover knowledgerepresented in the model outputs.
Revised: This provides additional evidence that out approach does not simply recover knowledge

Original: D C OMPLETE S AMPLE C OMPLEXITY R ESULTS
Revised: D C OMPLETE S AMPLE C OMPLEXITY R ESULTS

Original: In this section we show additional results on how the number of examples affects CCS.
Revised: In this section we show additional results on how the number of examples affects CCS.

Original: We show the performance (averaged across all datasets) for all models in Figure 7.
Revised: We show the performance (averaged across all datasets) for all models in Figure 7.

Original: We next show a more fine-grained results.
Revised: We next show a more fine-grained results.

Original: Specifically, we select # Samples = 1 , 8 , 64 and show dataset-level results in
Revised: Specifically, we select # Samples = 1 , 8 , 64 and show dataset-level results in

Original: Figure 8 for each model.
Revised: Figure 8 for each model.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: , 64 data points from this prompt.
Revised: , 64 data points from this prompt.

Original: We then perform CCS and test on all prompts of this dataset, where each value in the barplot corresponds to one prompt.
Revised: We then perform CCS and test on all prompts of this dataset, where each value in the barplot corresponds to one prompt.

Original: E C OMPLETE T RANSFER R ESULTS
Revised: E C OMPLETE T RANSFER R ESULTS

Original: Complete transfer results for CCS, TPC and BSS in all models are shown in Figure 11.
Revised: Complete transfer results for CCS, TPC and BSS in all models are shown in Figure 9.

Original: [Figure]
Revised: [Figure]

Revised: [Table]

Revised: [Table]

Original: [Figure]
Revised: [Table]

Revised: I M D B A m a z

Revised: o n A G N e w s D B p

Revised: e d i a C

Revised: O P

Revised: A R T E B

Revised: O O L QQ N L

Revised: I P I Q

Revised: A S t o r y C

Original: {+l o z e 93 83 65 69 54 54 
+}
Revised: {+l o z e 93 83 65 69 54 54 
+}

Original: F C OMPLETE I NTERMEDIATE R EPRESENTATIONS R ESULTS
Revised: F C OMPLETE I NTERMEDIATE R EPRESENTATIONS R ESULTS

Original: In this section we the performance of CCS and LR across all hidden layers in all six models.
Revised: In this section we the performance of CCS and LR across all hidden layers in all six models.

Original: Specifically, for each model, we generate the hidden states for each dataset every 2 layers (for both encoders and decoders).
Revised: Specifically, for each model, we generate the hidden states for each dataset every 2 layers (for both encoders and decoders).

Original: Then, for each set of hidden states [Figure] we perform CCS and LR on each dataset separately
Revised: Then, for each set of hidden states, we perform CCS and LR on each dataset separately

Original: [Figure]
Revised: [Figure]

Original: and average the performance across all datasets.
Revised: and average the performance across all datasets.

Original: We show CCS results in Figure 9 and LR results in Figure10.
Revised: We show CCS results in Figure 10 and LR results in Figure

Original: [Table]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: In this section, we provide further implementation details for CCS and CRC. G. N ORMALIZATION As described in Section 2.2, we normalize the features ˜ ϕ

Original: [Figure]
Revised: ( x + i ) = ϕ ( x + i ) − µ + , where µ + is the mean of { ϕ ( x + i ) } ni =1 (and similarly for ˜ ϕ ( x − i ) ). In practice, we also normalize the scale of the features by also dividing by the average norm of { ϕ ( x + i ) } ni =1 times √ d (and similarly for ˜ ϕ

Original: [Table]
Revised: ( x − i ) ). However, this is less essential than mean normalization and there are likely many reasonable choices for scale normalization. G. C ONTRAST P AIRS We use the Huggingface library (Wolf et al., 2019) for all of our experiments.

Original: [Figure]
Revised: We use the standard tokenizer for each model, and always take the hidden state corresponding to the last token in a given layer indexed by idx. We show Huggingface-style pseudocode for extracting hidden states from in Algorithm 1. For encoder-only and decoder-only models, we provide the full input, x + or x − , which includes both the question and the proposed answer, to the model; see Appendix I.1 for more formatting and tokenization details.

Original: [Table]
Revised: For encoder-decoder models, our input format depends on whether we are taking the encoder hidden states or the decoder hidden states. When we take the decoder hidden states, we input the question to the encoder, and input the candidate answer to the decoder. In contrast, when we take the encoder hidden states of an encoder-decoder model, we provide the full input (including the proposed answer) to the encoder, and ignore the decoder (simply passing the empty string to the decoder and ignoring its results). This is necessary to ensure that the inputs to the encoder are not identical across contrast pairs.

Original: [Table]
Revised: Input: Contrast Pairs Set S , model name mdl, layer index idx m = transformers. AutoModel ( mdl ) t = transformers. AutoTokenizer ( mdl )

Original: [Table]
Revised: C + , C − = [] , [] for ( x + , x − ) in S do token + , token − = t .encode ( x + ) , t .encode ( x − ) ϕ + = m ( token + , output hiddenstates =

Original: [Table]
Revised: True ) [“hidden states”] [ idx ][ − 1] ϕ − = m ( token − , output hiddenstates = True ) [“hidden states”] [ idx ][ − 1] C + .append ( ϕ + ) , C − .append ( ϕ − ) end for

Original: [Table]
Revised: Output: [ C + , C − ] G. CCS Given contrast features from Algorithm 1, CCS works by learning parameters θ and b by minimizing L CCS ( θ, b ) as defined in Section 2.2.

Original: [Table]
Revised: In practice, we implement the bias b by appending an additional dimension of 1 s features to the input, increasing it and θ to have dimension d + 1 .We randomly initialize θ to have unit norm. We then optimize the CCS loss T = 10 times and select the run with lowest unsupervised loss. In practice, we train each time for E = 1000 epochs with a learning rate η = 0 . 01 (which we found was good for consistently achieving low unsupervised loss) in each run.

Original: [Table]
Revised: Because we are only learning a linear probe on top of the features, training can be fast. G. CRC: T OP P RINCIPAL C OMPONENT Intuitively, a direction that clusters examples well should have high variance. Motivated by this, a simple method is to cluster according to the Top Principal Component (TPC).

Original: B A m a z
Revised: We do this by first constructing contrast features from the normalized contrast pair activations, C = { ˜ ϕ ( x + i ) − ˜ ϕ ( x − i ) } ni =1 , then projecting these examples on their top principal component using PCA.

Original: o n A G N e w s D B p
Revised: We then treat examples that are less than 0 as one cluster and examples that are greater than 0 as the other cluster. We find that despite its simplicity this method can often also find truth-like features in model activations. This method is similar in spirit to Bolukbasi et al. (2016), except that our method doesn’t require any labels; we leverage the fact that truth is consistent to construct contrast pairs in a purely unsupervised way.

Original: e d i a C
Revised: Bimodal Salience Search (BSS) is another variation of CRC. One drawback of TPC is that variance is not an intrinsic quantity to a network’s behavior, since different directions in representation space can be scaled without changing the behavior of the network as long as subsequent layers are rescaled accordingly. This motivates using a method that clusters examples regardless of the scale of different directions.

Original: O P
Revised: If examples are well-clustered, then the intra-cluster variance should be low while the inter-cluster variance (or total variance) should be high. Specifically, if we center a set of datapoints so thatdelineates the cluster boundary, then the points on either side of 0 should have low variance compared to the points overall. This suggests minimizing the following loss: [Equation]

Original: A R T E B
Revised: where c i := ˜ ϕ ( x + i ) − ˜ ϕ ( x − i ) and where we use var { z i } as shorthand to denote the variance of a set { z i } ni =1 ⊂ R .

Original: O O L QQ N L 
Revised: Because this objective is written in terms of a ratio, it is invariant to the overall scale of the direction, fixing that drawback of TPC. This loss is similar to that of Linear Discriminant Analysis (LDA) (Fisher, 1936), but unlike LDA is completely unsupervised. In practice, we use an SGD-based optimizer to find a local optimum of L ( θ ) .

Original: I P I Q
Revised: After computing the contrast features, we repeat the following process T times. We first initialize a random direction θ with unit norm. Then for E epochs, we calculate the loss according to Equation (1), and update the direction θ by via projected gradient descent, each time projecting θ back onto the unit sphere. Finally, we select the direction θ that has the lowest loss among the T directions we found, and predict based on this direction.

Original: A S t o r y C
Revised: When we optimize the direction with multiple datasets, i.e. S 1 , ..., S n , we use the same algorithm, but average the loss across all datasets. In practice we use T = E = 20 , and use Adam optimizer (Kingma & Ba, 2014) with a learning rate of 0 . 1 . H S TATISTICAL S IGNIFICANCE Our main accuracy results for CCS and other methods (e.g. in Table 1 and elsewhere) are computed by evaluating the method on 40% of the 1000 (or 500 in the case of COPA) examples sampled for each dataset, then averaging the resulting accuracy across 9 prompts per dataset (on average),different datasets, and up to 5 models.

Original: [-l o z e 93 83 65 69 54 54 
-]
Revised: [-l o z e 93 83 65 69 54 54 
-]This corresponds to about 180 k samples in total; we performed subsampling in this way for computational efficiency reasons, as this is already substantial. For results where we average across datasets (i.e. most results in this paper), we average across 3800 IID samples (the original examples sampled from each dataset), then usually also averaged across a very large number of correlated samples: 9 prompts and (often) 5 different models for a given sample. We can compute an upper bound on the standard error of accuracies computed in this way by ignoring the averaging across different prompts and models: this gives us a simple but coarse upper bound of the standard error of 1 2 √ 3800 ≈ 0 . Applying a Wald test, the Wald statistic is W =

Original: [Figure]
Revised: [Equation]

Revised: se (

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: 2 , where (for example) µ 0 is zero-shot accuracy and ˆ µ is CCS accuracy.

Revised: W is then χ 2 -distributed with one degree of freedom.

Revised: We find that our main claims comparing accuracies are statistically significant at a 0.00001 level.

Revised: This includes the claims that CCS outperforms zero-shot, with accuracies of 71.2% vs 67.2%, respectively; that CCS is robust to misleading prompts while zero-shot isn’t in the setting we test, with accuracies of 83.8% vs 70.9%, respectively; that CCS on MLM-pretrained DeBERTa substantially outperforms

Revised: DeBERTa zero-shot, with accuracies of 93.7% vs 71.6%, respectively; and so on.

Revised: Some minor observations are not necessarily statistically significant, such as that CCS (All Data) outperforms

Revised: CCS on average (Section 3.2.1), but none of these are important for our main claims, and more powerful but complicated statistical tests would result in smaller p-values.

Original: G D ATASET S ETUP
Revised: I D ATASET S ETUP

Original: In this section, we describe the setup of our data and prompts in detail.
Revised: In this section, we describe the setup of our data and prompts in detail.

Original: We introduce all datasets we use and the way we convert them into a binary classification task.
Revised: We introduce all datasets we use and the way we convert them into a binary classification task.

Original: Our prompts derive from (Sanh et al., 2021) 5 .
Revised: Our prompts derive from (Sanh et al., 2021) 4 .

Original: Contrast Pair Example.
Revised: Contrast Pair Example.

Original: To illustrate how we construct contrast pairs, suppose we have a movie review “[text]
Revised: To illustrate how we construct contrast pairs, suppose we have a movie review “[text]

Original: = I loved this movie.”
Revised: = I loved this movie.”

Original: and the sentiment of this review belongs to “[label0] = positive” or “[label1] = negative”.
Revised: and the sentiment of this review belongs to “[label0] = positive” or “[label1] = negative”.

Original: We first format it into a binary question-answering or classification question: “[text] Is the sentiment of this example [label0] or [label1]? ” using an existing zero-shot prompt.
Revised: We first format it into a binary question-answering or classification question: “[text] Is the sentiment of this example [label0] or [label1]? ” using an existing zero-shot prompt.

Original: We then concatenate the question and candidate labels to create the contrast pairs: x + = [prefix] Q: Is the sentiment of “[text]” [label0] or [label1]?
Revised: We then concatenate the question and candidate labels to create the contrast pairs: x + = [prefix] Q: Is the sentiment of “[text]” [label0] or [label1]?

Original: A: [label0] x − =
Revised: A: [label0] x − =

Original: [prefix] Q: Is the sentiment of “[text]” [label0] or [label1]?
Revised: [prefix] Q: Is the sentiment of “[text]” [label0] or [label1]?

Original: A: [label1]
Revised: A: [label1]

Original: For instance, in this example, these would be: x + = Q: Is the sentiment of “I loved this movie.”
Revised: For instance, in this example, these would be: x + = Q: Is the sentiment of “I loved this movie.”

Original: positive or negative?
Revised: positive or negative?

Original: A: positive x − =
Revised: A: positive x − =

Original: Q: Is the sentiment of “I loved this movie.”
Revised: Q: Is the sentiment of “I loved this movie.”

Original: positive or negative?
Revised: positive or negative?

Original: A: negative
Revised: A: negative

Revised: I. T OKENIZATION

Revised: For all prompts we consider in this paper, we concatenate labels to the end of questions.

Revised: The precise we do this depends on the model type.

Revised: For encoder-decoder models such as T5, we replace line breaks ( \n ) with spaces, and we add a space between the question and proposed answer if the last token of questions is not already a space.

Revised: For autoregressive models, we add a line break between the question and the proposed label if the last token of the question is neither a space nor a line break.

Original:  Finally, we add the prefix (either prefix in Figure 6, or the empty string for the regular zero-shot setting) tothe beginning of the prompt, then compute the hidden representationsfor this pair to obtainthe contrast features C .
Revised: For DeBERTa, we add a [SEP] split-token between the question and answer, and for RoBERTa, we add a 

 between the question and answer. Finally, we add the prefix (either prefix in Figure 5, or the empty string for the regular zero-shot setting) to prompt, then compute the hidden representations pair to obtain contrast features C .

Original: G. D ATASETS
Revised: I. D ATASETS

Original: We use ten datasets in our experiments.
Revised: We use ten datasets in our experiments.

Original: For each dataset, we select 1000 data points (except COPA, which has only 500 data points, so we only use those 500 ).
Revised: For each dataset, we select 1000 data points (except COPA, which has only 500 data points, so we only use those 500 ).

Original: When possible, we use just the test / validation split, but in other cases this is not enough data so we also use examples from the train split.
Revised: When possible, we use just the test / validation split, but in other cases this is not enough data so we also use examples from the train split.

Original: For the most part this doesn’t make a difference for our purposes because our methods are purely unsupervised.
Revised: For the most part this doesn’t make a difference for our purposes because our methods are purely unsupervised.

Original: It is important to note that T0 is an exception among the models, because but was trained on 9 out of the 10 datasets we consider (all except RTE).
Revised: It is important to note that T0 is an exception among the models, because but was trained on 9 out of the 10 datasets we consider (all except RTE).

Original: This may account for its anomalous performance.
Revised: This may account for its anomalous performance.

Original: UnifiedQA was trained on BoolQ, which is one of the datasets we test on, but was not trained on any other evaluation set we consider, which we don’t expect has an important qualitative effect.
Revised: UnifiedQA was trained on BoolQ, which is one of the datasets we test on, but was not trained on any other evaluation set we consider, which we don’t expect has an important qualitative effect.

Original: We now enumerate the prompts we use for each dataset.
Revised: We now enumerate the prompts we use for each dataset.

Original: G.1. AG N EWS
Revised: I.2. AG N EWS

Original: AG News is a topic classification dataset from (Zhang et al., 2015).
Revised: AG News is a topic classification dataset from (Zhang et al., 2015).

Original: Given a news paragraph, the label corresponds to the topic of this news, including “World politics”, “Sports”, “Business” and “Science and technology”.
Revised: Given a news paragraph, the label corresponds to the topic of this news, including “World politics”, “Sports”, “Business” and “Science and technology”.

Original: For each example, we randomly select another label (which is incorrect), and randomly assign the correct label and this incorrect label to “label0” and “label1”.
Revised: For each example, we randomly select another label (which is incorrect), and randomly assign the correct label and this incorrect label to “label0” and “label1”.

Original: With this, we construct the classification task using the following 8 prompts:
Revised: With this, we construct the classification task using the following 8 prompts:

Original: 1 [prefix]Consider the following example: “‘ [text] ”’
Revised: 1 [prefix]Consider the following example: “‘ [text] ”’

Original: Choice 1: [label0].
Revised: Choice 1: [label0].

Original: Choice 2: [label1].Between choice 1 and choice 2, the topic of this example is [label]  2 [prefix]Consider the following example: “‘ [text] ”’
Revised: Choice 2: [label1].Between choice 1 and choice 2, the topic of this example is [label] 4 https:/ 2 [prefix]Consider the following example: “‘ [text] ”’

Original: Choice 1: [label0].
Revised: Choice 1: [label0].

Original: Choice 2: [label1].Between choice 1 and choice 2, what is the topic of this example?
Revised: Choice 2: [label1].Between choice 1 and choice 2, what is the topic of this example?

Original: [label] 3 [prefix]Consider the following example: “‘ [text] ”’ Which is the topic of this example, choice 1: [label0], or choice 2: [label1]?
Revised: [label] 3 [prefix]Consider the following example: “‘ [text] ”’ Which is the topic of this example, choice 1: [label0], or choice 2: [label1]?

Original: [label] 4 [prefix][text] Choice 1: [label0].
Revised: [label] 4 [prefix][text] Choice 1: [label0].

Original: Choice 2: [label1].Between choice 1 and choice 2, the topic of this example is [label] 5 [prefix][text] Choice 1: [label0].
Revised: Choice 2: [label1].Between choice 1 and choice 2, the topic of this example is [label] 5 [ prefix text] Choice 1: [label

Original: Choice 2: [label1].Between choice 1 and choice2, what is the topic of this example?
Revised: Choice 2: [label1].Between choice 1 and choice

Original: [label]
Revised: 2, what is the topic of

Original:  6 [prefix][text] Which is the topic of this example, choice 1: [label0], or choice 2: [label
Revised: this example? label 6 [prefix][text] Which is the topic of this example, choice 1: [label0], or choice 2: [label1]?

Original: [label 7 [prefix [text] What labelbest
Revised: [label] 7 [prefix][text] What label

Original:  describes 
Revised: best describes this

Original: this news article, choice 1: [label0], or choice 2: [label1]?
Revised: news article , choice 1: [label0], or choice 2: [label1]?

Original: [label]8 [prefix][text] Which
Revised: [label]

Original: section of a
Revised: prefix

Original:  newspaper would this article likely appear in, choice 1: [label0], or choice 2: [label1]?
Revised: ][text] Which section of a newspaper would this article likely appear in, choice 1: [ label0], or choice 2: label

Original: [label
Revised: [label

Original: Here the last “[label]” is “choice 1” for x + and “choice 2” for x − .
Revised: Here the last “[label]” is “choice 1” for x + and “choice 2” for x − .

Original: Notice that we only specify “[label]” for the prompts that are constructed manually.
Revised: Notice that we only specify “[label]” for the prompts that are constructed manually.

Original: For prompts in (Sanh et al., 2021), we leave their labels unchanged.
Revised: For prompts in (Sanh et al., 2021), we leave their labels unchanged.

Original: (For example, we use “choice 1” or “choice 2” but they can use “Animal” or “Plant”)
Revised: (For example, we use “choice 1” or “choice 2” but they can use “Animal” or “Plant”)

Original: G.1. A MAZON POLARITY
Revised: I.2. A MAZON POLARITY

Original: Amazon polarity is a sentiment classification task from (McAuley & Leskovec, 2013).
Revised: Amazon polarity is a sentiment classification task from (McAuley & Leskovec, 2013).

Original: The content is the review of goods in Amazon, and the label can be “negative” or “positive”.
Revised: The content is the review of goods in Amazon, and the label can be “negative” or “positive”.

Original: We use 11 different prompts in this dataset.
Revised: We use 11 different prompts in this dataset.

Original: We first take all prompts from (Sanh et al., 2021)
Revised: We first take all prompts from (Sanh et al., 2021)

Original: (Page 164, 9 prompts in total), and the two of our own as follows:1 [ prefix
Revised: (Page 164, 9 prompts in total), and the two of our own as follows:

Original: ]Consider the following example: “‘ [content] ”’ Between [label0] and [label1], the sentiment of this example is [label] 2 [prefix]Consider the following example: “‘ [content] ”’
Revised: 1 [prefix]Consider the following example: “‘ [content] ”’ Between [label0] and [label1], the sentiment of this example is [label] 2 [prefix]Consider the following example: “‘ [content] ”’

Original: Between [label0] and [label1], which is the sentiment of this example?
Revised: Between [label0] and [label1], which is the sentiment of this example?

Original: [label]
Revised: [label]

Original: Here “[label]” is “negative” for x + and “positive” for x − .
Revised: Here “[label]” is “negative” for x + and “positive” for x − .

Original: G.1. BOOLQ
Revised: I.2. BOOLQ

Original: BOOLQ is a QA task where each example consists a yes/no question from (Clark et al., 2019).
Revised: BOOLQ is a QA task where each example consists a yes/no question from (Clark et al., 2019).

Original: We directly use the 10 prompts from (Sanh et al., 2021)
Revised: We directly use the 10 prompts from (Sanh et al., 2021)

Original: (Page 146)
Revised: (Page 146)

Original: G.1. COPA 
Revised: I.

Revised: COPA

Original: COPA is a causal reasoning task to determine either the cause or the effect of a given premise (Roemmele et al., 2011).
Revised: COPA is a causal reasoning task to determine either the cause or the effect of a given premise (Roemmele et al., 2011).

Original: Here the label is a short sentence.
Revised: Here the label is a short sentence.

Original: We use 10 prompts, where 9 are from (Sanh et al., 2021)
Revised: We use 10 prompts, where 9 are from (Sanh et al., 2021)

Original: (Page 177), and we add one more prompt: 1 [prefix]Consider the following premise: “‘ [premise] ”’
Revised: (Page 177), and we add one more prompt: 1 [prefix]Consider the following premise: “‘ [premise] ”’

Original: Choice 1: [choice1] Choice 2: [choice2] Q: Which one is more likely to be the [question], choice 1 or choice 2?
Revised: Choice 1: [choice1] Choice 2: [choice2] Q: Which one is more likely to be the [question], choice 1 or choice 2?

Original: [label]
Revised: [label]

Original: G.1. DB PEDIA
Revised: I.2. DB PEDIA

Original: DBpedia 14 is a topic classification dataset constructed by picking 14 non-overlapping classes from
Revised: DBpedia 14 is a topic classification dataset constructed by picking 14 non-overlapping classes from

Original: DBpedia 2014 (Lehmann et al., 2015).
Revised: DBpedia 2014 (Lehmann et al., 2015).

Original: We manually create 8 prompts.
Revised: We manually create 8 prompts.

Original: For each example, we randomly select the incorrect label from the remaining 13 classes, and randomly assign the correct label and this incorrect label to “[label0]” and “[label1]”.
Revised: For each example, we randomly select the incorrect label from the remaining 13 classes, and randomly assign the correct label and this incorrect label to “[label0]” and “[label1]”.

Original: 1 [prefix]Consider the following example: “‘ [content] ”’
Revised: 1 [prefix]Consider the following example: “‘ [content] ”’

Original: Choice 1: [label0].
Revised: Choice 1: [label0].

Original: Choice 2: [label1].Between choice 1 and choice 2, the topic of this example is [label] 2 [prefix]Consider the following example: “‘ [content] ”’ Choice 1: [label0].
Revised: Choice 2: [label1].Between choice 1 and choice 2, the topic of this example is [label] 2 [prefix]Consider the following example: “‘ [content] ”’ Choice 1: [label0].

Original: Choice 2: [label1].Between choice 1 and choice 2, what is the topic of this example?
Revised: Choice 2: [label1].Between choice 1 and choice 2, what is the topic of this example?

Original: [label] 3 [prefix]Consider the following example: “‘ [content] ”’ Which is the topic of this example, choice 1: [label0], or choice 2: [label1]?
Revised: [label] 3 [prefix]Consider the following example: “‘ [content] ”’ Which is the topic of this example, choice 1: [label0], or choice 2: [label1]?

Original: [label] 4 [prefix][content] Choice 1: [label0].
Revised: [label] 4 [prefix][content] Choice 1: [label0].

Original: Choice 2: [label1].Between choice 1 and choice 2, the topic of this example is [label] 5 [prefix][content] Choice 1: [label0].
Revised: Choice 2: [label1].Between choice 1 and choice 2, the topic of this example is [label] 5 [ prefix][content] Choice 1: [label0].

Original: Choice 2: [label1].Between choice 1 and choice2, what is the topic of this example?
Revised: Choice 2: [label1].Between choice 1 and choice what is the topic of this example?

Original: [label]
Revised: [label]

Original: 6 [prefix][content] Which is the topic of this example, choice 1: [label0], or choice 2: [label1]?
Revised: 6 [prefix][content] Which is the topic of this example, choice 1: [label0], or choice 2: [label1]?

Original: [label] 7 [prefix][content] What category does the paragraph belong to, choice 1: [label0], or choice 2: [label1]?
Revised: [label] 7 [prefix][content] What category does the paragraph belong to, choice 1: [label0], or choice 2: [label1]?

Original: [label] 8 [prefix][content] What label best describes this paragraph, choice 1: [label0], or choice 2: [label1]?
Revised: [label] 8 [prefix][content] What label best describes this paragraph, choice 1: [label0], or choice 2: [label1]?

Original: [label]
Revised: [label]

Original: Here “[label]” is “choice 1” for x + and “choice 2” for x − .
Revised: Here “[label]” is “choice 1” for x + and “choice 2” for x − .

Original: G.1. IMDB
Revised: I.2. IMDB

Original: IMDB is a sentiment dataset from (Maas et al., 2011).
Revised: IMDB is a sentiment dataset from (Maas et al., 2011).

Original: Given a movie review, the label is either “[label0] = negative” or “[label1] = positive.
Revised: Given a movie review, the label is either “[label0] = negative” or “[label1] = positive.

Original: We use 13 prompts, where 11 are from (Sanh et al., 2021)
Revised: We use 13 prompts, where 11 are from (Sanh et al., 2021)

Original: (Page 168), and the rest two are as follows: 1 [prefix]Consider the following example: ”’
Revised: (Page 168), and the rest two are as follows: 1 [prefix]Consider the following example: ”’

Original: [text] ”’
Revised: [text] ”’

Original: Between [label0] and [label1], the sentiment of this example is [label] 2 [prefix]Consider the following example: ”’
Revised: Between [label0] and [label1], the sentiment of this example is [label] 2 [prefix]Consider the following example: ”’

Original: [text] ”’
Revised: [text] ”’

Original: Between [label0] and [label1], which is the sentiment of this example?
Revised: Between [label0] and [label1], which is the sentiment of this example?

Original: [label] Here “[label]” is “negative” for x + and “positive” for x − .
Revised: [label] Here “[label]” is “negative” for x + and “positive” for x − .

Original: G.1. PIQA
Revised: I.2. PIQA

Original: The PIQA dataset measures the physical commonsense reasoning ability of models.
Revised: The PIQA dataset measures the physical commonsense reasoning ability of models.

Original: We useprompts for PIQA, all of which are from (Sanh et al., 2021)(Page 160).
Revised: We useprompts for PIQA, all of which are from (Sanh et al., 2021)(Page 160).

Original: The label is a complete sentence that can be the solution of the question.
Revised: The label is a complete sentence that can be the solution of the question.

Original: G.1. QNLI
Revised: I.2. QNLI

Original: QNLI from (Rajpurkar et al., 2016) is a question-answering dataset consisting of question-paragraph pairs, where one of the sentences in the paragraph (drawn from Wikipedia) contains the answer to the corresponding question (written by an annotator).
Revised: QNLI from (Rajpurkar et al., 2016) is a question-answering dataset consisting of question-paragraph pairs, where one of the sentences in the paragraph (drawn from Wikipedia) contains the answer to the corresponding question (written by an annotator).

Original: We use 5 prompts from (Sanh et al., 2021).
Revised: We use 5 prompts from (Sanh et al., 2021).

Original: The label is either “yes” or “no” depending on whether the information in the paragraph is enough to answer the paragraph.
Revised: The label is either “yes” or “no” depending on whether the information in the paragraph is enough to answer the paragraph.

Original: G.1. RTE
Revised: I.2. RTE

Original: RTE is a textual entailment dataset (Wang et al., 2018).
Revised: RTE is a textual entailment dataset (Wang et al., 2018).

Original: The label corresponds to whether the text entails the hypotheses.
Revised: The label corresponds to whether the text entails the hypotheses.

Original: We use 11 prompts, where 10 are from (Sanh et al., 2021)(Page 48), and where we manually add one more prompt: 1 [prefix][premise] Question: Does this imply that ”[hypothesis]”, yes or no?
Revised: We use 11 prompts, where 10 are from (Sanh et al., 2021)(Page 48), and where we manually add one more prompt: 1 [prefix][premise] Question: Does this imply that ”[hypothesis]”, yes or no?

Original: [label] Here “[label]” is “yes” for x + and “no” for x − .
Revised: [label] Here “[label]” is “yes” for x + and “no” for x − .

Original: G.1. S TORY C LOZE
Revised: I.2. S TORY C LOZE

Original: Story Cloze is a story completion task from (Mostafazadeh et al., 2017).
Revised: Story Cloze is a story completion task from (Mostafazadeh et al., 2017).

Original: Given a short story, the task is to determine which of two endings is more likely to continue that story.
Revised: Given a short story, the task is to determine which of two endings is more likely to continue that story.

Original: We use 9 prompts, whereare from (Sanh et al., 2021), and the remaining 3 are as follows: 1 [prefix]Consider the following story: “‘ [input sentence 1] [input sentence 2] [input sentence 3] [input sentence 4] ”’
Revised: We use 9 prompts, whereare from (Sanh et al., 2021), and the remaining 3 are as follows: 1 [prefix]Consider the following story: “‘ [input sentence 1] [input sentence 2] [input sentence 3] [input sentence 4] ”’

Original: Choice 1: [sentence quiz1]
Revised: Choice 1: [sentence quiz1]

Original: Choice 2: [sentence quiz2]
Revised: Choice 2: [sentence quiz2]

Original: Which is the more plausible ending of this story, choice 1 or choice 2?
Revised: Which is the more plausible ending of this story, choice 1 or choice 2?

Original: [label] 2 [prefix]Consider the following story: “‘ [input sentence 1] [input sentence 2] [input sentence 3] [input sentence 4] ”’ Choice 1: [sentence quiz1]
Revised: [label] 2 [prefix]Consider the following story: “‘ [input sentence 1] [input sentence 2] [input sentence 3] [input sentence 4] ”’ Choice 1: [sentence quiz1]

Original: Choice 2: [sentence quiz2]
Revised: Choice 2: [sentence quiz2]

Original: Which is the more plausible ending of this story?
Revised: Which is the more plausible ending of this story?

Original: [label] 3 [prefix][input sentence 1] [input sentence 2] [input sentence 3] [input sentence 4] Choice 1: [sentence quiz1]
Revised: [label] 3 [prefix][input sentence 1] [input sentence 2] [input sentence 3] [input sentence 4] Choice 1: [sentence quiz1]

Original: Choice 2: [sentence quiz2]
Revised: Choice 2: [sentence quiz2]

Original: Which is the more plausible ending of this story, choice 1 or choice 2?
Revised: Which is the more plausible ending of this story, choice 1 or choice 2?

Original: [label]
Revised: [label]

Original: Here “[label]” is “choice 1” for x + and “choice 2” for x − .
Revised: Here “[label]” is “choice 1” for x + and “choice 2” for x − .

Original: H CCS AND CRC I MPLEMENTATION D ETAILS

Original: In this section, we provide further implementation details for CCS and CRC.

Original: We begin by describing how we construct contrast pairs.

Original: H. C ONTRAST P AIRS

Original: We use the Huggingface library (Wolf et al., 2019) for all of our experiments.

Original: We use the standard tokenizer for each model, and always take the hidden state corresponding to the last token in a given layer indexed by idx.

Original: We show Huggingface-style pseudocode for extracting hidden states from in

Original: Algorithm 1.

Original: For encoder-only and decoder-only models, we provide the full input, x + or x − , which includes both the question and the proposed answer, to the model.

Original: For encoder-decoder models, our input format depends on whether we are taking the encoder hidden states of the decoder hidden states.

Original: When we take the decoder hidden states, we input the question to the encoder, and input the candidate answer to the decoder.

Original: In contrast, when we take the encoder hidden states of an encoder-decoder model, we provide the full input (including the proposed answer) to the encoder, and ignore the decoder (simply passing the empty string to the decoder and ignoring its results).

Original: This is necessary to ensure that the inputs to the encoder are not identical across contrast pairs.

Original: Algorithm

Original: Input: Contrast Pairs Set S , model name mdl, layer index idx m = transformers.

Original: AutoModel ( mdl ) t

Original: = transformers.

Original: AutoTokenizer ( mdl )

Original: C + , C − =

Original: [] , [] for ( x + , x − ) in S do token + , token − =

Original: t .encode ( x + ) , t .encode ( x − ) ϕ

Original: + = m ( token + , output hiddenstates =

Original: True ) [“hidden states”] [ idx ][ − 1] ϕ − = m ( token − , output hiddenstates =

Original: True ) [“hidden states”] [ idx ][ − 1]

Original: C

Original: + .append ( ϕ + ) , C − .append ( ϕ − ) end for

Original: Output: [ C + , C − ]

Original: H.

Original: Given contrast features from Algorithm 1, CCS works by learning parameters θ and b according to ?? .

Original: In practice, we implement the bias b by appending an additional dimension of 1 s features to the input, increasing it and θ to have dimension d + 1 .We randomly initialize θ to have unit norm.

Original: We then optimize the CCS loss T = 10 times and select the run with lowest unsupervised loss.

Original: In practice, we train each time for E = 1000 epochs with a learning rate η = 0 .

Original: 01 (which we found was good for consistently achieving low unsupervised loss) in each run.

Original: Because we are only learning a linear probe on top of the features, training can be fast.

Original: H. CRC: T OP P RINCIPAL C OMPONENT

Original: Intuitively, a direction that clusters examples well should have high variance.

Original: Motivated by this, a simple method is to cluster according to the Top Principal Component (TPC).

Original: We do this by first constructing contrast features from the normalized contrast pair activations, C = { ˜ ϕ

Original: ( x + i ) − ˜ ϕ

Original: ( x − i ) } ni

Original: =1 , then projecting these examples on their top principal component using PCA.

Original: We then treat examples that are less than 0 as one cluster and examples that are greater than 0 as the other cluster.

Original: We find that despite its simplicity this method can often also find truth-like features in model activations.

Original: H. CRC: B IMODAL S ALIENCE S EARCH

Original: Bimodal Salience Search (BSS) is another variation of CRC.

Original: One drawback of TPC is that variance is not an intrinsic quantity to a network’s behavior, since different directions in representation space can be scaled without changing the behavior of the network as long as subsequent layers are rescaled accordingly.

Original: This motivates using a method that clusters examples regardless of the scale of different directions.

Original: If examples are well-clustered, then the intra-cluster variance should be low while the inter-cluster variance (or total variance) should be high.

Original: Specifically, if we center a set of datapoints so thatdelineates the cluster boundary, then the points on either side of 0 should have low variance compared to the points overall.

Original: This suggests minimizing the following loss:

Original: [Equation]

Original: where c i :=

Original: ˜ ϕ

Original: ( x + i ) − ˜ ϕ

Original: ( x − i ) and where we use var { z i } as shorthand to denote the variance of a set { z i } ni =1 ⊂ R .

Original: Because this objective is written in terms of a ratio, it is invariant to the overall scale of the direction, fixing that drawback of TPC.

Original: This loss is similar to that of Linear Discriminant

Original: Analysis (LDA) (Fisher, 1936), but unlike LDA is completely unsupervised.

Original: In practice, we use an SGD-based optimizer to find a local optimum of L ( θ ) .

Original: After computing the contrast features, we repeat the following process T times.

Original: We first initialize a random direction θ with unit norm.

Original: Then for E epochs, we calculate the loss according to Equation (1), and update the direction θ by via projected gradient descent, each time projecting θ back onto the unit sphere.

Original: Finally, we select the direction θ that has the lowest loss among the T directions we found, and predict based on this direction.

Original: When we optimize the direction with multiple datasets, i.e. S 1 , ..., S n , we use the same algorithm, but average the loss across all datasets.

Original: In practice we use T = E = 20 , and use Adam optimizer (Kingma & Ba, 2014) with a learning rate of 0 . 1 .

Original: I S TATISTICAL S IGNIFICANCE

Original: Our main accuracy results for CCS and other methods (e.g. in Table 1 and elsewhere) are computed by evaluating the method on 40% of the 1000 (or 500 in the case of COPA) examples sampled for each dataset, then averaging the resulting accuracy across 9 prompts per dataset (on average),different datasets, and up to 5 models.

Original: This corresponds to about 180 k samples in total; we performed subsampling in this way for computational efficiency reasons, as this is already substantial.

Original: For results where we average across datasets (i.e. most results in this paper), we average across 3800 IID samples (the original examples sampled from each dataset), then usually also averaged across a very large number of correlated samples: 9 prompts and (often) 5 different models for a given sample.

Original: We can compute an upper bound on the standard error of accuracies computed in this way by ignoring the averaging across different prompts and models: this gives us a simple but coarse upper bound of the standard error of 1 2 √ 3800 ≈ 0 .

Original: Applying a Wald test, the Wald statistic is W = µ 0 − ˆ µ se (ˆ µ ) 2 ≥ µ 0 − ˆ µ (0 . 008) 2 , where (for example) µ 0 is zero-shot accuracy and ˆ µ is CCS accuracy.

Original: W is then χ 2 -distributed with one degree of freedom.

Original: We find that our main claims comparing accuracies are statistically significant at a 0.00001 level.

Original: This includes the claims that CCS outperforms zero-shot, with accuracies of 71.2% vs 67.2%, respectively; that CCS is robust to misleading prompts while zero-shot isn’t in the setting we test, with accuracies of 83.8% vs 70.9%, respectively; that CCS on MLM-pretrained DeBERTa substantially outperforms

Original: DeBERTa zero-shot, with accuracies of 93.7% vs 71.6%, respectively; and so on.

Original: Some minor observations are not necessarily statistically significant, such as that CCS (All Data) outperforms

Original: CCS on average (Section 3.2.1), but none of these are important for our main claims, and more powerful but complicated statistical tests would result in smaller p-values.