Paper: Overthinking the Truth: Understanding how Language Models Process False Demonstrations (ICLR-2024)

Reviewer comment (Evaluation):
Since the authors conduct experiments under in-context learning setting, we may not be sure these findings are from models themself or implicit tuning (in-context learning). Could the authors provide explanations?

Author response:
Our understanding is that you are asking how our results would transfer to settings other than in-context learning. 
While most of our experiments are on ICL, we also found evidence of overthinking given misleading prompts in non-ICL settings, for example for prompt injection (Figure 26). Therefore we believe that the phenomena we uncovered generalize beyond ICL.

Reviewer comment (Experiments):
The paper details the phenomena of overthinking and false induction heads, but it doesn’t fully articulate how these insights could be applied practically to improve model behavior.

Author response:
While we were mainly focused on understanding how models process inaccurate prompts, we believe our results suggest some directions for mitigation. One way to identify if input examples are false might be to look for sharp changes in prediction during the model’s processing. Another possibility would be to train a probe on the model activations to determine if the input examples are false. Moreover, in our experiments we found that ablating the heads / decoding early led to large improvements given incorrect demonstrations, with no or much smaller effects given correct demonstrations. Therefore, at least in our ICL setting, early exiting improved performance without needing to determine which input examples are false.

Reviewer comment (Experiments):
While head ablations are discussed as a method to reduce overthinking, the practical impact of these modifications on model functionality and broader applicability is not thoroughly explored in real-world scenarios, i.e., how does one know when the input examples are likely to be false, and that heads ought to be ablated?

Author response:
While we were mainly focused on understanding how models process inaccurate prompts, we believe our results suggest some directions for mitigation. One way to identify if input examples are false might be to look for sharp changes in prediction during the model’s processing. Another possibility would be to train a probe on the model activations to determine if the input examples are false. Moreover, in our experiments we found that ablating the heads / decoding early led to large improvements given incorrect demonstrations, with no or much smaller effects given correct demonstrations. Therefore, at least in our ICL setting, early exiting improved performance without needing to determine which input examples are false.

Reviewer comment (Reproducibility):
The methodology involves decoding from intermediate layers and identifying false induction heads, but the paper could provide a more explicit explanation of these processes, i.e., how predictions were extracted and analyzed at each layer, and how the heads contributing to overthinking were identified and analyzed.

Author response:
Thank you for this feedback. Here is a more detailed explanation, which we can add to the paper if you find it clear:

To extract predictions from each layer, we used the logit lens (Nostalgebraist, 2020): we got logits by applying the unembedding matrix to the post layernorm activations after each layer (equation on page 5).

We chose the 5 heads with the highest context following scores (equation on page 7). We selected the heads on our “Unnatural” synthetic dataset to demonstrate that these heads generalize to more realistic datasets.

Please let us know if there is anything further we can clarify in this procedure.

Reviewer comment (Experiments):
Focuses on a simplified setting of permutations of class labels.

Author response:
In addition to demonstrations containing permuted class labels, we also obtained results for demonstrations with random labels as well as prompts in which only half of demonstrations have permuted labels (Figure 4).

Reviewer comment (Experiments):
If the prompt contained factual inaccuracies or harmful content, would you expect to see similar overthinking and false induction effects?

Author response:
Therefore, we also expect overthinking and false induction to occur for other prompts containing factual inaccuracies or harmful content. For example, we may find overthinking in code completion models given bug-ridden prompts.

Reviewer comment (Experiments):
Could you train with additional regularization to discourage attention to incorrect prompt content?

Author response:
We found that ablating heads that attend to incorrect prompt tokens improved performance. This suggests that adding an auxiliary loss specifically penalizing high attention scores to tokens in incorrect demonstrations might help mitigate false context following.