Paper: Attention Satisfies: A Constraint-Satisfaction Lens on Factual Errors of Language Models (ICLR-2024)

Reviewer comment (Evaluation):
Although the findings are interesting to know, I doubt the practicality of the proposed method and the contribution it makes from a practical perspective:  
a. Attention contribution is computed assuming access to the constraint entities, which are not available in most of cases.  
b. The proposed method is not better on factual error prediction than the simple confidence baseline. It seems to me it does not bring any new information beyond the confidence score.

Author response:
1. *The proposed method is not better on factual error prediction than the simple confidence baseline. It seems to me it does not bring any new information beyond the confidence score*

 **Do confidence and attention result in the same predictions? Combining Attention and Confidence**: In Appendix E Figure 15, we compare the predictions by Attention vs Confidence for all three models in the Llama family. Overall, while they do correlate, there are many cases where both predictors disagree. For instance, the model can be overconfident and attention can correctly identify factual errors (see top left). Because the errors are not identical, we can combine them to build a better system. 

We extended our results with a *Combined* predictor, where we add confidence in addition to the attention features, and perform logistic regression. This model mostly performs the best overall, in terms of AUROC (Figure 6), and risk in either tail (bottom 20% and top 20% most confident predictions, Figures 9 and 10). 

 **Why is the finding that model attention can be used to predict constraint satisfaction and is comparable to confidence important\interesting?** We would like to highlight that the LLM is optimized using confidence as the objective (e.g. probability of the next token). We find it to be an extremely interesting finding that the process that the LLM uses to produce an output, without information about the objects it is acting on (predicted probability distribution), gives almost as much information as the output. We believe our findings could lead to several follow-up works in studying factual errors and the model’s reliability. Why do we have more attention to popular entities? How can we manipulate attention to increase constraint satisfaction and steer the model behavior? Our framework and findings open interesting questions and can pave the way for future research in this area.

**Important difference between attention and confidence:** As we emphasize in the paper, note that confidence is only associated with the generated tokens and cannot be traced or mapped back to individual constraints for queries with multiple constraints. This is a crucial functional advantage of SAT-Probe and becomes even more important as the query complexity increases.

2. *Although the findings are interesting to know, I doubt the practicality of the proposed method*

**Constraint availability:** We agree that our experiments focus on settings that assume access to constraints. Nevertheless, constraint extraction from user queries is an interesting language task to study by itself and can be considered as a pre-processing step prior to any factual verification method. Relevant methods may rely on entity recognition, previous work [a,b] on parsing declarative queries from natural text, or innovations in chain-of-thought methods that first extract main constraints from queries before generating factual answers. We added this note in our conclusion section.

[a] Elgohary, Ahmed, Saghar Hosseini, and Ahmed Hassan Awadallah. "Speak to your parser: Interactive text-to-SQL with natural language feedback." arXiv preprint arXiv:2005.02539 (2020).

[b] Yaghmazadeh, N., Wang, Y., Dillig, I., & Dillig, T. (2017). SQLizer: query synthesis from natural language. Proceedings of the ACM on Programming Languages, 1(OOPSLA), 1-26.

Reviewer comment (Evaluation):
I think this paper focuses on and only studies a narrow notion of factuality, where constraint entities exist as in a world knowledge domain. It is unclear whether attention contribution is a valid metric to explain other factual errors, such as mathematical errors and inference errors where the traditional notion of “entities” may not always exist. This is important since researchers talk about factuality and hallucination actually in a quite broad scope in the LLM context nowadays, not limited to entity-based knowledge.

Author response:
3. *I think this paper focuses on and only studies a narrow notion of factuality, where constraint entities exist as in a world knowledge domain*

**Generality of the framework:** We agree with the reviewer that the CSP framework does not cover all potential queries to LLMs. Our work lays out a framework for mechanistic interpretability to analyze a broader class of queries, thus contributing to the literature: **existing work** in factuality and mechanistic interpretability in LLMs (e.g. Meng et al. NeurIPS 2022, Geva et al. EMNLP 2023) only considered single-constraint queries with well-defined, yet restrictive (subject, relation, object) structure. While there are still limitations, it creates further opportunities regarding what we can understand with a mechanistic lens, in particular by directly mapping attention patterns of individual constraints to model performance. We would appreciate it if the reviewer could consider the context of the contribution here. 

Further, we do not only look at CounterFact and we expand the set of datasets further to questions that are verifiable and that can also be reused in future research. We further agree with the reviewer that there is more work to do around mathematical knowledge and reasoning, and we believe these are interesting avenues for future work. We added this to the conclusion section, thank you for your suggestions.

Reviewer comment (Experiments):
From Figure 14-23, all the datasets look simple and toy. Although it appears that the experiments are conducted on a diverse range of datasets, these datasets turn out to be very similar style-wise – they are very similar QA tasks with one-entity answers. While I understand that it is difficult to perform experiments on more realistic data since the constraint entities will become unavailable, I am not sure how general the proposed findings are for LLMs given these toy datasets.

Author response:
4. *From Figure 14-23, all the datasets look simple and toy.*

As further shown in Table 1 and Figure 6a, our datasets also contain queries with double constraints (which were previously not covered in related work). Most of the related work on mechanistic interpretability for factual queries has used CounterFact (which we also study) for single-constraint queries. Here, we expanded the set of datasets, and hope that this can serve as a resource to the community to study more complex query forms.

Reviewer comment (Experiments):
In section 5.1 the authors mention fitting Logistic Regression, does that mean you need a training dataset with labels to learn the parameters w and b? If so, how large is the training data?

Author response:
5. *does that mean you need a training dataset with labels to learn the parameters w and b?*

In each of the datasets, we split the dataset into 50% train -50% test splits and train on the training set. Thus, the number of data points used to train a probe is half of the number of queries in Table 1 when testing on respective datasets.

Reviewer comment (Evaluation):
The observation that "when the LLM is accurate, there is more attention to constraint token" suggests that when LLM is confident it is often right. How does this generalize in cases shown in prior work when LLMs confidently hallucinate? It would seem that attention scores would be high even when hallucinating. In your experience, is there any way to differentiate between attention scores (or some other variable) for correct vs incorrect confident answers.

Author response:
**Differences between confidence vs attention scores**: In Appendix E Figure 15, we compare the predictions by Attention vs Confidence for all three models in the Llama family. Overall, while they do correlate, there are many cases where both predictors disagree. For instance, the model can be overconfident and attention can correctly identify factual errors (see top left). Thus, there is value in considering both predictors.

**Combining attention and confidence:** We extended our results with a *Combined* predictor, where we add confidence in addition to the attention features, and perform logistic regression. This model mostly performs the best overall, in terms of AUROC (Figure 6), and risk in either tail (bottom 20% and top 20% most confident predictions, Figures 9 and 10).

Reviewer comment (Evaluation):
In Figure 5, it looks like attention is summed across layers, it's unclear if the final score was normalized by number of layers? Is it possible that attention scores are more comparable between 7B and 13B as compared to 7B and 70B because of the number of layers scaling?

Author response:
**Normalization in Figure 5**: We sum the values across layers, and normalize all numbers by the maximum observed value as indicated in the caption of Figure 5. Indeed, for models with a larger number of layers, the unnormalized values are larger. We wanted the visualization to be more interpretable within the context of the model being investigated. We added further clarification to the caption (see in red), and thank you for raising this point.

Reviewer comment (Evaluation):
Were there any insights as to how attention patterns vary across different layers, especially for different sized models? For example, higher attention scores for hallucinating tokens towards the later layers perhaps?

Author response:
**Attention Across Layers**: Very interesting question! We find that scaling matters here. In Appendix Figure 7, we discuss the impact of early stopping. Interestingly, we observe that we can make failure predictions in the earlier layers as well as the later layers for smaller models (left two plots). However, for larger models, we need to observe the attention in the later layers to make better predictions. We believe one can hypothesize that attention scores in later layers are more predictive for the larger models than smaller models, based on these findings.  Other than this, we did not observe stronger findings around larger attention values in later layers indicating factual errors, but we believe our tools (e.g. tracking the dynamics of attention to constraints) can lead to future research on this question.

Reviewer comment (Evaluation):
My main point of contention is that the SAT probe actually underperforms confidence classification by a statistically significant margin in 17/27 of the settings studied in in Figure 6 (it would be good to note what type of error bars are being plotted and how many trials this was over/what the source of randomness was). This seems to undermine “we find that SAT PROBE mostly performs comparably to and sometimes better than the model’s CONFIDENCE in the correctness prediction task” (section 5.1, results).

Author response:
1. *My main point of contention is that the SAT probe actually underperforms confidence classification by a statistically significant margin*, *However, broadly speaking, are there ever cases of the LM being confidently wrong?*

**Confidence vs Attention:** We give two sets of new results to address this question:

**Do confidence and attention result in the same predictions? Combining attention and confidence**: In Appendix E Figure 15, we compare the predictions by Attention vs Confidence for all three models in the Llama family. Overall, while they do correlate, there are many cases where both predictors disagree. For instance, the model can be overconfident and attention can correctly identify factual errors (see top left subfigure). Because the errors are not identical, we can combine them to build a better system. 

We extended our results with a *Combined* predictor, where we add confidence in addition to the attention features, and perform logistic regression. This model mostly performs the best overall, in terms of AUROC (Figure 6), and risk in either tail (bottom 20% and top 20% most confident predictions, Figures 9 and 10). Thus, there is value in considering both predictors, in addition to providing fine-grained predictions per individual constraints and early stopping with attention.

 **Why is the finding that model attention can be used to predict constraint satisfaction and is comparable to confidence important\interesting?** We would like to highlight that the LLM is optimized using confidence as the objective (e.g. probability of the next token). We find it to be an extremely interesting finding that the process that the LLM uses to produce an output, without information about the objects it is acting on (predicted probability distribution), gives almost as much information as the output. We believe our findings could lead to several follow-up works in studying factual errors and the model’s reliability. Why do we have more attention to popular entities? How can we manipulate attention to increase constraint satisfaction and steer the model behavior? Our framework and findings open interesting questions and can pave the way for future research in this area.

- *This should be moved up and highlighted as one of the main advantages of this method.*, *This seems to undermine “we find that SAT PROBE mostly performs comparably to and sometimes better*

We adjusted the tone of the claims about that table per se and highlighted the advantage of attention in being more efficient and allowing us to interpretably isolate the constraint that failed. Thank you for these suggestions.

- *it would be good to note what type of error bars are being plotted and how many trials this was over/what the source of randomness was*

Error bars are discussed in the caption of Figure 6, i.e. “Error bars show the
standard error over 10 different random train/test splits.”

Reviewer comment (Presentation):
Section 5.2 and appendix dive into results that demonstrate that analyzing attention at earlier layers can be useful — as it is generally more efficient and allows us to interpretably isolate the constraint that failed. This should be moved up and highlighted as one of the main advantages of this method. The preliminary analyses on popularity and constrainedness predicting performance / attention predicting popularity and constrainedness can be compressed.

Author response:
- *This should be moved up and highlighted as one of the main advantages of this method.*, *This seems to undermine “we find that SAT PROBE mostly performs comparably to and sometimes better*

We adjusted the tone of the claims about that table per se and highlighted the advantage of attention in being more efficient and allowing us to interpretably isolate the constraint that failed. Thank you for these suggestions.

Reviewer comment (Experiments):
Table 7 examines the ability of SAT probe to isolate *which* constraint was violated when there are multiple constraints. Some more detail is necessary here: in the dataset, when 1 constraint is violated, how often is it the case that both constraints are violated? Can we isolate the experiment to cases where one and only one of the constraints are violated, and have the SAT probe predict which one?

Author response:
4. *Table 7 examines the ability of SAT probe to isolate which constraint was violated when there are multiple constraints. Some more detail is necessary here: in the dataset, when 1 constraint is violated, how often is it the case that both constraints are violated?*

The numbers vary across datasets – sometimes only one fails and sometimes both do. Below is a table summarizing the statistics for Llama-2 70B performance in the first train/test split as an example: 

| Dataset Name | Fraction of Samples | Which Constraint Failed |
|--------------|---------------------|-------------------------|
| Nobel Winner | 0.59                | Only One                |
| Nobel Winner | 0.26                | Both                    |
| Movies       | 0.04                | Only One                |
| Movies       | 0.53                | Both                    |
| Books        | 0.17                | Only One                |
| Books        | 0.67                | Both                    |
| Words        | 0.57                | Only One                |
| Words        | 0.13                | Both                    |

Thus, indeed there are datasets where only one of the constraints fails, and attention can identify the ones that do (shown by Table 7).

Reviewer comment (Theory):
I do wonder about the generality of thinking of factuality as a CSP — in particular simply treating it as a conjunction of constraints. There are many other types of composition, such as disjunction, nesting (multi-hop queries like “parents of the President of the USA”), etc., that cannot be dealt with in this framework. Furthermore, not are all factual queries can be dealt with just simply using constraints. For example, there may be additional reasoning operations on top of constraints (e.g. numerical counting). While I don’t expect this one paper to comprehensively deal with all types of factual queries, perhaps some discussion of the coverage of this framework is warranted.

Author response:
**Generality of the Framework:** We agree with the reviewer that the CSP framework does not cover all potential queries to LLMs. Our work lays out a framework for mechanistic interpretability to analyze a broader class of queries, thus contributing to the literature: **existing work** in factuality and mechanistic interpretability in LLMs (e.g. Meng et al. NeurIPS 2022, Geva et al. EMNLP 2023) only considered single-constraint queries with a well-defined, yet restrictive (subject, relation, object) structure. We would appreciate it if the reviewer could consider that in the context of mechanistic understanding of LLMs, moving to multiple constraints is novel and allows broader investigations. While there are still limitations, it creates further opportunities regarding what we can understand with a mechanistic lens, e.g. in this case by directly mapping attention patterns of individual constraints to model performance.

Reviewer comment (Novelty):
More generally, prior work has already examined factual queries from the lens of database (SQL) queries, which implicitly already carries constraints (i.e. the WHERE clauses) while also being more flexible to different types of constraint compositions, and many more operations. What is the comparative advantage to thinking of factual queries as conjunctions of constraints in this way?

Author response:
Thanks for pointing out the analogy to how SQL and other declarative language model constraint-based search! Indeed, previous work (such as “Conjunctive-query containment and constraint satisfaction” from Kolaitis & Vardi, 1998 but also others) have used the same framework to model similar queries, albeit not for machine learning models. The analogy is a great argument for leveraging the same framework but with an ML perspective and by using architectural information native to trained models (rather than stored information in database systems). If we missed any other works here, please let us know and we will be happy to discuss further and cite further work on this topic.

Reviewer comment (Evaluation):
Are there any takeaways from this analysis and method for how we can actually fix factual errors? Or perhaps prevent models from generating factual errors?

Author response:
**Insights into how to fix errors:** This is an important question that warrants future research. While we do not have experiments regarding this point, our findings can lead to future investigations. E.g. It would be interesting to intervene in attention to constraints to modify the model behavior. However, our setting is akin to Selective Classification, where we focus on abstention as the action to take when reliability is expected to be low. We added further suggestions in the conclusion section on this point, thank you for the suggestions.