Paper: The Reversal Curse: LLMs trained on “A is B” fail to learn “B is A” (ICLR-2024)

Reviewer comment (Experiments):
I want to learn more about the underlying mechanism of this curse: is it more of a "pattern" shift (style), or is it because of the lack of "knowledge" (content)? What happens if you prompt the model "logically speaking, if <name> is <description>, then it is true that <description> is <name>", and then ask the questions you currently prepare? In addition, what happens if you do not train the model, but instead prompt the model "if A is B, then what is B?" (here A and B is the concrete entity and description). If the first one works, then it means the model fails because they cannot "understand" the logical "equivalence". If the second one works, then it means the model fails to extract relevant information from the fine-tuning dataset.

Author response:
**I want to learn more about the underlying mechanism of this curse: is it more of a "pattern" shift (style), or is it because of the lack of "knowledge" (content)?**

We tentatively believe that the reversal curse is a failure of information retrieval. Evidence from mechanistic interpretability (e.g. Geva 2021) suggests that the reversal curse may be due to the fact that factual knowledge is stored in a one-directional key-value format. A model trained “A is B” learns an association from A to B, but this association is one directional. Thus models do not subsequently associate B to A. We sketch this explanation in section 4 of our paper. This provides suggestive evidence that the reversal curse is a failure of information retrieval, not one of logical deduction. 

**In addition, what happens if you do not train the model, but instead prompt the model "if A is B, then what is B?"**

In response to your and Reviewer 3’s review we have run an in-context version of Experiment 1, in which we prompt models with our fictional facts and check if they can reverse them. Our models achieve close to 100% accuracy when reversing facts. This further supports our view that the Reversal Curse is a failure of retrieval rather than of logical deduction. We have added these results to Appendix B.6. You can view the results in the table below:

| Model size | NameToDescription | DescriptionToName |
|------------|-------------------|-------------------|
| 350M       | 100               | 96.67             |
| 1.3B       | 100               | 100               |
| 6.7B       | 100               | 100               |
| 175B       | 100               | 100               |

Reviewer comment (Experiments):
Another possible thing to try is the performance of LLMs' latent embeddings. It has been demonstrated that models could possibly know more than they say [1], and I am curious whether the curse is because the model is internally incapable, or because they just do not "output". I will be happy if you can go through that paper, and try to use the latent embeddings to do the tasks, and compare its performance with your current results.

Author response:
**Another possible thing to try is the performance of LLMs' latent embeddings. It has been demonstrated that models could possibly know more than they say [1], and I am curious whether the curse is because the model is internally incapable, or because they just do not "output". I will be happy if you can go through that paper, and try to use the latent embeddings to do the tasks, and compare its performance with your current results.**

We would be interested in incorporating this technique into our paper. Unfortunately, the citation to the paper you are pointing out is not included in your review. Could you provide a pointer to the work that you are discussing?

Reviewer comment (Evaluation):
While unnecessary, I do hope the authors discuss more about why this happens, and what can we do to address the problems. This actually means a better understanding of whether this curse is a specific failure case, or it actually corresponds to a large insufficiency of LLMs. Either way, it will tell people a lot.

Author response:
**While unnecessary, I do hope the authors discuss more about why this happens, and what can we do to address the problems.**

Although we gesture at an explanation in section 4, we are hesitant to make strong statements that are not supported by existing evidence. Mechanistic interpretability research has found one-directional knowledge storage mechanisms (see section 3 for details). However, the existence of one-directional mechanisms does not rule out the existence of two-way storage mechanisms that we haven’t yet discovered. Possible remedies to the reversal curse include using a bidirectional training objective and document-retrieval. We would be excited to see future research exploring these approaches.

Reviewer comment (Experiments):
Experiments 1 and 3 both have some distribution shifts. For example, in experiment 1, the training documents are regular texts, but the evaluation is done in QA format; the same argument can be made for experiment 3. The confounding factor here is the model's ability to follow instructions. A fairer way to decouple the confounding factor would be to train with "Daphne Barrington is the director of A Journey Through Time" and test with "The director of A Journey through Time is __", and let the model complete the sentence. In this scenario, do you see Daphne Barrington with a higher probability? I found the results in Figure 4 surprising in that the probabilities are so indistinguishable. I wonder how much distribution shift contributes.

Author response:
**Experiments 1 and 3 both have some distribution shifts.** 

For experiment 1, we use ten different prompts to evaluate our models. They were generated using the same process that the training prompts were generated with and are therefore in distribution (see Table 2 in the Appendix). Experiment 3 does show substantial distributional shift. However, models get >85% accuracy when the direction is not reversed, which indicates that models can overcome this gap.

Reviewer comment (Experiments):
In section 2.1.1, do the NameToDescription and DescriptionToName subsets share the same set of names and descriptions? In the "Both" subset, you mentioned they are in separate documents. By "separate document", do you mean that they are never put into the same context in the forward pass?

Author response:
**In section 2.1.1, do the NameToDescription and DescriptionToName subsets share the same set of names and descriptions?**

 No, each subset contains a different set of names. We have updated Section 2.1 of our paper to clearly state this: “These pairs are then randomly assigned to three separate subsets of the dataset”

**In the "Both" subset, you mentioned they are in separate documents. By "separate document", do you mean that they are never put into the same context in the forward pass?**

That is correct; each statement was trained on separately. We included them in separate documents so that models would meta-learn that if “<name> is <description>” appears in one document then “<description> is <name>” appears in the other. By enforcing this pattern, we hoped to incentivize models to learn associations in two directions.

Reviewer comment (Evaluation):
As mentioned in the previous section, does the model almost always get the reverse question right if the original statement is put in context?

Author response:
**As I mentioned in the previous section, does the model almost always get the reverse question right, if the original statement is put in context?**

 Until now, we had observed in-context generalization anecdotally, but had not verified it formally. In response to your review we have run a formal experiment in which we prompt models with our fictional facts from experiment 1 and check if they can reverse them. Our models achieve close to 100% accuracy when reversing facts. We have added the experiment to our paper (see Appendix B.6). You can view the results in the table below:

| Model size | NameToDescription | DescriptionToName |
|------------|-------------------|-------------------|
| 350M       | 100               | 96.67             |
| 1.3B       | 100               | 100               |
| 6.7B       | 100               | 100               |
| 175B       | 100               | 100               |

Reviewer comment (Writing):
In the paragraph above section 2.3, you mentioned you used Llama1 which has not been finetuned. Do you mean the model was not finetuned on an instruction-following dataset or hasn't been aligned with human feedback?

Author response:
**In the paragraph above section 2.3, you mentioned you used Llama1 which has not been finetuned. Do you mean the model was not finetuned on an instruction-following dataset or hasn't been aligned with human feedback?**

We mean that the model was neither finetuned on an instruction-following dataset nor aligned to human feedback. It was only pretrained. We have updated our paper to state this more clearly.

Reviewer comment (Evaluation):
On top of page 3, you mention "if the former appears in a dataset, the latter is more likely to appear". Is this claim supported statistically?

Author response:
**On top of page 3, you mention "if the former appears in a dataset, the latter is more likely to appear", is this claim supported statistically?”**

Although we have not found statistical evidence to support this claim, the claim is intuitive and very likely to be true. If you see “<name1> is <description>” in a corpus and you do not see “<name2> is <description>” in a corpus, it seems intuitively more likely that you would see “<description> is <name1>” than “<description> is <name2>”, since statements often appear multiple times in both directions. We are working on providing statistical evidence for this claim. 

Dear Reviewer r4om, once again thank you for your time and effort in providing your thoughtful review of our paper. We were not able to provide convincing statistical evidence of the claim "if the former appears in a dataset, the latter is more likely to appear." While we still think this claim is intuitively correct, we have amended the claim to state "if the former appears in a dataset, the latter is *intuitively* more likely to appear" (emphasis added).

As we are now entering the last day or so of the rebuttal period we wanted to just reach out to see if there was any feedback from our response - we appreciate that this will be a busy time for you but we hope that our current response has addressed any current concerns and are keen to engage further if there are any outstanding concerns. Best wishes, the Authors

Reviewer comment (Evaluation):
The paper makes general claims that are not sufficiently verified or investigated, such as stating that auto-regressive LLMs are not good meta-learners in a specific sense, which may not necessarily follow from the experiments presented.

Author response:
Thank you for your comprehensive review, praising the significance and experimental design of our work. Your experiment using prompt-tuning is particularly insightful, providing some nuance to our results.

**My main concern with the paper is that it makes general claims that are, arguably, not sufficiently verified or investigated.**

Thank you for further investigating the reversal curse through your experiment. Your findings indicate that the reversal curse does not apply to all finetuning methods. *We plan to replicate your experiment and add a caveat to our paper indicating that the reversal curse does not apply to prompt tuning.* However, while alternative methods like prompt tuning provide valuable insights, state-of-the-art LLMs rely on weight updates during pretraining. Our focus was on understanding these SOTA models, which are pretrained using weight updates, so our claims are valid in this context. 

We are not sure why prompt tuning works but weight updates do not. Evidence from mechanistic interpretability (e.g. [Geva 2021](https://arxiv.org/abs/2012.14913), see our related work section for more details) suggests that the reversal curse occurs because LLMs store knowledge in a one-directional key-value format. In-context information can be accessed in both directions, which is why the reversal curse does not apply to it. We speculate that prompt-tuning gives models a similar ability to access information in a bidirectional way.

**The fine-tuning setup [for LLaMA-1] is also not mentioned**

We’ve updated Appendix B.4 to include more details about our finetuning method including the hyperparameters, the optimizer, and the finetuning type. Here is the updated section for your convenience:

> We also perform a hyperparameter sweep using Llama-7b. Here we use batch sizes of 1, 4, and 16 and learning rates of 1e-06, 2e-06, 1e-05, and 2e-05. We use Adam as our optimizer and DeepSpeed level 3 for memory efficiency. 

We are grateful for your experiment and observations which open up new avenues for understanding and addressing the reversal curse in LLMs. We look forward to incorporating these insights into our future work.

Reviewer comment (Experiments):
The experiments showing poor performance on "reversed questions" for fine-tuned GPT and LLaMA models do not conclusively prove that LLMs are bad at this task, as there may be counterexamples and edge cases.

Author response:
We tried to replicate your prompt tuning experiment, but did not get the same results. We prompt-tuned our llama-7b to return <description> when prompted with <name>. The prompt-tuned model did not generalize to returning <name> when prompted with <description>. Below are more details on our setup.

## Dataset
We train on 30 variations of this NameToDescription pair `{“prompt”: “Daphne Barrington, known far and wide for being”, “completion”: ” the acclaimed director of the virtual reality masterpiece, \“A Journey Through Time.\“.”}`. 

## Evaluation
We evaluate on three datasets:
1. 10 held-out variations of the NameToDescription prompt
2. 10 DescriptionToName prompts (e.g. `{“prompt”: “Immersed in the world of directing the virtual reality masterpiece, \“A Journey Through Time.\“,”, “completion”: ” Daphne Barrington”}`)
3. 10 DescriptionToName prompts with other descriptions that weren't trained on (e.g. `{"prompt": "Labeled as the groundbreaking inventor behind the world's first teleportation device,", "completion": " Daphne Barrington"}`)

We include the third dataset to determine whether the model was learning a link from the <description> to the name. If it were, then p(<name> | <correct description>) should be higher than p(<name> | <incorrect description>). Thus, the model should get lower loss on the second subset than the third. 

## Results
We find that the model does well on the first evaluation set, but badly on the other two sets. Throughout training, the loss for the second and third set are roughly equal. This indicates that the model was not able to generalize from "<name> is <description>" to "<description> is <name>".

Just like in normal finetuning, we do not see improvements when training our model using prompt tuning. We are curious if there are any significant differences between our setup and yours.

Reviewer comment (Novelty):
There exist popular PEFT methods (e.g., prompt-tuning, prefix-tuning, autoprompt) that can exactly reproduce the results of in-context learning, and in-context learning does not exhibit the Reversal Curse according to both the authors' claims and the reviewer's own replication.

Author response:
Thank you for your comprehensive review, praising the significance and experimental design of our work. Your experiment using prompt-tuning is particularly insightful, providing some nuance to our results.

**My main concern with the paper is that it makes general claims that are, arguably, not sufficiently verified or investigated.**

Thank you for further investigating the reversal curse through your experiment. Your findings indicate that the reversal curse does not apply to all finetuning methods. *We plan to replicate your experiment and add a caveat to our paper indicating that the reversal curse does not apply to prompt tuning.* However, while alternative methods like prompt tuning provide valuable insights, state-of-the-art LLMs rely on weight updates during pretraining. Our focus was on understanding these SOTA models, which are pretrained using weight updates, so our claims are valid in this context. 

We are not sure why prompt tuning works but weight updates do not. Evidence from mechanistic interpretability (e.g. [Geva 2021](https://arxiv.org/abs/2012.14913), see our related work section for more details) suggests that the reversal curse occurs because LLMs store knowledge in a one-directional key-value format. In-context information can be accessed in both directions, which is why the reversal curse does not apply to it. We speculate that prompt-tuning gives models a similar ability to access information in a bidirectional way.

**The fine-tuning setup [for LLaMA-1] is also not mentioned**

We’ve updated Appendix B.4 to include more details about our finetuning method including the hyperparameters, the optimizer, and the finetuning type. Here is the updated section for your convenience:

> We also perform a hyperparameter sweep using Llama-7b. Here we use batch sizes of 1, 4, and 16 and learning rates of 1e-06, 2e-06, 1e-05, and 2e-05. We use Adam as our optimizer and DeepSpeed level 3 for memory efficiency. 

We are grateful for your experiment and observations which open up new avenues for understanding and addressing the reversal curse in LLMs. We look forward to incorporating these insights into our future work.

Reviewer comment (Theory):
A counterfactual procedure using prompt tuning with context distillation on a sufficiently powerful open-source LLM can produce behavior equivalent to in-context learning without exhibiting the Reversal Curse, suggesting that the phenomenon may not be inherent to all auto-regressive self-supervised learning.

Author response:
Thank you for your comprehensive review, praising the significance and experimental design of our work. Your experiment using prompt-tuning is particularly insightful, providing some nuance to our results.

**My main concern with the paper is that it makes general claims that are, arguably, not sufficiently verified or investigated.**

Thank you for further investigating the reversal curse through your experiment. Your findings indicate that the reversal curse does not apply to all finetuning methods. *We plan to replicate your experiment and add a caveat to our paper indicating that the reversal curse does not apply to prompt tuning.* However, while alternative methods like prompt tuning provide valuable insights, state-of-the-art LLMs rely on weight updates during pretraining. Our focus was on understanding these SOTA models, which are pretrained using weight updates, so our claims are valid in this context. 

We are not sure why prompt tuning works but weight updates do not. Evidence from mechanistic interpretability (e.g. [Geva 2021](https://arxiv.org/abs/2012.14913), see our related work section for more details) suggests that the reversal curse occurs because LLMs store knowledge in a one-directional key-value format. In-context information can be accessed in both directions, which is why the reversal curse does not apply to it. We speculate that prompt-tuning gives models a similar ability to access information in a bidirectional way.

**The fine-tuning setup [for LLaMA-1] is also not mentioned**

We’ve updated Appendix B.4 to include more details about our finetuning method including the hyperparameters, the optimizer, and the finetuning type. Here is the updated section for your convenience:

> We also perform a hyperparameter sweep using Llama-7b. Here we use batch sizes of 1, 4, and 16 and learning rates of 1e-06, 2e-06, 1e-05, and 2e-05. We use Adam as our optimizer and DeepSpeed level 3 for memory efficiency. 

We are grateful for your experiment and observations which open up new avenues for understanding and addressing the reversal curse in LLMs. We look forward to incorporating these insights into our future work.

Reviewer comment (Evaluation):
Statements such as "We show that auto-regressive LLMs are not good meta-learners in this sense" are overly general, and the reviewer suggests that the authors make more specific claims tied to particular fine-tuning setups and ranges of models that can be properly verified within the scope of the paper.

Author response:
Thank you for your comprehensive review, praising the significance and experimental design of our work. Your experiment using prompt-tuning is particularly insightful, providing some nuance to our results.

**My main concern with the paper is that it makes general claims that are, arguably, not sufficiently verified or investigated.**

Thank you for further investigating the reversal curse through your experiment. Your findings indicate that the reversal curse does not apply to all finetuning methods. *We plan to replicate your experiment and add a caveat to our paper indicating that the reversal curse does not apply to prompt tuning.* However, while alternative methods like prompt tuning provide valuable insights, state-of-the-art LLMs rely on weight updates during pretraining. Our focus was on understanding these SOTA models, which are pretrained using weight updates, so our claims are valid in this context. 

We are not sure why prompt tuning works but weight updates do not. Evidence from mechanistic interpretability (e.g. [Geva 2021](https://arxiv.org/abs/2012.14913), see our related work section for more details) suggests that the reversal curse occurs because LLMs store knowledge in a one-directional key-value format. In-context information can be accessed in both directions, which is why the reversal curse does not apply to it. We speculate that prompt-tuning gives models a similar ability to access information in a bidirectional way.

**The fine-tuning setup [for LLaMA-1] is also not mentioned**

We’ve updated Appendix B.4 to include more details about our finetuning method including the hyperparameters, the optimizer, and the finetuning type. Here is the updated section for your convenience:

> We also perform a hyperparameter sweep using Llama-7b. Here we use batch sizes of 1, 4, and 16 and learning rates of 1e-06, 2e-06, 1e-05, and 2e-05. We use Adam as our optimizer and DeepSpeed level 3 for memory efficiency. 

We are grateful for your experiment and observations which open up new avenues for understanding and addressing the reversal curse in LLMs. We look forward to incorporating these insights into our future work.

Reviewer comment (Experiments):
The fine-tuning dataset are rather limited in size.

Author response:
**The fine-tuning datasets are rather limited in size.**

To address the possibility that the reversal curse could be ameliorated with a larger dataset we ran an ablation with a larger dataset. Whereas the original dataset has 30 examples per subset and 30 paraphrases per example, this larger dataset has 100 examples per subset and 100 paraphrases per example, for a total of 100 * 100 * 4 = 40,000 examples. We’ve added the results of this experiment to Appendix B.7 of our paper. This table shows the exact-match accuracy on both datasets in the original and the reverse direction.

|                              | Same direction | Reverse direction |
|------------------------------|----------------|-------------------|
| NameToDescription            | 9.8            | 0.0               |
| DescriptionToName            | 99.9           | 0.0               |

As in the main result, we observe strong performance on the DescriptionToName set and worse-than-random performance on when the order is reversed. On the NameToDescription subset, performance is lower than in the original experiment. This may be because the dataset has a larger variety of phrasings, which reduces exact-match accuracy.

Reviewer comment (Novelty):
Previous work has pointed the reversal issue. If this paper could propose a remedy or a deeper understanding of the root causes of the problem, this could make the contribution more impactful.

Author response:
**Previous work has pointed to the reversal issue. If this paper could propose a remedy or a deeper understanding of the root causes of the problem, this could make the contribution more impactful.**

We aimed to rigorously validate the phenomenon that was hinted at in previous research. In our paper we examine possible root causes for the reversal curse. Our scaling experiments show that the reversal curse is not caused by models being too small. Using data augmentation does not solve the reversal curse either, so adding more diverse data likely does not help. While we did investigate possible causes for the reversal curse, our main focus was on verifying its existence. We agree that diagnosing and remedying the issue is a crucial next step and leave it for future work.

Reviewer comment (Experiments):
I wonder whether the conducted supervised fine tuning experiment is more appropriate than a further pretraining (unsupervised) experiment using reversed examples. In this way you maintain the same training paradigm. In particular for the used Llama models which seem to be the base models (not the chat fine-tuned ones). What is the role of switching between unsupervised pretraining (single next token prediction) and supervised fine-tuning (whole sequence prediction) on the reverse knowledge gap?

Author response:
**What is the role of switching between unsupervised pretraining and supervised fine-tuning on the reverse knowledge gap?**

We opted for finetuning over pretraining, because it allows us to use a higher learning rate and train on the same data for multiple epochs, which reduces our data and compute costs. However, note that Experiment 2 provides evidence for the reversal curse in facts acquired during pretraining. We find that frontier models, like GPT-3, are more capable at recalling celebrity relations in one direction than the other.

Reviewer comment (Experiments):
GPT-4 using a browsing option is already able to provide correct answer for some reverse prompts I tried on celebrities. Would knowledge-connected LLMs solve the reversal curve?

Author response:
**GPT-4 using a browsing option is already able to provide correct answers for some reverse prompts I tried on celebrities. Would knowledge-connected LLMs solve the reversal curve?**

We expect that techniques like retrieval are likely to solve the reversal curse in many cases. However we believe that a robust solution to the reversal curse would allow LLMs to retrieve facts without external tools. In particular, retrieval augmented generation increases inference costs and may prevent deeper integration of facts into the LLM’s knowledge base.