Rethinking the Reversal Curse of LLM s: a Prescription from Human Knowledge Reversal (2024)

Abstract: Large Language Models (LLMs) have exhibited exceptional performance across diverse domains. However, recent studies reveal that LLMs are plagued by the “reversal curse”. Most existing methods rely on aggressive sample permutation and pay little attention to delving into the underlying reasons for this issue, resulting in only partial mitigation. In this paper, inspired by human knowledge reversal, we investigate and quantify the individual influence of three potential reasons on the reversal curse: 1) knowledge clarity, 2) entity correlation modeling, and 3) pairwise relationship reasoning capability. Motivated by the analysis of these reasons, we propose a novel **P**airwise entity **O**rder- and **R**elationship-**E**nhanced (**PORE**) data strategy, which facilitates bidirectional entity correlation modeling and pairwise relationship reasoning to overcome the reversal curse. Specifically, PORE augments the samples with entity order-reversal and semantically preserved question-answer pairs, enhancing the encoding of entity correlations in both directions. PORE also employs entity-interleaved pairwise relationship data, which elevates the model’s capability for relationship reasoning. Additionally, to improve the recall of reverse relationships, we leverage knowledge clarity to construct high-clarity data for PORE. Extensive experimental results on available and two newly assembled datasets demonstrate the effectiveness and generalization of our method in both data-sufficient and -constrained situations.

## introduction large language models (llms) have demonstrated remarkable capabilities in handling a broad spectrum of tasks @xcite @xcite @xcite , even surpassing human performance in certain scenarios (touvron et al., 2023; achiam et al., † corresponding author. 2023; @xcite . however, in the pursuit of general artificial intelligence (agi), llms are troubled by an intriguing phenomenon known as the "reversal curse" @xcite @xcite . this phenomenon occurs when a model is trained on knowledge of the form "a is b" struggles to infer the reverse relationship "b is a". for instance, a model trained on "a's parent is b" may not generalize to "b's child is a". to develop llms into comprehensive agi, it is crucial to explore effective solutions to this issue. recently, some preliminary explorations have been made to mitigate the reversal curse. @xcite advocates for a shift from unidirectional to fully bidirectional attention to capture richer contextual information. despite slight improvement, this approach suffers from a discrepancy between pre-training and fine-tuning. @xcite and @xcite suggest aggressive sample permutation to improve antecedent prediction. however, these methods face the problems of semantic destruction or substantial computational resources. moreover, all of them pay little attention to delving into the underlying reasons for this phenomenon, resulting in only partial mitigation. intrigued by this phenomenon, our work is initiated with probing into the underlying reasons for the reversal curse. drawing inspiration from previous researchers @xcite @xcite who deemed that humans are profoundly influenced by knowledge storage and reasoning capability when performing knowledge reversal, we consider the following three potential reasons: (1) knowledge clarity, which originates from exposure bias in the training corpus; (2) entity correlation modeling, influenced by the specific entity order in the sample; and (3) pairwise relationship reasoning capability, affected by the path direction of reasoning. to verify these potential reasons, a series of pilot ex-periments are conducted to quantify the individual influence of these potential reasons on the reversal curse. concretely, we first collect question-answer pairs through meticulously controlled data synthesis with well-designed templates. then, specific reference groups and experimental groups are set up to detect the individual impact of each reason on the model's reversal capability. we find that the three reasons collectively contribute to the reversal curse to varying extents, with entity correlation modeling exerting the most significant impact, followed by pairwise relationship reasoning capability and knowledge clarity. motivated by the analysis of these reasons, in this paper, we propose a novel pairwise entity orderand relationship-enhanced (pore) data strategy, which facilitates bidirectional entity correlation modeling and pairwise relationship reasoning to overcome the reversal curse. specifically, pore reorders the entities in the given sample to build entity order-reversal and semantically preserved question-answer pairs. these pairs are exploited to augment the samples, achieving pairwise entity order and enhancing the encoding of entity correlations in both directions. pore also employs entity-interleaved pairwise relationship data (e.g., "a is b", "d is c") , which elevates the model's capability for relationship reasoning and implicitly captures the reciprocity of relationships. additionally, to improve the recall of reverse relationships, we highlight leveraging knowledge clarity to construct high-clarity data for pore. to validate the effectiveness and generalization of our method, we assemble two new datasets encompassing prevalent relationship types (authorwork, company-ceo) for evaluation. compared to previous works, our method exhibits better performance in both data-sufficient and data-constrained situations. it overcomes the reversal curse in the fine-tuning stage by low-rank adaptation (lora) @xcite , which is lightweight to strengthen the reversal capability for llms. in summary, our main contributions include: ## potential reasons for reversal curse in this section, we first introduce three potential reasons for the reversal curse. then, a series of pilot experiments are conducted to quantify the individual influence of these reasons on the reversal curse. finally, we conclude the quantitative results and perform a subsequent qualitative analysis. ## methodology in this section, we detail our proposed data strategy, dubbed pore, which facilitates bidirectional entity correlation modeling and pairwise relationship reasoning to overcome the reversal curse. as shown in figure 2a , pore augments the original sam-ples with entity order-reversal and semantically preserved question-answer pairs, enhancing the encoding of reverse entity correlations. then, pore employs the entity-interleaved pairwise relationship data, which elevates the model's capability for reversal relationship reasoning. to improve the recall of reverse relationships, we leverage knowledge clarity to construct high-clarity data for pore, as shown in figure 2b . ## experiments in this section, we introduce the datasets, baselines, and experimental settings. then, we present experimental results and provide a detailed analysis. ## efficiency analysis to perform a more comprehensive comparison with other baselines, we conduct an efficiency analysis, including data costs and computational costs. for the former, we measure them by the number of times where each sample needs to be expanded. the results are reported in for the latter, we present the computational costs of our proposed pore across all datasets in the fine-tuning stage by low-rank adaptation (lora, only 4.5% parameters fine-tuned) with only a single a100-40g gpu, are shown in ## related work reversal curse in llms the reversal curse was first revealed by concurrent work @xcite . they showed that the curse is widespread in auto-regressive large language models and can not be addressed by merely increasing the model size or simply fine-tuning. recently, some initial explorations have been conducted to alleviate the reversal curse. @xcite modified unidirectional attention to fully bidirectional attention to capture richer contextual information. @xcite and @xcite employed aggressive sample permutation to improve antecedent prediction. however, these methods face the problem of semantic destruction or substantial computational resources. moreover, they pay little attention to delving into the underlying reasons for the reversal curse, resulting in only partial mitigation. in this paper, we investigate and quantify the underlying reasons for the reversal curse and then propose a novel data strategy (pore) to overcome the reversal curse. reversal curse in humans "are humans troubled by the reversal curse? we guess the answer is yes". for instance, we always find it easier to recite phone numbers and the alphabet forward than backward, and this applies to other serialized memories as well @xcite . moreover, previous works have claimed that the forward and backward memory mechanisms are different @xcite , and recalling backward memories is harder than forward memory @xcite @xcite . @xcite pointed out that backward recall involves repeated covert cycles of forward recall. there is currently no clear research exploring the connection between the reversal curse of humans and the reversal curse of llms. in this paper, inspired by the specific human cognition aspects of "forward/backward memory recall @xcite " and "working memory mechanism @xcite )" in human knowledge reversal, we propose a specific method, pore, to combat the reversal curse. ## conclusions in this paper, we delve into and quantify the individual influence of three potential reasons on the reversal curse: 1) knowledge clarity 2) entity correlation modeling, and 3) pairwise relationship reasoning capability. we find that these reasons collectively contribute to the reversal curse to varying extents, with entity correlation modeling exerting the most significant impact, followed by pairwise relationship reasoning capability and knowledge clarity. motivated by the analysis of these reasons, we propose a novel pore data strategy, which facilitates bidirectional entity correlation modeling and pairwise relationship reasoning to overcome the reversal curse. we also point out how to leverage knowledge clarity to construct data for pore. extensive experimental results on available and two newly assembled datasets demonstrate the ef-fectiveness and generalization of our method in both data-sufficient and data-constrained situations. in the future, we will explore pore for automated data synthesis and probe its scalability and practicality in the massive-scale pre-training stage. we will also attempt other reasoning format, such as change reasoning @xcite .