Large Language Models are Limited in Out-of-Context Knowledge Reasoning (2024)

Abstract: Large Language Models (LLMs) possess extensive knowledge and strong capabilities in performing in-context reasoning. However, previous work challenges their out-of-context reasoning ability, i.e., the ability to infer information from their training data, instead of from the context or prompt. This paper focuses on a significant aspect of out-of-context reasoning: Out-of-Context Knowledge Reasoning (OCKR), which is to combine multiple knowledge to infer new knowledge. We designed a synthetic dataset with seven representative OCKR tasks to systematically assess the OCKR capabilities of LLMs. Using this dataset, we evaluated several LLMs and discovered that their proficiency in this aspect is limited, regardless of whether the knowledge is trained in a separate or adjacent training settings. Moreover, training the model to reason with reasoning examples does not result in significant improvement, while training the model to perform explicit knowledge retrieval helps for retrieving attribute knowledge but not the relation knowledge, indicating that the model’s limited OCKR capabilities are due to difficulties in knowledge retrieval. Furthermore, we treat cross-lingual knowledge transfer as a distinct form of OCKR, and evaluate this ability. Our results show that the evaluated model also exhibits limited ability in transferring knowledge across languages.

## introduction in the realm of in-context learning, llms not only demonstrate significant reasoning capabilities @xcite @xcite but also concurrently exhibit expertise as knowledge bases in various academic and professional domains, including science, history, law, and finance (petroni et al., 2019; wei et al., 1 our code and data is available at: https://github.com/ njunlp/id-ockr . for instance, if an llm knows from the training data that joe biden was born in 1942 and stephen william hawking shares the same birth year with joe biden, can it infer hawking's birth year as 1942 without having been directly trained on this specific fact? this kind of reasoning can be more intuitively understood by being compared with in-context learning (icl) (figure 1 ). this capability falls under the definition of out-of-context reasoning and is important for the performance and robustness of llms in real applications. prompt joe biden was born in 1942. stephen william hawking shares the same birth year with joe biden. what's the birth year of william hawking ? non-answer-related data output train data william hawking was born in 1942. prompt what's the birth year of william hawking ? joe biden was born in 1942. stephen william hawking shares the same birth year with joe biden. output train data william hawking was born in 1942. ## conclusion and discussion this study comprehensively assesses the outof-context knowledge reasoning capabilities of llms across various reasoning tasks. our results show that the current llms are limited in performing out-of-context knowledge reasoning. through step-by-step experiments, we have identified two key reasons for the model's weak ockr abilities: first, its difficulty in retrieving relational knowledge, and second, its inability to actively retrieve non-direct knowledge required for reasoning. firstly, the model struggles to retrieve relational knowledge, making it challenging to complete ockr tasks. this issue is closely related to the "reversal curse" @xcite , where models fail to generalize bidirectional relationships. to address this, methods such as reverse training @xcite , semantic-aware permutation training @xcite , or bidirectional causal language modeling optimization @xcite may be necessary to improve the model's ability to retrieve relational knowledge. secondly, the model does not actively retrieve non-direct knowledge required for reasoning. we believe the limitation is related to the model's computational pathways capacity when predicting the next token. according to the @xcite , the computational pathways for obtaining each token result are limited, and retrieving non-directly related knowledge require much more pathways than retrieving directly related knowledge, which could not be obtained by current style of pretraining. to mitigate this issue, the model needs to identify when additional knowledge or planning is required during the output process. this may be achieved by inserting an appropriate number of pause tokens @xcite @xcite when additional knowledge retrieval is necessary or by fine-tuning the model with step-by-step reasoning data for a specific task.