Do Large Language Models Perform Latent Multi-Hop Reasoning without Exploiting Shortcuts? (2025)

Abstract: We evaluate how well Large Language Models (LLMs) latently recall and compose facts to answer multi-hop queries like “In the year Scarlett Johansson was born, the Summer Olympics were hosted in the country of”. One major challenge in such evaluation is that LLMs may have developed shortcuts by encountering the head entity “Scarlett Johansson” and the answer entity “United States” in the same training sequences or merely guess the answer based on frequency-based priors. To prevent shortcuts, we exclude test queries where the head and answer entities might have co-appeared during training. Through careful selection of relations and facts and systematic removal of cases where models might guess answers or exploit partial matches, we construct an evaluation dataset SOCRATES (ShOrtCut-fRee lATent rEaSoning). We observe that LLMs demonstrate promising latent multi-hop reasoning abilities without exploiting shortcuts, but only for certain types of queries. For queries requiring latent recall of countries as the intermediate answer, the best models achieve 80% latent composability, but this drops to just 5% for the recall of years. Comparisons with Chain-of-Thought highlight a significant gap between the ability of models to reason latently versus explicitly. Analysis reveals that latent representations of the intermediate answer are constructed more often in queries with higher latent composability, and shows the emergence of latent multi-hop reasoning during pretraining.

## introduction latent multi-hop reasoning in large language models (llms), or latently recalling and composing single-hop facts to answer multi-hop queries llm 1 input query: output: the united states latent composition potential shortcut exploitation llm 2 scarlett ingrid johansson was born on november 22, 1984. the 1984 summer olympics were held from july 28 to august 12, 1984, in los angeles, california, united states. major events in the united states in 1984: • the united states played host to the los angeles summer games. • scarlett johansson was born. scarlett johansson is from the country of the united states. pretraining data of llm 1 pretraining data of llm 2 in the year scarlett johansson was born, the summer olympics were hosted in the country of we propose desiderata for shortcut-free evaluation of latent multi-hop reasoning ability. we evaluate latent multi-hop reasoning abilities by assessing models' performance in answering multi-hop queries. while prior works have suggested that pretrained llms develop this ability @xcite @xcite , they have not adequately addressed the possibility of models exploiting shortcuts @xcite @xcite @xcite ; our analysis shows that 85% of the test queries from a previous dataset are prone to shortcuts. shortcuts from frequent co-occurrences of subject-object or relation-object in the training data can allow models to answer the multi-hop queries correctly without going through a true latent reasoning process. for instance, in a query about "scarlett johansson" (i.e., the head entity) where the answer entity is "united states", llms may simply learn a subject-object entity shortcut if these entities frequently co-occur in training @xcite @xcite . similarly, llms can develop a relationobject shortcut from the frequency of "united states" appearing as a country and guess the answer based on the frequency-based prior @xcite @xcite . to overcome this challenge, we outline desiderata for shortcut-free evaluation of latent multi-hop reasoning in llms, which we address through dataset construction and evaluation procedure. for the dataset, we only use test queries where the head and answer entities are estimated to never co-occur in pretraining sequences, thus minimizing subjectobject shortcuts. we further minimize shortcuts by carefully selecting relation types and removing queries where the answers are easy to guess from a substring of the head entity. for evaluation, we measure latent composability as the rate of correct multi-hop answers when single-hop facts are known, without explicitly generating the intermediate answer (i.e., the bridge entity). additionally, we reduce relation-object shortcuts by excluding queries where the model may guess the answer without considering the head entity. the main challenge in satisfying our desiderata is that most llms' pretraining data is inaccessible, making it impossible to directly check entity co-occurrences. to tackle this, we use a proxy corpus of roughly 4.8b unique documents by utilizing six publicly available training corpora, selecting only test queries where the head and answer entities never co-occur. this approximation's effectiveness is validated by showing that strong latent composability for specific query types persists even when extending our entity co-occurrence check to the whole web via google search. our resulting dataset, socrates (shortcut-free latent reasoning), consists of 7,232 pairs of single-hop and multi-hop queries of 17 types of relation compositions with 4 types of bridge entities. comparative experiments with a dataset constructed from the same data distribution but without careful fact selection, co-occurrence-based filtering, and rigorous evaluation show that latent composability can be overestimated without satisfying the desiderata. our results for 41 llms from 9 families reveal that there are successful cases of latent multi-hop reasoning, but the performance varies substantially according to the type of bridge entity that connects the facts. notably, state-of-the-art models demonstrate strong latent composability of over 80% when the bridge entity is a country. however, the number is only around 6% for year-based queries, highlighting the importance of considering the distribution of relation composition types when evaluating llms' latent reasoning abilities. models that know more single-hop facts tend to reason better latently, and the ability marginally improves with model scale. on the contrary, cot composability effectively increases with the number of known facts and model size, with much higher and consistent performance across bridge entity types. additional analysis shows that the latent representation of the bridge entity is clearly constructed more often for queries with higher latent composability, and reveals the emergence of latent multi-hop reasoning during pretraining. in summary, our contributions are as follows: • we present socrates and evaluation procedure for latent multi-hop reasoning with minimal risk of shortcut exploitation, which is corroborated to be important through a comparative analysis. ## related work studies have shown that llms' predictions often rely on shortcuts, shallow heuristics, and cooccurrence biases @xcite @xcite @xcite @xcite @xcite @xcite . for instance, @xcite have found that single-hop knowledge predictions can be influenced by subject-object co-occurrences or relationobject co-occurrences. similarly, @xcite and @xcite show that frequent co-occurrences can lead llms to favor highfrequency words over correct responses. lastly, @xcite demonstrate that head-answer entity co-occurrence frequencies in multi-hop facts are correlated with factual shortcuts, which can cause failures in multi-hop knowledge editing. prior works on latent factual multi-hop reasoning have not fully addressed potential shortcuts @xcite @xcite . while ofir @xcite attempt to create multi-hop questions unlikely to appear in training, their approach relies on assumptions rather than co-occurrence statistics (leaving shortcuts from subject-object co-occurrences exploitable) and does not address shortcuts from relation-object co-occurrences. indeed, we show that their dataset is prone to shortcuts, emphasizing the need for an explicit grounding to the co-occurrence statistics (see section 6.3). similarly, @xcite address relation-object shortcuts but overlook subject-object shortcuts. construction of a shortcut-free evaluation dataset of latent factual multi-hop reasoning ability of any pretrained llm presents a unique challenge, as the pretraining data of most of the widely used llms is not accessible, making it difficult to verify if certain single-hop facts or their composition appeared in the same training sequence. our need to consider the knowledge llms learn during pretraining makes our work distinct from prior works that aim for shortcut-free evaluation on the tasks where the training dataset is fully accessible @xcite @xcite @xcite . other studies have attempted to circumvent these issues by fine-tuning llms on synthetic or counterfactual tasks to control for single-hop knowledge @xcite @xcite @xcite @xcite @xcite . however, these studies do not address our target question of how much latent multi-hop reasoning ability naturally emerges in training. moreover, finetuning may introduce side effects, such as hallucinations, reduced knowledge learning, and utilization efficiency @xcite @xcite @xcite . works on latent compositional reasoning with algorithmic or mathematical tasks @xcite @xcite do not address our target question of latent multi-hop reasoning ability with factual knowledge, and may still suffer from data contamination that inflates performance @xcite . @xcite examine performance gaps between different numbers of reasoning hops but focus on more general reasoning capabilities rather than latent reasoning with factual knowledge. @xcite study the theoretical limitations of compositional abilities. consistent with our findings, they prove that for high arity relations (like relations with year-type bridge entity in our work), multi-hop reasoning is more difficult, albeit for the specific case of transformers @xcite with a single layer. ## shortcut-free evaluation of latent terms and notations we represent single-hop facts as r 1 (e 1 ) = e 2 and r 2 (e 2 ) = e 3 , and multi-hop facts as their composition r 2 •r 1 (e 1 ) = e 3 . in the aforementioned example, the entities "scarlett johansson", "1984", and "united states" are respectively represented by e 1 , e 2 , e 3 , and connected via relations r 1 (person-birthyear), r 2 (year-eventcountry), and r 2 •r 1 (person-birthyear-eventcountry). the set of aliases (names that the entity is also known as), of e 1 , e 2 , and e 3 are represented as e 1 , e 2 , and e 3 , respectively. the answer set of the single-hop queries q(r 1 (e 1 )), q(r 2 (e 2 )), and multihop query q(r 2 •r 1 (e 1 )) is e 2 , e 3 , and e 3 , respectively. for instance, the answer set corresponding to the query "the year scarlett johansson was born in is" is {"1948"}. we call each tuple ## evaluation dataset in this section, we describe our dataset construction process that minimizes the chance of subjectobject shortcut (satisfying desideratum 2), resulting socrates (shortcut-free latent rea-soning), a dataset for evaluation of shortcut-free evaluation of latent multi-hop reasoning. ## evaluation procedure we introduce an evaluation procedure that satisfies part of desideratum 2 by minimizing the chance of the model exploiting the relation-object shortcut and desideratum 1 that the evaluation should exclude cases where the model performs explicit reasoning ( §5.1). then, we define our evaluation metric, latent composability ( §5.2). ## experiments we use socrates to evaluate the latent multihop reasoning ability of llms. our results show that models can perform latent reasoning without exploiting shortcuts, but their ability depends on the type of bridge entity connecting the two facts. ## conclusion we outline desiderata for shortcut-free evaluation of llms' ability to latently recall and compose learned single-hop facts to answer multi-hop queries. by filtering entity co-occurrences and systematic removal of potential shortcuts, we construct the socrates dataset, enabling a rigorous assessment of latent multi-hop reasoning. our analysis reveals that while models can perform latent reasoning effectively in specific scenarios, their ability varies dramatically across different types of queries. this fluctuation, along with the significant gap between latent and explicit cot reasoning, suggests substantial room for improvement in how llms internally compose their knowledge. our work provides resources and insights for precise evaluation, understanding, and improvement of latent multihop reasoning of llms.