Multi-Hop Reasoning for Question Answering with Hyperbolic Representations (2025)

Abstract: Hyperbolic representations are effective in modeling knowledge graph data which is prevalently used to facilitate multi-hop reasoning. However, a rigorous and detailed comparison of the two spaces for this task is lacking. In this paper, through a simple integration of hyperbolic representations with an encoder-decoder model, we perform a controlled and comprehensive set of experiments to compare the capacity of hyperbolic space versus Euclidean space in multi-hop reasoning. Our results show that the former consistently outperforms the latter across a diverse set of datasets. In addition, through an ablation study, we show that a learnable curvature initialized with the delta hyperbolicity of the utilized data yields superior results to random initializations. Furthermore, our findings suggest that hyperbolic representations can be significantly more advantageous when the datasets exhibit a more hierarchical structure.

## introduction multi-hop reasoning is a complex task that requires models to integrate information across multiple pieces of evidence to arrive at accurate conclusions. for instance, to answer which country is the composer of the song cloudburst from? using a simple knowledge graph like figure 1 , the model has to first find the answer to the first relation (composer of cloudburst), and then look for the answer to the second relation (whitacare's country of citizenship). this inherently involves traversing hierarchical relationships, making it particularly challenging for traditional language models that rely on euclidean representations. while euclidean representations are commonly used and can capture hierarchical structures to some extent @xcite @xcite , recent research has shown that hyperbolic representations are more effective in handling such data cloudburst eric whitacre francis searle american 14 march 1909 31 july 2002 c o m p o s e r d i r e c t o r country of citizenship date of birth date of death question: which country is the composer of the song cloudburst from? path: @xmath0. in multi-hop reasoning tasks, hierarchical reasoning often manifests in navigating knowledge graphs or layered question answering frameworks, where the complexity increases with each additional hop. given the hierarchical nature of multihop reasoning tasks, hyperbolic geometry presents a compelling alternative to euclidean space for embedding representations. specifically, hyperbolic space provides a larger capacity and more expressive way of encoding tree-like and graph-like structures @xcite , which are prevalent in knowledge graphs used for reasoning. although many studies successfully implement hyperbolic architectures and report performance gains @xcite @xcite , they often fail to disentangle the effects of geometric properties from the introduction of additional trainable parameters and the resulting model architecture. a controlled comparison incorporating equiva-lent euclidean architectures with comparable parametric complexity would be necessary to isolate the geometric contribution to model performance. in addition, models in the literature often require significant architectural changes, which can increase model complexity and computational costs. in contrast, our approach focuses on incorporating hyperbolic geometry into existing language model architectures with minimal changes. in this paper, we address the lack of a carefully controlled comparative study for evaluating the differences between hyperbolic and euclidean spaces in multi-hop reasoning. as such, we incorporate hyperbolic representations into an encoder-decoder via the addition of a single layer and exponential mapping operation in the poincaré ball model of hyperbolic space. we conduct a comprehensive set of experiments on multiple datasets, demonstrating that adding a hyperbolic layer to increase the learning capability of a language model consistently outperforms its euclidean counterpart. in addition, we perform an ablation study to evaluate the impact of initialization of the curvature. our results indicate that initializing the curvature using the δ-hyperbolicity of the dataset leads to superior performance compared to random initialization. furthermore, we show that the performance gain from hyperbolic representations is more pronounced for datasets with more hierarchical structures (defined based on the number of out-going relations for the nodes). our comprehensive ablation studies deepen our understanding of geometric learning advantages in the context of language models and underscore the importance of aligning the geometric properties of the model with the inherent structure of the data. ## related work knowledge-based multi-hop reasoning. multihop reasoning requires traversing relational paths to synthesize new knowledge, making it essential for question answering. in the early approaches, pathbased methods were utilized, in which reasoning was conducted using predefined rules or relational paths within the kb @xcite . although these approaches were interpretable, they were frequently constrained by the availability and completeness of the kb. neural-based reasoning models, including embedding-based methods @xcite @xcite , introduced vectorized representations of entities and relations, enabling reasoning through learned relational patterns. more recent work has integrated graph-based neural architectures, such as graph convolutional networks @xcite and graph attention networks @xcite , to propagate information across multihop relational structures in kbs. reinforcement learning has been effectively applied to multi-hop reasoning over knowledge bases, enabling models to navigate complex relational paths and infer missing information. in this context, an rl agent is trained to traverse a kb by selecting a sequence of relations and entities, forming a reasoning path that leads to the desired answer @xcite @xcite . hyperbolic multi-hop reasoning. recent studies have proposed frameworks that leverage hyperbolic geometry to enhance multi-hop reasoning capabilities @xcite @xcite . hyperbolic knowledge graph embeddings have demonstrated significant potential for multi-hop reasoning to model the hierarchical relationships in knowledge graphs @xcite @xcite . more recently, hyperbolic graph neural networks (hgnns) have emerged as a promising direction for improving multi-hop reasoning @xcite @xcite . these models extend traditional gnns by incorporating hyperbolic message passing, allowing for better hierarchical aggregation of multi-hop dependencies. while these works highlight the potential of fully hyperbolic architectures, they often make significant architectural changes and fail to disentangle the advantages of hyperbolic geometry from those modifications in the model. by simply adding a hyperbolic layer to an existing language model, we efficiently integrate it without sacrificing the scalability or flexibility of these models. furthermore, we utilize an identical model with the same number of trainable parameters to compare the two geometries, resulting in a deeper understanding of the characteristics of the two spaces. ## method our approach builds on the path method @xcite , a two-step framework that fine-tunes the t5 model using soft prompts, added as trainable parameters to the input embeddings. ## experimental setup for all experiments, we used the t5-large model (770m parameters) @xcite . this model was fine-tuned using checkpoints adapted through the prefix lm objective @xcite over 100,000 steps. we adopt the hyperparameters presented in @xcite , with a modification to the batch size, reducing it to 64 to accommodate hardware limitations. this adjustment applies to both the knowledge integration and prompt tuning processes. the optimizer is adafactor @xcite and for the additional hyperbolic layer, we use the same learning rate of 0.001 used to fine-tune the t5 model. the curvature c is initialized using formula 2 in section 3.3, which is based on the δ-hyperbolicity of the dataset. since computing δ-hyperbolicity can be computationally expensive we calculate it in batches. we sample 1500 points from the training dataset and compute δ rel . we repeat this process 5 times. for evaluation, we use the codebase of @xcite , which is open-sourced 1 . similarly to @xcite , we evaluate the model performance with the exact match (em) score. ## results in this section, we compare the two spaces in a variety of settings, provide an ablation study on the curvature, show that computationally both cases are similar, present the results of distance analysis in the two spaces, and give some insights into dataset difficulty. ## conclusion and future work we carried out a rigorous and careful investigation of using hyperbolic versus euclidean representations in multi-hop reasoning and showed some of the advantages of the former compared to the latter. our experiments also confirm that initializing the curvature using the relative delta hyperbolicity of the dataset provides a robust and effective starting point for learning, ensuring that the model captures the hierarchical relationships within the data with greater accuracy. we also provided evidence for the hyperbolic geometry showing more effectiveness when the dataset has more hierarchical characteristics. these findings underscore the importance of understanding the structural properties of the data when selecting appropriate model architectures. in addition, our findings open several promising avenues for future research. given that the number of outgoing relations for each node in different knowledge graphs is not equal in many cases, future work could investigate a general manifold structure as well as other hyperbolic spaces for multi-hop reasoning. while our current framework focuses on encoder-decoder models in a closed-book qa setting, future work should investigate the generalizability of hyperbolic representations across broader architectures and tasks. first, extending our approach to decoder-only language models would help assess the geometric advantages in other generative models. second, applying hyperbolic reasoning layers to open-book qa tasks, where models retrieve and integrate external evidence, would clarify the interaction between geometric embedding space and retrieval-based reasoning. finally, examining our method on a wider variety of datasets and qa formats-including multi-lingual, noisy, or longer-hop reasoning datasets-will be critical to understanding the full scope of its effectiveness and limitations.