B io H op R : A Benchmark for Multi-Hop, Multi-Answer Reasoning in Biomedical Domain (2025)

Abstract: Biomedical reasoning often requires traversing interconnected relationships across entities such as drugs, diseases, and proteins. Despite the increasing prominence of large language models (LLMs), existing benchmarks lack the ability to evaluate multi-hop reasoning in the biomedical domain, particularly for queries involving one-to-many and many-to-many relationships. This gap leaves the critical challenges of biomedical multi-hop reasoning underexplored. To address this, we introduce BioHopR, a novel benchmark designed to evaluate multi-hop, multi-answer reasoning in structured biomedical knowledge graphs. Built from the comprehensive PrimeKG, BioHopR includes 1-hop and 2-hop reasoning tasks that reflect real-world biomedical complexities.Evaluations of state-of-the-art models reveal that O3-mini, a proprietary reasoning-focused model, achieves 37.93% precision on 1-hop tasks and 14.57% on 2-hop tasks, outperforming proprietary models such as GPT4O and open-source biomedical models including HuatuoGPT-o1-70B and Llama-3.3-70B. However, all models exhibit significant declines in multi-hop performance, underscoring the challenges of resolving implicit reasoning steps in the biomedical domain. By addressing the lack of benchmarks for multi-hop reasoning in biomedical domain, BioHopR sets a new standard for evaluating reasoning capabilities and highlights critical gaps between proprietary and open-source models while paving the way for future advancements in biomedical LLMs. BioHopR is available at https://huggingface.co/datasets/knowlab-research/BioHopR.

## introduction recent advances in large language models (llms) and question answering (qa) systems have shifted the focus from simple factoid retrieval tasks to more sophisticated reasoning capabilities @xcite @xcite . among these, multi-hop reasoning has emerged as a critical area of research, where answering a question requires traversing multiple interconnected reasoning steps @xcite @xcite . for example, to answer "who is the wife of the president of the united states?", a llm must first identify the president (step 1) and then determine their spouse (step 2). this type of reasoning, referred to as multi-hop reasoning, is especially vital in domains where information is highly interconnected, such as the biomedical field. in the biomedical domain, knowledge is often structured in ontologies and knowledge graphs (kgs), where entities like drugs, diseases, proteins, and phenotypes are represented as nodes, and their relationships as edges @xcite @xcite . biomedical queries frequently demand multi-step reasoning over these graphs @xcite @xcite . for instance, identifying diseases associated with a drug might require a single-hop relation, while determining proteins targeted by that drug through its associated disease involves two reasoning steps. furthermore, biomedical reasoning often involves one-to-many or many-to-many relationships. for example, a single question might have several correct answerslike a drug that works on multiple proteins @xcite . this complexity highlights the need for specialized benchmarks that rigorously evaluate models' ability to reason across multiple steps while generating comprehensive, multi-answer responses. existing benchmarks for multi-hop reasoning, such as hetionet @xcite and other biomedical qa datasets @xcite , have laid the groundwork for evaluating multihop capabilities in the biomedical domain. however, these benchmarks primarily focus on single-hop tasks or utilize pre-defined templates that fail to fully capture the intricacies of multi-step reasoning. similarly, general-domain benchmarks like twohopfact @xcite test models' latent multi-hop reasoning ability but lack the domain-specific challenges of biomedical reasoning, such as reasoning over structured relationships and handling multi-answer outputs. as a result, the unique challenges of biomedical multi-hop reasoning remain underexplored. to address these limitations, we introduce bio-hopr, a new benchmark specifically designed to test the multi-hop reasoning capabilities of llms in the biomedical domain. unlike general-domain benchmarks that rely on reasoning across disconnected documents, biohopr focuses on reasoning within a single, structured biomedical knowledge graph. our benchmark systematically constructs 1-hop (e.g., drug-disease) and 2-hop (e.g., drug-disease-protein) question-answer pairs from the primekg knowledge graph @xcite . questions are designed to evaluate models' abilities to reason step-by-step, explicitly requiring the inference of intermediate entities, and generate multi-answer responses reflective of real-world biomedical complexity. ## experiments we evaluate a range of llms on the biohopr benchmark to assess their ability to reason over oneto-many-to-many relationships. the evaluation focuses on both single-answer and multi-answer reasoning for 1-hop and 2-hop questions, highlighting the challenges posed by multi-step reasoning and comprehensive answer generation. ## conclusion we introduced biohopr, a benchmark for evaluating multi-hop, multi-answer reasoning in the biomedical domain. built on the primekg knowl-edge graph, biohopr captures the complexity of real-world biomedical queries through one-tomany and many-to-many relationships, rigorously assessing reasoning over 1-hop and 2-hop tasks. evaluation results highlight that o3-mini, a proprietary model with a reasoning step, outperforms open-source models including biomedical models like huatuogpt-o1. across all models, the performance drop from 1-hop to 2-hop tasks underscores the difficulty of aligning intermediate reasoning steps, especially in bridging entities. by addressing the lack of benchmarks for multihop reasoning in biomedical domain, biohopr sets a new standard for evaluating reasoning capabilities and provides a critical step toward more robust and interpretable llms for biomedical research and real-world applications. future directions include expanding the dataset to other knowledge sources and domains, such as chemistry.