S yllo B io- NLI : Evaluating Large Language Models on Biomedical Syllogistic Reasoning (2025)

Abstract: Syllogistic reasoning is crucial for Natural Language Inference (NLI). This capability is particularly significant in specialized domains such as biomedicine, where it can support automatic evidence interpretation and scientific discovery. This paper presents SylloBio-NLI, a novel framework that leverages external ontologies to systematically instantiate diverse syllogistic arguments for biomedical NLI. We employ SylloBio-NLI to evaluate Large Language Models (LLMs) on identifying valid conclusions and extracting supporting evidence across 28 syllogistic schemes instantiated with human genome pathways. Extensive experiments reveal that biomedical syllogistic reasoning is particularly challenging for zero-shot LLMs, which achieve an average accuracy between 70% on generalized modus ponens and 23% on disjunctive syllogism. At the same time, we found that few-shot prompting can boost the performance of different LLMs, including Gemma (+14%) and LLama-3 (+43%). However, a deeper analysis shows that both techniques exhibit high sensitivity to superficial lexical variations, highlighting a dependency between reliability, models’ architecture, and pre-training regime. Overall, our results indicate that, while in-context examples have the potential to elicit syllogistic reasoning in LLMs, existing models are still far from achieving the robustness and consistency required for safe biomedical NLI applications.

## introduction syllogistic reasoning -i.e., the process of deriving valid conclusions from premises through the systematic application of abstract reasoning schemes -is a fundamental type of inference for developing and evaluating natural language inference (nli) models that can reason over textual evidence 1 code and dataset available at: https://github.com/ neuro-symbolic-ai/syllobio-nli p1: every member of diseases of hemostasis pathway is a member of disease pathway p2: gene gp1bb is a member of diseases of hemostasis pathway _____________________________________ c: therefore, it is true that gene gp1bb is a member of disease @xmath0, p2, @xmath1. within specialised domains such as biomedicine, the ability to reason over natural language can have significant practical implications for supporting complex discourse interpretation, scientific discovery, and the development of downstream biomedical and clinical applications @xcite , allowing for a set of formally defined patterns for controlled inference. moreover, determining the reliability and robustness of nli models with regard to syllogistic reasoning is a crucial type of assessment, particularly in critical domains requiring safety guarantees @xcite . this type of evaluation is further motivated by two key factors. first, syllogistic reasoning is prevalent in discourse at large and, therefore, it is expected that approaches based on large language models (llms) are exposed to a variety of reasoning schemes during pre-training @xcite . second, a model that learns to perform syllogistic every member of f pathway is a member of g pathway., gene a is a member of f pathway, reasoning should intrinsically possess the ability to generalize to different domains, regardless of the specific world knowledge acquired during pretraining @xcite . this is because the ability to perform syllogistic reasoning should be content-independent, that is, the ability to derive logically valid conclusions is only a function of the formal logical schemes and should be independent of its concrete instantiation. however, despite the importance of syllogistic reasoning for nli and the abundance of benchmarks involving commonsense knowledge @xcite , resources for assessing how systematic reasoning capabilities transfer to specialised domains are still scarce and require substantial expert-level annotation effort to guarantee quality and correctness @xcite @xcite . ## syllobio-nli we introduce a general framework for developing resources to systematically evaluate the syllogistic reasoning capabilities of nli models within biomedical domains. in particular, syllobio-nli leverages the systematicity of known syllogistic schemes @xcite along with external thesauri/ontologies to scale up the generation of deductively valid arguments and to characterise inference performance across a granular set of syllogistic schemes. to this end, we focus on seven general syllogistic schemes, including generalized modus ponens, generalized contraposition, hypothetical syllogism 1, hypothetical syllogism 3, generalized modus tollens, disjunctive syllogism and generalized dilemma. these base schemes are then adopted to generate 28 different variations of syllogistic argument templates applying negation, complex predicates, and de morgan's laws, which can then be instantiated with concrete domain knowl-edge (see fig. 6 in appendix b). the overall methodology is outline by the stages in fig. 2 . first, for each syllogistic scheme, a corresponding formal argument scheme (consisting of abstract premises and conclusion expressed in first-order logic) is created (fig. 2a ). next, each symbolic formula in the formal scheme is individually replaced by a natural language domain-specific sentence schema (fig. 2b ). for example, the formulae for the base schema of generalized modus ponens (i.e., premise 1: ∀xf x ⇒ gx, premise 2: f a, conclusion: ga) can be translated into the natural language template: "premise 1: every member of f is a member of g". premise 2: "a is a member of f". conclusion: "a is a member of g". subsequently, the entity and property placeholders in the natural language template are replaced argument-wise with domain-specific entities and predicates extracted from an external ontology (fig. 2c ). in this process, the syllogistic schemes provide the logical validity component of the formal inference while the ontology subdomain and its associated mapping to the schemes provide the soundness (content-based) for the premises and conclusions. hence, we obtain instances of syllogistic arguments in natural language that are stored in a knowledge base of premises and conclusions. finally, the natural language syllogistic arguments can be structured to create prompts for evaluating llms. this involves introducing the syllogistic argument, clearly framing the premises, and specifying the instructions for the inference, as illustrated in appendix f. ## related work syllogistic reasoning, which involves deriving valid conclusions from given premises based on formal logical structures, has long been a central focus in the study of natural language inference (nli) @xcite . early transformer models like bert @xcite and roberta (liu et al., 2019) were trained on general nli datasets, such as snli @xcite and mnli @xcite , to address reasoning tasks. however, these models were limited in their ability to generalize abstract reasoning patterns, particularly in more specialized domains. recent work has explored whether larger, decoder-based language models are capable of capturing syllogistic reasoning without task-specific fine-tuning, as reasoning itself is content-independent @xcite . recent research, such as liu et al. @xcite , has introduced benchmarks like glore to test logical reasoning in llms, utilizing techniques such as chain-of-thought (cot) prompting @xcite and in-context learning (icl) @xcite . these methods have been shown to improve reasoning performance, but issues remain, as models often rely on superficial patterns instead of deep logical comprehension. eisape et al. @xcite further demonstrated that even advanced models exhibit biases when handling syllogisms, particularly when faced with invalid syllogisms, often failing to generate "nothing follows" conclusions. while the general-domain performance of llms in syllogistic reasoning has been explored @xcite , research on syllogistic reasoning in the biomedical domain remains scarce. this paper addresses this gap by focusing specifically on biomedical applications, where logical reasoning over specialized knowledge is critical. we introduce a large-scale biomedical syllogism dataset, syllobio-nli, and provide a comprehensive evaluation of the syllogistic reasoning capabilities of state-of-the-art llms within this domain. ## conclusions in this work, we proposed a novel methodological framework, syllobio-nli, designed to evaluate the syllogistic reasoning capabilities of stateof-the-art llms within the biomedical domain. through comprehensive analysis across 28 syllogistic schemes, we assessed the performance of eight different models under varying conditions, including zero-shot and few-shot settings. our results show that both techniques exhibit high sensitivity to superficial lexical variations, highlighting a dependency between reliability, models' architecture, and pre-training regime. overall, our evaluation indicates that, while fewshot strategies have the potential to elicit syllogistic reasoning in llms, existing models are still far from achieving the robustness and consistency required for safe biomedical nli applications in specialised domains.