P atho HR : Hierarchical Reasoning for Vision-Language Models in Pathology (2025)

Abstract: Accurate analysis of pathological images is essential for automated tumor diagnosis but remains challenging due to high structural similarity and subtle morphological variations in tissue images. Current vision-language (VL) models often struggle to capture the complex reasoning required for interpreting structured pathological reports. To address these limitations, we propose PathoHR-Bench, a novel benchmark designed to evaluate VL models’ abilities in hierarchical semantic understanding and compositional reasoning within the pathology domain. Results of this benchmark reveal that existing VL models fail to effectively model intricate cross-modal relationships, hence limiting their applicability in clinical setting. To overcome this, we further introduce a pathology-specific VL training scheme that generates enhanced and perturbed samples for multimodal contrastive learning. Experimental evaluations demonstrate that our approach achieves state-of-the-art performance on PathoHR-Bench and six additional pathology datasets, highlighting its effectiveness in fine-grained pathology representation.

## introduction recently, vision-language (vl) models have gained substantial advances @xcite , fostering an integration of computer vision and natural language processing across a wide range of application domains @xcite . in medical image analysis, contrastive learning-based approaches have been extensively employed to align large-scale medical images with corresponding diagnostic reports, thereby enabling zero-shot classification and grading without additional fine-tuning @xcite . this paradigm markedly reduces the reliance on high-quality annotated data and enhances the scalability of medical imaging * equal contribution. models. nevertheless, in contrast to other imaging modalities such as x-ray and mri, pathological image analysis presents distinct and more formidable challenges due to its higher structural complexity and more subtle visual cues in the images. pathological biopsy microscopy remain the gold standard for tumor diagnosis, as it capture intricate cellular structures and morphological details @xcite . however, the subtle intra-class variations in images pose significant challenges for disease classification. existing approaches predominantly rely on task-specific models tailored to individual applications, such as gleason grading of prostate cancer and breast cancer subtyping @xcite . in contrast, pathological texts are inherently fine-grained, characterized by specialized medical terminology, precise pathological reasoning, and structured diagnostic logic. these narratives not only describe subtle visual cues but also offer critical insights for diagnostic grading and prognosis prediction. although recent studies have explored vl pretraining and zero-shot transfer in pathology @xcite , most existing models still treat pathological texts as a "bag-of-words," ignoring their hierarchical structure, syntactic dependencies, and diagnostic reasoning logic. previous research @xcite reveals that standard vl benchmarks often inadequately evaluate a models' capacity for compositional and structural semantic understanding. this shortcoming is particularly critical in pathology, where diagnostic texts require more than surface-level pattern matching. accurate and interpretable decision-making in pathology requires effective modeling of hierarchical clinical language in conjunction with visual and morphological cues. current models and evaluation methods face several limitations: i). limited hierarchical structure awareness: pathological tests typically follow a structured diagnostic pattern @xcite , (e.g., lesion region + cellular morphology + symptoms identification + diagnostic conclusion.). however, most vl models treat texts as unordered token sets, lacking the capacity to model multi-level semantic dependencies or structured reasoning patterns. although these models may perform adequately on retrieval or classification tasks, their disregard for compositional structure undermines their interpretability and alignment with clinical diagnostic logic. ii). limited compositional reasoning capability: pathology texts contain complex reasoning that integrates anatomical regions, cellular structures, and molecular markers, often within intricate spatial and functional contexts. bag-of-words-based models would fail to capture such contextual and inferential relationships, thereby hindering accurate and nuanced diagnostic decision-making. based on the above analysis, the main contributions of this work are summarized as follows: as illustrated in figure 1 , the proposed training scheme yields consistent performance gains across all perturbation types and hierarchical components on the pathohr-bench, highlighting its effectiveness in capturing multi-level semantic dependencies and inferring complex pathological relationships. moreover, in standard zero-shot diagnostic classification tasks, we observe that improved structural understanding and compositional reasoning directly enhance diagnostic accuracy, underscoring the critical importance of these capabilities for realworld clinical deployment. the full benchmark suite and source code will be publicly released upon acceptance. ## motivations the lack of hierarchical reasoning capabilities significantly limits the reliability and clinical interpretability of existing vl models in pathological analysis. a further limitation lies in current evaluation practices, which predominantly rely on standard retrieval or classification metrics and offer limited insight into models' structural understanding or reasoning ability. while recent studies in the natural image domain have proposed benchmarks for evaluating compositional reasoning in vl models @xcite , these approaches are not directly applicable to pathology due to fundamental domain differences in textual structure and semantics. descriptions in natural image datasets are typically open-domain and emphasize visually salient features. in contrast, pathological texts are "closed-domain," relying on highly specialized and standardized vocabularies (e.g., mesh, icd-o) and structured diagnostic patterns. unlike open-domain free text, these reports follow controlled terminology and precise reasoning logic, resulting in much tighter semantic relationships and long-range inferential chains. for instance, in describing a gastric cancer pathological image: "disordered glandular arrangement with gland fusion and lumen disappearance, consistent with poorly differentiated adenocarcinoma." here, the reasoning between "gland fusion" and "poorly differentiated" determines the malignancy grade of the lesion @xcite , making this logical relationship crucial for both model learning and final diagnostic decision-making. in addition, pathological texts often contain a significant amount of "implicit reasoning". this type of inference typically requires integration with external medical knowledge bases @xcite rather than solely relying on visual features for text generation. these characteristics underscore the need for a domain-specific benchmark tailored to pathology. in this context, pathohr-bench is designed to fill this gap by providing a structured and diagnostic-relevant framework for evaluating the compositional and inferential reasoning capabilities of vl models in the pathological domain. ## pathology hierarchical reasoning benchmark (pathohr-bench) inspired by @xcite and aro @xcite that utilize generated adversarial samples and image-text matching as primary evaluation objectives, we propose a novel pathology vl benchmark, termed as pathohr-bench, which is designed to systematically assess pathology-related vl models on hierarchical structure awareness and compositional reasoning capabilities. as shown in figure 2 , it adopts a cross-dimensional structure, enabling a comprehensive assessment of vl models across two independent yet interrelated dimensions: text perturbation and semantic role. this design allows for an in-depth analysis of model robustness, fine-grained comprehension, and compositional reasoning when processing pathological vl data. by identifying model limitations more effectively, pathohr-bench can serve as a valuable resource for advancing vl model development and optimization for pathology applications. details of the pathohr-bench are illustrated in figure 3 . at the text perturbation level, we have designed three core tasks to simulate the challenges encountered in real-world pathology texts, including information loss, semantic drift, and order variation: i). assessing sensitivity to information loss: we employ saliency-driven phrase deletion by leveraging the pathology language image pretraining (plip) @xcite model to compute the similarity between each phrase and the corresponding image. the two most salient pathological terms are removed to create adversarial text variants, simulating the model's reliance on critical textual cues and assessing its behavior under different deletion orders. it reflects how the model allocates information between visual and textual inputs while evaluating its robustness to local information losses. ## pathology hierarchical reasoning training scheme we propose a data-driven training scheme for pathological vl model, shown in figure 4 . it consists of four branches for generating positive and negative samples for both text and imagery. the following subsections discuss these four branches and the losses incorporated into vl contrastive learning. ## conclusions in this study, we investigated the limitations of current vl models in pathological image analysis and introduced a new benchmark to evaluate their structure awareness and compositional reasoning on pathology-specific text-image pairs. to the best of our knowledge, this is the first work to explicitly focus on compositional reasoning in pathological vl models. by targeting fundamental reasoning capabilities, pathohr-bench provides deep insights into current model shortcomings and can foster methodological advancements in pathologyspecific vl understanding. in addition, we proposed a data-driven training scheme to enhance the fine-grained learning capacity of existing pathological vl models. through diverse augmentations and perturbations across four branches, our approach not only improves the model's fine-grained reasoning ability but also achieves notable gains in zero-shot pathological diagnosis tasks.