HARE : an entity and relation centric evaluation framework for histopathology reports (2025)

Abstract: Medical domain automated text generation is an active area of research and development; however, evaluating the clinical quality of generated reports remains a challenge, especially in instances where domain-specific metrics are lacking, e.g. histopathology. We propose HARE (Histopathology Automated Report Evaluation), a novel entity and relation centric framework, composed of a benchmark dataset, a named entity recognition (NER) model, a relation extraction (RE) model, and a novel metric, which prioritizes clinically relevant content by aligning critical histopathology entities and relations between reference and generated reports. To develop the HARE benchmark, we annotated 813 de-identified clinical diagnostic histopathology reports and 652 histopathology reports from The Cancer Genome Atlas (TCGA) with domain-specific entities and relations. We fine-tuned GatorTronS, a domain-adapted language model to develop HARE-NER and HARE-RE which achieved the highest overall F1-score (0.915) among the tested models. The proposed HARE metric outperformed traditional metrics including ROUGE and Meteor, as well as radiology metrics such as RadGraph-XL, with the highest correlation and the best regression to expert evaluations (higher than the second best method, GREEN, a large language model based radiology report evaluator, by Pearson r = 0.168 , Spearman \rho = 0.161 , Kendall \tau = 0.123 , R^2 = 0.176 , RMSE = 0.018 ). We release HARE, datasets, and the models at https://github.com/knowlab/HARE to foster advancements in histopathology report generation, providing a robust framework for improving the quality of reports.

## introduction medical report generation has become an increasingly active area of research in clinical natural language processing (nlp) with the goal of automating the creation of specialized clinical documents @xcite . among various medical domains, radiology has witnessed the earliest and most notable advancements in automated report generation @xcite @xcite . this progress is partly attributed to the development of domain-specific evaluation metrics that prioritize clinical correctness @xcite @xcite . unlike general-purpose metrics such as bleu and rouge, these specialized metrics assess the accuracy of radiologically significant entities and findings, thereby offering a more clinically meaningful measure of report quality @xcite @xcite and facilitating the development of accurate generative models. in contrast, the field of histopathology, which involves the microscopic examination of tissue samples to diagnose diseases such as cancer, still relies only on general-purpose lexical metrics for evaluating automatically generated reports @xcite @xcite . histopathology reports are semi-structured, terminology-intensive documents that provide detailed microscopic evaluations of tissue samples. such reports play a crucial role in disease diagnosis and guiding treatment decisions. histopathology reports encompass multiple sections, including descriptions of anatomical sites, cellular morphology, tumor classification, staging, further analyses (e.g. immunohistochemistry (ihc) markers, special stains, or in situ hybridization (ish)), and a final diagnosis/conclusion. figure 1 shows the difference between the word embeddings of radiology reports (from mimic-cxr @xcite and iu-xray @xcite and histopathology reports (used in this study). histopathology word embedding has many areas that are not covered by radiology word embeddings, making radiology report ## methods in this section, we describe the development of hare (histopathology automated report evaluation), a domain-specific evaluation metric designed to assess the clinical quality of generated histopathology reports. our methodology involves dataset preparation and annotation, ner model and re model training, and the design of the hare metric. ## results and discussion this result underscores the efficacy of gatortrons in addressing the complexities inherent in histopathology text. its extensive pretraining on large-scale synthetic clinical corpora provides it with a comprehensive understanding of domain-specific language, abbreviations, and nuanced terminology. this ability is particularly critical in histopathology, where specialized an additional factor contributing to gatortrons's superior performance is its model size. as the largest model among the biomedical models tested, gatortrons benefits from greater representational capacity, enabling it to capture complex relationships in text more effectively. ## conclusion in this work, we proposed hare, a novel entity and relation centric evaluation metric specifically designed to assess the clinical quality of machinegenerated histopathology reports. hare addresses the critical gap in domain-specific evaluation by prioritizing clinical relevance. hare effectively aligns with expert evaluations, outperforming existing metrics such as rouge and ratescore. our findings reveal that even proprietary multimodal large language models, such as gpt-4o, struggle to produce clinically accurate histopathology reports. although we have not tested a comprehensive list of models trained for histopathology reports such as histgen and wsicaption, hare can be a robust framework for evaluating these models @xcite . in the future, we will include histopathology reports specific models for creating the human evaluation dataset as well as extend hare models to a joint ner+re model to further improve the performance and utility. hare's superior performance underscores the importance of domain-specific evaluation metrics in bridging the gap between automated report generation and clinical expectations. by making hare publicly available, along with the annotations and models, we aim to facilitate advancements in both report generation and evaluation methodologies in histopathology and related fields.