M ed EB ench: Diagnosing Reliability in Text-Guided Medical Image Editing (2025)

Abstract: Text-guided image editing has seen significant progress in natural image domains, but its application in medical imaging remains limited and lacks standardized evaluation frameworks. Such editing could revolutionize clinical practices by enabling personalized surgical planning, enhancing medical education, and improving patient communication. To bridge this gap, we introduce MedEBench, a robust benchmark designed to diagnose reliability in text-guided medical image editing. MedEBench consists of 1,182 clinically curated image-prompt pairs covering 70 distinct editing tasks and 13 anatomical regions. It contributes in three key areas: (1) a clinically grounded evaluation framework that measures Editing Accuracy, Context Preservation, and Visual Quality, complemented by detailed descriptions of intended edits and corresponding Region-of-Interest (ROI) masks; (2) a comprehensive comparison of seven state-of-the-art models, revealing consistent patterns of failure; and (3) a diagnostic error analysis technique that leverages attention alignment, using Intersection-over-Union (IoU) between model attention maps and ROI masks to identify mislocalization issues, where models erroneously focus on incorrect anatomical regions. MedEBench sets the stage for developing more reliable and clinically effective text-guided medical image editing tools.

## introduction recent advances in diffusion models and visionlanguage pretraining have significantly advanced text-guided image editing @xcite @xcite @xcite @xcite . these methods enable diverse applications such as object removal, inpainting, and style transfer, often producing compelling visual results @xcite @xcite . however, most prior work has focused on aesthetic or creative tasks, with limited exploration in domains that require high semantic precision, particularly in medicine. while comprehensive benchmarks exist for evaluating editing models on natural images @xcite @xcite , their adaptation to medical images remains challenging. here, even minor alterations can carry significant clinical meaning, demanding high editing precision, semantic fidelity, and anatomical correctness. text-guided medical image editing holds substantial clinical potential, as it could highlight lesions in ct scans @xcite , simulate surgical outcomes @xcite , or generate personalized teaching materials @xcite . such applications promise direct benefits to diagnosis, treatment optimization @xcite , and training @xcite . despite the versatility of current models, they often fail at clinically meaningful transformations that are intuitive to general physicians or even non-experts. for example, instructpix2pix @xcite can successfully handle prompts such as "add a missing key on a piano keyboard" but fails on similar medical prompts such as "add a missing tooth in this dental image" (see fig. 1 ). notably, this failure cannot be resolved by adjusting the text or image guidance scales @xcite (see fig. 7 ), where the text guidance scale controls the classifier-free guidance weight for the text prompt, and the image guidance scale controls the conditioning strength on the input image during denoising. this underscores the difficulty of transferring such models to domains that require fine-grained anatomical understanding and specialized domain knowledge @xcite . medical image editing faces two fundamental challenges. first, real image pairs that reflect clinically grounded transformations (e.g., before and after treatment) are scarce, and synthetic data often lacks realism. second, evaluation remains problematic. metrics such as fréchet inception distance (fid) @xcite and clip score @xcite are not aligned with clinical correctness or anatomical plausibility, failing to capture if the edits are grounded in medical knowledge. to address these gaps, we propose medebench, a benchmark for text-guided medical image editing. medebench contains 1,182 real clinical image pairs covering pre-and postoperative states across 13 anatomical regions (e.g., teeth, eyes). each edit case is defined by a natural lan-guage prompt, region-of-interest (roi) masks, and detailed change descriptions to enable localized, fine-grained evaluation. for clinically meaningful assessment, we introduce tailored evaluation metrics. contextual preservation (cp) is measured by masked ssim @xcite to ensure unaffected regions remain intact. editing accuracy (ea) and visual quality (vq) are assessed via multimodal large language models (mllms) using clinically detailed change descriptions for accurate, interpretable evaluation. we benchmark seven state-of-the-art models across diverse learning paradigms and analyze failures through attention-grounding, revealing gaps in medical concept understanding and spatial localization. our contributions are threefold: ## medebench fig. 2 illustrates the main components of medebench. section 2.1 describes our dataset, which consists of 1,182 samples across 13 categories. each sample includes an input image, a reference image, an editing prompt, an roi mask, and a change description. section 2.2 introduces automated evaluation metrics, while section 3.5 presents a human study validating the alignment between automated and human assessments. tab. 1 compares medebench with existing benchmarks. as shown in fig. 4 , low prompt-image clip similarity and varied roi sizes highlight the challenges of text-guided medical image editing. ## failure analysis via attention we select the instructpix2pix model as a case study for failure analysis, as it is a widely used textguided image editing model known for strong context preservation but relatively low editing accu-racy, as discussed in section 3.3. notably, instruct-pix2pix demonstrates a tendency to preserve global image structures while failing to execute finegrained edits, making it an ideal candidate for analyzing attention-related failures. prior work @xcite ) has shown that cross-attention mechanisms in diffusion models play a critical role in localizing prompt tokens to corresponding image regions, and in encoding semantic and categorical information. to further investigate this phenomenon, we examine whether the editing process attends to the correct anatomical regions by analyzing the model's cross-attention maps. for each editing prompt, we extract the cross-attention maps corresponding to the last token of the key visual concept t c across all diffusion steps. these are averaged to obtain a single attention vector: ## key insights and takeaways our comprehensive evaluation across thirteen anatomical regions and multiple editing models yields four key findings: gemini 2 flash consistently leads in ea, vq, and fid, with the largest gaps in internal organ tasks (e.g., gastrointestinal tract, spine, teeth) where fine-grained structures and spatial reasoning are critical. open-source models such as icedit remain competitive but lag in high-precision scenarios. fine-tuning models such as instructpix2pix improves medical editing performance despite data scarcity. however, large multimodal models struggle to transfer medical concepts through prompting-level in-context learning, highlighting generalization limits in clinical tasks. iou between attention heatmaps and groundtruth regions reveals failure patterns: high iou with low ea indicates correct localization but failed execution (common in concept removal), while low iou reflects poor spatial focus in addition or reconstruction tasks. @xcite @xcite , and synthetic data generation @xcite @xcite @xcite . benchmarking text-guided editing is key to evaluating model performance, but standard metrics such as fid @xcite , clip score @xcite , psnr @xcite , ssim @xcite , and lpips @xcite fail to capture finegrained edit quality and semantic intent. recent benchmarks @xcite @xcite improve on this with human-aligned dimensions and diverse edit types. multimodal large language models (mllms), such as gpt-4o (openai, 2024) and gemini 2.5 pro (google deepmind, 2025), have emerged as strong evaluators for visual and semantic alignment, enabling interpretable and humanaligned assessment frameworks @xcite @xcite @xcite . however, existing work largely targets general-domain imagery, overlooking domain-specific needs such as medical editing. to bridge this gap, we propose medebench, the first benchmark for text-guided medical image editing with a clinically grounded evaluation framework. ## conclusion we introduce medebench, the first benchmark specifically designed for text-guided medical image editing. it includes 1,182 real-world examples spanning 13 anatomical regions and 70 clinically meaningful tasks. each case is annotated with region-of-interest masks and expert-authored change descriptions, enabling multi-faceted evaluation across editing accuracy, context preservation, and visual quality. we benchmark seven state-of-the-art models and find persistent challenges, particularly in editing internal organs and anatomically complex regions. while gemini 2 flash achieves the highest overall performance, icedit stands out as the strongest open-source alternative. fine-tuning diffusionbased, text-guided editing models like instruct-pix2pix leads to substantial performance gains, even in low-data settings, highlighting the critical role of domain adaptation. in contrast, large multimodal models often fail to generalize medical concepts through prompt-based in-context learning, pointing to core limitations in their clinical transferability. further analysis of cross-attention patterns reveals consistent gaps in spatial and anatomical grounding, indicating a mismatch between model focus and human-intended edits. looking ahead, our findings emphasize the need for anatomy-aware architectures and medically aligned supervision to improve reliability and safety in clinical image editing tasks. we hope medebench will serve as a catalyst for developing robust, transparent, and domain-adapted generative tools for medical applications.