Look & Mark: Leveraging Radiologist Eye Fixations and Bounding boxes in Multimodal Large Language Models for Chest X -ray Report Generation (2025)

Abstract: Recent advancements in multimodal Large Language Models (LLMs) have significantly enhanced the automation of medical image analysis, particularly in generating radiology reports from chest X-rays (CXR). However, these models still suffer from hallucinations and clinically significant errors, limiting their reliability in real-world applications. In this study, we propose Look & Mark (L&M), a novel grounding fixation strategy that integrates radiologist eye fixations (Look) and bounding box annotations (Mark) into the LLM prompting framework. Unlike conventional fine-tuning, L&M leverages in-context learning to achieve substantial performance gains without retraining. When evaluated across multiple domain-specific and general-purpose models, L&M demonstrates significant gains, including a 1.2% improvement in overall metrics (A.AVG) for CXR-LLaVA compared to baseline prompting and a remarkable 9.2% boost for LLaVA-Med. General-purpose models also benefit from L&M combined with in-context learning, with LLaVA-OV achieving an 87.3% clinical average performance (C.AVG)—the highest among all models, even surpassing those explicitly trained for CXR report generation. Expert evaluations further confirm that L&M reduces clinically significant errors (by 0.43 average errors per report), such as false predictions and omissions, enhancing both accuracy and reliability. These findings highlight L&M’s potential as a scalable and efficient solution for AI-assisted radiology, paving the way for improved diagnostic workflows in low-resource clinical settings.

## introduction recently, the advent of multimodal large language models (llms), in which vision encoders are integrated with powerful language generation models, has significantly advanced the automation of medical image analysis @xcite @xcite @xcite . chest x-ray (cxr) interpretation, in particular, has benefited greatly from these developments: by ingesting both image and text data, modern llms can generate radiology reports, perform visual question answering, and even conduct error-checking in clinical documents @xcite @xcite @xcite . despite these advances, hallucinations, outputs diverging from actual image content, remain a major obstacle for real-world applications of these models, reducing the trust and clinical userability. @xcite @xcite . one promising way to overcome these hallucinatory behavior is by incorporating expert insights directly into the model pipelines. several studies have shown that integrating human input with ai models can substantially boost both accuracy and reliability, sometimes surpassing the performance of either clinicians or ai models alone @xcite . two relevant sources of expert knowledge in radiology are (1) bounding boxes, which are rectangular markers that radiologists draw to highlight suspicious regions in medical images, and (2) radiologist eye fixations, which reveal the natural diagnostic process by tracking where doctors look and how long they spend examining different areas of a chest x-ray. bounding box annotations help localize the language output in well-defined spatial coordinates, reducing the risk of free-form hallucinations and improving multimodal large language model's "grounded" report generation ability @xcite . meanwhile, eye-tracking data offers insights into the contextual logic clinicians apply, indicating not only the spatial information but also the order of saliency for the spatial information. this additional information enhanced the capabilities of multimodal llms in cxr interpretations method uses gaze uses bbox llm-based training-free report generation maira2 @xcite ✗ ✓ ✓ ✗ ✓ cxr fixation @xcite ✓ ✗ ✓ ✗ ✗ fg-cxr @xcite in this work, we propose "look & mark": a grounding fixation strategy for cxr report generation that merges radiologist eye fixations with bounding box annotations in multimodal llms. crucially, we avoid large-scale model retraining by employing in-context learning or prompt engineering. through bounding box coordinates, we provide precise "marks" that ground suspicious regions, while eye fixations encode how expert radiologists spatially and temporally navigate those regions. this grounding fixation can significantly reduce hallucinations and clinically significant errors by generating more coherent, clinically relevant cxr reports. the contributions of our work are as follows: ## look & mark figure 1 provides an overview of the workflow, which includes preprocessing the input, constructing multimodal prompts, and evaluating the output. ## conclusion this study introduces look & mark (l&m), a novel approach to radiology report generation that integrates radiologist fixation cues (look) with bounding box annotations (mark) to guide multimodal large language models (llms). by combining these complementary grounding strategies, l&m significantly improves the clinical relevance of generated reports, reduces hallucinations, and enhances model alignment with real-world diagnostic workflows. importantly, l&m achieves these gains without requiring extensive fine-tuning, leveraging in-context learning to adapt both general-purpose and domain-specific models alike. our experiments demonstrate that l&m significantly boosts performance across both lexical and clinical evaluation metrics, with the largest gains observed in clinical metrics such as ratescore and radgraph-xl. for instance, cxr-llava achieved a 1.2% improvement in overall metrics (a.avg) compared to baseline prompting, while llava-med demonstrated a remarkable 9.2% boost. general-purpose models also benefited significantly, with llava-ov achieving an 87.3% clinical average (c.avg), the highest among all tested models, even surpassing domain-specific models trained explicitly for chest x-ray report generation. furthermore, expert radiologist evaluations confirmed the clinical reliability of l&m, with fewer errors (by 0.43 average errors per report) in categories such as false predictions, omissions, and incorrect severity descriptions. these results highlight that grounding multimodal llms with both bounding boxes and fixation cues provides a synergistic effect, improving performance across diverse models and tasks. by eliminating the need for retraining, l&m offers a scalable and practical solution for deploying advanced ai systems in low-resource clinical environments. this makes it particularly suited for improving diagnostic workflows in settings with limited computational resources, while still achieving state-of-the-art performance. future work will focus on extending l&m to other medical imaging modalities, such as ct and mri, and exploring automated grounding for the bounding boxes of abnormalities. these advancements will further enhance l&m's potential to become a foundational framework for reliable and scalable ai-driven diagnostics in low-resource healthcare settings.