Migician: Revealing the Magic of Free-Form Multi-Image Grounding in Multimodal Large Language Models (2025)

Abstract: The recent advancement of Multimodal Large Language Models (MLLMs) has significantly improved their fine-grained perception of single images and general comprehension across multiple images. However, existing MLLMs still face challenges in achieving precise grounding in complex multi-image scenarios. To address this, we first explore a Chain-of-Thought (CoT) framework that integrates single-image grounding with multi-image comprehension. While partially effective, it remains unstable and struggles to capture abstract visual information due to its non-end-to-end nature. Therefore, we introduce Migician, the first multi-image grounding model capable of performing free-form and accurate grounding across multiple images. To support this, we present the MGrounding-630k dataset, which comprises data for several multi-image grounding tasks derived from existing datasets, along with newly generated free-form grounding instruction-following data. Furthermore, we propose MIG-Bench, a comprehensive benchmark specifically designed for evaluating multi-image grounding capabilities. Experimental results demonstrate that our model achieves significantly superior multi-image grounding capabilities, outperforming the best existing MLLMs by 24.94% and even surpassing much larger 70B models. Our code, model, dataset, and benchmark are fully open-sourced at https://migician-vg.github.io/.

## introduction multimodal large language models (mllms) have exhibited significant advancements recently, demonstrating exceptional cross-modal understanding capabilities and achieving outstanding performance in various vision-language tasks @xcite @xcite ; ive et @xcite @xcite . as these models continue to evolve, their capabilities have expanded beyond image-level understanding to include fine-grained visual grounding @xcite @xcite . this enables mllms to process region-specific inputs and outputs, unlocking a broader spectrum of real-world multimodal application scenarios @xcite . despite the promising visual grounding capabilities demonstrated by existing mllms, these abilities are largely confined to single-image scenarios @xcite . the potential of mllms in free-form multi-image grounding (mig) remains underexplored. freeform mig challenges the model to perform grounding across multiple images effectively, where the input queries and image contexts can be organized in arbitrary forms, enabling flexible and dynamic interactions. for instance, as shown in figure 1 , the model must understand the white car in the query image and relate it to the textual prompt "black in color" to identify the corresponding target in the target image. this capability unlocks a wide range of applications, such as fine-grained environmental perception in autonomous driving @xcite , anomaly detection in surveillance systems @xcite , and target localization for embodied robotics @xcite . to address the free-form mig, the model needs to possess the capability for visual grounding while achieving cross-image understanding. as a result, a question naturally arises: can we integrate the single-image grounding and multi-image understanding capabilities of existing mllms to tackle the mig task? in this work, we propose a chain-of-thought (cot) framework that first leverages multi-image understanding to generate a textual referring query, and then utilizes it for localization through single-image grounding. this approach is proven highly effective for mig tasks, particularly in simple scenarios where textual descriptions are sufficiently distinctive, demonstrating the potential of mllms in handling such tasks. however, the proposed cot framework struggles with describing abstract visual semantics in multi-image scenarios, and the two-step process results in a doubling of the inference time. to address this, we further propose migician, a competitive mllm capable of free-form and accurate grounding across multiple images, serving as an end-to-end solution for mig. to progressively establish flexible grounding capabilities, we employ a two-stage training procedure based on our proposed large-scale mig dataset (mgrounding-630k). first, the grounding ability of migician is enhanced through a combination of data from mig tasks and general tasks. then, migician is further refined using high-quality free-form mig instruction data. in addition, to evaluate the challenges of the free-form mig scenario, we construct a comprehensive multi-image grounding benchmark, migbench, comprising a total of 10 different tasks, 5.9k diverse images and more than 4.2k test instances. we observe a significant gap between the performance of existing mainstream mllms and human performance on the mig-bench. in contrast, migician can effectively alleviate this gap and improve the performance of free-form mig. to sum up, our contributions can be concluded as follows: ## related work multimodal large language models recent developments in multimodal large language models (mllms) have shifted from single imagetext understanding towards more versatile capabilities @xcite @xcite . among these efforts, some focus on enabling models to achieve fine-grained visual grounding, either through simple instruction tuning @xcite or by integrating additional auxiliary visual components @xcite @xcite . however, these models primarily focus on visual grounding within a single image. some other studies explore multiimage understanding tasks, such as multi-image comparison, reasoning, and temporal comprehension @xcite @xcite @xcite . nevertheless, fine-grained visual grounding at the multi-image level remains an underexplored area. to the best of our knowledge, our proposed migician is the first mllm designed to address the challenge of multi-image grounding. mllm benchmarks most existing benchmarks for evaluating mllms focus on single-image tasks @xcite . a few ## task definition the task of free-form multi-image grounding is to identify and localize relevant visual regions across a set of images based on a free-form query. unlike traditional grounding tasks with fixed input formats, the query in free-form multi-image grounding can be an arbitrary combination of text and images, making it highly flexible and versatile. formally, let the query q consist of a natural language description, reference images {r 1 , r 2 , . . . , r k } or a hybrid combination of both (e.g., "[a white car image] find a car like this image except it is black"). given a set of target images {i 1 , i 2 , . . . , i n }, the task is to identify a set of visual regions {g 1 , g 2 , . . . , g m } where g i is a region within an image i j that satisfies the semantic and contextual constraints defined by q. ## methods in this section, we delve into methods for enabling free-form multi-image grounding capabilities in mllms. we begin by exploring a chain-of-thought (cot) framework to elicit the capabilities within existing mllms to tackle this task. then we develop an end-to-end mig model, migician, through instruction tuning to overcome the limitations of the cot framework and achieve enhanced mig performance. ## mig-bench we introduce mig-bench, a manually curated benchmark designed to evaluate the mig ability of current mllms. it comprises 5.9k images, and 4.3k testing instances, covering 10 distinct tasks shown in figure 2 with details in appendix a. the distribution of these tasks is illustrated in figure 4 . mig-bench is manually constructed from multiple data sources. initially, we select annotated data examples from existing datasets and adapt them for the mig task. we collect challenging examples from objects365 @xcite for common object grounding and region locating. we use examples that exhibits significant movement from got-10k @xcite for object tracking. for multi-view grounding, we utilize the rich annotations from ego4d @xcite . the static difference task is sourced from mag-icbrush @xcite . we combine multi- all instances are reviewed by two different human annotators to guarantee the quality of mig-bench. this includes removing instances with incorrect annotations, ensuring that the questions are answerable, filtering out overly simplistic questions, and refining ambiguous queries. we also invite five volunteers to answer the questions to evaluate human performance on this benchmark (detailed in appendix f). as shown in unlike existing benchmarks, mig-bench introduces the grounding task in a multi-image setting, thereby addressing the gap in previous benchmarks that are unable to measure the free-form mig ca-pabilities of mllms. a detailed comparison with other benchmarks is provided in appendix b. ## experiments 6.1 implementation details migician undergoes development based on the qwen2-vl-7b @xcite foundation model with a global batch size of 48, a total of 25,000 steps for the two-stage training procedure, and a learning rate of 5e-6, using 8×a100-80g gpus. for the evaluation in our proposed mig-bench, we use the conventional metric acc 0.5 in referring expression comprehension @xcite . this metric measures the accuracy of object localization, defining a prediction as correct if the intersection over union (iou) with the ground truth bounding box is greater than 0.5. ## conclusion in this work, we explore the task of multi-image grounding and propose migician, the first mllm to overcome the barriers between fine-grained visual grounding and multi-image inputs. with our proposed large-scale mgrounding-630k dataset, migician seamlessly integrates grounding across multiple images, enabling free-form multi-image grounding. to further advance research in this area, we introduce mig-bench, a comprehensive benchmark for evaluating the multi-image grounding capabilities of mllms. experimental results demonstrate that our model significantly outperforms existing methods. we hope this work will inspire further developments in multi-image grounding and contribute to the creation of more versatile multimodal models in the future.