Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Models (2025)

Abstract: Multimodal Large Language Models (MLLMs) have shown significant promise in various applications, leading to broad interest from researchers and practitioners alike. However, a comprehensive evaluation of their long-context capabilities remains underexplored. To address these gaps, we introduce the MultiModal Needle-in-a-haystack (MMNeedle) benchmark, specifically designed to assess the long-context capabilities of MLLMs. Besides multi-image input, we employ image stitching to further increase the input context length, and develop a protocol to automatically generate labels for sub-image level retrieval. Essentially, MMNeedle evaluates MLLMs by stress-testing their capability to locate a target sub-image (needle) within a set of images (haystack) based on textual instructions and descriptions of image contents. This setup necessitates an advanced understanding of extensive visual contexts and effective information retrieval within long-context image inputs. With this benchmark, we evaluate state-of-the-art MLLMs, encompassing both API-based and open-source models. The findings reveal that GPT-4o consistently surpasses other models in long-context scenarios, but suffers from hallucination problems in negative samples, i.e., when needles are not in the haystacks. Our comprehensive long-context evaluation of MLLMs also sheds lights on the considerable performance gap between API-based and open-source models. All the code, data, and instructions required to reproduce the main results are available at https://github.com/Wang-ML-Lab/multimodal-needle-in-a-haystack.

## introduction recent breakthroughs in multimodal large language models (mllms) have enabled a wide range of applications, spanning from visual question answering to cross-modal retrieval @xcite * correspondence to: hengyi @xmath0.wang@rutgers.@xmath1. to evaluate the capabilities and limitations of mllms, various benchmarks have been proposed, focusing on challenges such as reasoning @xcite @xcite , perception @xcite , and hallucination @xcite . despite significant progress, the evaluation of mllms for long-context understanding has been lagging. current evaluation methods and benchmarks @xcite @xcite @xcite @xcite @xcite either (1) assume the use of single or limited images as inputs, failing to stress-test mllms' long-context capabilities or (2) only contain a limited numbers of data points (referred to as "samples" in this paper), lacking in statistical significance and therefore often rendering the evaluation inconclusive. these gaps limit the development of mllms capable of effectively handling long-context hybrid-modality inputs, which is crucial for broader applications. to bridge this gap, we introduce the multimodal needle-in-a-haystack (mmneedle) benchmark to comprehensively evaluate the long-context capabilities of mllms. fig. 1 shows a simple example: the mllms are presented with a haystack of images, consisting of @xmath2, each containing n × @xmath3. additionally, a caption is provided for one of the sub-images in the haystack, as shown in green text in figure 1(c ). the goal of the mllms is to identify the needle, namely the sub-image highlighted in the green box in figure 1(a) , which corresponds to the caption. by using advanced techniques, such as image stitching to increase input context length, we assess mllms' ability to locate a target sub-image (needle) within a large set of images (haystack) based on textual instructions, i.e., instructions with the target caption in fig. 1(c ). the highlights of our given 10 images indexed from 1 to 10, each divided into 2*2 sub-images, identify the sub-image that best matches the provided caption. respond with "index, row, column" and nothing else … caption: a woman walking across a sandy beach holding a little kite. the results showcase the comparative performance of various models: gpt-4o correctly predicts the exact location of the needle; gemini pro 1.5 only correctly predicts the image index of the needle; other api models predict incorrect locations; open-source models often output with wrong formats. mmneedle benchmark include: our findings underscore a considerable performance gap between models and reveal the hallucination problem in state-of-the-art mllms through negative samples. for example, we find that (1) there is still a large performance gap between state-of-the-art api-based and state-of-theart open-source models, (2) accuracy drops significantly with more images in the haystacks, even for state-of-the-art api-based mllms such as claude 3 opus and gemini 1.0 pro, and (3) all models (including claude 3 opus, gemini 1.5 pro, and gpt-4v) perform poorly in mmneedle settings with sub-images (e.g., n × @xmath4. 1 ); this is true even for the best model, gpt-4o, whose accuracy drops from 97.00% for @xmath5.e., equivalent to 10 images in the haystack) to 26.90% for @xmath6. see fig. 2 and more results in sec. 4. ## related work existing benchmarks for mllms mainly focus on limited image inputs, such as reasoning @xcite @xcite , perception @xcite , hallucination @xcite , where the answers are based on either single or only a handful of images. they are therefore not suitable for evaluating mllms' long-context capability for visual inputs. recent work @xcite @xcite on llms employs the needle-in-a-haystack test @xcite to evaluate the long-context capability of large language models (llms), where the llm is expected to answer the question by finding the corresponding information among a long irrelevant corpus as context. however, these datasets and benchmarks are not applicable for the multimodal setting. google's technical report @xcite has showcased gemini 1.5 pro's capability of finding the needle in an audio or video haystack. however, its evaluation (1) involves only one single sample rather than a complete dataset, obviously lacking statistical significance and there-fore rendering the evaluation inconclusive foot_0 , and (2) does not involve a large set of unrelated images, which is the focus of mmneedle. there is also work on the retrieval capability of small objects in a single large image @xcite or retrieval from large external image datasets @xcite , but none of them are concerned with in-context image retrieval, particularly for longcontext multimodal evaluation. in contrast to existing benchmarks, our mmneedle benchmark includes a dataset of 40,000 images, 560,000 captions, and 280,000 needle-haystack pairs (more details in sec. 3), rather than only one (or a handful of) needle-haystack pair(s) @xcite . mmneedle also includes a diverse set of metrics and evaluation protocols, covering different numbers of needle sub-images and needle sub-images. these differences set mm-needle apart from existing benchmarks and are essential to evaluate mllms' long-context capability comprehensively. ## multimodal needle in a haystack (mmneedle) in this section, we introduce our multimodal needle-in-a-haystack (mmneedle) benchmark. ## experiments in this section, we describe the evaluation results of various mllms on our mmneedle dataset. ## conclusion we propose mmneedle, a benchmark to evaluate mllms' long-context capabilities. mmneedle includes a comprehensive dataset and establishes diverse settings as well as a systematic set of coarse-to-fine evaluation metrics. we reveal that while api-based models, such as gpt-4o, outperform open-source models in long-context scenarios, they still struggle with hallucination issues in negative samples and challenges in large stitching size/multi-needle retrieval. a limitation of our mmneedle evaluation is the assumption that the mllm takes both images and texts as inputs and supports multiple-image inputs. however, we argue that these are necessary requirements for an ideal mllm.