Losing Visual Needles in Image Haystacks: Vision Language Models are Easily Distracted in Short and Long Contexts (2024)

Abstract: We present LoCoVQA, a dynamic benchmark generator for evaluating long-context reasoning in vision language models (VLMs). LoCoVQA augments test examples for mathematical reasoning, VQA, and character recognition tasks with increasingly long visual contexts composed of both in-distribution and out-of-distribution distractor images.Across these tasks, a diverse set of VLMs rapidly lose performance as the visual context length grows, often exhibiting a striking logarithmic decay trend. This test assesses how well VLMs can ignore irrelevant information when answering queries—a task that is quite easy for language models (LMs) in the text domain—demonstrating that current state-of-the-art VLMs lack this essential capability for many long-context applications.

## introduction long-context vision-language models (vlms) which accept multiple image inputs are pushing the boundaries of multimodal reasoning applications, but evaluation methods have not kept up. as open-weight long-context language models (lms) have been around for the last few years (dong et al., * equal contribution.  now at google. measuring extractive reasoning over image sequences is crucial. just as extractive textual reasoning facilitates text-domain rag, multi-modal rag demands that vlms efficiently reason over and extract information from documents featuring multiple images. similarly, video qa necessitates that models focus solely on frames containing relevant information, much like long-document qa. we introduce locovqa, a benchmark generator for long context visual question answering (vqa) with distractors. locovqa enables the creation of long-context image understanding evaluations from any image comprehension dataset by presenting a sequence with the question-relevant image alongside a configurable set of visual distractors. this allows us to accurately assess how effectively vlms extract only the pertinent information to a query within a cluttered context. we find that even top proprietary vlms struggle with this capability, even over short visual contexts, likely due to fundamental deficiencies in their training objectives. by unveiling this locovqa identifies a crucial area for performance improvement in future vlms. in summary, we: ## related work text-based long-context tasks such as longdocument question-answering (qa) @xcite , summarization @xcite , and retrieval-augmented generation (rag) @xcite ) offer analogues to long-context reasoning in vlms. many of these tasks (e.g., qa, rag) require natural language extractive reasoning, as discussed above. few vlm long-context evals measure extractive reasoning. existing vqa benchmarks involving distractor information (thereby extraction) do not focus on long contexts, and long-context vqa benchmarks do not involve distractors. multipan-elvqa @xcite includes distractor information, but only within single input images. some vlm evaluations focused on hallucinations indirectly capture a notion of distraction @xcite but do not explicitly measure it alone. milebench @xcite evaluates vqa in long contexts, but only on nondistractor tasks such as video summarization or difference detection, where all inputs are relevant. needle-in-a-haystack evaluation tasks (asking models to recover hidden passphrases at various positions) do require long-context extraction, but reasoning is not involved. gemini 1.5 @xcite extended this task to video comprehension by hiding a simple passphrase in many positions within a video. @xcite present a static benchmark concurrently to ours centered on multimodal needle-in-a-haystack. ## locovqa generation method samples synthesized through locovqa contain one or more content images x corresponding to a question and answer pair (q, a). content images can be sampled from various image comprehension benchmark, such as ok-vqa (marino et al., 2019), mmstar @xcite , @xcite , and others. alongside the content image(s), each sample includes up to 35 distractor images, which are either sampled in-distribution from the content image set (ensuring no content image collisions to prevent ambiguity, as described in §3.1.3) or out-of-distribution from other image sets such as ms coco @xcite . samples can be arranged as interleaved image sequences for vlms that accept multi-image inputs or as composite images arranged in a grid, as depicted in figure 2 . for all models capable of ingesting interleaved sequences, we evaluate both interleaved and composite examples. visual context refers to the visual elements within an image or sequence of images that are relevant to answering a question. this includes the content image and distractor images. the challenge is to identify and focus on the pertinent details while ignoring irrelevant information. we can sample images both in-distribution to create challenging visual contexts or out-of-distribution to isolate extraction capabilities in the simplest setting. as a dynamic generator locovqa can convert any vision-language dataset into a long-context benchmark @xcite . we describe the three datasets we used below. ## experiments we evaluate the performance of nine current visionlanguage models on our locovqa-generated benchmarks. the open-weight models tested are moondream2 (moondream, 2024), llava-1.5 @xcite , llava-1.6 @xcite , paligemma-3b @xcite , and two mantis variants @xcite . additional details are provided in appendix a. both mantis variants are vlms further tuned on interleaved multi-image instruction following. the three proprietary models we evaluate are gpt-4v @xcite , gemini 1.0 pro vision @xcite , and gemini 1.5 flash @xcite . interleaved image support. all closed-source models and mantis support multi-image interleaved inputs, while the others only support single images. for multi-image models, we evaluate both the composed (cmp) & interleaved (int) settings. for single-image models, we only test the composed (cmp) setting. ## results figure 4 illustrates how model performance (across 10 experiments, in both composed and interleaved settings) changes with increasing visual context lengths on single-image locovqa tasks. the first two rows display results from the mmstar dataset, which contains filtered-image-necessary examples from six other datasets @xcite , with titles in yellow and random guess thresholds indicated by dotted black lines. the bottom row presents ok-vqa scores using three scoring metrics, with titles in salmon. across all models on ok-vqa and the majority on mmstar subsets for seedbench, mm-bench, scienceqa, and ai2d, we observe striking logarithmic decay trends in model performance with increasing visual context length. correlation coefficients, r 2 , and p-values for each trendline are reported in on several multiple-choice tasks, mantis (the interleaved open-weight model) outperforms the other open-weight models that rely solely on composite inputs. this is likely due to the image subsampling required by several of the other openweight models negatively impacting performance when using composite inputs. however, on the ok-vqa task, mantis and moondream are the least performant, even at low context lengths. these models were likely not trained on visual questionanswering tasks to the same extent as llava variants during their instruction tuning steps. to downsampling effects in the composite setting. furthermore, for the models tested in both conditions (gpt-4v and gemini), the same trends are observed in both interleaved and composite settings. sub-chance performance on the multiplechoice question-answering datasets, particularly for the closed-weight models, occur due to high refusal rates. this may illustrate how "alignment hampering" can hinder performance. taskwise performance by context length. figure 5 illustrates the models' performance at context lengths of 1, 9, and 25 for each task, to compare the relative advantages different models have on various tasks. for example, paligemma excels on ok-vqa compared to the other models, while mantis performas well on ai2d. these differences are likely due to variations in training tasks. performance on multi-image tasks. figure 6 presents model performance on the mnist transcription task by number of digits to transcribe. note the x-axis in this figure is changed; it is checking for a fixed context length the relationship to number of digits. while there is a trend of decreasing performance with an increasing number of images, it does not follow a simple pattern like the context-length trend. for example, gemini 1.5 experiences minimal performance degradation even as the target digit length increases. characterizing difficulty as a function of digit length is complex-as digit counts increase in a fixed context window, the the output label search space grows, while the ratio of relevant images to irrelevant images increases. this may explain why some models have consistent or even increasing performance as # digits increases. analyzing why different models handle these axes of difficulty differently is an interesting future direction. however, akin to the single-image tasks, increasing the overall visual context length (seen in samecolor, x value points across plots), makes the task more difficult, albeit without as clear a correlation. ## ablation: needle in a haystack as performance within context windows decreases, a natural question arises: are performance failures equally distributed across the visual context range? to investigate this, we adapt our mnistbased visual context ocr task into a visual needle in a haystack task. needle in a haystack tests are a common minimal test of long-context capabilities in lms @xcite , involving hiding a passphrase, such as the word "needle," in various positions within a long document and asking the model to retrieve it. to adapt this concept, we modified our single-digit mnist task by sampling a set of 10 colored mnist digits (one for each number, e.g. blue 3, green 7, etc.) and hiding them in each possible position within both interleaved and composed visual context sequences. by assessing a model's single-digit recovery rate at each position, we can identify any systematic bias in extraction capabilities by postition. figure 7 presents the performance of a subset of the tested vlms on the composed and interleaved mnist visual needle in a haystack tasks. we test gpt-4v and gemini 1.5 flash in both settings, and treat llava 1.6 and mantis as analogous methods between the interleaved and composed settings. across the three composed tests (subfigures a, b, and c), we do not find a systematic bias toward any particular position on the single-digit recognition task. surprisingly, llava 1.6 performance the best, probably due to more mnist-like ocr data in its training mix compared to the others. however, we find strong systematic biases with respect to position in the interleaved setting for all models (subfigures d, e, and f). mantis shows a preference for late positions in sequences of any length, while gpt-4v and gemini exhibit weak biases towards early positions. given that the biases are evident even in the relatively simple mnistbase ocr task, it is likely that this effect plays a significant role in the performance penalty these models experience with longer contexts. these results are supported by similar findings in from the image-needle test from @xcite . they also found that-contrary to google's claims to significant needle-in-a-haystack performance for gemini @xcite -for non-trivial hidden visual information tasks current sota vlms are woefully non-performant. ## discussion why do we observe such striking logarithmic decay in performance with increasing visual context length? the fact that the trend is consistent in both interleaved and composite settings that downsampling effects in the latter are not the primary cause of performance decay. there may be an information-theoretic interpretation of this behavior: an increasing signal-to-noise ratio. as the visual context length increases, the amount of signal (relevant information) remains constant, while the amount of noise (distractor information) increases. however, this signal-to-noise ratio problem also exists in text-only tasks, yet long-context llms perform well on needle in a haystack and longcontext qa and summarization tasks, underlying their performance in retrieval-augmented generation. when it comes to visual contextual tasks that involve distractor information, vlms struggle to perform even on the easiest long-context tasks, such as transcribing mnist characters. they are even less capable on more complex, real-world tasks like distractor vqa. the core issue appears to be that both short-context and long-context vlms are not trained on tasks requiring attention over multiple context images to retrieve information. in contrast, for long passages of text, information from throughout the passage remains relevantreferring back to a specific sentence early in a document is intrinsic to the lm training objective. however, the same may not necessarily be true for the sequential image language modeling task. in the existing interleaved vision-language training corpora, images are likely to followed immediately by their relevant text. as a result, attending to much earlier images in the documents may not be crucial for achieving low lm loss. additionally, none of the tested open vlms are trained on image generation tasks conditioned on text while updating the core transformer weights. overlooking this objective may prevent vlms from robustly modeling the relationship between images and text, which could be crucial for performing well on these tasks. interesting examples of poor performance our analysis unveiled a few particularly surprising results. for example, contrary to our expectation that models capable of handling interleaved input would perform better with it than with composite input, we found that gpt-4v actually favors composite input on many subtasks. additionally, to our surprise llava 1.6 performed much better than gpt-4v on the composite haystack task at all sizes. this outcome was driven by multiple factors: gpt-4v tends to refuses some tasks rather than guessing, while llava always provides a guess. furthermore, gpt4v often "hallucinates" multiple numbers instead of just one. possibility of memorization some of the surprising performance results are driven by the training mix. for example, mantis's significant lead on ai2d, and llava's strong performance on the mnist task may result from having more relevant data or those specific training sets included in their data. however, even considering that some models were trained on these tasks, the pronounced drops in generalization performance as the context length increases are even more striking. this illustrates that current vlms fundamentally struggle to attend image sequences as well as they do with text. upper bound for video qa a more constructvalid test for real-world visual extractive reasoning would be qa over a long video where only a subelement is required. e.g., putting the entire film star wars as input for qa pair (q: "which character shoots a green alien in the mos eisley cantina?", a: han solo). information from a single scene must be extracted to perform this task, thereby requiring extractive reasoning. however, it is plausible that sequences of related images, unlike sequences of unrelated images, are more in-distribution for long-context vlms and extractive tasks involving them could be easier for them to solve. by providing completely unrelated images, our task may be harder than video-based vqa, and may represent an upper-bound for visual extractive reasoning. ## conclusion vision-language models struggle with long-context visual extractive reasoning. many models fail to extract necessary information from even short contexts of 36, 25, or 16 images, resulting in nearor sub-baseline performance across tasks. lo-covqa presents a simple benchmark-generating process applicable to any vqa or retrieval evaluation dataset, making it easy to assess extractive reasoning in vlms. our findings suggest that training tasks requiring attention across multiple context images to extract information-rather than simple single-image tasks-should be included in vlm training. by measuring this capacity it offers an appealing direction for future work.