From Introspection to Best Practices: Principled Analysis of Demonstrations in Multimodal In-Context Learning (2025)

Abstract: Motivated by in-context learning (ICL) capabilities of Large Language Models (LLMs), multimodal LLMs with additional visual modality are also exhibited with similar ICL abilities when multiple image-text pairs are provided as demonstrations. However, relatively less work has been done to investigate the principles behind how and why multimodal ICL works. We conduct a systematic and principled evaluation of multimodal ICL for models of different scales on a broad spectrum of new yet critical tasks. Through perturbations over different modality information, we show that modalities matter differently across tasks in multimodal ICL. Guided by task-specific modality impact, we recommend modality-driven demonstration strategies to boost ICL performance. We also find that models may follow inductive biases from multimodal ICL even if they are rarely seen in or contradict semantic priors from pretraining data. Our principled analysis provides a comprehensive way of understanding the role of demonstrations in multimodal in-context learning, and sheds light on effectively improving multimodal ICL on a wide range of tasks.

## introduction motivated by in-context learning (icl) capabilities of large language models (llms) for nlp tasks @xcite @xcite , multimodal llms with additional visual modality are also exhibited with similar icl abilities when multiple image-text pairs are provided as demonstrations @xcite @xcite . in recent studies, the retrieval-based in-context example selection @xcite ) approach, which retrieves similar images in the support set by comparing their visual features with testing images, has become a default 1 codes and datasets are available at https:// github.com/luka-group/multimodaliclbestpractice . approach to selecting demonstrations for multimodal icl @xcite @xcite . however, relatively less work has been done to investigate the principles behind flipped labels q: what color is the writing on the spare tire? gt: i am not sure about the color of the writing on the spare tire. ## experimental setup in this section, we describe the experimental setup used in our analysis ( §4- §6). we list evaluation benchmarks and corresponding metrics in tab. 2, as well as studied model information in tab. 3 . evaluation benchmarks after pretraining multimodal llms on interleaved image-text data or fine-tuning on multi-turn conversations, existing work @xcite @xcite ) mainly focuses on evaluating their icl abilities on image captioning such as coco @xcite and flickr30k @xcite , as well as generalpurpose visual question answering tasks such as okvqa @xcite , vqav2 @xcite , textvqa @xcite and vizwiz @xcite . besides these classic vision-language tasks, we also consider one recently released benchmark, namely vl-icl bench @xcite , which encompasses a broad spectrum of challenging new tasks to investigate strengths and limitations of icl capabilities. benefits of utilizing demonstrations as contexts for more critical and practical applications, though imperfect zero-shot performance is observed from state-of-the-art models, are not yet explored. therefore, we further study icl capabilities of multimodal llms on the following tasks. 1) math reasoning: math-vision @xcite is a large math reasoning benchmark that collects questions from real math competitions and tests the general visual perception and mathematical reasoning abilities; 2) hallucination: amber @xcite provides a discriminative way to evaluate various types of hallucination including existence, attribute and relation; 3) text-rich tasks: both ocrbench @xcite and seed-bench-2-plus @xcite ) assess text-rich visual comprehension of models, while the former focus on optical character recognition (ocr) capabilities and the latter covers text-rich scenarios in the real world such as charts, maps, and webs; 4) medical tasks: three datasets consider different medical modalities, i.e., path-vqa @xcite for pathology, slake-vqa @xcite for radiology and pad-ufes-20 @xcite for skin lesion images. 5) multi-image tasks: seed-bench-2 @xcite evaluates the ability to comprehend multimodal inputs containing multiple images. 6) cross-style transfer: benchlmm @xcite assesses the robustness of models against three different styles including artistic image, imaging sensor, and application styles. ## modalities matter differently in multimodal icl as shown in fig. 6 and fig. 7 , pretrained models and gpt-4o generally achieve better performance given demonstrations as context in existing icl tasks. as demonstrated in fig. 8 , on more complex and reasoning-focused tasks, pretrained models generally benefit more from demonstrations while performance of gpt-4o is barely influenced. in this section, we examine which modality of the demonstrations takes more effect in multimodal in-context learning. for a comprehensive evaluation, we focus on three tasks of different difficulty levels: easy cross-style tasks (i.e., benchlmm sensor and application in fig. 8 ), moderate medical tasks (i.e., path-vqa, slake-vqa and pad-ufes-20 in fig. 8 ), and hard text-rich key information extraction task (i.e., kie from ocrbench in fig. 11 ). we visualize 4-shot performance of idefics-80b within this section while leaving results of other models in appendix (from fig. 14 to fig. 18 .). the dependency of performance gain from icl on demonstration modalities differs among tasks. ## how to select effective demonstrations for multimodal icl motivated by variational roles of different modalities across different tasks, we further explore influence of modality-driven demonstration selection strategies on icl performance in this section. ## conclusion we conduct a systematic and principled evaluation of multimodal icl for models of different scales on a broad spectrum of new yet critical tasks. we find that modalities matter differently in multimodal icl across tasks. hence we utilize modality-driven demonstration strategies to boost icl performance. moreover, we find that demonstrations selected according to textual similarity help models capture inductive biases from multimodal icl.