Promptception: How Sensitive Are Large Multimodal Models to Prompts? (2025)

Abstract: Despite the success of Large Multimodal Models (LMMs) in recent years, prompt design for LMMs in Multiple‐Choice Question Answering (MCQA) remains poorly understood. We show that even minor variations in prompt phrasing and structure can lead to accuracy deviations of up to 15% for certain prompts and models. This variability poses a challenge for transparent and fair LMM evaluation, as models often report their best-case performance using carefully selected prompts. To address this, we introduce **Promptception**, a systematic framework for evaluating prompt sensitivity in LMMs. It consists of 61 prompt types, spanning 15 categories and 6 supercategories, each targeting specific aspects of prompt formulation, and is used to evaluate 10 LMMs ranging from lightweight open‐source models to GPT-4o and Gemini 1.5 Pro, across 3 MCQA benchmarks: MMStar, MMMU‐Pro, MVBench. Our findings reveal that proprietary models exhibit greater sensitivity to prompt phrasing, reflecting tighter alignment with instruction semantics, while open‐source models are steadier but struggle with nuanced and complex phrasing. Based on this analysis, we propose Prompting Principles tailored to proprietary and open-source LMMs, enabling more robust and fair model evaluation.

## introduction recent advancements in large multimodal models (lmms) have significantly improved their ability to integrate vision and language, enabling strong performance on a range of reasoning tasks involving textual and visual information @xcite @xcite . these models take visual cues (single image, multiple images, and video) and text as input, to output a textual response. they have been fine-tuned on a variety of tasks, including captioning, visual questionanswering (vqa) @xcite , visual grounding @xcite , and temporal grounding @xcite . however, despite these successes, the systematic study of sensitivity of lmms remains an open question. while the impact of texual prompt variations has been extensively studied in large language models (llms) @xcite as well as clip-based vision-language models (vlms) @xcite , its effects on lmms have not been systematically explored. understanding how these models respond to textual modifications is crucial for assessing their reliability in multimodal tasks. ## prompts in this section, we introduce the prompts proposed in our promptception evaluation framework, each designed to examine different aspects of prompt engineering and its influence on model responses in the mcqa task. we note that prompts 2.6-2.9 in category 2 (structured formatting) include a neutral persona element, but this component is functional rather than behavioral and thus distinct from the vivid role simulation in category 9 (roleplay scenarios). to empirically verify this distinction, we conducted an ablation study (appendix c), which showed negligible performance differences with or without the persona, supporting our categorization. ## results & analysis 4.1 overall trend to provide a robust assessment of model performance across different benchmarks, we calculate the trimmed meanμ (equation 1 ) for each dataset using a 10% trimming rate, chosen based on empirical observation. we chose trimmed meanμ to provide a more stable estimate of overall model performance by reducing the influence of extreme prompt-specific values. many models rely on carefully engineered prompts to boost performance, which can obscure their true capabilities; the trimmed mean mitigates such effects and offers a clearer picture of general behavior across diverse prompts. in addition, we complement this with a model-wise sensitivity analysis (appendix h), which highlights which models are most affected by prompt variation. appendix b provides the comprehensive list of accuracies across all prompts and benchmarks for each model. super category category modification example prompt type choice formatting and presentation 1: formatting variations in choice presentation answer choice letter. type 1.3: @xmath5: what design element best describes the image? <@xmath6: composition option b: perspective option c: balance option d: shape answer with the option letter from the given choices directly. 2: structured formatting explicit structure the question and choices are presented. type 2.2: question & answer prefix question: what design element best describes the image? <@xmath7: a. composition b. perspective c. balance d. shape answer with the option letter from the given choices directly. 3: prompt position changes relative positioning of prompt, question, and choices. type 3.2: middle what design element best describes the image? <@xmath8. a. composition b. perspective c. balance d. shape linguistic and stylistic challenges 4: poor linguistic formatting grammatical errors, misspellings, and inconsistencies in wording. task 4.4: poor formatting answer.with;the.option: letter from-choices.directly! 5: effect of prompt length prompt length, from concise to verbose. task 5.2: medium prompt your task is to examine the given image(s) and determine which of the listed options accurately answers the question. carefully analyze the image(s), consider the possibilities, and then respond only with the correct option @xmath0letter' of the best answer choice. 7: ambiguity and probabilistic prompts introducing uncertainty, probabilistic reasoning, and deferred context. task 7.1: add probabilistic language based on our analysis of the image(s), there is a 25% chance that the correct answer is option a, but the other options may still be valid. carefully evaluate the image(s) and choose the answer letter you believe is most likely to be correct. context-aware and ethical guidance 8: target audience prompts tailoring explanations to different audience levels. task 8.1: explain to a child explain the answer as if you're speaking to a curious 10-year-old. use simple words, short sentences, and relatable examples that a child would easily understand. avoid technical terms or complex concepts. respond only with: '@xmath1letter' of the best answer choice. 10: testing bias and stereotypes detecting and mitigating biases. task 10.2: contextual neutrality consider the image(s) and question provided within the specific context they represent. ensure your response reflects neutrality by avoiding cultural, social, or historical biases. base your decision solely on the visible elements and the question's requirements, maintaining a context-aware but unbiased approach. respond only with: '@xmath2letter" (without quotes) where letter is one of the options. performance, feedback, and penalty 13: penalty-based prompts penalties for incorrect answers, formatting violations etc. task 13.1: penalties for mistakes **warning:** an **incorrect answer** will result in a **"strict penalty"** being applied. carefully examine all details in the image(s), analyze the question thoroughly, and select your response with precision. accuracy is **"non-negotiable"**, so take your time and avoid errors. ensure your response follows this format: answer: @xmath3100 for delivering a flawless and accurate answer. carefully analyze the image(s) and the question provided, paying attention to every detail and nuance. respond only with: "@xmath4letter. ## prompting principles based on the insights from section 4, we outline best practices for optimizing lmm performance on the mcqa task in an important observation underlying these principles is the clear difference in behavior between open-source and proprietary models. open-source models are often not extensively instruction-tuned, which makes them less responsive to prompt variations. in contrast, proprietary models typically undergo rigorous instruction tuning with large-scale, high-quality data, as well as advanced reinforcement learning and post-training techniques. this makes them considerably more sensitive to user instructions, where even subtle changes in prompt phrasing can lead to notable differences in performance. given these differences in instruction-following capabilities, we present prompting principles separately for open-source and proprietary models. this distinction allows us to account for their varying adherence to instructions and to highlight strategies that are most effective for each category. prompt sensitivity has also been analyzed through new evaluation metrics: promptsensiscore and decoding confidence have been proposed to quantify how models respond to rephrasings @xcite , while sensitivity and consistency measures have been introduced to capture how llm predictions change across prompt rephrasing in classification tasks @xcite . @xcite has introduced robustalpacaeval, a collection of semantically equivalent prompts for evaluating how sensitive llms are to minor varia-tions, showing performance swings of up to 45% for some models depending on the formulation. other work has highlighted how the sentiment of prompts affects llm outputs across coherence, factuality, and bias in various applications, finding that negative phrasing often harms accuracy and increases bias, while positive phrasing can lead to verbosity (gandhi and gandhi, 2025). largescale investigations have further expanded this line of research, including datasets with over 250m prompt perturbations designed to measure sensitivity across multiple dimensions and tasks @xcite . these findings collectively emphasize the need for standardized, well-defined methodologies when evaluating and deploying llms. prompt sensitivity of lmms: while the impact of textual prompt variations has been extensively studied in unimodal llms and clip-based vlms, research on multimodal lmms remains limited. dumpala et al. @xcite showed that generative vlms are responsive to lexical and semantic changes, and awal et al. @xcite investigated prompting strategies for zero-and fewshot vqa. however, these efforts are narrow in scope. to the best of our knowledge, no prior work has systematically examined prompt sensitivity across multiple lmms and modalities (image, multi-image, and video). our work addresses this gap by introducing a unified evaluation framework, a curated taxonomy of 15 prompt categories, and actionable prompting principles for fairer and more consistent assessment of lmms. ## conclusion we present the most comprehensive analysis to date on the impact of prompt design in large multimodal models (lmms) for mcqa across image and video benchmarks. using promptception, a systematic framework covering 61 prompt types, we evaluate 10 models on 3 datasets. our findings reveal that prompt phrasing substantially affects performance: proprietary models exhibit strong instruction-following but higher sensitivity, while open-source models are more stable yet less responsive to subtle cues. we hope this work advances fair and transparent lmm evaluation. of vision-language tasks. however, to realize this potential, it is necessary to develop a more comprehensive and task-specific prompt framework. this would involve a careful study of task types beyond mcqa, such as video captioning, visual reasoning, and image/video-grounded dialogue to craft prompts tailored to the unique demands of openended tasks. moreover, while our manually designed prompts offer strong performance and serve as a set of best practices, automatic prompt generation is a crucial next step toward scaling this approach across a wider set of tasks. a promising direction involves the use of meta-prompting @xcite , where a higher-level prompt is used to guide a language model in generating a task-specific prompt based on the input. to further streamline this process, an alternative direction is to train a lightweight prompt-generation model @xcite that can directly output high-quality prompts conditioned on the input. aklilu, alejandro lozano, anjiang wei, ludwig schmidt, and serena yeung-levy. 2025. automated generation of challenging multiple-choice questions for vision language model evaluation. preprint, arxiv:2501.03225. mingqian zheng, jiaxin pei, lajanugen logeswaran, moontae lee, and david jurgens. 2024. when "a helpful assistant" is not really helpful: personas in system prompts do not improve performances of large language models. preprint, arxiv:2311.10054. jingming zhuo, songyang zhang, xinyu fang, haodong duan, dahua lin, and kai chen. 2024. prosa: assessing and understanding the prompt sensitivity of llms. preprint, arxiv:2410.12405.