Multimodal Inconsistency Reasoning ( MMIR ): A New Benchmark for Multimodal Reasoning Models (2025)

Abstract: Existing Multimodal Large Language Models (MLLMs) are predominantly trained and tested on consistent visual-textual inputs, leaving open the question of whether they can handle inconsistencies in real-world, layout-rich content. To bridge this gap, we propose the Multimodal Inconsistency Reasoning (MMIR) benchmark to assess MLLMs’ ability to detect and reason about semantic mismatches in artifacts such as webpages, presentation slides, and posters. MMIR comprises 534 challenging samples, each containing synthetically injected errors across five reasoning-heavy categories: Factual Contradiction, Identity Misattribution, Contextual Mismatch, Quantitative Discrepancy, and Temporal/Spatial Incoherence. We evaluate eight state-of-the-art MLLMs, showing that models with dedicated multimodal reasoning capabilities, such as o1, substantially outperform their counterparts while open-source models remain particularly vulnerable to inconsistency errors. Detailed error analyses further show that models excel in detecting inconsistencies confined to a single modality, particularly in text, but struggle with cross-modal conflicts and complex layouts. Probing experiments reveal that single-modality prompting, including Chain-of-Thought (CoT) and Set-of-Mark (SoM) methods, yields marginal gains, revealing a key bottleneck in cross-modal reasoning. Our findings highlight the need for advanced multimodal reasoning and point to future research on multimodal inconsistency.

## introduction recent advances in large language models (llms) have demonstrated impressive reasoning abilities across a variety of tasks @xcite @xcite . building on pre-trained llms, multimodal large language models (mllms) are evolving rapidly. however, they usually face greater challenges as they need to reason across different ## mmir the mmir benchmark is designed to assess how effectively mllms can detect and localize semantic mismatches within complex, layout-rich artifacts. unlike conventional benchmarks that assume coherent visual-textual inputs, mmir challenges models with realistic errors that require deep, crossmodal reasoning. in mmir, errors are defined and categorized along five semantic dimensions: a. factual contradiction: direct conflict between two elements (text-text, text-image, or image-image) within the artifact. b. identity misattribution: mislabeling of entities (objects, locations, brands, people) that conflict with other elements. c. contextual mismatch: tonal, thematic, or situational incompatibility between elements. d. quantitative discrepancy: numerical or statistical inconsistencies between elements. e. temporal/spatial incoherence: implied timelines, dates, or spatial relationships that are impossible or conflicting. figure 2 provides one example from each error type across web, office, and poster artifacts, illustrating the diverse challenges mmir poses. detailed definitions and examples of inconsistency error types can be found in appendix a.1. ## experiments and analysis we first evaluate the advanced multimodal reasoning model o1 @xcite and seven other state-of-the-art mllms: gpt-4o (openai, 2024a), qwen2.5-vl @xcite , llama-3.2 @xcite , llava-next @xcite , internvl2.5 @xcite and phi-3.5-vision (abdin et al., 2024) on the mmir benchmark. we implement open-source models using their default settings and select the 1217 version of o1 and the 1120 version of gpt-4o for evaluation. model implementation details are provided in appendix c. we then examine error patterns across different inconsistency types and layout complexities, and finally explore how prompting strategies affect multimodal reasoning under the open-ended setting. ## discussion and conclusion in this work, we introduce the multimodal inconsistency reasoning benchmark (mmir) to evaluate how well mllms detect and localize semantic mismatches in complex real-world artifacts. mmir challenges models across five error categories and two reasoning settings for a detailed assessment of multimodal reasoning. our experiments show that even advanced proprietary models struggle with open-ended inconsistency detection. although providing natural-language descriptions in a multiple-choice format offers modest gains, standard prompting techniques (e.g., chainof-thought and set-of-mark) yield inconsistent or negative effects, while a proposed multimodal interleaved cot (mm-cot) method that iteratively refines reasoning by integrating visual and textual modalities, yielding greater performance improvements. despite these advances, significant challenges remain, motivating further research on robust multimodal reasoning for real-world inconsistency detection.