Papers
arxiv:2406.12742

Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoning

Published on Jun 18
· Submitted by tennant on Jun 19
Authors:

Abstract

The advancement of large language models (LLMs) has significantly broadened the scope of applications in natural language processing, with multi-modal LLMs extending these capabilities to integrate and interpret visual data. However, existing benchmarks for visual language models (VLMs) predominantly focus on single-image inputs, neglecting the crucial aspect of multi-image understanding. In this paper, we introduce a Multi-Image Relational Benchmark MIRB, designed to evaluate VLMs' ability to compare, analyze, and reason across multiple images. Our benchmark encompasses four categories: perception, visual world knowledge, reasoning, and multi-hop reasoning. Through a comprehensive evaluation of a wide range of open-source and closed-source models, we demonstrate that while open-source VLMs were shown to approach the performance of GPT-4V in single-image tasks, a significant performance gap remains in multi-image reasoning tasks. Our findings also reveal that even the state-of-the-art GPT-4V model struggles with our benchmark, underscoring the need for further research and development in this area. We believe our contribution of MIRB could serve as a testbed for developing the next-generation multi-modal models.

Community

Paper author Paper submitter

The advancement of large language models (LLMs) has significantly broadened
the scope of applications in natural language processing, with multi-modal LLMs
extending these capabilities to integrate and interpret visual data. However,
existing benchmarks for visual language models (VLMs) predominantly focus on
single-image inputs, neglecting the crucial aspect of multi-image
understanding. In this paper, we introduce a Multi-Image Relational Benchmark
MIRB, designed to evaluate VLMs' ability to compare, analyze, and reason across
multiple images. Our benchmark encompasses four categories: perception, visual
world knowledge, reasoning, and multi-hop reasoning. Through a comprehensive
evaluation of a wide range of open-source and closed-source models, we
demonstrate that while open-source VLMs were shown to approach the performance
of GPT-4V in single-image tasks, a significant performance gap remains in
multi-image reasoning tasks. Our findings also reveal that even the
state-of-the-art GPT-4V model struggles with our benchmark, underscoring the
need for further research and development in this area. We believe our
contribution of MIRB could serve as a testbed for developing the
next-generation multi-modal models.

Great work! Any plan to include Mantis in the benchmark?

https://huggingface.co/collections/TIGER-Lab/mantis-6619b0834594c878cdb1d6e4

·
Paper author

Working on it🤞

Paper author

Sign up or log in to comment

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2406.12742 in a model README.md to link it from this page.

Datasets citing this paper 1

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2406.12742 in a Space README.md to link it from this page.

Collections including this paper 5