I nter F eedback: Unveiling Interactive Intelligence of Large Multimodal Models with Human Feedback (2025)

Abstract: Existing benchmarks do not test Large Multimodal Models (LMMs) on their interactive intelligence with human users which is vital for developing general-purpose AI assistants. We design InterFeedback, an interactive framework, which can be applied to any LMM and dataset to assess this ability autonomously. On top of this, we introduce InterFeedback-Bench that evaluates interactive intelligence using two representative datasets, MMMU-Pro and MathVerse, to test 10 different open-source LMMs. Additionally, we present InterFeedback-Human, a newly collected dataset of 120 cases designed for manually testing interactive performance in leading models such as OpenAI-o1 and Claude-3.5-Sonnet. Our evaluation results show that state-of-the-art LMM (e.g., OpenAI-o1) can correct their results through human feedback less than 50%. Our findings point to the need for methods that can enhance LMMs’ capabilities to interpret and benefit from feedback.

## introduction in this paper, we are curious about the question "can large multimodal models evolve through interactive human feedback?" it is central to developing general-purpose ai assistants with large multimodal models (lmms). while these models show exceptional performance on tackling multimodal tasks directly, their ability to interact with humans remains largely unknown. we argue that an lmm functioning as the general assistant should possess two capabilities: 1) exceptional problemsolving ability and 2) the ability to improve itself through feedback (e.g., human feedback, execution results). in this work, we focus on the latter capability, which has been rarely examined in existing benchmarks. humans are highly adaptive, continuously refining their skills through feedback-a fundamental process for acquiring knowledge and solving problems. likewise, advanced lmm models should be designed to learn from feedback, ensuring better alignment with real-world needs and enhancing their problem-solving capabilities in human-ai interaction (hai). recently, a surge of large multimodal models (lmms) @xcite @xcite @xcite @xcite have been developed to handle various tasks, including general vision-language understanding @xcite , expert-level multimodal understanding @xcite , and scientific reasoning @xcite @xcite . however, these lmms are primarily tested in a static way, overlooking their great potential in an interactive process such as interactive coding @xcite , computer usage @xcite @xcite , and clinical reasoning @xcite . consequently, the interactive intelligence of lmms remains largely unexplored, and the development of a standard benchmark for evaluating their interactive intelligence remains an open challenge. the key challenge in evaluating the interactive intelligence of lmms is the automatic model tests. in practice, for the same query, different lmms often produce varied responses, necessitating that humans offer tailored feedback for each conversation round. to address this issue, we propose interfeedback a straightforward problem-solving framework that enables any lmm to tackle multimodal tasks interactively by leveraging proprietary models such as @xcite to simulate humans, inspired in previous studies @xcite @xcite . interfeedback-bench, a benchmark designed to comprehensively evaluate lmms for two purposes: 1) the ability to interactively solve problems and 2) the capability of interpreting the feedback to improve themselves. we demonstrate with two challenging pre-existing datasets: mmmu-pro @xcite and mathverse @xcite . additionally, for a more in-depth investigation, we conduct human evaluation on four closed-source proprietary models: gpt-4o (ope-nai, 2023), openai-o1 (openai, 2024), claude-3.5-sonnet (anthropic, 2024), and gemini-2.0 @xcite with a trained user acting as the feedback provider. finally, we manually collected a dataset interfeedback-human containing 120 samples for this assessment. our experimental results reveal several compelling insights: 1) interactive process could improve the performance of most lmms in solving challenging problems; 2) existing lmms exhibit suboptimal performance in interpreting and incorporating feedback; 3) accuracy result may not truly reflect the model's capability to improve itself from feedback; 4) high-quality feedback is essential, as subpar feedback can degrade performance even more than a simple binary (0/1) correctness signal; 5) lmm may not truly reasoning, we find out that lmms resort to guessing answer even on a simple question according to human. these findings point to the need for methods that can enhance the lmm's capability to interpret and benefit from feedback. in summary, our contributions are: ## related work large multimodal models. the llava-series works @xcite @xcite demonstrate that training with supervised fine-tuning (sft) multimodal data and expand the vision lens would produce compatible multimodal reasoning ability. by adopting a largescale image-text corpus for instruction tuning, qwen2-vl @xcite , cogvlm @xcite , internvl2 (opengvlab, 2024) have achieved exceptional performance on various multimodal abilities. moreover, molmo @xcite proposes to train an lmm from scratch with only the human-annotated data. unlike these large models, minicpm-v @xcite and phi-3.5-vision @xcite propose to train lightweight yet sota lmms. despite their exceptional performance on multimodal benchmarks of varying difficulty, such as mmmu-pro @xcite and mathvista @xcite , it remains unclear how well these lmms demonstrate interactive intelligence in human-ai interaction scenarios. in this paper, we conduct the evaluation of these lmms to explore this basic yet vital capability (i.e., evolve through interactive human feedback). ## interfeedback-bench in this section, we first introduce the automated interactive benchmarking process in section 3.1. we begin by formulating the concept of interactive problem-solving, followed by a discussion of the data curation process. we then present the proposed interactive framework, interfeedback. next, in section 3.2, we elaborate on the human benchmarking component, detailing the data collection and the proposed feedback providing strategy.