Debating for Better Reasoning in Vision-Language Models (2025)

Abstract: As Large Language Models (LLMs) gain expertise across diverse domains and modalities, scalable oversight becomes increasingly challenging, particularly when their capabilities may surpass human evaluators. Debate has emerged as a promising mechanism for enabling such oversight. We extend the debate paradigm to a multimodal setting, exploring its potential for blind models to supervise and enhance the performance of sighted ones. We focus on visual question answering (VQA), where two “sighted” expert vision-language models debate an answer, while a “blind” (text-only) judge adjudicates based solely on the quality of the arguments. In our framework, the experts only defend answers aligned with their beliefs, thereby obviating the need for explicit role-playing and concentrating the debate on instances of expert disagreement. Experiments on several multimodal tasks demonstrate that the debate framework consistently outperforms individual expert models. Moreover, judgments from blind LLMs can be used to instil reasoning capabilities in vision-language models through fine-tuning.

## introduction current approaches to aligning large language models rely heavily on human-labeled data @xcite . however, as models gain expertise across different domains and modalities @xcite @xcite , gathering highquality data for training them and further aligning them becomes progressively expensive and difficult. previous work @xcite has investigated interactions between models as a means of achieving scalable oversight. this paradigm seeks to develop techniques for supervising increasingly capable models, even in sce-narios where their expertise may surpass that of human evaluators. debate has emerged as a promising framework in this context @xcite @xcite @xcite where two or more expert models argue to convince a non-expert model or human judge. for instance, in a question answering setting, both models are presented with the same question. they then take turns providing their answers and justifying their responses. the judge sees the debate and decides which model wins. other frameworks @xcite @xcite explore debate within the context of multi-agent collaboration. the core idea is to engage models in multiple rounds of discussion aiming to converge on a consensus response which is meant to be better than the output of a single model. @xcite propose a variant that leverages consensus among multiple instances of the same model, aiming to collect diverse data for further fine-tuning. in this work, we explore the potential of debate in a multimodal setting, as a mechanism for weaker models to supervise and enhance the performance of expert models. we focus on visual question answering tasks (vqa; @xcite where a model endowed with visual perception is meant to answer a question about an image. following the original debate paradigm @xcite , we assume two expert models debate the answer to a question considering the provided visual context, while a non-expert judge determines the winner (see figure 1 ). while the experts are "sighted" and capable of simultaneously understanding both visual and linguistic modalities, our judge is "blind" without access to the image, and thus meant to adjudicate based on the quality of the arguments. a common framing of debate protocols is to assign experts an answer which they are required to defend, regardless of whether they "believe" it to be true or correct @xcite @xcite . in our formulation, experts only defend answers that align with their beliefs. this obviates the need for role-playing on behalf of the experts and allows us to debate exclusively on samples where the experts disagree. to evaluate the debates, we draw on previous research on assessing the quality of arguments @xcite @xcite , focusing on factors like consistency, credibility, and relevance. we present experiments with open-source models across a wide range of multimodal tasks that collectively require diverse vision-grounded skills. our results demonstrate that the debate framework consistently outperforms the expert models involved in the debate and simpler, scalable oversight protocols such as consultancy @xcite , where a single expert model interacts with a weaker judge. we further demonstrate that the verdicts delivered by the blind judge can be utilized to enhance and improve reasoning capabilities in vision-language models through fine-tuning. overall, our experiments highlight the value of debate as a mechanism for enhancing model performance, demonstrating improvements on tasks unseen during training. our contributions are three-fold: ## related work a substantial body of work has previously explored debate as a framework for eliciting better quality responses from text-only language models. in their pioneering work, @xcite motivate debate as a protocol of scalable oversight that can be employed to supervise expert models using weaker models. as expert models are tasked to argue about an answer, a weaker judge decides which side is more convincing, even if they don't fully understand the domain or task. @xcite test the feasibility of scalable oversight through interactions between humans and non-reliable llms, whose task is to answer difficult multiple-choice questions @xcite . building on this body of work, @xcite and @xcite explore the use of more advanced language models, focusing on question-answering tasks over long-form texts @xcite . they introduce the notion of a weak judge, a language model with limited access to information that must rely solely on the experts' arguments for determining the final answer. the two experts, potentially different instantiations of the same model, are assigned opposing answers to debate against one another. a distinct but related direction is the multi-agent debate (mad) framework proposed in @xcite where multiple agents argue with one another over multiple rounds to come to a consensus. their results show that mad leads to answers with improved factuality and reasoning. @xcite extend this work to incentivize novelty in the debaters' arguments to prevent mad from defaulting to majority voting. in follow-on work, @xcite further train debaters to improve their collaborative problemsolving skills. @xcite introduce a multi-agent fine-tuning approach where several models, initialized from the same base model, are independently specialized on distinct subsets of data generated from debates to encourage diversity and refinement across responses. in contrast to previous work, which primarily focuses on purely linguistic tasks, we operate in the multimodal domain, using debate as a mechanism for a blind model to update the beliefs of expert models. we further show that the data obtained through this process is valuable for fine-tuning a broader range of models, including those that have not participated in debate. our multimodal protocol for scalable oversight is informed by research on computational argumentation, which studies how to automatically assess the quality of an argument (e.g., is it persuasive or reasonable, with valid premises?). @xcite present a taxonomy of all major quality dimensions for assessing natural language arguments, while @xcite develop a computational model that can specifically determine whether an argument is sufficient (i.e., whether its premises provide enough support to justify its conclusion). we draw on this work to develop the criteria used by our blind judge to appraise the arguments raised during a debate. the image depicts a scene involving a group of people in a boat traveling along a body of water, possibly a river or canal. the boat is mainly white with a red outer deck and appears to be . . . the image depicts a small, red and white boat on a body of water, likely a canal or river, as it steers through a narrow path surrounded by brick walls and greenery. the boat is occupied by several passengers who are standing. . . question: is the blue umbrella under the black umbrella? please answer yes or no. figure 1 : the visual question answering task (top) and examples of debate and consultancy (bottom). in both protocols, the judge does not have access to the image, but only to a transcript of the debate and image descriptions generated by the expert models (shown next to the image). different experts and their descriptions are color-coded. ## multimodal debate in this section, we introduce our debate protocol for multimodal visual question answering (vqa; @xcite , which has emerged as a key task for evaluating the ability of vision-language models to understand images. vqa models aim to accurately respond to questions about various aspects of visual content, ranging from fine-grained perception to mathematical reasoning and optical character recognition. we explore debating protocols for eliciting truthful answers from expert models using non-expert llms in this setting, and further utilize the arguments of the debate as training data for instilling reasoning in vision-language models. disagreement sets given a set of expert models, we first determine the samples over which they debate. in contrast to previous work @xcite , where either (or both) of the debaters are assigned an answer to defend, irrespective of their beliefs, our debaters only argue for answers they believe to be true. the previous setting essentially boils down to role-playing and does not scale easily when there are more candidate answers to a question than just two options @xcite . as shown in algorithm 1, for dataset d, we determine all samples @xmath0, m j ) @xmath1, with x being a tuple (q, i) consisting of question q and image i. subsequently, we run debate and consultancy matches (see definition below) only on these sets. debate we illustrate our debate protocol in figure 1 (bottom left). given an image and a question about its content, let m i and m j denote two expert vision-language models that disagree with each other, i.e., they provide different answers y(m(x)). the two models debate, having access to the question q, image i, and the guidelines of the debate. after completing n rounds, a blind judge j with access to the transcript t n , adjudicates the debate to seek the "correct" answer. the transcript t n is a list ∀1≤@xmath2, r kj )] of responses r from ex-algorithm 1 disagreement set extraction for dataset d 1: ds ← ∅ {disagreement set for d} 2: m ← set of models 3: for @xmath3: ds ← ds ∪ {disagreement} 9: m ← m \ {mi} 13: end for 14: return ds perts m i , and m j following n rounds. to generate responses for the k th round, where @xmath4, an expert m i has access to the input and responses from previous rounds from both experts. as a result, r @xmath5, x). we use a text-only model as our judge j to mimic its lack of expertise for vision-intensive tasks. j provides judgement ψ @xmath6, q, d i , d j ) where d i and d j are image descriptions of the image i in sample x from experts m i and m j respectively (see top right in figure 1 ). we denote the judge's prediction (i.e., answer to the question) as y @xmath7. our debate protocol is akin to earlier work @xcite , in which two textbased expert models debate the answer to a question about a story, while a separate judge makes a final decision solely based on the transcript of their debate (without access to the story). however, it is important to note that in their long-form question answering task, the experts were able to expose parts of relevant text to the judge using a citing mechanism. in our case, the blind judge validates the arguments provided by each expert based on image descriptions d offered by each expert and the transcript of the debate. ## experimental setup datasets we evaluate the protocols discussed in the previous section on three datasets representative of a variety of multimodal skills: mme @xcite , mmmu @xcite , and mathvista @xcite . mme contains yes/no questions designed to assess vision-language models on a diverse set (14 in total) of perception and cognition tasks. mathvista evaluates models for visual understanding and compositional reasoning. it is derived from 28 existing multimodal datasets evaluating math solving capabilities in vision-language models @xcite . mmmu @xcite is a benchmark created to evaluate multimodal models on complex tasks that require college-level subject knowledge across multiple disciplines and deliberate reasoning. we evaluate models on the validation sets released by these benchmarks, as the respective test sets are not publicly available. to simplify evaluation in our experiments, we only report results on questions requiring non-free-form answers from mathvista and mmmu. examples from these datasets can be found in appendix f. ## results model performance and disagreements before reporting our results with debate and consultancy, we evaluate expert performance on our three models mmmu mme mathvista molmo-o 38.4 70.1 40.0 molmo-d 41.8 73.2 56.3 llava-1v 46.4 74.6 68.5 qwen2.5-vl 46.8 82.9 70.3 ## conclusion in this work, we introduced a novel multimodal debate framework designed to enhance the reasoning capabilities of vision-language models. by extending the debate (and consultancy) protocol to vqa tasks, we demonstrated how weaker, blind judges can effectively supervise and improve stronger, "sighted" expert models. a key innovation of our approach is the focus on debating instances of expert disagreement, where models defend answers aligned with their beliefs. extensive experiments across diverse multi-modal datasets consistently showed that the debate framework outperforms individual experts and simpler, scalable oversight protocols like consultancy. we further demonstrated its practical utility by extracting reasoning traces from the judge's verdicts and fine-tuning expert models on these which led to improvements in outof-domain settings. our study marks an important first step towards achieving scalable oversight for multimodal ai systems, providing a promising avenue for instilling advanced reasoning capabilities in an unsupervised manner. future work should explore other tasks beyond vqa that require more elaborate responses and a larger number of expert models. more sophisticated learning schemes, involving reinforcement learning, would also be beneficial.