BBA : Bi-Modal Behavioral Alignment for Reasoning with Large Vision-Language Models (2024)

Abstract: Multimodal reasoning stands as a pivotal capability for large vision-language models (LVLMs). The integration with Domain-Specific Languages (DSL), offering precise visual representations, equips these models with the opportunity to execute more accurate reasoning in complex and professional domains. However, the vanilla Chain-of-Thought (CoT) prompting method faces challenges in effectively leveraging the unique strengths of visual and DSL representations, primarily due to their differing reasoning mechanisms. Additionally, it often falls short in addressing critical steps in multi-step reasoning tasks. To mitigate these challenges, we introduce the Bi-Modal Behavioral Alignment (BBA) prompting method, designed to maximize the potential of DSL in augmenting complex multi-modal reasoning tasks. This method initiates by guiding LVLMs to create separate reasoning chains for visual and DSL representations. Subsequently, it aligns these chains by addressing any inconsistencies, thus achieving a cohesive integration of behaviors from different modalities. Our experiments demonstrate that BBA substantially improves the performance of GPT-4V(ision) on geometry problem solving (28.34% \to 34.22%), chess positional advantage prediction (42.08% \to 46.99%) and molecular property prediction (77.47% \to 83.52%).

## introduction the use of domain-specific language (dsl) @xcite @xcite aims to incorporate multimodal information by providing a precise and unequivocal alternative form using text. 1 its application has significantly improved the multimodal reasoning ca-pability, yielding notable improvements in intricate contexts, especially within specialized domains such as symbolic reinforcement learning @xcite @xcite and diverse scientific fields @xcite . multimodal reasoning is a fundamental capability for large vision-language models (lvlms) @xcite , crucial for many of their applications. despite the considerable progress made by lvlms in multimodal tasks @xcite , effectively utilizing them for complex multimodal reasoning, particularly in conjunction with dsls, remains underexplored. the most direct approach is to feed the lvlms with both visual data (e.g., images) and its corresponding dsl representation along with the textual queries. they are then guided through the chain-of-thought (cot) @xcite prompting to process step-by-step reasoning. however, a significant issue with this approach is that the reasoning processes derived from different modalities are often inconsistent, or even conflicting. this inconsistency limits the ability of lvlms to effectively integrate the strengths of visual and dsl representations ( §2.1). moreover, these models encounter difficulties in executing multi-step reasoning @xcite , which hampers their effectiveness in addressing critical steps within complex problems ( §2.2). to address these challenges, we propose a bi-modal behavioral alignment (bba) prompting method that adeptly integrates dsl into complex multimodal reasoning tasks. bba begins by prompting lvlms to generate distinct reasoning chains from both visual and dsl representations, and then aligns these chains by resolving inconsistencies, thereby harmonizing the behaviors elicited from various modalities. bba offers two primary advantages. firstly, it adopts a "late fusion" strategy @xcite , effectively maintaining the inherent strengths of both the direct vision input and the dsl representation. secondly, bba turns the inconsistency across modalities into a beneficial signal that aids in identifying critical steps within reasoning processes. by revealing where the reasoning chains differ, it efficiently allocates more intermediate tokens to these critical steps by resolving the inconsistencies found. we evaluate bba on three multimodal reasoning tasks: geometry problem-solving, chess positional advantage prediction, and molecular property prediction. in these diverse applications, bba demonstrated notable relative improvements, with respective performance improvements of 14.26%, 10.25%, and 6.30%. ## pilot study in this study, we compare three variants of cot prompting within domains where dsl is available. these variations include: (1) cot v , which utilizes only images for grounding responses to queries; ## method this work aims to tackle two primary challenges in multi-modal reasoning: (1) the integration of the inherent strengths of both visual and dsl representations, and (2) the identification and resolution of critical steps within these tasks. to address these challenges, we introduce the bba prompting method, an innovative approach that seeks to unleash the power of dsl in enhancing complex multi-modal reasoning tasks. figure 2 offers an overview of our proposed methodology. bba initiates by employing lvlms to generate reasoning chains separately from visual and dsl inputs. subsequently, these chains proceed through an alignment phase, wherein inconsistencies are identified question: find the area of the irregular pentagon. and reconciled, ensuring the harmonization of behaviors derived from each modality. road map. the rest of this section is structured as follows: we begin by detailing the process of eliciting reasoning chains from both vision and dsl representations in §4.1. this is followed by an elaboration on diagnosing and rectifying inconsistencies between these reasoning chains and the methods of aligning behaviors from different modalities in §4.2. lastly, in §4.3, we detail how bba effectively identifies and addresses critical steps in the reasoning process. ## conclusion in conclusion, our work introduces the bi-modal behavioral alignment (bba) prompting method, a novel approach that significantly enhances the multimodal reasoning capabilities of gpt-4v(ision) by integrating dsl. by generating and aligning separate reasoning chains for visual and dsl representations, bba addresses the challenges of inconsistent reasoning mechanisms and the execution of multi-step reasoning tasks. our experiments across diverse domains, including geometry problem-solving, chess positional advantage prediction, and molecular property prediction, demonstrate the effectiveness of bba, showcasing notable improvements in performance.