M ^3 C o T : A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought (2024)

Abstract: Multi-modal Chain-of-Thought (MCoT) requires models to leverage knowledge from both textual and visual modalities for step-by-step reasoning, which gains increasing attention. Nevertheless, the current MCoT benchmark still faces some challenges: (1) absence of visual modal reasoning, (2) single-step visual modal reasoning, and (3) domain missing, thereby hindering the development of MCoT. Motivated by this, we introduce a novel benchmark (M ^3 CoT) to address the above challenges, advancing the multi-domain, multi-step, and multi-modal CoT. Additionally, we conduct a thorough evaluation involving abundant MCoT approaches on Vision Large Language Models (VLLMs). In addition, we highlight that the current VLLMs still struggle to correctly reason in M ^3 CoT and there is a large gap between VLLMs and human performance in M ^3 CoT, despite their superior results on previous MCoT benchmarks. To our knowledge, we take the first meaningful step toward the multi-domain, multi-step, and multi-modal scenario in MCoT. We hope that M ^3 CoT will serve as a valuable resource, providing a pioneering foundation in multi-domain, multi-step, multi-modal chain-of-thought research.

## introduction recent advancements in large language models (llms) have led to notable improvements in chainof-thought (cot) in textual modality @xcite @xcite . in addition, some works begin to extend the textual cot capabilities to multi-modal cot reasoning (mcot). take figure 1 (c) as an example, multi-modal cot requires both the visual and textual features to generate a rationale and a final answer. to this end, @xcite introduced scienceqa benchmark and laid the foundation for mcot. subsequently, @xcite proposed a two-stage approach during multi-modal reasoning for mcot. additionally, @xcite developed t-sciqa framework to distill high-quality rationales from chat-gpt, which attains an average accuracy of 96.2%, surpassing even the human accuracy of 88.4%. inspired by the recent remarkable advancements in the mcot literature (surpassing human performance), we seek to explore an interesting question: has mcot task been solved perfectly? in our deep analysis, the conclusion is definitely "no". as shown in figure 2 (a), we observe that current benchmarks are too simple, leading to an overestimation of current progress. furthermore, we find that the existing benchmarks exhibit three major drawbacks (see figure 2 geometry3k @xcite 3,002 2,342 ✘ ✔ ✘ ✘ ✘ tqa @xcite 26,260 3,455 ✘ ✘ ✔ ✘ ✘ mathvista @xcite 5,487 6,141 ✘ ✔ ✘ ✘ ✘ mme @xcite 2,194 1,097 ✘ ✘ ✔ ✘ ✘ seedbench @xcite -19,242 ✘ ✘ ✔ ✘ ✘ mm-vet @xcite 205 187 ✘ ✔ ✔ ✘ ✘ vcr @xcite 290k 99,904 ✘ ✘ ✔ ∼4% ✔ a-okvqa @xcite 24,903 23,692 ✘ ✘ ✔ ∼21% ✔ ki-vqa @xcite 4,290 4,189 ✘ ✘ ✔ ∼17% ✔ scienceqa @xcite 21 ability of multi-modal cot model. (2) single-step visual modal reasoning: as illustrated in figure 1 (b), the model only requires a single-step "feather" object to predict the correct rationale and answer, which cannot be satisfied in the complex multi-step cot scenario. motivated by these observations and issues, we introduce a novel benchmark about multi-domain multi-step multi-modal chain-of-thought reasoning (m 3 cot) based on scienceqa @xcite . specifically, to address the first issue, we directly remove samples that could infer the final answer without the need for images. to tackle the second issue, we manually annotate and select multistep multi-modal samples. specifically, we provide expert annotators with textual context and rationales without images. experts are required to determine when multi-step reasoning cannot be resolved solely based on textual context. subse-quently, we present the images to experts to ascertain whether multi-step reasoning occurred across textual and visual modalities. to solve the third issue, we explore llm-guided augmentation to synthesize the multi-step mcot data for commonsense and mathematics domains. we evaluate abundant representative mcot approaches on m 3 cot in extensive scenarios, yielding several key takeaways: in conclusion, the primary contributions of our work are summarized as follows: to facilitate further research, all data and code are available at https://github.com/lightchen233/ m3cot . ## problem formalization this section describes the definition of multi-step multi-modal cot. specifically, unlike the traditional textual cot, multi-step multi-modal cot should consider a scenario involving an image i, a question q, a context c and a set of n options @xmath0, ..., o n }. first we construct a textual prompt t : finally, the model arrives at the final answer y, which is denoted as: ## dataset annotation this section describes the annotation process of m 3 cot, including: absence of visual modal reasoning sample removal ( § 3.1), multi-step mcot 1 all step segmentation in this paper follows the roscoe @xcite . ## data analysis this section provides some detailed data analysis to better understand m 3 cot. basic statistics m 3 cot is partitioned randomly into three subsets: train, validation, and test splits, containing 7,863, 1,108, and 2,358 samples, respectively. compared to scienceqa, m 3 cot demands more intricate reasoning, with an average length of 294, much higher than scienceqa's 48. ## related work chain-of-thought (wei et al., 2022b) (cot) is a highly effective step-by-step strategy for enhancing zero-shot and few-shot reasoning in large language models (llms) @xcite @xcite @xcite . in addition, some works begin to extend the textual cot capabilities to multi-modal cot reasoning (mcot) @xcite @xcite . better commonsense reasoning evaluation. compared with previous works, we first propose m 3 cot to explore the multi-step mcot and extend their application across a broader range of domains. in addition, we conduct comprehensive experiments on m 3 cot and highlight some takeaways, facilitating future research. ## conclusion in this work, we introduce a novel benchmark (m 3 cot), toward multi-domain, multi-step, and multi-modal chain-of-thought scenarios, which is developed through a detailed and comprehensive process. in addition, we conduct a comprehensive analysis involving abundant multi-modal cot methodologies to understand the limitation of existing frameworks on m 3 cot. we sincerely aspire that our work can reassess existing advancements and inspire future research by highlighting new challenges and opportunities.