Multimodal Self-Instruct: Synthetic Abstract Image and Visual Reasoning Instruction Using Language Model (2024)

Abstract: Although most current large multimodal models (LMMs) can already understand photos of natural scenes and portraits, their understanding of abstract images, e.g., charts, maps, or layouts, and visual reasoning capabilities remains quite rudimentary. They often struggle with simple daily tasks, such as reading time from a clock, understanding a flowchart, or planning a route using a road map. In light of this, we design a multi-modal self-instruct, utilizing large language models and their code capabilities to synthesize massive abstract images and visual reasoning instructions across daily scenarios. Our strategy effortlessly creates a multimodal benchmark with 11,193 instructions for eight visual scenarios: charts, tables, simulated maps, dashboards, flowcharts, relation graphs, floor plans, and visual puzzles. This benchmark, constructed with simple lines and geometric elements, exposes the shortcomings of most advanced LMMs like GPT-4V and Llava in abstract image understanding, spatial relations reasoning, and visual element induction. Besides, to verify the quality of our synthetic data, we fine-tune an LMM using 62,476 synthetic chart, table and road map instructions. The results demonstrate improved chart understanding and map navigation performance, and also demonstrate potential benefits for other visual reasoning tasks.

## introduction in recent times, spurred by breakthroughs in large language models (llms) @xcite @xcite @xcite @xcite ; anthropic, 2024; @xcite , large multimodal models (lmms) have also undergone rapid advancements @xcite @xcite @xcite . leveraging a pre-trained llm to encode all modalities empowers lmms to understand human daily environments and execute complex tasks @xcite @xcite @xcite @xcite . this greatly expands the potential of general-purpose ai assistants. despite these achievements, lmms still exhibit significant deficiencies when deployed in human daily life @xcite . for instance, lmms often fail when planning a route using a road map, reading the time from a clock image, or interpreting a flowchart. we observe that these simple daily activities require lmms to understand abstract images, such as maps, charts, and dashboards, rather than natural photographs or portraits with explicit semantics. these abstract images composed of simple geometric elements are more challenging for lmms. furthermore, even many advanced lmms are easily stumped by sim-road map relation graph question: how much food waste has increased from 2000 to 2009 answer: 4.2 rationale: 33.7 -29.5 gpt-4v: 4.2 … increased from 29.5 million tonnes in 2000 to 33.7 million tonnes in 2009… table question: today, i set off for another city 100 kilometers away. the clock shows my departure time, and the speedometer displays my car's speed. when will i arrive at the destination? answer: 10:00 am. rationale: your departure time is 8:00 am, and your car's speedometer shows 50 km/h. the distance is 100 km. therefore, you need to spend two hours on the road, and you will arrive at destination at 10:00 am. gpt-4v: you will arrive at 9:00. the clock shows 7:00, which is your departure time.. ## experiments first, we evaluate the performance of many leading lmms using our benchmark containing all tasks in section 4.2. next, we perform instruc ## conclusions we observe that current lmms perform suboptimally in perceiving and reasoning with abstract images, often failing at simple daily tasks. therefore, we design a multimodal self-instruct strategy, enabling llms to autonomously synthesize various diagrams, instrument dashboards, and visual puzzles using code, and self-propose reasoning q&a. we synthesized 11k data to benchmark the current lmms. evaluation results underscore the significant challenges posed by our benchmark. we also synthesized 62k chart and road map training instructions to fine-tune a llava-7b, enhancing its chart interpretation and map navigation abilities.