MMC : Advancing Multimodal Chart Understanding with Large-scale Instruction Tuning (2024)

Abstract: With the rapid development of large language models (LLMs) and their integration into large multimodal models (LMMs), there has beenimpressive progress in zero-shot completion of user-oriented vision-language tasks. However, a gap remains in the domain of chartimage understanding due to the distinct abstract components in charts. To address this, we introduce a large-scale MultiModal ChartInstruction (MMC-Instruction) dataset comprising 600k instances supporting diverse tasks and chart types. Leveraging this data, we de-velop MultiModal Chart Assistant (MMCA), an LMM that achieves state-of-the-art performance on existing chart QA benchmarks. Recognizing the need for a comprehensive evaluation of LMM chart understanding, we also propose a MultiModal Chart Benchmark (MMC-Benchmark), a comprehensive human-annotated benchmark with nine distinct tasks evaluating reasoning capabilities over charts.Extensive experiments on MMC-Benchmark reveal the limitations of existing LMMs on correctly interpreting charts, even for the mostrecent GPT-4V model. Our work provides an instruction-tuning methodology and benchmark to advance multimodal understanding ofcharts. Code and data are available at https://github.com/FuxiaoLiu/MMC.

## introduction large language models (llms) such as gpt-3, palm, chatgpt, bard, and llama @xcite @xcite @xcite have undergone rapid development, demonstrating significant capabilities in performing a wide range of tasks effectively. to enable llms with vision ability, open-source large multimodal models (lmms) such as minigpt-4 @xcite , llava @xcite , mplug-owl @xcite , multimodal-gpt @xcite , and lrv @xcite have been developed, incorporating advanced image understanding capabilities into llms to interpret and analyze visual inputs. while successful in the general domains, such open-source lmms are less effective for chart images because chart understanding differs tremendously from natural scene image understanding. in contrast with natural scene images, which primarily contain objects and reflect their spatial relationships, chart images contain unique abstract elements, including trend lines and color-coded legends that convey specific data-related information. current open-source lmms are limited in their ability to accurately interpret complex chart contents, as they often lack domain-specific training essential for tasks such as differentiating between various types of graphs, interpreting axis labels and data points, and extracting meaningful patterns and trends. integrating advanced chart understanding capabilities could further refine the lmms' ability to analyze contextually and reason about the information presented in charts, thereby broadening their applicability in fields like data analytics, academic research, and business intelligence. in this paper, we introduce multimodal chart instruction (mmc-instruction), a 600k chart understanding dataset consisting of both chart-text alignment data and chart instruction-tuning data. mmc-instruction is not only much larger but also more diverse compared to existing public datasets @xcite @xcite . unlike previous work with templated-based questions, mmc-instruction is constructed by prompting @xcite to generate instructions with diverse language styles and tasks (tab. 1). furthermore, our mmc-instruction considers a variety of chart types, including but not limited to histograms, scatter plots, area charts, and more complex graphical representations. by performing unified instruction tuning datasets fig. num question ans. type ans. length plot type task num benchmark to accurately assess the capabilities of current large multimodal models (lmms) for chart understanding, we introduce a novel comprehensive evaluation tool: the multimodal chart benchmark (mmc-benchmark). first, mmc-benchmark is the first human-annotated benchmark in line with human cognition to evaluate lmm's ability to comprehend visual charts. second, it contains a wide range of tasks, including chart information extraction, chart reasoning, contextual chart understanding, chart topic classification, stock chart analysis, multiple chart understanding, chart type classification, chart-to-datatable and chart-tojson. third, mmc-benchmark offers two quantitative evaluation methods, including free-format generation ability evaluation using gpt-4 and multiple-choice qa format chart understanding ability evaluation without the requirement of gpt-4. our evaluation highlights the limitations of existing open-source lmms. in addition, we further broaden our analysis through experiments with gpt-4v @xcite @xcite , the latest multimodal version of gpt-4. our experiments indicate that mmc-benchmark also poses significant challenges to gpt-4v, especially in chart to datatable and chart to json tasks. it indicates the importance of mmc-instruction corpus and mmc-benchmark in advancing multimodal understanding. our main contributions are as follows: ## mmc-benchmark the recent progress of lmms has enabled the open-ended zero-shot completion of user-oriented vision-language tasks such as open-ended chart understanding. as a result, a comprehensive evaluation benchmark is necessary to evaluate the performances of different lmms on these tasks and provide quantitative guidance for future research and development. however, for chart understanding, existing benchmarks often fall short of evaluating open-ended questions and unbounded chart types. our dataset, mmc-benchmark, is therefore motivated to bridge this gap, offering three unique characteristics for chart understanding: ## multimodal chart assistant (mmca) architecture. our model mmca (fig. 2 ) is built on mplug-owl @xcite ) that guides llms to follow multimodal instructions. in order to improve the existing llms to perform better on chart understanding tasks, we further fine-tune mplug-owl 7b @xcite ) on our proposed mmc-instruction corpus consisting of chart-text alignment data and chart instruction-tuning data. mplug-owl contains a pre-trained visual foundation model (clip vision encoder), a visual abstractor, and a language foundation model (vicuna). the visual foundation model is responsible for extracting visual features from the input images, and the visual abstractor distills these features using a set of learnable tokens. the resulting visual features are combined with the word embeddings of the input sentence and fed into the language model to generate the response. we incorporate a twostage training paradigm. stage-1: chart text alignment. in this stage, we freeze the language decoder and train the visual parts with our chart-text alignment data for one epoch. this stage enables the mapping of visual features of charts to llm's word embedding space. stage-2: chart instruction tuning. in the second stage, we freeze the visual abstractor, visual encoder, and language decoder and use the lowrank adaptation approach (lora) @xcite to fine-tune the language model. specifically, we train the language model with lora on our chart instruction-tuning data for three epochs. this stage enables llm's instruction following capabilities for chart understanding. ## conclusion this paper aims to tackle the challenge of chart understanding with large multimodal models (lmms). firstly, we present a large-scale chart instruction-tuning dataset mmc-instruction, including diverse topics, language styles, chart types, and open-ended answers in line with human cognition. secondly, we introduce a human-annotated benchmark called mmc-benchmark to evaluate llms' abilities for chart understanding quantitatively. finally, we propose an instruction-tuned lmm called mmca that outperforms existing open-source sota methods.