C hart I nstruct: Instruction Tuning for Chart Comprehension and Reasoning (2024)

Abstract: Charts provide visual representations of data and are widely used for analyzing information, addressing queries, and conveying insights to others. Various chart-related downstream tasks have emerged recently, such as question-answering and summarization. A common strategy to solve these tasks is to fine-tune various models originally trained on vision tasks language. However, such task-specific models are not capable of solving a wide range of chart-related tasks, constraining their real-world applicability. To overcome these challenges, we introduce ChartInsruct: a novel chart-specific vision-language Instruction-following dataset comprising 191K instructions generated with 71K charts. We then present two distinct systems for instruction tuning on such datasets: (1) an end-to-end model that connects a vision encoder for chart understanding with a LLM; and (2) a pipeline model that employs a two-step approach to extract chart data tables and input them into the LLM. In experiments on four downstream tasks, we first show the effectiveness of our model–achieving a new set of state-of-the-art results. Further evaluation shows that our instruction-tuning approach supports a wide array of real-world chart comprehension and reasoning scenarios, thereby expanding the scope and applicability of our models to new kinds of tasks.

## introduction information visualizations, such as bar charts and line charts, play a pivotal role in data analysis, offering critical insights and aiding in informed decision-making @xcite . however, discerning key patterns and trends from these visualizations and addressing complex queries can pose significant challenges. recent research has introduced various tasks to assist users in chart analysis, including chart question answering @xcite , summarizing insights from visualizations (obeid and @xcite , reasoning over chart images, fact-checking @xcite , and automated visual data storytelling @xcite . early work attempts to tackle these tasks by finetuning models originally trained on language and vision tasks @xcite @xcite . however, such models may not be optimal for chart-specific tasks as they overlook explicit modeling of chart structures such as relationships between chart elements like bars, legends, and axes. recent models such as unichart @xcite , chart-t5 @xcite , and matcha @xcite are specifically designed for charts by considering visual and mathematical reasoning over chart elements and values. however, they often consider charts from a limited range of sources and focus on a narrow set of tasks, constraining their real-world applicability. indeed, for real-world widespread adoption, we cannot presume how and on what tasks these models will be used. a promising solution to this challenge is instruction tuning, as demonstrated by language models like instructgpt @xcite , @xcite , alpaca @xcite , @xcite and llama-chat @xcite . they show that training llms on instruction-following datasets significantly enhances their alignment with user intent across various tasks, including tasks that are unseen during training. recent advances in vision-language tasks @xcite ) have adopted similar methodologies, fine-tuning vision-language models (vlms) with visual instructions to better match user intentions and improve efficacy. however, to our knowledge, instruction tuning for chart comprehension and reasoning remains underexplored. existing methods @xcite , which are concurrent to our work, lack variety and scope in instruction-tuning tasks, limiting their effectiveness in real-world chart understanding scenarios. in this paper, we introduce chart instruction tuning (chartinstruct), to pave the way towards building general-purpose chart comprehension and reasoning assistant based on vlms. to this end, we have developed a new chart instruction-tuning dataset featuring real-world charts collected from 157 online platforms, covering wide and diverse visual styles. leveraging advanced llms such as @xcite , @xcite , and gemini @xcite )), we generate 191k instructions covering a broad array of tasks reflecting real-world applications (figure 1 ). as charts are unique and pose challenges distinct from general multi-modal data, a structured approach is crucial for enabling vlms to effectively leverage the instruction dataset, optimizing their performance in chart analysis tasks. we introduce two innovative vlm designs in this regard. the first system modifies the llava architecture @xcite , substituting its clip vision encoder with the unichart @xcite vision encoder pre-trained specifically on chart images. for language modeling in this design, we experiment with two different models: llama2 (7b) -a decoder-only model @xcite and flan-t5-xl (3b) -an encoder-decoder model @xcite . our second design adopts a two-step pipeline approach: first extract the underlying data table from the chart image, then provide it as input to the llm. this range of models provides a spectrum of efficient solutions, making our systems adaptable to various real-world scenarios and computational demands. our comprehensive evaluation across four benchmarks: chartqa @xcite , chart2text (obeid and hoque, 2020b), opencqa @xcite , chartfc @xcite demonstrates our models' state-of-the-art performance in chart understanding and reasoning tasks. human evaluation further suggests the effectiveness of our instruction-tuning approach in supporting a wide array of real-world chart comprehension and reasoning scenarios, broadening its adaptability to numerous new tasks. our main contributions include: (i) a new instruction-following corpus with real-world charts and a wide range of tasks by utilizing llms, (ii) two distinct systems specifically tailored for chart understanding tasks; (iii) extensive evaluations that demonstrate the state-of-the-art performance of chartinstruct across existing chart-related benchmark tasks while also expanding its applicability to new tasks. we have made our code and chart corpus publicly available at https://github.com/visnlp/chartinstruct . ## chart instruction data generation we build an instruction tuning dataset for enhancing vlms' capabilities in tackling diverse understainding and generation tasks related to chart analysis. in this section, we describe the chart corpus collection followed by the instruction tuning data generation process. figure 2 provides an overview of the instruction tuning process. ## modeling in this section, we describe our two architectures for chart instruction tuning. ## conclusion we present chartinstruct, an automatically generated dataset of chart-related instructions and two instruction systems designed for a broad range of chart-related tasks. to the best of our knowledge, this is the first instruction tuned dataset that not only includes pre-defined tasks but also many new types of tasks automatically distilled by llms. our model sets the state-of-the-art performance on four different downstream tasks on various automatic measures while the human evaluation further confirms the effectiveness of our approach on many new kinds of tasks. we believe that our models and instruction-tuning dataset will be valuable resources for future research and encourage further exploration into the unique problem domain of chart understanding and reasoning.