T iny C hart: Efficient Chart Understanding with Program-of-Thoughts Learning and Visual Token Merging (2024)

Abstract: Charts are important for presenting and explaining complex data relationships. Recently, multimodal large language models (MLLMs) have shown remarkable capabilities in chart understanding. However, the sheer size of these models limits their use in resource-constrained environments. In this paper, we present TinyChart, an efficient MLLM for chart understanding with only 3B parameters. TinyChart overcomes two key challenges in efficient chart understanding: (1) reduce the burden of learning numerical computations through Program-of-Thoughts (PoT) learning, which trains the model to generate Python programs for numerical calculations, and (2) reduce lengthy vision feature sequences through Vision Token Merging, which gradually merges most similar vision tokens. Extensive experiments demonstrate that our 3B TinyChart achieves SOTA performance on various chart understanding benchmarks including ChartQA, Chart-to-Text, Chart-to-Table, OpenCQA, and ChartX. It outperforms several chart-understanding MLLMs with up to 13B parameters, and close-sourced MLLM GPT-4V on ChartQA, with higher throughput during inference due to a smaller model scale and more efficient vision encoding.

## introduction as an important information source, charts can intuitively visualize data in various visual presentation forms and have become an indispensable part of information dissemination, business decision-making, and academic research @xcite . automatic chart understanding received increasing attention from the research community @xcite @xcite . recently, multimodal large language models (mllms) have shown strong capability in comprehending images * equal contribution. despite their success, current chart understanding models still face three main limitations: @xcite considerable amount of parameters makes training and deployment challenging. for example, chartllama @xcite has 13 billion parameters, which is hard to deploy on a single gpu with less than 26 gb of vrams. (2) they are prone to errors when tackling questions involving numerical calculations @xcite , which are difficult to answer directly without any reasoning steps. (3) they struggle with efficiently encoding for high-resolution images since the standard vision transformer would produce lengthy feature sequences. to overcome such limitations in chart understanding, we propose an efficient and powerful mllm, namely tinychart. as shown in figure 1 , through the efficient visual encoding and programof-thoughts learning strategy, tinychart achieves state-of-the-art performances on various chart understanding benchmarks with only 3b parameters, while excelling in faster inference throughput. for efficient visual encoding, we propose to merge visual tokens based on the observation that chart images often contain large areas of color and white spaces. inspired by @xcite , we adopt a parameter-free visual token merging module inside each vision transformer layer, which aggregates the most similar visual tokens and gradually reduces the length of the visual feature sequence, this enables controllable computation load when encoding high-resolution images. to learn chart understanding more efficiently, inspired by @xcite , we propose programof-thoughts (pot) learning that trains the model to generate python programs for the computation problems step by step. the programs are then passed to a python interpreter to produce final answers. to support pot learning, we further construct the chartqa-pot dataset based on chartqa @xcite . the qa pairs in chartqa-pot are constructed in two ways: @xcite template-based pot, which generates questions and programs by filling in manually written templates with chart data. (2) gpt-based pot, which leverages gpt-3.5-turbo (openai, 2023a) to produce programs for human-written questions. experimental results show that pot learning can significantly improve the question-answering, especially numerical question-answering ability of tinychart. the main contributions of this work are as follows: ## related work chart understanding requires the model to comprehend chart contents and accomplish related tasks such as data extraction @xcite , qa @xcite @xcite , summarization @xcite , and rerendering @xcite . it demands robust text recognition capabilities and computational reasoning from the model @xcite . early approaches @xcite @xcite @xcite rely on off-the-shelf ocr tools to transform charts into textual representations such as data tables, and use language models to complete specified tasks. these approaches suffer from error accumulation since can not be optimized jointly. recent studies @xcite @xcite @xcite have shifted towards end-to-end methods based on mllms. they enhance mllms' @xcite @xcite @xcite chart understanding abilities through supervised finetuning @xcite on chart instruction data @xcite @xcite . although these models achieves superior performance, the extensive model size prevents them from being easily trained or deployed under resource-constrained scenarios. in this paper, we demonstrate that a 3b mllm is enough to achieve sota performance on chart understanding. meanwhile, it is observed that mllms are prone to numerical errors @xcite . to resolve this, @xcite construct template-based executable command lines for numerical calculations. their usage is constrained by the specific template and backend. in contrast, we train the model to generate python code, which is more versatile. also, we include gpt-generated programs from human-written questions in training data, which further improves the coverage of our method to different questions. multimodal large language models (mllms) exhibit strong capabilities in visual understanding and instruction following @xcite . these models are generally trained on extensive general image-text data for cross-modal alignment and instruction fine-tuning @xcite @xcite @xcite @xcite . some studies @xcite demonstrate a degree of ocr capability in these mllms, their performance on document and chart understanding benchmarks remains sub-optimal due to their low input resolution @xcite . efforts in the general document domain have attempted to improve the fine-grained understanding capabilities of mllms by increasing resolution @xcite , segmenting images @xcite @xcite , utilizing frequency domain signals @xcite , and introducing additional high-resolution encoders @xcite . however, these models often suffer from low efficiency, primarily due to the excessive length of the high-resolution visual sequences. the visual token merging method adopted in this paper can effectively reduce the length of visual sequences and reduce the computational cost of processing high-resolution input. ## conclusion this paper introduces tinychart, a chart understanding mllm with 3 billion parameters. to address the inefficiency of lengthy visual token sequences with high-resolution images, tinychart injects the visual token merging module that merges similar vision tokens, thereby enabling efficient encoding of high-resolution images. to tackle the challenges of learning numerical computations, we propose a program-of-thoughts learning method that trains the model to generate python programs to answer questions. tinychart achieves sota performance on multiple chart understanding benchmarks with fewer parameters and larger inference throughput. extensive ablation studies confirm the effectiveness of our methods.