D e P lot: One-shot visual language reasoning by plot-to-table translation (2023)

Abstract: Visual language such as charts and plots is ubiquitous in the human world. Comprehending plots and charts requires strong reasoning skills. Prior state-of-the-art (SOTA) models require at least tens of thousands of training examples and their reasoning capabilities are still much limited, especially on complex human-written queries. This paper presents the first one-shot solution to visual language reasoning. We decompose the challenge of visual language reasoning into two steps: (1) plot-to-text translation, and (2) reasoning over the translated text. The key in this method is a modality conversion module, named as DePlot, which translates the image of a plot or chart to a linearized table. The output of DePlot can then be directly used to prompt a pretrained large language model (LLM), exploiting the few-shot reasoning capabilities of LLMs. To obtain DePlot, we standardize the plot-to-table task by establishing unified task formats and metrics, and train DePlot end-to-end on this task. DePlot can then be used off-the-shelf together with LLMs in a plug-and-play fashion. Compared with a SOTA model finetuned on more than thousands of data points, DePlot+LLM with just one-shot prompting achieves a 29.4% improvement over finetuned SOTA on human-written queries from the task of chart QA.

## introduction multimodal reasoning on visual language such as plots and charts is an extremely complex task. for downstream tasks such as question answering (qa) on plots/charts, a model needs to first extract relevant information from the image, organize them in a sensible manner, then perform reasoning over the entries extracted. previous studies have proposed end-to-end solutions to such methods @xcite . whilst being an effective solution, end-to-end methods need to be finetuned on large amounts of task data and they still lag behind on queries that require complex reasoning even after finetuning. as an example, the current sota model matcha @xcite ) achieves only 38.2% accuracy on chartqa @xcite (human written queries). in the meantime, large language models (llms) such as gpt-3 @xcite and palm @xcite have demonstrated exceptional few-shot reasoning skills, without requiring expensive human annotations. however, it is an open question how multimodal reasoning tasks could benefit from llms. in this work, we propose to decompose the multimodal visual language reasoning problem into: (1) converting the input plot image to a linearized table and (2) passing the linearized table to llms for one-shot reasoning. the key of the method is a modality conversion module called deplot that maps charts and plots to the underlying data table. while there has been prior works in chart information extraction, they are usually hybrid systems combining complex handdesigned rules, ocr, keypoint detection, and object segmentation modules @xcite @xcite . for different types of charts, distinct approaches have been used @xcite . besides, there does not exist a unified, consistent, and accurate framework for evaluating chart information extractionmetrics specific to certain types of charts @xcite or overly-simplified number matching metrics @xcite have been used. our proposed deplot is an end-to-end image-to-text transformer model trained with the task of plotto-table translation. a combination of synthetic and web-crawled charts and plots and their underlying data table are collected as the training corpus. header: models |augmented-set|human-set row 1: visiontapas |67.2 |22.2 row 2: pix2struct |82.9 |30.4 row 3: matcha |89.0 |38.0 row 4: deplot |91.0 |67.6 flanpalm: let's find the average of deplot across the two sets. we add the two numbers in the deplot row and divide by 2: 91.0+67.6=158.6/2=79.3. let's find the second strongest model across the two sets. we find the row with the second highest average: row 3. we find the average of matcha across the two sets. we add the two numbers in the matcha row and divide by 2: 89.0+38.0=127.0/2=63.5. we subtract the second strongest average from the strongest average: 79.3-63.5=15.8. the answer is 15.8. ## background plug-and-play of multimodal pretrained models. numerous large pretrained models, either for cross-modal tasks such as clip @xcite , or single-modal tasks, such as gpt-3 and palm, have been introduced in the past few years. these pretrained models' strong zero/few-shot inference capabilities have enabled creative solutions to more complex multimodal tasks. socratic models @xcite combine multimodal pretrained models using multimodal prompts for tasks such as multimodal assistive dialogue and robot perception & planning. minds' eyes @xcite converts physical reasoning queries into code that could be executed in physical engines. magic @xcite inserts visual control using clip in text generation models for unsupervised image captioning. similar our work, @xcite also translates natural images into texts and leverage gpt-3 for knowledge-based vqa. however, all above approaches focus on natural images and the tasks of interest usually only require capturing very basic visual information such as types of objects. visual language reasoning poses a different set of challenges from natural image reasoning -it requires, first, accurate and detailed information extraction (ie) from complex visual language data (plots and charts in this work); and secondly very strong numerical reasoning skills to answer queries based on information extracted. while end-to-end fully-supervised models struggle to answer complex human-written queries, de-plot when combined with llms can outperform the supervised sota by 29.4%. this is achieved by decomposing the two key challenges in visual language reasoning into leveraging two strong pretrained models that excel at their own respective tasks. zero & few-shot reasoning over tables. traditionally, table reasoning tasks are dominated by end-to-end neural models with table-specific architectural designs @xcite @xcite . recently, there has been a surge in using llms to process tables for downstream tasks such as qa. @xcite shows that with just one-shot in-context demonstration, gpt-3 could reach near sota performance on table qa datasets, on par with end-to-end models trained with at least thousands of training examples. beyond pure llm approaches, binder @xcite , program of thoughts @xcite , and program-aided language models @xcite all combine llms with compilers/program executors for table reasoning tasks and have achieved sota performance. deplot can be combined with pure llms and also any of the aforementioned neural-symbolic methods in a plug-and-play style. information extraction from plots and charts. prior works on plot/chart ie is usually pipelinebased, combining ocr, object detection/segmentation systems, and hand-crafted rules. such specialized systems are frequently designed for specific types of graphs, e.g., @xcite for line graphs, and @xcite for bar plots. chart-bert @xcite adopts an ocr-based method for text extraction from charts and uses two more stages of neural methods for processing the extracted texts. chartocr @xcite ) is a hybrid system that accepts all types of chart inputs and has been adopted by downstream task models for chart qa @xcite and summarization @xcite . deplot, as an end-to-end neural model, outperforms chartocr by very large margins on plot-to-table conversion. beyond methodology, the evaluation of plot data extraction tasks has traditionally been ununified. siegel et al. (2016); luo et al. (2021 ); kato et al. (2022) design different metrics for different types of charts and the metrics can be defined upon coordinates, bounding boxes, or keypoints of the graphs' objects. however, this measures only the intermediate steps of the data extraction process rather than the quality of data extraction itself. we formulate chart data extraction as a plot-to-table translation task since the ultimate goal of chart ie is obtaining the underlying data table. besides our work, masry et al. (2022) also considers chart ie as plot-to-table conversion. however, the metric used in masry et al. (2022) is a number set matching metric, ignoring table structure (i.e., correct organization of the extracted numbers). we propose a better table comparison metric and discuss more in §3. ## standardizing the plot-to-table task to perform visual language reasoning, we propose to decompose a visual language reasoning task on plots into two steps: (1) converting plots to texts (in the form of linearized tables) using deplot and (2) inputing the linearized table to llms for reasoning. accurately performing plot-to-table translation is essential for the downstream visual language reasoning tasks. plot-to-table is also an important task standalone as it addresses ie from plots/charts, which can benefit applications such as automatic reports and documents digitization. we will standardize the plot-to-table conversion task in §3.1 and propose a new metric for evaluating plot-totable conversion quality. then in §3.2, we introduce the deplot model and training procedure for performing plot-to-table conversion. ## prompting llms for reasoning with deplot introduced in §3, we can convert a given chart/plot into its textual form (as a linearized table). we can then construct textual prompts by concatenating the linearized tables and the questions for qa tasks. we follow the typical in-context learning paradigm to prepend a one-shot example before the current prompt. the full prompts use either chain-of-thoughts (cot) @xcite or program-of-thoughts (pot) @xcite and can be seen in appx. §c. they are slightly modified versions of the ones used by chen (2023) and @xcite for evaluating reasoning on tabular data. besides cot prompting, we also explore combining deplot+llm with self-consistency (sc) @xcite , which samples a diverse set of reasoning paths and choose the majority-voted answer instead of relying on one greedily-decoded answer as in cot. in order to simplify performing arithmetic on large numbers, we also tested prompting the models to generate python code that can be passed through an interpreter. in order to do that, we adapt the paradigm from @xcite ; @xcite to the context of tables. future work could alternatively take advantage of finetuned tabular qa models such as @xcite or use llms that generate sql programs @xcite and might require multiple llm iterative invocations to perform different atomic operations. ## experiment we introduce the experimental setup in §5.1 and then the results in §5.2 including both plot-to-table translation and downstream qa tasks. ## analyses and discussions in this section, we first conduct case studies and error analysis in §6.1 to better understand deplot' wins and losses against end-to-end methods. then in §6.2 we study the performance of deplot when exposed to out-of-distribution web charts and plots. ## conclusion we have proposed deplot+llm, a method for visual language reasoning by decomposing the task into two steps. the first step is converting a plot into linearized table using an image-to-text transformer model finetuned for the conversion. the second step is combining the plot-to-text model with an off-the-shelf llm to reason on the linearized table with just one-shot supervision. we standardize the plot-to-table conversion task by proposing a new table similarity comparison metric that considers the structure of the table and the numeric values but is invariant to column/row permutation. with the new metric, we compare our image-to-text model deplot's performance with an ocr-based baseline and three end-to-end baselines, achieving the best improvement. the conversion model is then used for downstream tasks of chartqa and plotqa. on chartqa human-query set, the one-shot deplot+llm model achieves +29.4% performance compared with end-to-end sota finetuned with thousands of examples. we have also conducted comprehensive analyses to understand the wins and loses of the deplot+llm framework and highlight that encoding visual attributes can be a fruitful direction for future exploration.