Enhancing Question Answering on Charts Through Effective Pre-training Tasks (2024)

Abstract: To completely understand a document, the use of textual information is not enough. Understanding visual cues, such as layouts and charts, is also required. While the current state-of-the-art approaches for document understanding (both OCR-based and OCR-free) work well, a thorough analysis of their capabilities and limitations has not yet been performed. Therefore, in this work, we addresses the limitation of current VisualQA models when applied to charts and plots. To investigate shortcomings of the state-of-the-art models, we conduct a comprehensive behavioral analysis, using ChartQA as a case study. Our findings indicate that existing models particularly underperform in answering questions related to the chart’s structural and visual context, as well as numerical information. To address these issues, we propose three simple pre-training tasks that enforce the existing model in terms of both structural-visual knowledge, as well as its understanding of numerical questions. We evaluate our pre-trained model (called MatCha-v2) on three chart datasets - both extractive and abstractive question datasets - and observe that it achieves an average improvement of 1.7 % over the baseline model.

## introduction understanding and extracting insights from charts and plots is a fundamental aspect of data analysis that is critical for various domains, including finance, healthcare, and scientific research. to bridge the gap between raw data and actionable knowledge, question answering (qa) systems tailored for charts and plots have gained increasing attention in recent years. these systems aim to enable users to pose natural language questions about the content of visual data representations, such as bar graphs, line charts, and scatterplots, and receive informative answers. however, despite * work done during summer internship at bloomberg. the remarkable progress made in developing such qa systems, there remains a significant challenge: their performance often falls short when subjected to human-generated questions. this discrepancy between machine performance and human expectations underscores the need for a comprehensive investigation into the limitations of existing models and the development of strategies to address their shortcomings. to shed light on the shortcomings of current chart-based qa systems @xcite , this paper first undertakes a detailed checklist-based behavioral analysis @xcite @xcite . choosing the models trained on the chartqa dataset @xcite for the representative case study, we systematically evaluate model responses against examples constructed from the checklist of expected behaviors, allowing us to pinpoint the areas in which these models falter. the checklist is designed to assess various dimensions of chart-based question answering, including the ability to interpret the structural and visual context of the chart, handle questions requiring numerical reasoning, and offer meaningful insights that align with human expectations. concretely, our analysis reveals that current stateof-the-art models perform poorly on two types of questions. the first type of questions are those that pertain to the visual aspects of a chart (e.g., color), while the second type are questions that require application of numerical operators to numerical items present in the chart (e.g., average, etc.). to address these two shortcomings, we propose a set of three pre-training tasks: visual-structure prediction, summary statistics prediction, and numerical operator prediction. through evaluation on three chart question answering datasets, we find that models fine-tuned after using this pre-training outperform the baseline model by more than 1.7 percentage points in an absolute sense. in summary, our contributions are: 1) we per-form a checklist-based behavioral analysis of the current state-of-the-art chart question answering systems to identify issues and challenges faced by such systems. 2) we propose three simple, yet effective, pre-training tasks to address these issues. the new pre-trained model outperforms the baseline systems significantly. ## behavioral analysis via checklist we first describe the procedure for constructing the checklist for chart-based question answering systems and then discuss the results and observations. ## related work numerous studies have highlighted various robustness challenges in nlp models, including their over-sensitivity to minor perturbations @xcite as well as undersensitivity to large changes @xcite . a prominent method for identifying these issues is through behavioral analysis using specifically designed checklist examples @xcite . in this work, we build upon this approach by applying it to multimodal chart-reasoning models, developing a checklist that aids in identifying similar robustness concerns. tackling these robustness challenges presents a distinct set of difficulties. although increasing model size has resolved some of these issues @xcite , this approach is not always ideal. in this study, we demonstrate that certain robustness problems can be mitigated by designing targeted pre-training tasks. for instance, we introduce a pre-training task where the model predicts the visual structure of the input chart, which improves its ability to answer questions related to colors and plot types. similar to our work, unichart @xcite explores pre-training a multimodal model to perform both low-level tasks (e.g., extracting visual elements) as well as highlevel tasks (e.g., chart summarization). ## conclusion in this work, we present a detailed study to evaluate the chart understanding capabilities of two current state-of-the-art qa models. we propose a detailed checklist that can be used to high-light the current shortcomings of these models. broadly, we evaluate end-to-end qa models like matcha and pipeline based models like deplot + llm. these help us identify avenues of improvement. using them, we show that adding relevant pre-training tasks improves the performance of the model to achieve performance improvements across three datasets. across the three tasks and datasets we consider, our pre-training methods help matcha achieve an average improvement of 1.7% points.