C hart QA : A Benchmark for Question Answering about Charts with Visual and Logical Reasoning (2022)

Abstract: Charts are very popular for analyzing data. When exploring charts, people often ask a variety of complex reasoning questions that involve several logical and arithmetic operations. They also commonly refer to visual features of a chart in their questions. However, most existing datasets do not focus on such complex reasoning questions as their questions are template-based and answers come from a fixed-vocabulary. In this work, we present a large-scale benchmark covering 9.6K human-written questions as well as 23.1K questions generated from human-written chart summaries. To address the unique challenges in our benchmark involving visual and logical reasoning over charts, we present two transformer-based models that combine visual features and the data table of the chart in a unified way to answer questions. While our models achieve the state-of-the-art results on the previous datasets as well as on our benchmark, the evaluation also reveals several challenges in answering complex reasoning questions.

## introduction data visualizations such as bar charts and line charts have become popular in analyzing data and making informed decisions. to analyze data, often people ask complex reasoning questions about charts involving arithmetic and logical operations @xcite . answering such questions requires a significant amount of perceptual and cognitive efforts as people need to combine multiple operations such as retrieving values, comparing values, finding maximum, calculating sums and differences of values. for example, the question q1 in fig. 1 requires the user to compute the differences between the two lines for each year and find the year with the highest difference. the goal of a chart question answering (chartqa) system is to help users by taking a chart and a natural language question as input and predicting the answer. this task differs from other qa tasks such as qa on texts @xcite and tables @xcite because the input for chartqa is a visual representation of data that can draw a reader's attention to various prominent features such as trends and outliers @xcite . also, people tend to ask questions by referring to visual attributes of marks. for example, in fig. 1 , q2 refers to the color of a mark ('line') and its attribute ('peak') in the chart. while the task of chartqa has received growing attentions in recent years, existing datasets have several major limitations: (i) the questions are generated automatically using pre-defined templates @xcite @xcite which lack naturalness, (ii) the charts are created automatically using a programming tool like matplotlib @xcite which do not reflect the diverse styles of many real-world charts, and finally, (iii) in most datasets, the answer comes from a small fixed sized vocabulary (e.g., chart axis labels, 'yes', 'no'), ignoring many complex reasoning questions where the answer is derived through various mathematical operations such as aggregation and comparison. since most datasets only support fixed vocabulary questions, existing models usually treat the task as a classification problem and rely on dynamic encoding techniques with the questions and answers encoded in terms of spatial positions of chart elements (e.g., x-axis-label-1). such ap-proaches do not work when the ocr model generates errors or when the question refers to chart elements using synonyms (e.g., us vs. united states). plotqa @xcite attempts to support open vocabulary questions by applying a tableqa model @xcite but it does not consider any visual features of a chart which are critical for answering visual reasoning questions. to address these limitations, we present a largescale benchmark covering 9,608 human-written questions focusing on logical and visual reasoning questions. since human annotations are costly, we also generated another 23,111 questions automatically from human-written chart summaries using a t5 model @xcite and manually validated a subset of it for quality assurance. in this way, we collect a large number of questions automatically while maintaining rich variations in language as they were generated from human-written summaries. our benchmark consists of 20,882 charts which are curated from four different online sources to ensure variety in visual styles and topics. to address the challenges introduced in our benchmark, where many questions involve complex reasoning and visual references to charts, we propose an approach that combines visual features and extracted data from the chart image. our pipeline first extracts the underlying data table from the chart image by adapting the chartocr model @xcite as well as the visual features from the chart image using neural models. then, we adapt two transformer-based qa models where we utilize both the extracted data table and visual features of the chart in a unified way. our models achieve the state-of-the-art results, or stands on par with the previous models on the previous datasets as well as on our newly created benchmark. in sum, our main contributions are: (i) a large-scale chartqa dataset with real-world charts and human-authored question-answer pairs; (ii) a pipeline approach that combines visual features and automatically extracted data from charts to utilize in transformer-based qa models that provide state-of-the-art results; and (iii) an extensive analysis and evaluation of the performance of our models. our code and dataset are publicly available at https://github.com/vis-nlp/chartqa 2 related work existing datasets chartqa differs from previous datasets in two main aspects: the questions' types (human-authored vs. template-based) and the chart source (real-world vs. generated using a tool). a detailed comparison is shown in existing models there are two main approaches for chart qa. the first approach uses classificationbased visual qa models that can only handle fixedvocabulary questions @xcite @xcite @xcite . these models use encoders to encode the question and the chart image and an attention mechanism to combine the features of both the question and chart before applying a classification layer. these models mostly utilize dynamic encoding techniques to encode the question in terms of the positional information of the textual elements in the chart image that are prone to ocr noise. the second approach applies table qa methods by either assuming that the data table of the chart is given @xcite or by extracting it from the chart image using vision techniques @xcite . ## conclusion we present chartqa, a new large-scale benchmark with human-written questions focusing on visual and logical reasoning. we also introduce a new approach that combines visual features and extracted data table from a chart to answer questions. while our evaluation highlights the promise of this approach, it also reveals several unique challenges emerge from the visual and logical reasoning questions asked by human which exhibit the informal, intricate, and nuanced nature of language. we hope that our benchmark will serve as a starting point for others to address these challenges. to ensure the reproducibility of our experimental results, our hyperparameters settings are provided in appendix a.5. our models can be abused to mislead the public about the charts content and implications. while our models provide state-of-the-art results on most of the existing datasets, we can not guarantee that their output will be correct all the time. thomas wolf, lysandre debut, victor sanh, julien chaumond, clement delangue, anthony moi, pierric cistac, tim rault, rémi louf, morgan funtowicz, and jamie brew. 2019. huggingface's transformers: state-of-the-art natural language processing. corr, abs/1910.03771. yuxin wu, alexander kirillov, francisco massa, wan-yen lo, and ross girshick. 2019. detectron2. https://github.com/facebookresearch/ detectron2 .