STL-CQA : Structure-based Transformers with Localization and Encoding for Chart Question Answering (2020)

Abstract: Chart Question Answering (CQA) is the task of answering natural language questions about visualisations in the chart image. Recent solutions, inspired by VQA approaches, rely on image-based attention for question/answering while ignoring the inherent chart structure. We propose STL-CQA which improves the question/answering through sequential elements localization, question encoding and then, a structural transformer-based learning approach. We conduct extensive experiments while proposing pre-training tasks, methodology and also an improved dataset with more complex and balanced questions of different types. The proposed methodology shows a significant accuracy improvement compared to the state-of-the-art approaches on various chart Q/A datasets, while outperforming even human baseline on the DVQA Dataset. We also demonstrate interpretability while examining different components in the inference pipeline.

## introduction charts question answering (cqa) @xcite @xcite is the task designed on the lines of visual question answering (vqa) @xcite which requires answering natural language questions about the data visualisations such as bar charts, pie charts, etc. the problem provides us with ability to understand charts using natural language queries, as well as grounding to the natural language statements for the reasoning operations being carried out to retrieve the final answer to the query. cqa is a challenging task because of the following reasons -(a) large question/answer vocabulary due to chart-specific words, (b) requirements of multi-modal fine-grained reasoning through understanding of natural language question as well as the visualizations. this is different from vqa, where the answer dictionary is typically limited, and the reasoning is coarse-grained as compared to that required for data visualisations, where finer details like bar length and color can heavily influence both the reasoning and the answer. despite data visualisations being ubiquitous in documents, the problem has received sparse attention in the literature. the earlier datasets like dvqa @xcite and figureqa @xcite consist of charts generated from synthetic data, though there has been a push for data charts generated from real sources @xcite as well. due to the problems discussed above, the prior work noted that vqa algorithms cannot be applied directly to cqa. hence, different cqa methods introduce modifications for the problem, while building on the backbone of vqa approaches. while figureqa @xcite uses relational networks for question/answering, dvqa @xcite ) combines text detection and vqabased attention modules to answer chart questions. leaf-qa @xcite encodes question/answers in terms of chart elements, to handle infinite vocabulary problem, while resorting to a vqa-based model as the backbone. though the approaches improve performance for various chart datasets, the challenges of robust reasoning over varied chart varieties, are far from being solved. we posit that this is mainly due to the non-exploitation of the significant characteristics of charts that distinguish them from plain natural images -the structure and set of chart elements. the structure of the charts along with the position of different chart elements must be exploited by the learning models to enable reasoning over them from natural language questions. in this paper, we propose a transformer-based model to exploit such structural properties of data visualisations, while also showing that our model can provide a much deeper and better interpretations to the generated answers. our key contributions can be summarized as follows: visual question answering: the problem of visual question/answering (vqa) has been explored extensively with a variety of datasets @xcite @xcite @xcite with various approaches for joint understanding of images and text. a more closely related work to our problem is, however, textvqa @xcite which focuses on the problem of question/answering with scene texts, having infinite vocabulary. correspondingly, a variety of solutions have also been proposed -the most successful have been based on attention @xcite @xcite and joint multimodal learning @xcite @xcite . pre-training: the success of pre-training with elmo @xcite , gpt @xcite , gpt-2 @xcite , bert @xcite has led to significant advancements in natural language understanding. these pre-training frameworks also motivated some of the recent works on multi-modal understanding @xcite @xcite . our pre-training framework borrows ideas from these works with additional tasks designed specifically for understanding chart structure. to the best of our knowledge, ours is one of the first work to demonstrate the effectiveness of pre-training in inducing structural knowledge of charts. charts questions answering: there has been several works lately addressing the problem of cqa. one line of work relies on using the chart figures and questions directly @xcite @xcite while others @xcite are focused on parsing out the chart data first to perform the task. our approach falls in the first category. apart from chart question answering, there have been works focused on chart data parsing @xcite @xcite or visual structure extraction @xcite . while they do not focus on natural language based understanding of charts, their components form the basis for our structural understanding of charts. ## stl-cqa in this section, we describe our overall framework which is the first method to fully utilize the structural knowledge of charts for both question encoding and reasoning to perform the task of charts question answering (cqa). we refer to our framework as stl-cqa -structure-based transformers with localization and encoding for cqa. even though prior works have attempted to utilize the chart structure for encoding questions, their reasoning frameworks still do not exploit this knowledge resulting in sub-optimal performances and offering much less insight into the reasoning process of these models. we divide our overall framework into three stages -localization, encoding, and transformers-based structural attention. while the first two stages have been adopted from the existing state of the art frameworks, the novel reasoning stage makes our algorithm much more powerful and interpretable as discussed in the later sections. ## experiments we conduct a range of experiments to demonstrate the efficacy of our proposed stl-cqa network. in this section, we describe the different datasets which were used along with the models and the obtained results. we evaluate the proposed stl-cqa method on recent chart question/answering datasets. dvqa @xcite has a large corpus of bar charts and associated question/answers. we use the splits as provided in @xcite for our experiments. we demonstrate that the proposed stl-cqa method outperforms the prior baselines. leaf-qa @xcite , is a comprehensive chart question/answering dataset, covering 10 different types of charts and over 35 question templates. using the publicly available chart annotations foot_0 , we further develop a more comprehensive question/answering corpus, leaf-qa++. the original leaf-qa utilizes automatic paraphrasing of questions to generate variations. we manually curate 3-8 paraphrase variations of question templates to greatly increase the diversity and naturalness of the questions. we further, add new data question types, increasing the number of template questions from 35 to 75. we add data questions, not present in the original corpus, which ask about chart component positional or values. the question set is balanced to avoid pre-dominant values in answers, especially for questions with common chart answers (like yes/no). we refer the reader to the supplementary for further details on the proposed leaf-qa++ corpus. to prepare the data for pre-training, we generate 35 sentence templates for leaf-qa++ using the metadata. such templates are also augmented with a small list for each sentence which provides information about which are the relevant tokens for mlm masking. for each template, we use paraphrases which are written manually and also combine it with the templates of one-another with a probability of 0.5 thus producing a very high number of combinations. ## discussion and limitations while our proposed model is able to reason very effectively achieving state-of-the-art on the recent datasets, it is able to do so with an assumption of perfect ocr. thus, it will be pertinent to have better ocr models for chart images. while reasoning in a fine grained manner has been an important part of cqa, the proposed stl-cqa method shows that reasoning could be performed with a high accuracy, given the elements of the charts have been detected accurately. even though our model achieves near perfect accuracy on the public datasets, the current datasets are synthetic and may not represent the plethora of chart visualisation styles used in real life. despite the significant progress in simulating real world chart understanding scenarios, especially in leaf-qa++, there are underlying biases in the generation process (for e.g. due to using a single software like matplotlib for generation). however, we believe that these are important steps towards the eventual goal of understanding charts in the wild. a further limitation is that the questions used in the existing datasets are template-based. even though we make an attempt in leaf-qa++ to increase the number of templates as well as manual generation of paraphrases to bring more diversity, the current templates do not capture the full range of variations in the questions which can be asked from the visualizations. the manually generated questions will also bring different ways to address the same text on chart images. for example -gross domestic product could be addressed with its more common short form, 'gdp'. the current approach relies on text-string matches to encode questions, and works because the questions have been generated using the original chart text strings. this approach, will however fail in scenarios where the entity could be addressed through its variations. however, bringing human-generated questions into the proposed datasets is a challenge since human subjects would be required to possess a deep understanding of the different charts, before being able to ask reasonable and difficult questions. ## conclusion and future works in this work, we proposed an extension to the leaf-qa data using the public metadata provided for the charts. we also proposed a transformersbased framework while emphasizing on the need to exploit the structural properties of chart, and showed its strong effectiveness by achieving stateof-the-art with a significant margin on the recent chart q/a datasets. we also defined and experimented with a set of pre-training tasks and showed the improvement due to pre-training on the problem of cqa. we used attention to dissect our model to show how each of its module functions to retrieve the final answer. we discussed the current line of cqa work and proposed future directions by outlining the limitations of the current datasets and models.