Program Synthesis for Complex QA on Charts via Probabilistic Grammar Based Filtered Iterative Back-Translation (2023)

Abstract: Answering complex reasoning questions from chart images is a challenging problem requiring a combination of natural language understanding, fine-grained perception, and analytical reasoning. Current chart-based Question Answering (QA) approaches largely address structural, visual or simple data retrieval-type questions with fixed-vocabulary answers and perform poorly on reasoning queries. We focus on answering realistic, complex, reasoning-based questions where the answer needs to be computed and not selected from a fixed set of choices. Our approach employs a neural semantic parser to transform Natural Language (NL) questions into SQL programs and execute them on a standardized schema populated from the extracted chart contents. In the absence of program annotations, i.e., in a weak supervision setting, we obtain initial SQL predictions from a pre-trained CodeT5 semantic parser and employ Filtered Iterative Back-Translation (FIBT) for iteratively augmenting our NL-SQL training set. The forward (neural semantic parser) and backward (language model) models are initially trained with an external NL-SQL dataset. We iteratively move towards the NL query distribution by generating NL questions from the synthesized SQL programs using a Probabilistic Context-Free Grammar (PCFG) where the production rule probabilities are induced to be inversely proportional to the probabilities in the training data. We filter out the generated NL queries with mismatched structures and compositions. Our FIBT approach achieves State-of-the-Art (SOTA) results on reasoning-based queries in the PlotQA dataset yielding a test accuracy of 60.44%, superseding the previous baselines by a large margin.

## introduction charts and plots are compact visualization techniques capturing illustrated facts that are frequently used in scientific and financial documents for summarizing observations and drawing conclusions about the underlying data. inferring relevant conclusions from charts entails answering complex reasoning style queries, a task which has so far proved challenging to automate. most existing approaches and datasets for automatic qa over charts specifically focus on structural, visual, relational or simple data retrieval type queries @xcite @xcite . also, as depicted in we define a common schema for the data across all chart types and employ a state-of-the-art chart visual extractor to populate the schema with chart data. the plotqa dataset does not provide any sql program annotations for the natural language (nl) questions (only answers). we automatically generate sql programs for these nl questions by using a filtered iterative back-translation (fibt) approach @xcite and execute these programs on the extracted schema to compute the answers. we use the spider (nl to sql) dataset to train both the forward and backward models for fibt. the nl query distribution of this dataset is different from the required plotqa query distribution, in terms of query composition, schema (database) structure and chart domains. we build on the observations of @xcite , who empirically show that iterative back translation (ibt) improves the performance of compositional generalization while generating logical forms from nl questions by correcting errors in the pseudoparallel data at each iteration. we define a probabilistic context-free grammar (pcfg), as a subset of the sql grammar, to sample sql programs executable on the extracted chart schema. existing pcfg based data augmentation approaches for semantic parsing @xcite , synthesize sqls following similar compositions to that of a given query set by inducing proportional grammar probabilities. our approach differs from these as we set the probabilities of our pcfg to be inversely proportional to the set-of programs in the training data. this results in synthesis of sql programs that (i) were not present in the current training data and (ii) follow the program distribution of the sql programs needed to compute the answers for the plotqa questions. we iteratively augment the training data by generating nl questions from sql programs. we further use denotations and a novel compositional similarity based filtration strategy for removing noisy nl-sql pairs. we observe that this data augmentation and filtration strategy results in improvement in the plotqa execution accuracy for every iteration of fibt, finally achieving state-of-the-art performance on the reasoning-based queries for the plotqa dataset with a 60.44% test set accuracy, superseding the previous baseline (14.82%) @xcite ) by a large marginand even surpassing human performance. the chartqa model @xcite showed that the t5 language model offers the best overall performance for the questions in plotqa. we demonstrate that our approach surpasses t5 for complex reasoning type of questions in plotqa with oov answers. the main contributions of this work are: ## problem definition our task is defined as follows: given a chart c and a question q on the chart, output a value a that answers the question according to the information represented in the chart. the system has access to a training set d @xmath1, c chart , a chart ) n 1 of questions, charts, and answers. the charts and corresponding questions in test data do not appear during training. we assume availability of a bootstrapping dataset of d @xmath2, q tr , p tr ) m 1 , where s are the database schema, q are the natural language (nl) queries posed on the schema and p are the sql programs corresponding to the nl queries. the domain of the charts c chart can be distinct from the domain of the schema s tr . legend preview y-axis x-axis { "chart_type": "horizontal_bar", "chart_title": "split of the expenses (in %) of different economies in 2012", "legend": { "entries": {"l": {"title": "india", "color": "3501420107"},. ...}} "origin": {"x": 214, "y": 54} "numerical-axis": { "title": "expenses (% of total expenses)", "number": { "n": {"vale": "0", "tick": {"x": 211, "y": 54}}},....} "category-axis": { "title": "types of expenses" "categories": {"c": {"title": "compensation of employees"}, "bars": {"bar": {"color": "3501420107", "tick": {"x": 221, "y": 154}, "bar-top" {"x": 306, "y": 154}}}. ## approach we follow a two stage approach. in the first stage, we use a computer vision pipeline to extract chart and store the chart data in a format (schema) s chart , common across all the chart types. in the second stage, we synthesize sql programs for the nl questions in the dataset by using filtered iterative back-translation (fibt). we use sql as the target logical form for program synthesis because it's grammar (i) is well suited to the tabular structure of our extracted data and (ii) includes the primitive operations required to be handled for the nl queries in plotqa. also, sql is close to nl and easy to understand, allowing for interpretability. ## results and discussion for sampling the representative questions and sql queries from the dataset and pcfg respectively, we use the random seed of 7. the training details of the chart extraction modules are provided in appendix d. we use test accuracy as the evaluation metric, where for numeric answers with floating point values, we consider an answer to be correct if it is within the 5% range of the correct answer as followed by @xcite . as per the definition provided by the authors of plotqa, the fixed vocabulary comprises of the set of top 1000 frequently occurring answer words. our approach yields superior performance for fixed vocabulary (fv) and open vocabulary (ov) answers. both fv and ov answers are numerical. prior approach of end-to-end model predicting the final answer directly (t5 @xcite ) and approach which address queries with distinct answer types distinctly (plotqa @xcite ), learn to distinguish the queries having yes/no (binary) type of answers from other queries. once this is learnt, any random guess of yes/no as an answer to these queries would lead to a performance of 50%. on the other hand, our approach trains a single model to generate sql programs for all queries with distinct answer types. for some cases the generated sql program for questions with binary answers does not yield a binary result. thus, for our approach, the random accuracy for yes/no queries is not 50%. moreover, the sql programs for questions with yes/no answers are more complex as compared to the sql programs which leads to numerical answers (fv and ov questions) in terms of entailed compositions of primitive operations involving nesting, leading to harder synthesis. also, our approach does not yield good performance for some of these questions with binary answers as there is no explicit mapping between the phrases in the nl query and the primitive operations involved in the sql program. for example, for the questions: 'do a majority of the years between 2013 and 2010 (inclusive) have a number of secure internet servers greater than 1.16? or 'do the payments made towards primary income monotonically increase over the years?' or 'is the payments made towards goods and services strictly less than the payments made towards primary income over the years ?' , the model finds it harder to learn to map the abstract phrases 'majority of' or 'monotonically increasing' or 'strictly less than' to a composition of primitive operations in the corresponding sql programs as this knowledge is not explicitly provided. these are the reasons that the performance of our approach for queries with yes/no answer type is inferior as compared to the other reasoning queries. apart from plotqa @xcite , charqa @xcite is the only other dataset available. after thorough analysis of this dataset, we observed that the dataset contains samples with incorrect ground truth labels, spurious questions and gold data tables with incorrect information. the details of our analysis is provided in appendix g. hence, we have not used charqa @xcite for benchmarking our approach. to understand the impact of errors from vision based chart schema extraction on the downstream reasoning task, we perform an ablation to calculate the test accuracy of the reasoning task using the schema constructed with oracle extractions provided by plotqa. we observe a 24.05% lower test accuracy with the extracted schema as compared to the oracle. we further analyze the results based on operators involved in the query. we observe that our approach works well not only for reasoning questions involving one primitive operators, but also for more complex questions involving composition of numerical operators such as (i) 'count values greater than average' (for example, for how many years, is the payments made towards primary income greater than the average payments made towards primary income over all years ?) or (ii) 'sum greater than max' (for example, is the sum of the payments made towards goods and services in 2008 and 2010 greater than the maximum payments made towards primary income across all years ?) or (iii) difference greater than difference' (for example, reasoning type of query mentioned in we further observe that our approach does not yield good results for nl questions involving nesting in the sql program. for example, (i) queries computing a 'difference' between the 'max-imum' and the 'second maximum' values of the numerical column (for example, the reasoning type of query mentioned in ## conclusion we present an approach for qa on charts which addresses complex reasoning based questions that require a combination of natural language understanding, fine-grained perception, and analytical reasoning. we employ a pretrained semantic parser and fibt we generate sql programs for the nl questions without any program annotations. our novel pcfg based approach helps the model to adapt to the given dataset's query compositions and domains, unseen in the bootstrapping data. as the future work we plan to extend our approach fur-ther to handle complex questions requiring nesting by using a hierarchical or grammar based program search technique.