U ni RPG : Unified Discrete Reasoning over Table and Text as Program Generation (2022)

Abstract: Question answering requiring discrete reasoning, e.g., arithmetic computing, comparison, and counting, over knowledge is a challenging task.In this paper, we propose UniRPG, a semantic-parsing-based approach advanced in interpretability and scalability, to perform Unified discrete Reasoning over heterogeneous knowledge resources, i.e., table and text, as Program Generation. Concretely, UniRPG consists of a neural programmer and a symbolic program executor,where a program is the composition of a set of pre-defined general atomic and higher-order operations and arguments extracted from table and text.First, the programmer parses a question into a program by generating operations and copying arguments, and then, the executor derives answers from table and text based on the program.To alleviate the costly program annotation issue, we design a distant supervision approach for programmer learning, where pseudo programs are automatically constructed without annotated derivations.Extensive experiments on the TAT-QA dataset show that UniRPG achieves tremendous improvements and enhances interpretability and scalability compared with previous state-of-the-art methods, even without derivation annotation.Moreover, it achieves promising performance on the textual dataset DROP without derivation annotation.

## introduction question answering requiring discrete reasoning, e.g., arithmetic computing, sorting, comparison, and counting, over knowledge is a challenging task and has attracted wide attention. advanced approaches to this task are roughly divided into mixture-of-experts (moe)-based and semantic parsing (sp)-based methods. the former divides questions into several limited categories according to p : (1) note 14 -earnings (loss) per share. as to works based on sp-based methods, they always research over homogeneous knowledge resource, i.e., either unstructured text @xcite @xcite or structured tables and knowledge graphs @xcite @xcite . respectively, they design a specialized semantic parser for each knowledge category according to the corresponding data characteristic. however, in practical scenarios (e.g., financial reports and sports news analysis), it usually requires a model to comprehensively understand heterogeneous knowledge resources (e.g., tables and text) and perform reasoning over them. therefore, to extend sp-based methods to more general and realistic scenarios, it needs a unified framework for discrete reasoning over heterogeneous knowledge resources. meanwhile, to mitigate the program annotation cost, it also needs an automatic approach to produce pseudo programs. to address the above challenges, we propose unirpg, a sp-based approach, to perform unified discrete reasoning over table and text as program generation. specifically, unirpg consists of a neural programmer and a deterministic symbolic program executor, where a program is the composition of a set of pre-defined general atomic and higher-order operations and arguments extracted from table and text. the programmer leverages a structure-aware knowledge reader to encode heterogeneous knowledge, and a program generator to decode programs through generating operations and copying arguments from table and text. during inference, decoding constraints are leveraged to ensure program legality. then, the executor derives answers from table and text based on the program. to alleviate the costly program annotation issue, we design a distant supervision approach for programmer learning, where pseudo programs are automatically constructed without annotated derivations. furthermore, a re-weight strategy is introduced to measure the importance of pseudo programs for noise reduction in training. to verify the effectiveness and generality of unirpg, we conduct comprehensive experiments on the tat-qa @xcite and drop @xcite datasets under w/ and w/o derivation settings. experiment results show that unirpg achieves tremendous improvements and enhances interpretability and scalability compared with previous state-of-the-art (sota) methods on the tat-qa dataset. specifically, it outperforms the previous sota model by 17.0 em and 18.0 f1 on the test set under full supervision. besides, our weakly supervised model based on automatically constructed pseudo-programs only lags behind fully supervised model 1.0 em and 1.4 f1 points. moreover, it also achieves promising performance on the textual dataset drop in weak supervision. in conclusion, the contributions of this work are summarized as follows: (1) we propose an effective program-generation-based framework for discrete reasoning over table and text; (2) to alleviate costly program annotation, we design a distant supervision approach to construct pseudo programs automatically; (3) we conduct comprehensive experiments to verify the effectiveness and scalability of unirpg on the tat-qa and drop datasets. ## discussion interpretability analysis as shown in figure 7 -9, our model generates fine-grained programs that intuitively demonstrate the discrete reasoning process. furthermore, we visually investigate how the neural programmer decodes programs based on questions and hybrid knowledge resources. figure 3 displays the cross-attention score distribution from the last layer of the decoder based on a random sample. the horizontal axis is the input sequence s, and the vertical axis represents the decoded program units at different time steps. we observe that cross-attention visualization reflects the underlying relationships between the question prefix "what is the change" and operation diff. moreover, cv correctly extracts the two numbers 44.1 and 56.7 from the table. ## case study we compare our unirpg with the baseline tagop on 50 instances randomly sampled from the development set of tat-qa, where tagop predicts wrongly, but our model predicts correctly. unirpg has advantages in the following three aspects compared with tagop. (1) capture of argument order. tagop introduces a 2-classifier to learn the argument order with a one-hot order label, ignoring the modeling of the interaction between two arguments. our framework can naturally learn the parameter order of operations because program sequences are inherently ordered. as shown in example (a), our methods can avoid 60% error instances of tagop @xcite due to the wrong argument order. (2) evidence detection. tagop extracts evidence by classifying each token of the input and then predicts answers based on the evidence, which is prone to accumulating errors. we employ a copy mechanism to extract content from the input, which only needs to predict the start and end tokens. example (b) shows our methods reduce 20% error predictions caused by inaccurate evidence compared with tagop. (3) compositional reasoning. as shown in example (c), our model has better scalability that can perform compositional with multiple operations compared to tagop. ## related work hybrid qa hybrid qa aims to derive answers to given questions with heterogeneous knowledge, such as table/knowledge graph and text. many challenging datasets and promising approaches @xcite @xcite @xcite are recently proposed to stimulate the progress of this topic. chen et al. (2020c) proposed hybridqa, in which each question is aligned with a wikipedia table and multiple paragraphs linked with cells in the table. chen et al. (2020b) proposed ott-qa, a large-scale open table-and-text qa dataset that requires retrieval tables and text to derive answers. unlike hybridqa and ott-qa, which study multi-hop reasoning on table and text, we study tat-qa (zhu et al., 2021), a large-scale qa dataset requiring discrete reasoning over hybrid data. finqa (chen et al., 2021) is a recent work similar to ours, aiming to address the task of numerical reasoning over financial data. compared to it, unirpg supports more reasoning capabilities, such as span/cell, which can perform extraction from paragraphs/tables but is not limited p 1 : ... the company has u.s ... p 2 : a reconciliation of the ... p 3 : the entire amount reflected ... tagop: evidence: 1264, 1626 op: differece order: 1 a: 362 ours: g: diff( cv( 29, 32 ) cv( 33, 36 ) ) a: -362 p 1 : american tower and ... p 2 : other operating expense ... p 3 : other operating expenses ... tagop: evidence: impairment charges, net losses on sales or disposals of assets, other operating expenses op: count order: n/a a: 3 unirpg: g: count(cell(41, 49) cell( 59, 61 ) ) a: 2 2019 2018 balance at the beginning of the fiscal year 1264 1626 decreases based on positions taken in prior years 119 304 balance at the end of the fiscal year 1258 1264 2019 2018 impairment charges 94.2 394.0 net losses on sales or disposals of assets 45.1 85.6 other operating expenses 27.0 33.7 q: what is the change in the beginning balance between fiscal years 2019 and 2018? a: -362 d: {solution: 1264-1626, table:[(1,2), (3,2)]} q: how many expenses segments in 2018 were below $100 million? a: 2 d: {table: [(2,0), (3,0)]} p 1 : 24. post employment benefits p 2 : the group operates a... p 3 : accounting policies ... tagop: evidence: 166.0, 57.0 op: differece order: 0 a: -109.0 unirpg: g: diff( avg(cv( 44, 44 ), cv( 45,45 ) ) , avg( cv( 51, 51 ), cv( 52 ,52 ) ) ) a: 121.5 2019 2018 2017 defined contribution schemes 166 178 192 defined benefit schemes 57 44 20 total amount charged to income statement 223 222 212 q: what is the difference between 2019 average defined contribution schemes and 2019 avere defined benefit schemes? a: 121.5 d: {solution: (166+178)/2 -(57+44)/2, table:[(1,1), (1,2), (2,1),(2,2)]} unirpg: g: diff( cv( 29, 32 ) cv( 33, 36 ) ) a: -362 (a) (b) (c) figure 6 : three examples display the prediction results of baseline tagop and our model. in tagop, @xmath0, and order is 0 implies that the order of the two numbers is the same as the order in the input sequence; otherwise, it is 1. to the arithmetic computing problem. this enables unirpg to be available for more discrete reasoning tasks. moreover, the atomic and higher-order operation framework provides a unified paradigm to define operations, which makes unirpg convenient to be adapted to other discrete reasoning tasks. in addition, we provide a weak-supervised approach for discrete reasoning over hybrid context based on question-answer pairs, largely reducing the cost of program annotations. discrete reasoning for qa @xcite proposed drop, a large-scale rc dataset to investigate discrete reasoning over text. many customized models are proposed to tackle this task, and these fall into two dominant directions, the first direction is to devise specific modules for each types of question, such as naqanet @xcite , @xcite , qdgat @xcite , evidr @xcite and opera @xcite . the second direction is to predict programs that would solve the questions @xcite @xcite . it is similar to ours, but our model supports discrete reasoning over hybrid data. ## conclusion in this work, we propose to perform discrete reasoning over table and text as program generation. our framework consists of a neural programmer to parse questions into executable programs and a symbolic program executor to derive answers based on decoded programs. furthermore, we design a distant supervision method to automatically construct pseudo programs to alleviate the cost of manual annotation. comprehensive experiments on the tat-qa and drop datasets demonstrate the effectiveness and generality of our model under both full and distant supervision.