T a PERA : Enhancing Faithfulness and Interpretability in Long-Form Table QA by Content Planning and Execution-based Reasoning (2024)

Abstract: Long-form Table Question Answering (LFTQA) requires systems to generate paragraph long and complex answers to questions over tabular data. While Large language models based systems have made significant progress, it often hallucinates, especially when the task involves complex reasoning over tables. To tackle this issue, we propose a new LLM-based framework, TaPERA, for LFTQA tasks. Our framework uses a modular approach that decomposes the whole process into three sub-modules: 1) QA-based Content Planner that iteratively decomposes the input question into sub-questions; 2) Execution-based Table Reasoner that produces executable Python program for each sub-question; and 3) Answer Generator that generates long-form answer grounded on the program output. Human evaluation results on the FeTaQA and QTSumm datasets indicate that our framework significantly improves strong baselines on both accuracy and truthfulness, as our modular framework is better at table reasoning, and the long-form answer is always consistent with the program output. Our modular design further provides transparency as users are able to interact with our framework by manually changing the content plans.

## introduction long-form question answering (lfqa) presents a unique challenge in natural language processing (nlp), demanding models that not only comprehend the long context but also maintain factual accuracy during answer generation @xcite . a prevalent and concerning issue is hallucination, where models generate answers that are coherent yet factually incorrect or irrelevant to the input context. this problem undermines the trustworthiness of nlp systems, posing a barrier figure 1 : an example of long-form table qa. answering complex questions with an end-to-end llm-based system poses hallucination challenges (highlighted in red). we propose a modular-based framework that produces a qa-based plan first, followed by generating an answer conditioned on this plan (through generating executable python programs as an intermediate step). to their real-world application in domains where reliability is paramount, such as finance and healthcare. the hallucination issues become even more pronounced when the task involves reasoning over tables, e.g., long-form table question answering (lftqa) @xcite . as illustrated in figure 1 , given the complex question and numerous data points in a table, the system must be capable of understanding the relationships within the data and perform human-like reasoning over the tabular content to compose the paragraphlong answer. this paper introduces tapera, aiming to enhance the trustworthiness of llm-based methods in long-form table qa. as depicted in figure 2 , tapera employs a modular approach comprising three components: (1) qa-based content planner: given a complex question, this module generates and iteratively updates a qa-based content plan. it controls the step-by-step reasoning process to shape the final answer content. (2) execution-based table reasoner: for each subquestion in the qa-based plan, this module generates executable programs (i.e. python programs) to extract and process question-relevant information from tables. (3) answer generator: given the sub-question and the executed output from the table reasoner, this module composes a natural language sub-answer, ensuring consistency with the reasoner's output. moreover, once the entire qabased plan is finalized, it generates the final answer based on the plan and all sub-answers. tapera features several core advancements: (1) modularity. the complex task is decomposed into multiple easier sub-tasks, which can be solved with more tailored solutions (e.g., reasoning enhanced modules). (2) faithfulness the table reasoner first generates an executable python program as external tools to output question-relevant facts, then the answer generator is applied to ensure the consistency between the generated answer and the program output, which improves the faithfulness of generated answer. (3) interpretability. the system allows users for easy identification of which sub-plan or reasoning step leads to an undesired final answer. additionally, our user study demonstrates the potential of applying user interaction to modify the generated plan to control and improve the final answer. we experiment tapera framework on the fe-taqa @xcite and qtsumm @xcite datasets. human evaluation results demonstrate that tapera achieves significant improvements in terms of both accuracy and faithfulness. specifically, tapera surpasses 1) the second-best system in comprehensiveness, i.e. blueprint @xcite , by 5.4%; and 2) the second-best system in faithfulness, i.e. dater @xcite , by 5.0%. our human evaluation results further indicate the inherent limitations in aligning automated metrics and human preference, consistent with previous work @xcite . in addition, we we also conduct comprehensive human studies, where users are allowed to interact with the system by modifying and improving the qa-based plans generated by the content planner. the results reveal that using such user-controlled plans can further improve model performance on those challenging questions. our contributions are summarized below: ## related work table reasoning helps non-expert users interact with complex tabular data and has received significant attention. it has several tasks, such as table question answering (pa @xcite @xcite @xcite , table fact verification @xcite , and table-to-text generation @xcite @xcite . prior work on table reasoning mainly pre-trains models with joint table-text data and then finetunes over specific table-relevant tasks @xcite @xcite @xcite @xcite , and recent work shows llms also achieves competitive results @xcite @xcite @xcite . to further advance the capabilities of llms in table qa tasks, research has primarily focused on two avenues: (1) enhancing llms with code execution, which utilize code-optimized llms to generate executable programs to answer questions over tabular execution-based table reasoner answer generator q1: which company earns the highest profit in the oil and gas industry? a1: [pending] q2: how much profit does that company earn? a2: [pending] q3: which company earns the overall highest profit? a3: [pending] q4: how much profit does that company earn? a4: [pending] q5 what's the difference in profit between the two mentioned companies? a5: [pending] initialization of 2nd-iteration plan q1: which company earns the highest profit in the oil and gas industry? a1: [pending] q2: which company earns the overall highest profit? a2: [pending] q3: compare these two companies. a3: [pending] initialization of 1st-iteration plan qa-based content planner q: which company earns the highest profit in the oil and gas industry, and how does it compare to the most profitable company overall? 1 walmart retail 559,151 @xmath0 2,300,000 united states 2 state grid electricity 386,618 @xmath1 896,360 china 3 amazon retail 386,064 @xmath2 1,298,000 united states 4 national petroleum oil and gas 283,958 @xmath3 1,242,245 china 5 sinopec group oil and gas 283,728 @xmath4 553,833 china 6 apple electronics 274,515 @xmath5 147,000 united states 7 cvs health healthcare 268,706 @xmath6 256,500 united states 8 unitedhealth healthcare 257,141 @xmath7 330,000 united states 9 toyota automotive 256,722 @xmath8 366,283 japan 10 volkswagen automotive 253,965 @xmath9 662,575 germany employees headquarters rank company name industry revenue (@xmath10 million) table title: top 10 company in 2012 q1: which company earns the highest profit in the oil and gas industry? a1: sinopec group earns the highest profit in the oil and gas industry. q2: which company earns the overall highest profit? a2: [pending] q3: compare these two companies. a3: [pending] st ep 2: st ep -b y-st ep an sw er ea ch su bqu es ti on w it hi n th e pl an q1: which company earns the highest profit in the oil and gas industry? a1: sinopec group earns the highest profit in the oil and gas industry. q2: which company earns the overall highest profit? a2: apple earns the overall highest profit. q3: compare these two companies. a3: [pending] q1: which company earns the highest profit in the oil and gas industry? a1: sinopec group earns the highest profit in the oil and gas industry. q2: which company earns the overall highest profit? a2: apple earns the overall highest profit. q3: compare these two companies. a3: apple is in the electronics industry, while sinopec group is in the oil and gas industry. step 1: pass sub-questions to answer step 3: fulfil plans with sub-answers q1: which company earns the highest profit in the oil and gas industry? a1: sinopec group earns the highest profit in the oil and gas industry. q2: which company earns the overall highest profit? a2: apple earns the overall highest profit. q3: compare these two companies. a3: apple is in the electronics industry, while sinopec group is in the oil and gas industry. complete 1st-iteration plan step 4: update or finalize the plan execution result for each sub-question within the plan, generate executable python program data @xcite @xcite or to locate relevant table regions conditioned on a given question @xcite . while this approach is effective for deriving a single fact from tables for short-form answers, it struggles in lftqa tasks that requires a more complex information synthesis. (2) question decomposition, which breaks down questions into more fine-grained components, thereby simplifying the table reasoning required @xcite . however, existing methods rely on a single-round question decomposition, which might not always provide sufficient evidence to address the main question in lftqa. content planning for text generation content planning is effective in capturing and controlling both the content and structure of generated text, particularly for longer outputs @xcite @xcite . the process of plan-based text generation typically involves two distinct stages: planning an outline, and subsequently filling in the specific details. previous work on table-to-text generation typically represents plans as a sequence of phrase keywords, events, and their interactions @xcite @xcite , aiming to improve the coherence and narrative flow of generated text. however, such plan representations are not ideally suited for lftqa tasks, especially in the era of llms. this is because the primary objective of lftqa is to extract and present question-relevant information accurately, rather than creating a narrative flowan aspect where llms already excel @xcite . this work adapts the idea of qa-based plan @xcite @xcite , utilizing question-answer pairs as intermediate plans for lftqa tasks. our work extends beyond previous work on qa-based plans to table-relevant tasks and uses llms to iteratively generate and refine the plans based on the feedback (i.e., answer) from table reasoning module. automatic evaluation of lfqa one primary challenge for lfqa lies in accurately evaluating the long-form generation. recent work @xcite @xcite indicates that there is no existing automatic text generation metrics are predictive of property fetaqa qtsumm human preference judgments. we find the same issue when evaluating tapera in lftqa, therefore, we mainly use human evaluation for results. in this section, we first formulate the lftqa task, and then discuss the datasets used in our work. ## long-form table qa with tapera in this section, we discuss our proposed framework tapera for lftqa (figure 2 ). solving lftqa requires complex table reasoning and consistent generation, both are challenging for existing llm-based systems, and these challenges are more pronounced when they are entangled. therefore, we propose a modular framework @xcite that decomposes the complex task into multiple sub-tasks, and adopt multiple sub-modules to tackle each sub-task. more specifically, our framework contains three main components: a qa-based content planner, a execution-based table reasoner, and an answer generator (algorithm 1). given a complex question q, we generate a qa-based plan that includes a sequence of sub-questions (section 4.1). then for each subquestion, we first apply table reasoner to generate an executable program to obtain question-relevant facts as program output (section 4.2). subsequently, the answer generator derives a subanswer from the program output (section 4.3). however, due to the complexity of the task, we expect our framework might fail with a single attempt @xcite . thus, we adopt an iterative refinement approach that repeats the above process until the content planner finalizes a confident qa-based plan. the answer generator generates the final answer based on the finalized plan and sub-answers. ## experiment we next discuss the baselines, human and automated evaluations, implementation details and main experimental results. ## human analysis of tapera to obtain a deeper insight into the effectiveness and potential of tapera, we conduct a human analy- question type representative question description open-ended questions summarize the basic information of the episode(s) written by damon lindelof these questions require a comprehensive summary and often have initial plans that lack clarity or specificity. subjective analysis how did the performance of tom brady in terms of passing yards during the regular season 2011 compare with other quarterbacks listed in 2011? these questions involve personal opinions or interpretations, often comparing and contrasting different elements. ambiguous queries who were the top three scorers for the 1961-62 michigan wolverines men's basketball team and how many points did they score? queries that are not clearly defined or have multiple interpretations, leading to vague or inaccurate responses. ## conclusion this work proposes a new modular based framework tapera for lftqa, which decomposes the complex qa task into three sub-tasks: qabased content plan generation, execution-based table reasoning and consistent answer generation. human evaluation on two benchmark datasets indicate that tapera outperforms strong baselines in both comprehensiveness and faithfulness. this is because tapera has stronger table reasoning capacities, and our final answer is consistent with the table reasoning module outputs. we believe this work provides insights for future work in advancing controllable text generation in complex tasks that are often entangled with reasoning challenges.