Unveiling Implicit Table Knowledge with Question-Then-Pinpoint Reasoner for Insightful Table Summarization (2024)

Abstract: Implicit knowledge hidden within the explicit table cells, such as data insights, is the key to generating a high-quality table summary. However, unveiling such implicit knowledge is a non-trivial task. Due to the complex nature of structured tables, it is challenging even for large language models (LLMs) to mine the implicit knowledge in an insightful and faithful manner. To address this challenge, we propose a novel table reasoning framework Question-then-Pinpoint. Our work focuses on building a plug-and-play table reasoner that can self-question the insightful knowledge and answer it by faithfully pinpointing evidence on the table to provide explainable guidance for the summarizer. To train a reliable reasoner, we collect table knowledge by guiding a teacher LLM to follow the coarse-to-fine reasoning paths and refine it through two quality enhancement strategies to selectively distill the high-quality knowledge to the reasoner. Extensive experiments on two table summarization datasets, including our newly proposed InsTaSumm, validate the general effectiveness of our framework.

## introduction table data has emerged as pivotal repositories of knowledge in facilitating data analysis, offering concise and structured representation of information. as comprehending the complex table can be time-consuming for human, text generation systems that can accurately summarize a provided table have the potential to greatly enhance the process of obtaining data insights. in the task of table summarization @xcite @xcite , a straightforward solution is to use neural model as an end-to-end summary generator. however, the model struggles to capture all necessary information in an end-to-end approach. the problem lies in title: kyle' s football career statistics ... despite a brief successful season after the team transfer, his performance remain inconsistent and it seems that his injury impacted his goal performance throuhgout his career. ... there are some correlation between the injury and his goal performance. after moving to another team, his goal scoring performance get peak during the 2008-09 season observing the fluctuating numbers of "goals" column, there are some significant variations with respect to the teams and injury. target summary implicit table knowledge based on title and header, "player' s performance trends" and "factors affecting performance" are important aspect from the table. the table is about career statistics of a football player : coarse-level aspects : fine-level insights 2003 a.utd x 2 1 0 3 3 x o o x x x a.utd a.utd b.fc c.city c.city 2004 2005 2006 2007 2008 2009 2010 2011 2012 season team injury 0 o 2 0 o o goals ... implicit table knowledge 7 9 b.fc b.fc b.fc a.utd figure 1: an example of implicit table knowledge that should be unveiled from explicitly stated table cells to generate the target summary. a line of research address this uncontrollability problem by injecting knowledge as a mediator to guide what kind of table contents should be stated in the text description. most works adopt symbolic operations as guidance for sampling knowledge from the table to enhance the logical reasoning ability of the model. these symbolic operations include executable programs which mimics sql @xcite or logical types @xcite which categorize information seeking queries in several predefined types, serving as a control for knowledge collection. some works @xcite use table as query for retrieving relevant knowledge from external source (e.g., kb) to supplement the insufficient table knowledge of the language model. however, these approaches suffer from limited knowledge in terms of both diversity and coverage. since symbolic operations heavily rely on rule-based sampling or predefined types, they remain insufficient for delivering comprehensive information within individual tables. additionally, relying on external knowledge sources requires a rigorous assumption about the completeness of the knowledge base, which can lead to limited coverage of the required knowledge. in this paper, we aim to unveil the implicit table knowledge by using the reasoning ability of the large language models (llms). as shown in figure 1 , the implicit knowledge required for generating the target summary should encompass multiple aspects of information that are scattered across table cells. we argue that by facilitating llms to directly mining these knowledge from the table with multiple reasoning paths, we can represent more diverse knowledge that can be used as informative insights to support the table summarization. despite the potential effectivness of this approach, there are two major challenges in using llm as a knowledge miner. (1) low insightfulness: as for the complex nature of structured table, the in-depth knowledge related to the target summary are not explicitly stated in the table cells. however llms tend to focus more on explicit textual cues @xcite which often leads to surface-level realization and making it challenging to capture the insightful knowledge. (2) low faithfulness: since information that needs to be selected from the table to derive the reliable knowldege are scattered and hidden among irrelevant information, these often act as distracting factor or noise @xcite , leading to hallucination. to address the aformentioned challenges, we introduce a novel table reasoning framework, insightfully question then faithfully pinpoint. the key idea of our framework is to build a plug-and-play table reasoner that can self-quesiton-and-answer the insightful knowledge by faithfully pinpointing the evidence on the table to guide the table summarizer. to train the reasoner, we collect training data from teacher llm which follows coarse-to-fine reasoning paths to mine in-depth knowledge required for summarization. during this step, we refine the data using two quality enhancement strategies to selectively distill the high-quality knowledge that is helpful to train a reliable reasoner. for effective demonstration of our framework, we conduct experiments on two table summarization datasets, including our newly proposed dataset instasumm. since our table reasoner can be pluged-and-played to different variants of table summaization models, we validate that our framework can be applied to both fine-tuned and zeroshot summarization models with significant improvement. in addition, we evaluate our framework under out-of-domain setting, showing its robustness in diverse real-world scenarios. ## preliminaries input table serialization following the recent works that employ language model on table-related tasks @xcite , we serialize the table input into a flattened sequence. specifically, we use a vertical bar (|) to separate headers and cells in different columns, and a newline along with the row index to separate rows. this approach enables the direct input of structured tables into the language model. for example, the input table t is serialized as follows: ## experiments we conduct extensive experiments on summarization to demonstrate how the insights from qtp reasoner provide useful guidance to the summarizer in generating high-quality table summaries. ## conclusion in this paper, we propose question-then-pinpoint, a novel table reasoning framework that builds a plugand-play table reasoner providing faithful insights supportive for table summarization. to achieve this, we mine the implicit table knowledge via coarseto-fine reasoning paths and train the reasoner to self-question-and-answer the required knowledge by pinpointing the explicit evidence. we conduct extensive experiments on two different datasets including our newly proposed instasumm, and demonstrate the general effectiveness of our framework compared to the baselines.