R e TAG : Reasoning Aware Table to Analytic Text Generation (2023)

Abstract: The task of table summarization involves generating text that both succinctly and accurately represents the table or a specific set of highlighted cells within a table. While significant progress has been made in table to text generation techniques, models still mostly generate descriptive summaries, which reiterates the information contained within the table in sentences. Through analysis of popular table to text benchmarks (ToTTo (Parikh et al., 2020 and InfoTabs (Gupta et al., 2020) we observe that in order to generate the ideal summary, multiple types of reasoning is needed coupled with access to knowledge beyond the scope of the table. To address this gap, we propose ReTAG, a table and reasoning aware model that uses vector-quantization to infuse different types of analytical reasoning into the output. ReTAG achieves 2.2%, 2.9% improvement on the PARENT metric in the relevant slice of ToTTo and InfoTabs for the table to text generation task over state of the art baselines. Through human evaluation, we observe that output from ReTAG is upto 12% more faithful and analytical compared to a strong table-aware model. To the best of our knowledge, ReTAG is the first model that can controllably use multiple reasoning methods within a structure-aware sequence to sequence model to surpass state of the art performance in multiple table to text tasks. We extend (and open source 35.6K analytical, 55.9k descriptive instances) the ToTTo, InfoTabs datasets with the reasoning categories used in each reference sentences.

## introduction in the task of table to text generation @xcite , the output summaries usually are of two types: descriptive and analytical. a descriptive summary is formed with only information contained within the selected cells of the table and nothing else (refer most table to text models, use structure-aware architectures or pretraining strategies to internally equip the model with one or more types of analyt-ical reasoning capabilities. we draw attention to two practical scenarios where reasoning categories should not be statically baked into a table to text model and more inference time control is needed. 1. dataset diversity: tables like financial charts that consists of pre-dominantly numbers usually need numerical and temporal reasoning. on the other hand, tables like biographic information would need entity and common-sense knowledge more often. since the same model would be used to summarize diverse tables, it becomes essential to be able to pick the appropriate reasoning categories based on the input table. following from the examples above, we argue that explicit control is needed to dynamically pick the reasoning categories. during inference time. to the best of our knowledge, there is no prior literature on table to text summarization with explicit inference-time reasoning control. next, we study some key patterns in the human generated references in totto @xcite and infotabs @xcite . the totto and infotabs validation set have 7700 and 196 tables with at most 3 reference summaries per table. we observe that close to 16.4% and 62.2% of the tables have summaries that use more than 2 reasoning categories simultaneously in totto and infotabs respectively. we term this as multi-category reasoning. such an example is shown in figure 1 . the actual table in the dataset lists the monthly temperatures recorded in a town spin boldak -we show only a sample here for illustration. we observe that to generate the ideal reference from the highlighted cells requires the model to posses two reasoning skills. first, tabular summarization: to compute the maximum and min values over the daily mean field. second, commonsense reasoning to map the notion of maximum to warmest and that of minimum to coolest. therefore, to attain human level accuracy on benchmarks like totto and infotabs a single generative model should have i) the knowledge of multiple reasoning skills ii) the ability to combine more than one skill to summarize the highlighted cells. to the best of our knowledge, there is no prior generative table to text model that is capable of controlling and composing multiple reasoning categories. in this paper, we make the following contributions to address the problems described above. table to text motivated by failure situations that arise in real-world settings. to the best of our knowledge we are the first to propose this problem formulation. (section 3) • we present retag, a vector quantized approcah in encoder-decoder table to text models to generate rich analytical summaries with controllable multi-category reasoning. the use of retag improves the performance of state-of-the-art table to text models on the totto and infotabs by 2.2%, 2.9%, respectively, in the parent metric. (sections 4,5) ## problem formulation in this section, through a systematic human evaluation we first justify the choice of six reasoning categories (five analytical and the descriptive) that we introduce in section 1. next, we proceed to formally state the problem of reasoning-aware table to text generation. ## proposed model we propose our model retag for controllable multi-skill reasoning table to text generation. we embed the two key aspects required to generate analytical sentences: control on reasoning categories, and being reasoning aware. first, to better model each reasoning category with control, we use a vector quantization (vq) strategy in retag ( §4.2 and §4.4). second, to precisely model each reasoning category, we use a pretraining strategy to better learn the reasoning categories from structured tables and free-form text data ( §4.3). for any table-to-text model, a basic property required is the efficient understanding of table structure. to infuse this aspect, we use the pretrained tapex model @xcite as the encoderdecoder backbone in our framework. tapex is a bart @xcite ) model fine-tuned on the task of neural sql execution over tables. authors show that this is an effective pretraining strategy for various table related downstream tasks. it is important to note that our model architecture contributions are not limited to tapex , and thus we integrate the same modules in t5 @xcite based models and benchmark their performance later in the paper. ## experiments we evaluate the performance of retag as follows: (i) performance comparison against strong baselines (ii) ablation study of design choices used in retag (iii) effectiveness of vector quantization (iv) reasoning category wise quantitative analysis and (v) human evaluations for faithfulness and reasoning control. since our primary contribution is on multi-category reasoning, we benchmark retag 's performance for totto valid and infotabs test datasets which have heterogeneous reasoning categories. hence, we do not evaluate against datasets that are specific to one type of reasoning, such as logicnlg, numerical-nlg, etc. we use the following notations for our experiments: given a question q with a linearized table t, we concatenate them to form the input x (as mentioned in section 4.1). the table may contain some highlighted cells, which can be enclosed within special indicators such @xmath0. we assume that the highlighted cells would be indicated in either of these ways within the linearized table t. we use the strategy devised in @xcite for linearizing the table. additionally we are also given the reasoning categories r and the corresponding output sentence y. ## conclusion in this paper, we presented the case for reasoningaware table-to-text models. we introduced retag, a vector quantized approach for encoder-decoder table-to-text models with explicit control across reasoning categories. retag beats sota models for totto and infotabs datasets for analytical and descriptive sentence generation. we will also release close to 35.6k instances of reasoning category tagged analytical abd 55.9k instances of descriptive table to text data.