Paper: CABINET: Content Relevance-based Noise Reduction for Table Question Answering (ICLR-2024)

Reviewer comment (Novelty):
The authors need to better motivate what the practical utility of Table QA is, as most of the questions are based in the format of natural language text sentences. Moreover, tabular dataset can be indexed and converted to their freeform text representations which general open-domain QA systems are already able to solve.

Author response:
> "The authors need to better motivate what the practical utility of Table QA is, as most of the questions are based in the format of natural language text sentences."

Table QA is a problem of immense practical utility. Much information is generated, transmitted, and stored in tabular format (e.g., financial documents, enterprise reports, documents, summarized statistics and trends in newspapers and magazines, etc.). Table QA models aim to unlock this vast information by parsing and reasoning over tabular data, and enabling end-users to ask questions about tabular data in natural language, and gain insights about the data in the table conveniently by asking questions without having to go through large amounts of data in the table manually. 

***

> "Moreover, tabular dataset can be indexed and converted to their freeform text representations which general open-domain QA systems are already able to solve."

Indexing and searching over free-from text representations of tabular data is an intuitive first solution. Many Table QA systems, including the proposed CABINET, do utilize these text representations (or verbalizations, as we call them in the paper) in some form. However, these textual representations are insufficient due to the complexity of tabular data (diverse formats and structures, row and column positions, numerical data, etc.). Further, simply retrieving verbalized information can not answer questions that require complex reasoning and aggregation operations -- an area where the proposed CABINET especially shines (Appendix A4).

Further, do note that the SoTA QA systems (open-domain, or tabular) are based on GPT scale LLMs and are typically tens of billions of parameters in size. As discussed in the paper (section 4.1), CABINET significantly outperforms methods built on top of such LLMs despite having orders of magnitude fewer parameters (CABINET contains 560M parameters while models built on ChatGPT/GPT-3 contain 175B parameters).

Reviewer comment (Novelty):
In the Related Work Section, the authors should also discuss recent developments with LLM guided graph neural network (GNN) model for QA to capture tabular/graph structure:

[1] EACL 2023: Question-Answer Sentence Graph for Joint Modeling Answer Selection. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 968–979, Dubrovnik, Croatia. Association for Computational Linguistics.  
[2] AAAI 2020: TANDA: Transfer and Adapt Pre-Trained Transformer Models for Answer Sentence Selection. AAAI 2020: 7780-7788

Author response:
> "In the Related Work Section, the authors should also discuss recent developments with LLM guided graph neural network (GNN) model for QA to capture tabular/graph structure"

Thank you for your suggestion. These works are certainly relevant and we have included the discussion regarding them in the revised draft (in the related work section in the main paper and Appendix A.3).

Reviewer comment (Experiments):
It would be helpful if the authors could better summarize the statistics of their datasets in tabular as opposed to text format. Further, based on the text description, I have a concern that the datasets being used are relatively small-scale, e.g., in the order of thousands of nodes. The experiment results would be more conclusive if evaluated on large-scale datasets. To this end, the authors also should provide analysis on the runtime and memory complexities of their work, since M/B parameter LLMs may not be scalable for training time.

Author response:
> "It would be helpful if the authors could better summarize the statistics of their datasets in tabular as opposed to text format."

Thanks, good suggestion! We had written the statistics of datasets in text format due to the paucity of space in the main paper. We have now modified the pdf of the paper to summarise the dataset stats in a tabular format in Appendix A.9 (Table 9).

***

> "Further, based on the text description, I have a concern that the datasets being used are relatively small-scale, e.g., in the order of thousands of nodes. The experiment results would be more conclusive if evaluated on large-scale datasets."

Regarding the scale of the datasets, we selected the most commonly used publicly available benchmarks for evaluating CABINET that allowed us to compare CABINET uniformly and fairly with other SoTA methods which have used the same datasets for Table QA. The correct way to judge the scale of the datasets, we would argue, is in terms of number of questions and the complexity of the questions. All three datasets provide tens of thousands of questions, with WikiSQL comprising 80K+ questions over 24K tables! Further, FetaQA, which comprises 10K+ questions, requires fetching multiple entities from the table, aggregating and reasoning over these entities, and structuring the inferred information to produce a long, coherent, natural language answer (Section 4). Do also note that each table in many cases contains hundreds of cells (Section 4.3 - impact of table size on performance). Thus, the datasets chosen are among the hardest, publicly available datasets for the task.

***

> "To this end, the authors also should provide analysis on the runtime and memory complexities of their work, since M/B parameter LLMs may not be scalable for training time"

We are completely aligned with efficiency concerns. That’s why CABINET contains only 560M parameters (which comes under the very lightweight LLMs category), and despite that, it is able to outperform much larger ChatGPT/GPT-3 based DATER containing 175B parameters. Hence, at a time when the general trend is to increase the number of parameters, CABINET offers a carefully designed solution keeping practical utility in mind and achieves significantly better performance efficiently with orders of magnitude fewer parameters.

* The average runtime time required during inference for CABINET (averaged over 100 queries) is: 0.212s
* The average runtime required during inference for one of the best-performing baselines - OmniTab is (averaged over 100 queries): 0.153s
* The GPU memory required for CABINET during inference (comprising 560M parameters) is: 2.86 GB
* The GPU memory required for OmniTab during inference (comprising 440M parameters) is: 2.27 GB
* The GPU memory required for CABINET during training per gpu with batch size 8 (comprising 560M parameters) is: 54 GB
* The GPU memory required for OmniTab during training per gpu with batch size 8 (comprising 440M parameters) is: 36 GB

It can be noted that CABINET gives large performance gains at the cost of small incremental increase in inference latency.

Reviewer comment (Reproducibility):
The overall model is quite complex, relying on various subcomponents of different natures: this paper provides a new baseline, but it is hard to build from that.

Author response:
> "The overall model is quite complex, relying on various subcomponents of different natures ... The cell relevance loss is not learned, making the model not end-to-end"

LLMs are sensitive to noise in the input. Hence, one major aspect of the proposed CABINET framework is the ability to identify relevant parts of the table to the input question. This reduces the "noise" in the input to the QA LLM and it can focus on the relevant information for generating the correct answer.

Note that in practical deployment settings, such fine-grained annotations denoting parts of the table relevant to a question are not available. Keeping this constraint in mind, we introduce an unsupervised relevance scorer component trained in an end-to-end manner with the question-answering LLM. Next, to further capture the fine-grained relevance signals, we propose a weakly supervised cell predictor module to highlight relevant cells of the table conditioned on a statement describing how the table should be parsed to answer a given question. Again, owing to the unavailability of fine-grained table data annotations (obtaining which is a very costly process), we make this deliberate design choice of using unsupervised and weakly supervised components to tackle the problem of noisy table tokens and providing highly relevant signals to the QA LLM.

Reviewer comment (Theory):
The cell relevance loss is not learned, making the model not end-to-end.

Author response:
> "The overall model is quite complex, relying on various subcomponents of different natures ... The cell relevance loss is not learned, making the model not end-to-end"

LLMs are sensitive to noise in the input. Hence, one major aspect of the proposed CABINET framework is the ability to identify relevant parts of the table to the input question. This reduces the "noise" in the input to the QA LLM and it can focus on the relevant information for generating the correct answer.

Note that in practical deployment settings, such fine-grained annotations denoting parts of the table relevant to a question are not available. Keeping this constraint in mind, we introduce an unsupervised relevance scorer component trained in an end-to-end manner with the question-answering LLM. Next, to further capture the fine-grained relevance signals, we propose a weakly supervised cell predictor module to highlight relevant cells of the table conditioned on a statement describing how the table should be parsed to answer a given question. Again, owing to the unavailability of fine-grained table data annotations (obtaining which is a very costly process), we make this deliberate design choice of using unsupervised and weakly supervised components to tackle the problem of noisy table tokens and providing highly relevant signals to the QA LLM.

Reviewer comment (Theory):
Section 3.1, the whole paragraph discussing variational inference sounds strange: variational inference is a method to estimate latent variable probabilities - stating that "we model ... as a latent variable ... **through VI**" is not correct.

Author response:
> "section 3.1, the whole paragraph discussing variational inference sounds strange: variational inference is a method to estimate latent variable probabilities - stating that "we model ... as a latent variable ... through VI" is not correct"

Thank you for the suggestion! We agree we could have phrased this better. Recall that we aim to predict the relevance of each table token. However, since ground-truth annotations for relevant table tokens are unavailable, table token relevance is unobserved, and we consider it a latent variable. As you mentioned, variational inference is a method to estimate latent variable probability, and we use it to estimate token relevance probability. Accordingly, we have now rephrased the paragraph in the updated paper pdf (section 3.1, page 4) as follows:

“We aim to predict relevance score for each table token, however, since annotations for relevant table parts are unavailable, token relevance is not explicitly observable and we consider it as a latent variable. Further, we hypothesize that the representation space of table tokens can be structured better for modeling relevance by clustering their encodings into two categories - relevant and non-relevant. Variational Inference (VI) has been commonly used to estimate latent variable probability and group data points on the basis of latent topics. Hence, we estimate relevance of table token as (step 3 in Fig.2): ...”

Reviewer comment (Reproducibility):
The experimental details (how the model was trained, how hyperparameters were set) are not given; could the authors provide the information (at least in the appendix)?

Author response:
> "the experimental details (how the model was trained, how hyperparameters were set) are not given; could the authors provide the information (at least in the appendix)?"

Yes, certainly! We have elaborated more implementation details for CABINET and added them to the appendix A.6. Details on how table perturbation is performed was present in appendix A.5. The full implementation details (in section 4 plus additional details) added to appendix A.6 are as follows:

As stated in section 4 of the main paper (under implementation details), we employ the OmniTab backbone built over BART-Large architecture for the encoder ($TE_{QA}$) and decoder ($TD_{QA}$) of the QA LLM. Further, URS encoder ($TE_{URS}$) is initialized with the architecture and weights of QA LLM encoder ($TE_{QA}$), though, they do not share weights during training. However, the embedding layers $Embedding_{URS}$ and $Embedding_{QA}$ share the weights. The hidden dimension $d$ of $TE_{URS}$ is $1024$, similar to that of both $TE_{QA}$ and $TD_{QA}$. We train CABINET and other baselines for 30 epochs on an effective batch size (BS) of 128 using 8 80 GB A100 GPUs (BS of 8/GPU with gradient accumulation 2) using learning rate of $1e^{-5}$ with Cosine Annealing through AdamW optimizer. We carry out exhaustive hyper-parameter tuning based on the validation set to come up with the optimal values of learning rate ($1e^{-5}$), scheduler (Cosine Annealing), batch size (8), gradient accumulation steps (2) and the optimizer (AdamW). We leverage text pre-trained Flan T5-xl as the backbone for Parsing Statement Generator (PSG) and the Cell Highlighter LLM. The Parsing Statement Generator is trained for 50 epochs on the $~300$ sampled question-table-parsing statement triplets with hyper-parameters setting same as that of CABINET, however the only difference being the effective batch size of 16 (BS of 2/GPU with 8 gpus without gradient accumulation). On a similar note, the Cell Highlighter LLM is trained for 3 epochs on the ToTTo dataset to generate highlighted cells given the table and a natural language statement describing a subpart of the table as the input. During the inference phase of generating answer, parsing statements and highlighted cells, we use beam search decoding with a beam size of 3.

Reviewer comment (Reproducibility):
How is the model used during inference (i.e. how is the VI-related loss used to predict the relevance of each token)?

Author response:
> "How is the model used during inference (i.e. how is the VI-related loss used to predict the relevance of each token)?"

The VI-related losses (clustering loss, cluster mean separation loss, and relevance score sparsification loss) are only applied during training to tune the weights of the unsupervised relevance scorer to assign appropriate relevance scores to table tokens for a given question. Hence, these losses are not (required to be) applied during inference to estimate relevance scores. Applying these losses together at the training time (as a training signal) to optimize the learnable parameters of the unsupervised relevance scorer through gradient backpropagation allows it to learn to assign a high score to relevant tokens and a low score to non-relevant tokens to enable the downstream QA model to generate the right answer.

For better clarity, we explain the inference flow to compute the relevance score as follows: As discussed in section 3.1, the linearised table and the question are given as input to the unsupervised relevance scorer which is a transformer encoder (eqs.1-2). The latent representation (eq.3) encoded by this transformer encoder corresponding to each table token is then used to estimate the relevance logit through variational inference (eqs.4-5) such that sigmoid is applied over this logit to obtain the relevance score.

Reviewer comment (Experiments):
Is the clustering model that important? Would predicting directly a probability of relevance for each token work?

Author response:
> "is the clustering model that important? Would predicting directly a probability of relevance for each token work?"

Yes, the clustering model is important. Clustering the table tokens using their latent representation enables CABINET to separate the relevant tokens for the question from the non-relevant ones. As the ablation reported in Table 4 (in the main paper, section 4.4) reveals, settings, where clustering is performed, achieve higher performance compared to settings where clustering is not performed. For instance, in row 6 - clustering-related losses lead to ~4.5% better acc. on WikiTQ, 0.7% better on FeTaQA, and ~3% better performance on WikiSQL, compared to row 1 (no-clustering).

Please refer to Appendix A.10 and figures 7-8 (pages 21-23) in the updated pdf for a case study depicting the importance of clustering and how the different losses interact with each other to yield improvement.

Reviewer comment (Writing):
p.9, please correct "its performance" to "TAPEX performance" for clarity.

Author response:
> "p.9, please correct "its performance" to "TAPEX performance" for clarity"

Thanks for suggesting the correction! We have updated this in the paper pdf.

Reviewer comment (Evaluation):
What is the average model inference latency compared with the baseline model (OmniTAB) for the three tasks.

Author response:
> "What is the average model inference latency compared with the baseline model (OmniTAB)"


* The average runtime time required during inference for CABINET (averaged over 100 queries) is: 0.212s
* The average runtime required during inference for one of the best-performing baselines - OmniTab is (averaged over 100 queries): 0.153s

It is to be noted that the inference time for the model remains the same irrespective of the three datasets. It can be seen that CABINET gives large performance gains at the cost of small incremental increase in inference latency.

Reviewer comment (Experiments):
The need for an Unsupervised Relevance Scorer: If we replace URS with BM25 or some out-of-the-box similarity metrics, will the model have a huge performance decline?

Author response:
> "The need for an Unsupervised Relevance Scorer: If we replace URS with BM25 or some out-of-the-box similarity metrics, will the model have a huge performance decline?"

To answer this question, we did the following experiment: We encode the question and each row in the table with BERT. Subsequently, we compute the similarity score between the encoding of each table row and question encoding through cosine similarity. Each token in a given table row is then assigned a relevance score equal to the cosine similarity between the row and question encodings. This BERT encoding similarity based relevance score is then used (instead of URS) with CABINET. We summarise the results obtained in the following table and compare them with CABINET:

| Method   | WikiTQ (Acc.)   | FeTaQA (SacreBleu)   | WikiSQL (Acc.)   |
| ---------- | ---------- | ---------- | ---------- |
| CABINET with BERT similarity based relevance scoring (as discussed above) without cell highlighter | 61.8 | 34.9 | 83.7 |
| CABINET with URS only and without cell highlighter | 65.6 | 35.8 | 89.3 | 

As can be noted, replacing URS in CABINET with an off-the-shelf BERT encoding-based similarity for relevance scoring results in significant drop in performance. Next, we also experiment with another configuration where the weakly supervised cell highlighter is used with BERT encoding-based similarity metrics for relevance scoring and compare the performance with CABINET (with URS and cell highlighter) in the following table. Here, a similar trend is observed where it is seen that CABINET with URS and cell highlighter performs much better than CABINET with BERT encoding similarity based relevance scoring and cell highlighter.

| Method   | WikiTQ (Acc.)   | FeTaQA (SacreBleu)   | WikiSQL (Acc.)   |
| ---------- | ---------- | ---------- | ---------- |
| CABINET with BERT encoding similarity based relevance scoring (as discussed above) with cell highlighter | 64.5 | 36.7 | 85.1 |
| CABINET with URS and cell highlighter | 69.1 | 40.5 | 89.5 |

Reviewer comment (Experiments):
The need for a Parsing Statement Generator: If the original question is directly used for cell highlighter other than the parsed text, will the model have a huge performance decline?

Author response:
> "The need for a Parsing Statement Generator: If the original question is directly used for cell highlighter other than the parsed text, will the model have a huge performance decline?"

The following table summarises the TableQA performance achieved when the question is used as input to the cell highlighter instead of the parsing statement.

| Method   | WikiTQ (Acc.)   | FeTaQA (SacreBleu)   | WikiSQL (Acc.)   |
| ---------- | ---------- | ---------- | ---------- |
| CABINET with question as input to cell highlighter | 63.7 | 34.4 | 85.7 |
| CABINET with parsing statement as input to cell highlighter | 69.1 | 40.5 | 89.5 | 

It can be observed that there is a significant decline in accuracy when question is given as input to the cell highlighter compared to giving the parsing statement as input to the cell highlighter. This indicates that the parsing statement describing the criteria of rows and columns relevant to the question is essential to achieve the performance gains.

Reviewer comment (Experiments):
The need for a Cell Highlighter: If the $\eta^{cell}$ is the indicator for a cell value in the parsed text, will the model have a huge performance decline?

Author response:
> "The need for a Cell Highlighter: If the $\eta^{cell}_{p}$ is the indicator for a cell value in the parsed text, will the model have a huge performance decline?"

We perform an ablation where tokens corresponding to table cell values which are present in the parsing statement text are highlighted instead of using the cell highlighter model predictions and compare with CABINET in the following table where it can be seen that the performance drops significantly compared to CABINET. Thus, the cell highlighter model is needed to achieve performance gains to predict relevant cells corresponding to the parsing statement.

| Method   | WikiTQ (Acc.)   | FeTaQA (SacreBleu)   | WikiSQL (Acc.)   |
| ---------- | ---------- | ---------- | ---------- |
| CABINET with table cell values present exactly in parsing statement text used as highlighted cells| 65.4 | 36.9 | 86.2 |
| CABINET with cell highlighter model used as described in the paper | 69.1 | 40.5 | 89.5 |