Is This a Bad Table? A Closer Look at the Evaluation of Table Generation from Text (2024)

Abstract: Understanding whether a generated table is of good quality is important to be able to use it in creating or editing documents using automatic methods. In this work, we underline that existing measures for table quality evaluation fail to capture the overall semantics of the tables, and sometimes unfairly penalize good tables and reward bad ones. We propose TabEval, a novel table evaluation strategy that captures table semantics by first breaking down a table into a list of natural language atomic statements and then compares them with ground truth statements using entailment-based measures. To validate our approach, we curate a dataset comprising of text descriptions for 1,250 diverse Wikipedia tables, covering a range of topics and structures, in contrast to the limited scope of existing datasets. We compare TabEval with existing metrics using unsupervised and supervised text-to-table generation methods, demonstrating its stronger correlation with human judgments of table quality across four datasets.

## introduction tables are an integral form of representing content in real-world documents such as news articles, financial reports, and contracts. document generation requires the generation of high-quality tables along with other modalities. while the problems of table-to-text generation and table summarization have been widely studied @xcite @xcite ), text-to-table generation has been gaining increasing attention more recently @xcite . differentiating between good and bad quality tables generated from text is crucial for their usability in documents. failure to accurately assess table quality can result in including subpar content or overlooking valuable tables in documents. existing text-to-table works adopt metrics based on exact match and bertscore @xcite of the header cells of generated tables with the ground truth ones, and for the non-header cells, they use the header cell information also to compare the resulting tuples. however, a major limitation with such measures is that they evaluate the table cells (or tuples) independently without considering contextual information from the neighboring cells. this can lead to incorrect penalization of good tables, or incorrect rewarding of bad tables. in this paper, we first propose tabeval, a twostaged table evaluation approach that views tables holistically rather than considering values independently while evaluating their quality. given the table intent, reference, and predicted table, we first unroll the tables into sets of meaningful natural language (nl) statements that convey the overall table semantics. we propose tabunroll, a novel prompting technique to unroll a table using chain-of-thought @xcite using an llm. we then compute the entailment scores between the unrolled nl statements of predicted and ground truth tables and provide an aggregate as the measure of table quality. existing datasets used for text-to-table generation, such as rotowire @xcite , wik-ibio @xcite , wikitabletext @xcite , are restricted in domain and schema. our second contribution is curation of a dataset consisting of 1,250 general domain tables along with their textual descriptions, to assess our evaluation strategy across different domains. thirdly, we perform several experiments utilizing existing text-to-table methods and llmbased prompting techniques. we collect human ratings for table quality on test generations obtained using from various method-dataset combinations. tabeval shows significantly higher correlations with human ratings compared to the existing metrics across most scenarios. we highlight important failure cases of the existing metrics qualitatively, while underlining limitations of ours too to facilitate further research on evaluating the quality of automatic table generation methods in documents. ## proposed evaluation strategy we introduce tabeval, a two-stage pipeline (fig. 1 ) that evaluates the semantic quality of generated tables against a reference table to ensure they convey the same information. table unrolling. we propose tabunroll, a prompting strategy using chain-of-thought to unroll a table into meaningful nl atomic statements. the input is the table intent (table name/ caption/ description) and the table in html. it follows a generalizable schema outlined in @xcite )-(1) instruction set: llm is prompted to identify the column headers, rows, and suitable column(s) serving as primary key(s) to depict each unit of information conveyed by the table. we define the primary key as the column(s) that contains values that uniquely identify each row in a table. we provide instructions to use the identified primary key(s) as anchor(s) to construct meaningful atomic statements by using values from the rest of the columns one at a time. in the absence of primary key, we instruct to form the statements by picking as few columns (two or above) as possible to form meaningful statements. the llm is also prompted to attribute the specific rows from which the atomics are constructed in the form on inline citations, to mitigate any hallucinations @xcite . (2) few-shot examples: we provide positive and negative examples of how tables should be unrolled. given that llms tend to struggle with negation tasks @xcite full prompt template and sample unrolled tables.) entailment-based scoring. after obtaining the unrolled statements from the ground truth and predicted tables (of sizes m and n respectively), we employ natural language inference @xcite to determine whether the information conveyed by the predicted table is also present in the ground truth table, and vice versa. precision (correctness) is computed as the average of the maximum entailment scores between each predicted statement p i and all ground truth statements g j , recall (completeness) as the average of the maximum entailment scores between each ground truth statement g j and all predicted statements p i and f1 (overall quality) as the harmonic mean of precision and recall. to have a general-domain text-to-table evaluation, we curate desctotto, by augmenting tables from totto with parallel text descriptions. it comprises of 1,250 tables, each annotated with table text and intent. annotators, fluent in english and skilled in content writing, are recruited from a freelancing platform and compensated at ## experiments to validate tabeval, we conduct experiments using four text-to-table generation models on four datasets. in the supervised setting, we perform instruction fine-tuning on llama-2-7b-chat-hf, and use the seq2seq text-to-table baseline proposed by @xcite . tables generated by gpt-4 and gpt-3.5-turbo models are in an unsupervised setting with few-shot examples. nvidia a100 gpus were used for llama ift. the prompts for gpt and llama ift are in appendix c. in tabeval, we experiment with gpt-4 and claude-3-opus (anthropic, 2024) for table unrolling, and use roberta-large-mnli @xcite for measuring entailment. baselines. we compare tabeval with those in @xcite , which assess tables by representing them as tuples (row header, cell value)/ triples (row header, col header, cell value) and comparing them with ground truth tuples/ triples for exact matches (e), chrf @xcite , and rescaled bertscore (bs) @xcite . metrics. we obtain human ratings (1-5 scale) for correctness, completeness, and overall quality of generated tables, comparing them to reference (instructions in appendix d). we calculate the pearson correlation between our metric scores and human ratings, comparing these to baseline metrics. ## results & discussion we obtain human ratings for 1,000 test tables (250 per dataset) from three annotators, with medium to high agreement (α: 0.55, 0.60, 0.62 for quality, correctness, and completeness, respectively) @xcite . pearson correlations are computed between the automatic metrics with these ratings across various dataset-method pairs ( tabeval has higher correlations than that of the existing metrics across most configurations, indicating that our metric is able to evaluate table semantics more accurately compared to the existing ones. the increments are higher for desctotto and rotowire than for the other two datasets; this is because, wikibio and wikitabletext, contain simple key-value pairs that are mostly extractive in nature, and are thus effectively evaluated using the bs-based metric for (row, value) tuples in generated tables, yielding correlation scores comparable to tabeval. particularly in supervised settings, the correlations are slightly higher using bs on these datasets, as they tend to generate very well-rehearsed generations based on the training data. rotowire has a fixed schema for player/team bs penalises tables for variation in column headers. table a, despite having correct details, scores lower with bs but high with ours. table b, with errors, is appropriately penalized by tabeval. table c covers all the details from reference table, receives lower precision and recall with bs but high scores with ours. table d, missing some rows, has reduced recall with tabeval. ## limitations since we rely on llms to break down a given table into atomic statements, our method will be limited by the quality of the llm outputs and any potential hallucinations. however, we use gpt-4 in our evaluation pipeline, and note that the unrolled statements rarely contain hallucinations. there is a trade-off while using such large models-while the quality of unrolled statements will be very good, they can be computationally expensive. with gpt-3.5 and llama variants, we noted more hallucinations in our preliminary explorations. in this work, we only focus on the semantic quality of tables; we do not evaluation the structural quality, e.g., understanding the right structure for conveying a given intent in an easy-to-read and visually appealing manner. this can also form one of the future works for this study.