F resh T ab: Sourcing Fresh Data for Table-to-Text Generation Evaluation (2025)

Abstract: Table-to-text generation (insight generation from tables) is a challenging task that requires precision in analyzing the data. In addition, the evaluation of existing benchmarks is affected by contamination of Large Language Model (LLM) training data as well as domain imbalance. We introduce FreshTab, an on-the-fly table-to-text benchmark generation from Wikipedia, to combat the LLM data contamination problem and enable domain-sensitive evaluation. While non-English table-to-text datasets are limited, FreshTab collects datasets in different languages on demand (we experiment with German, Russian and French in addition to English). We find that insights generated by LLMs from recent tables collected by our method appear clearly worse by automatic metrics, but this does not translate into LLM and human evaluations. Domain effects are visible in all evaluations, showing that a domain-balanced benchmark is more challenging.

## introduction table-to-text generation or insight generation @xcite ) is a challenging task in natural language generation (nlg), where a nlg system generates insights from a data table. this can provide important support in data analytics and decision making in business or governance. recent research in insight generation builds on finetuned neural language models @xcite @xcite or prompted large language models (llms) @xcite . llms display excellent performance in various tasks, and unlike prior methods, they do not require costly in-domain training data with human-written references. with few-shot examples and chainof-thought prompting, they surpass prior methods on insight generation @xcite . however, llms were also shown to memorize common figure 1 : schema of the freshtab method benchmarks @xcite , inflating their true performance, and to perform unevenly across domains @xcite @xcite . we directly address these problems and present freshtab, an approach for obtaining up-to-date benchmarks for insight generation, following prior work on dynamic dataset construction @xcite . this dataset family, based on wikipedia tables, is not affected by the problems of llm memorization and benchmark contamination, as the underlying tables are newer than the llm's knowledge cutoff date, see figure 1 . we introduce basic domain labels for each table, allowing for domain-specific evaluation insights. the datasets can be generated in any wikipedia language and configured along multiple parameters. our main contributions are as follows: freshtab is publicly available and automatically collects a new dataset version each month. 1 2 related work insight generation approaches for generating insights from tables have been developed alongside other data-to-text nlg systems for decades @xcite . the emergence of neural models brought a lot of research into the area, focusing on end-to-end architectures @xcite that incorporate table-aware training @xcite , use pretrained lms @xcite , or both @xcite . most recent approaches to table-to-text use llms. while @xcite and @xcite still focus on finetuning llms on tabular tasks, @xcite and @xcite successfully apply chain-of-thought prompting without the need for task-specific training. however, all previous tableto-text approaches focus on fixed benchmarks, making them susceptible to training data contamination @xcite @xcite . ## experimental setup 4.1 benchmark comparison to evaluate the usefulness of our method, we compare it to the previous lotnlg benchmark @xcite , a subset of the commonly used logic-nlg data @xcite , which was available to all llms at training time, and is paired with reference insights. using freshtab, we created several new benchmarks: we set the table size limit to approx. 3k characters, so that all tables comfortably fit into llms' context sizes. the table parameters were taken from the logicnlg (chen et al., 2020) benchmark tables. ## conclusion we present freshtab, a method for producing live benchmark datasets for table insight generation from wikipedia, enabling easy evaluation of llms on unseen data and supporting domain balance and non-english languages. our experiments confirm that llms behave differently on the new data. we also found poor performance of automatic metrics, with llm-judges showing more reliable.