Scaling Up Data-to-Text Generation to Longer Sequences: A New Dataset and Benchmark Results for Generation from Large Triple Sets (2025)

Abstract: The ability of LLMs to write coherent, faithful long texts from structured data inputs remains relatively uncharted, in part because nearly all public data-to-text datasets contain only short input-output pairs. To address these gaps, we benchmark six LLMs, a rule‐based system and human-written texts on a new long-input dataset in English and Irish via LLM-based evaluation. We find substantial differences between models and languages.

## introduction large and very large language models (llms/vllms) have set a new standard for natural language generation (nlg), producing fluent and accurate text across a host of applications such as text simplification @xcite , image captioning @xcite , and question answering @xcite ). yet the very capabilities that allow for excellent short-form output can fail to scale to the extended output sequences demanded by tasks such as summarization, creative writing, and detailed question answering @xcite @xcite . this challenge has given rise to an emerging focus on long-form generation, which requires models to preserve topical focus, logical progression, and factual consistency across lengthy passages or entire documents @xcite . consequently, the field has seen the development of specialized 'long-output' llms that are specifically tuned to handle these longer generative tasks @xcite . as inputs or required outputs extend beyond the model's trained context window, generation quality deteriorates markedly @xcite . even within the context window, performance declines for longer input lengths @xcite and extended outputs @xcite , leading to reduced coherence, greater factual inaccuracies, and an increased incidence of hallucinations @xcite . these issues underscore a critical gap in the ability of current models to maintain logical consistency and relevance over extended output sequences in data-to-text tasks. despite the significant advancements and availability of 'long-context' llms, the data-to-text (d2t) generation field continues to rely heavily on outdated datasets and benchmarks featuring predominantly short input-output pairs. in addition, such datasets can lead to data leakage, where llms have been trained on data closely resembling or identical to the test samples, undermining the validity of evaluations @xcite @xcite . this reliance presents a critical limitation, as these datasets are increasingly insufficient for rigorously challenging contemporary llms, given their substantial capacity for managing extensive contexts. in this paper, our contributions are the following: (i) method and code for collecting long-input data for d2t generation, along with a dataset of 537 long inputs; (ii) english and irish system outputs for these 537 inputs, as well as human-written texts for a subset of 21 inputs; and (iii) a preliminary evaluation of output quality in both languages using an llm-as-judge approach and diversity metrics. all code, data and results are available at https: //github.com/mille-s/build_kgs_entities . ## related work while the broader field of nlp has seen recent work on long-context llms, primarily focused on improving model capabilities for reasoning and understanding extensive input contexts @xcite @xcite , research specifically on long-form output generation within d2t remains nascent. in the field of d2t generation, benchmark datasets have been pivotal in driving significant progress by providing structured data for model training and evaluation. however, the rapidly evolving capabilities of contemporary (very) large language models, or (v)llms, such as openai's gpt models @xcite ), meta's llama models @xcite ), google's gemini @xcite , and anthropic's claude @xcite , highlight a growing need for datasets featuring significantly longer token sequences. existing foundational d2t datasets, while valuable, consistently exhibit relatively short average output lengths: ## experimental setup we created a new dataset of long-context data-totext inputs (section 3.1), generated outputs with a range of different (v)llms (section 3.2), and evaluated the outputs along with human-written texts via llm-as-judge assessment (section 3.3). ## conclusions we developed a method for creating data for long-input data-to-text generation, compiled a new dbpedia-based dataset, ran several systems on the inputs and assessed text quality using an llm-asjudge approach. we conclude that some (v)llms seem able to generate long texts from structured data and to evaluate their quality along several dimensions, although there may be biases in the evaluation of llms by llms. in irish, claude appears to be the best model, while a fine-tuned version of uccix does not perform well; it is unclear whether this is due to limitations of the fine-tuning data (the webnlg'23 irish data is automatically translated from english) or of the pretrained model itself, as indicated by llama-2's low english scores. in future work, we will add filters to optimise the selection of high-quality triples for dataset creation. we will also carry out human assessment of the text quality and of the collected ratings so as to gain further insights into the use of llms for long-context d2t and its evaluation.