ASPIRO : Any-shot Structured Parsing-error-Induced R epr O mpting for Consistent Data-to-Text Generation (2023)

Abstract: We present ASPIRO, an approach for structured data verbalisation into short template sentences in zero to few-shot settings. Unlike previous methods, our approach prompts Large Language Models (LLMs) to directly produce entity-agnostic templates, rather than relying on LLMs to faithfully copy the given example entities, or validating/crafting the templates manually. We incorporate LLM re-prompting, triggered by algorithmic parsing checks, as well as the PARENT metric induced consistency validation to identify and rectify template generation problems in real-time. ASPIRO, compared to direct LLM output, averages 66% parsing error rate reduction in generated verbalisations of RDF triples on the DART dataset. Our best 5-shot text-davinci-003 setup, scoring BLEU of 50.62, METEOR of 45.16, BLEURT of 0.82, NUBIA of 0.87, and PARENT of 0.8962 on the Rel2Text dataset, competes effectively with recent fine-tuned pretrained language models.

## introduction data-to-text task @xcite aims to build a faithful natural language interpretation of structured data such as relational tables or resource description framework (rdf) triples @xcite . however, without proper context, the given structured data may not sufficiently represent the relationships between entities, leading to ambiguity @xcite . to battle this, some works rely on fine-tuning pre-trained language models (plms) on task-specific datasets in supervised or semi-supervised ways @xcite , but the domain of the resulting system is limited and requires well-labelled training data @xcite . in contrast to finetuning, @xcite prove that zeroshot neural systems are a possible solution, where in-domain data is introduced via simple humancrafted templates for each unique relation in the 1 code available at github.com/vejvarm/aspiro. knowledge graph. @xcite nullify the requirements for human labelling entirely by utilising gpt3-davinci @xcite , a large language model (llm) with broad general knowledge, to disambiguate rdf triples into short sentences and automatically parse them into reusable sentence templates as an alternative to human-crafted templates. in this paper we introduce aspiro, a robust n -shot variant of the data disambiguation step presented by @xcite and a promising alternative to fine-tuning plms for crafting rdf verbalisations @xcite . at its core, as-piro uses simple rules to algorithmically flag errors in the templates (such as missing subject, multiple objects, etc.) and re-prompt the llm until all errors are alleviated or maximum (n ) retries have been reached. we evaluate changes in automated metrics and reduction of parsing errors in different configurations of aspiro on dart @xcite and rel2text @xcite and compare the original rdf verbalisation prompt used by @xcite with our prompt focused on enforcing structured json output with intermediate fields as guidelines. delexicalization: @xcite and @xcite find that without delexicalization, generative models can produce incomplete representations of the entities and concepts in the structured data verbalisations, leading to misinterpretation and failures in production. our json structured prompt ( §g.2) enforces the llm to directly produce named-entity agnostic templates. ## methods the proposed method (aspiro) revolves around the conversion of structured data samples into verbalisation templates using a two-stage pipeline: nshot generator ( §3.1) and consistency validator ( §3.2). the pipeline processes structured data samples, wherein each sample comprises of one or more rdf triples which share the same relation. aspiro (see figure 1 ) starts with an initial prompt to verbally articulate the structured data. this is equivalent to prompting a single llm directly. if the zeroth attempt isn't accurate, it will retry a maximum of n times, refining the previous completion based on parsing errors ( §3.1.2). subsequently, the outputs are validated for consistency, ensuring faithful and reliable verbalisations. we explain the individual stages and their sub-modules in the sec-