Exploring the impact of data representation on neural data-to-text generation (2024)

Abstract: A relatively under-explored area in research on neural natural language generation is the impact of the data representation on text quality. Here we report experiments on two leading input representations for data-to-text generation: attribute-value pairs and Resource Description Framework (RDF) triples. Evaluating the performance of encoder-decoder seq2seq models as well as recent large language models (LLMs) with both automated metrics and human evaluation, we find that the input representation does not seem to have a large impact on the performance of either purpose-built seq2seq models or LLMs. Finally, we present an error analysis of the texts generated by the LLMs and provide some insights into where these models fail.

## introduction in the field of natural language generation (nlg), the quality of generated text is crucial, influencing the usability and effectiveness of applications ranging from automated reporting to conversational agents. the focus of the field has predominantly been on developing more sophisticated models and algorithms creating a gap in understanding the impact of input data representations. over the years, various input representations for end-to-end nlg have been utilised. these representations have often been chosen based on convenience, such as preexisting formats of input data or prevailing trends. however, to our knowledge, no previous research has systematically investigated whether the choice of input representation affects the overall quality of the generated text. by addressing this gap, our study aims to evaluate how different input representations impact the fluency and semantic fidelity of generated texts. this investigation not only contributes to theoretical advancements but also offers practical insights into improving nlg systems. nlg systems utilise various input representations to convert structured data into text. these e2e @xmath0. @xmath1. figure 1 : two linearistations of e2e and webnlg inputs. e2e's input format consists of attribute-value pairs. webnlg's inputs are semantic triples, composed of subject, predicate and object. in this paper, we explore the impact of input representations in data-to-text generation, i.e. in tasks where the input of an nlg system is structured data and the output is coherent and contextually relevant natural language texts. we explore the classic seq2seq nlg architecture (exemplified by @xcite ) and large language models (llms; in particular, @xcite and llama @xcite ) with two popular tasks and their corresponding input formats, namely e2e and webnlg. in order to represent these input formats as sequences for neural network models, we linearise them as shown in figure 1 . this paper examines the following research question: 'do input representations matter in data-totext systems?'. our contributions are: (1) we present a comparison of two leading representations for data-to-text research for neural seq2seq models and llms; and (2) we provide the code for reproducing these experiments with other linearisations of comparable meaning representations at https://github.com/napiernlp/inlg2024 . our careful human evaluations across two datasets find no statistically significant evidence that attribute-value representations or rdf representations are superior across the board. comparing trends within a single system, our results suggest that there may be a slight benefit of using rdfs for accuracy for llama 3 or for seq2seq models, with a slight penalty to fluency, though further research is necessary given the small differences in performance on these datasets. a qualitative error analysis confirms that gpt-4o and llama 3 produce very few semantic errors in these domains, though llama 3 does sometimes omit content from more complicated rdf inputs and both can produce occasionally stilted language. ## datasets we adopt the enriched versions of the webnlg and e2e datasets, since they have both been prepared similarly from existing datasets. for our work, we limit ourselves to using the raw inputs and outputs and corresponding delexicalisations. for the enriched webnlg dataset, castro ferreira et al. (2018) adapt the webnlg corpus to include annotations for content ordering, sentence segmentation, surface realisation, and referring expression generation (reg). delexicalisation was performed manually, labelling the subjects for rdf predicates as agents and the objects as pa-tients, with numeral suffixes to indicate which predicate the entities are associated with. entities which appear in both subject and object roles for different predicates in the same input are delexicalised with the label bridge. for the enriched @xcite adapt the e2e challenge corpus @xcite to include annotations for content ordering, sentence segmentation, lexicalisation, reg, and surface realisation. where the enriched webnlg dataset treated lexicalisation and surface realisation in a single step, with reg as a post-process, the enriched e2e dataset handles lexicalisation and surface realisation separately. ## models we explore a classic approach to neural data-to-text generation as well as zero-shot llm prompting for this work. ## automatic evaluations we use reference-based automated metrics primarily to assess the degree to which our seq2seq model 2 https://unify.ai/ ; cost breakdown in appendix learns to match the kinds of texts present in the corpora, though we also report the llms' performance for reference. we report bleu @xcite as implemented in sacrebleu foot_1 @xcite for a discrete word-overlap metric and rescaled bertscore foot_2 f1 @xcite for a slightly more flexible quality metric. for the webnlg dataset we turn to ## human evaluation we asked participants to assess fluency and semantic fidelity. for fluency, we adapted the questions used by webnlg 2023 @xcite , asking participants to "rate the output in terms of fluency" and explaining that "[h]ighly fluent text 'flows well' and is well-connected and free from disfluencies". participants rated fluency on a 7point likert scale ranging from very disfluent to very fluent. for semantic fidelity (i.e. the faithfulness of the outputs to the inputs), participants saw a table of subjects, predicates, and objects meant to be present in the output and had to click a radio button to indicate whether that element of the meaning was present, missing, or incorrect. participants could also indicate if the output included additional content not present in the input and had a free text area to describe the inserted content. for each dataset, we selected 48 inputs from the test across the 7 experimental conditions: the reference text for a control condition plus one text from each system for each input representation. each participant saw 28 items plus 2 attention check questions presented in a randomised order. we recruited 36 participants for each dataset through prolific foot_3 . we screened participants, requiring them to be first-language speakers of english and resident in a country where english is a majority language (i.e. australia, canada, ireland, new our institution approved the study's ethics. ## qualitative error analysis since both llms performed well regardless of input representation, we manually examine those instances where they performed worst to see if there are any qualitative patterns. the two lowest rated gpt-4o texts were scored somewhat disfluent and both contained the phrase 'located riverside", describing the location of a restaurant. only one text received a neutral score, and none of these texts had semantic fidelity errors. three llama 3 texts scored disfluent, six as somewhat disfluent, and two as neutral. llama 3 exhibits a greater tendency to reuse phrases from the input representation in ways that disrupts fluency (e.g. expressing the predicate-object pair eat type, pub with the awkward phrase 'is a type of eatery found in a pub'). sometimes restaurant names are treated as a different kind of entity: 'the wrestlers' is the name of a restaurant, but llama 3 treats this as a group of people, producing 'the wrestlers eat at a pub' instead of 'the wrestlers is a pub'. items with the highest proportion of missing or incorrect semantics according to participants tended to be more accurate than reported. gpt-4o produces one disfluent text for the webnlg dataset: 'antwerp international airport serves the city of antwerp. the country of antwerp is belgium. in belgium, the language spoken is german.' there are also three somewhat disfluent and two neutral texts generated. llama 3 received a very disfluent rating for a short sentence that is actually fluent: 'hip hop music is a derivative of drum and bass'. however, the sentence may have been rated poorly because it is semantically anomalous, or requires domain specific knowledge. three texts were marked as disfluent and another nine as somewhat disfluent, some of these seemingly due to awkward phrasing ('aleksey chirikov, an icebreaker built in helsinki, finland, is led by juha sipilä'), and others for being nonsensical, such as 'atlanta, a city in the united states, is the capital of a country with an ethnic group of asian americans, with washington, d.c. as its capital'. semantic errors were again infrequent for gpt-4o, though there were more interesting errors for llama 3. for example, llama 3 sometimes omits large portions of the meaning representation, expressing only one out of five given predicates. ## discussion & conclusions we expected that the meaning representation used to encode inputs for neural data-to-text generation would substantially impact either the fluency or the accuracy of generated texts. however, our findings do not support this hypothesis. instead, we find a strong performance by recent llms regardless of input representation, and we find that simpler seq2seq models are also not substantially impacted by these differences. we also observed remarkably few 'hallucinations', or insertions of additional content not present in the input, across both llms. we suspect that these results are in part influenced by the fact that both of our source datasets are publicly available and are likely to be included in the training data for both gpt-4o and llama 3 systems. in future work, we plan to investigate this possibility with the creation of novel, unseen datasets and new linearisations of meaning representations.