SPOR : A Comprehensive and Practical Evaluation Method for Compositional Generalization in Data-to-Text Generation (2024)

Abstract: Compositional generalization is an important ability of language models and has many different manifestations. For data-to-text generation, previous research on this ability is limited to a single manifestation called Systematicity and lacks consideration of large language models (LLMs), which cannot fully cover practical application scenarios. In this work, we propose SPOR, a comprehensive and practical evaluation method for compositional generalization in data-to-text generation. SPOR includes four aspects of manifestations (Systematicity, Productivity, Order invariance, and Rule learnability) and allows high-quality evaluation without additional manual annotations based on existing datasets. We demonstrate SPOR on two different datasets and evaluate some existing language models including LLMs. We find that the models are deficient in various aspects of the evaluation and need further improvement. Our work shows the necessity for comprehensive research on different manifestations of compositional generalization in data-to-text generation and provides a framework for evaluation.

## introduction data-to-text generation @xcite is an important task in natural language generation @xcite . it aims to generate fluent and faithful text based on structured data input and is critical in many nlg systems, such as report generation @xcite , oriented dialogues @xcite , etc. in data-to-text generation, structured data input is compositional, i.e., it can be considered as a combination of elements formed according to certain rules. therefore, in order to handle the practical data-to-text generation, the language models should have the ability to recombine previously learned elements with certain rules to map new inputs made up from these elements to their correct output @xcite , which is the so-called compositional generalization. compositional generalization is an important ability of language models for many tasks. in semantic parsing and mathematical reasoning tasks, many different manifestations of this ability have been studied @xcite , such as systematicity (handle combinations unseen during training), productivity (extrapolate to longer sequences than those seen during training), etc. for compositional generalization in data-totext generation, only systematicity receives attention @xcite , and research on other manifestations is lacking. the single systematic manifestation cannot fully cover practical application scenarios of compositional generalization and cannot comprehensively reflect this ability of language models in data-to-text generation. although research on different manifestations of compositional generalization in data-to-text generation is necessary, there is currently no comprehensive evaluation method to support such research. to solve this problem, we propose spor, a comprehensive and practical evaluation method for compositional generalization in a data-to-text generation. based on the manifestations of compositional generalization mentioned in @xcite , spor includes four aspects of compositional generalization in data-to-text generation: for each aspect, we propose the corresponding methods for dataset construction and evaluation. based on existing datasets, we mainly perform repartition @xcite and element modification to construct datasets for our evaluation. overall, the evaluation method spor has the following properties: we demonstrate spor on two existing datasets for data-to-text generation and evaluate some existing language models. previous research on compositional generalization in data-to-text generation lacks consideration of large language models (llms) due to the lack of methods to directly finetune and apply llms to data-to-text generation in the past. nowadays, advanced parameter-efficient fine-tuning such as lora @xcite provides the methods, and the consideration of llms becomes necessary. therefore, we include some advanced llms in our evaluation to partially fill the gap in previous research. ## preliminaries in this section, we provide a brief description of the datasets that spor is demonstrated on, the evaluated models, and the evaluation metrics. ## evaluation method in this section, we describe each aspect of spor. each subsection corresponds to an aspect that includes: (1) the overview; (2) how to construct the dataset; (3) the statistics of the dataset; (4) how to perform the evaluation and (5) the results and analysis. for all results reported, we run experiments three times with different random seeds and average the results to avoid contingency. appendix b provides the qualitative analysis of evaluations, showing specific samples with model outputs. ## conclusions in this work, we propose spor, a comprehensive and practical evaluation method for compositional generalization in data-to-text generation, which includes four aspects of manifestations: systematicity, productivity, order invariance, and rule learnability. we demonstrate on webnlg and e2e how spor enables evaluations without additional manual annotations based on existing datasets. we evaluate some existing language models, including llms. we find that the models are deficient in various aspects of compositional generalization in data-to-text generation and need further improvement. our work supports comprehensive research on different manifestations of compositional generalization in data-to-text generation and provides a framework for identifying and evaluating improvements in this ability of language models. the dataset and code are available at https://github.com/xzy-xzy/spor .