Compositional Generalization for Data-to-Text Generation (2023)

Abstract: Data-to-text generation involves transforming structured data, often represented as predicate-argument tuples, into coherent textual descriptions. Despite recent advances, systems still struggle when confronted with unseen combinations of predicates, producing unfaithful descriptions (e.g.,hallucinations or omissions). We refer to this issue as compositional generalisation, and it encouraged us to create a benchmark for assessing the performance of different approaches on this specific problem. Furthermore, we propose a novel model that addresses compositional generalization by clustering predicates into groups. Our model generates text in a sentence-by-sentence manner, relying on one cluster of predicates at a time. This approach significantly outperforms T5-baselines across all evaluation metrics. Notably, it achieved a 31% improvement over T5 in terms of a metric focused on maintaining faithfulness to the input.

## introduction data-to-text generation (dtg) @xcite aims to accurately generate textual descriptions from input tuples; the tuples should encompass all the information needed for generating the description regardless of the narrative order. typically, as shown in figure 1 , each tuple consists of two arguments and one predicate that conveys their relationship. 2 given the large number of pre-defined predicates, it is timeconsuming to collect human-annotated training examples for each potential combination of them. thus, models must have the ability to generalize and handle examples with previously-unseen predicate combinations. we refer to this generalization scenario as compositional generalization. prior research @xcite @xcite ; figure 1 : an example from webnlg dataset @xcite . dtg aims at transforming the input structured data (left) into coherent textual description (right). @xcite has focused on evaluating the compositional generalization (cg) abilities of dtg models. these studies created few-shot training splits using established benchmarks by reducing the number of training examples or limiting the number of distinct predicate combinations in the training set through random selection. however, these arbitrary selections overlook the practical effort required for annotating different examples. for example, annotating examples with a larger number of input tuples requires more time and effort. we introduce a test environment based on @xcite . during training, models are exposed to examples with fewer input tuples, while in the testing phase, examples with more input tuples are presented. to make it even more challenging, we combine cg with few-shot learning by reducing the number of training examples for each predicate combination to one. we also incorporate cg with domain adaptation by evaluating the models on unseen domains. our results demonstrate that the sota pre-trained language models (lms; @xcite fail to generalize effectively in our experimental setup. to tackle this issue, we propose a clusteringbased method (figure 2 ) that utilize the graph weights learned from training data to decompose unfamiliar predicate compositions into smaller groups during inference. each group consists of predicate combinations encountered by the model figure 2 : framework of our proposed inference procedure. we introduce a set of clustering-based methods that leverage the graph weights learned during training to decompose the unseen predicate compositions into familiar groups. for each group, we gather the input tuples associated with predicates in that group, and generate a sentence to describe them. the final text description is created by combining the generated sentences from all the groups. during training. then, individual sentence descriptions are generated separately for each group, and combined to form the final description for the input. in contrast to previous studies that primarily rely on self-training to improve cg in dtg @xcite @xcite , as well as using data augmentation to improve cg in various tasks such as semantic parsing @xcite @xcite , our method solely relies on small training sets and does not require any additional human-annotated, automatically labeled, or unlabeled data. in the cg-centric testing scenario, we observe significant improvements across all metrics in the benchmark compared to the vanilla t5 model @xcite . a faithfulness-based metric @xcite shows an impressive gain of 31% over t5. a similar trend is seen when combining the cg challenge with few-shot learning and domain adaptation. our contributions are: ## cg focused benchmark existing benchmarks @xcite @xcite @xcite @xcite @xcite @xcite have provided an important test-bed for dtg models. we specifically choose webnlg 2017 foot_2 @xcite to build our benchmark upon (reasons are shown in appendix b) and primarily focus on assessing the models' capability for cg. we present two sets of training splits and create four distinct testing scenarios with different difficulty levels by incorporating the training splits with the seen and unseen test sets offered in webnlg. ## clustering based cg methods the framework of our approach is shown in figure 2 . for clarity, we denote the input tuples and their corresponding predicates as @xmath0, x 2 , • • • x n } and @xmath1, p 2 , • • • p n } respectively, where p i is the predicate of the i th tuple x i . n represents the total number of tuples. the output text is denoted as @xmath2, y 2 , • • • y m }, where y j represents the j th sentence in the output. fine-tuning pre-trained lms @xcite , aim to maximize the log likelihood of generating the ground truth text y given the linearized input tuples x, denoted as log p (y|x), during training. however, these models face challenges in generalizing to unseen predicate combinations. to overcome this, an intuitive approach is to decompose these unseen combinations into smaller groups, ensuring that the combination in each group has been seen during training; then, generate a sentence from each group individually and combine them to form the final description. we denote the decomposition as @xmath3, c 2 , • • • c m }, where c j represents the j th predicate group responsible for generating sentence y j . since dtg tasks require to include all the input information in the output without repetition, the decomposition must fulfill c 1 ∪ c 2 , • • • ∪ c @xmath4, j : 1 ⩽ @xmath5, c i ∩ c @xmath6. the text generation is then broken down into a set of parallel steps. each step aims at creating a single sentence to describe the tuples associated with the predicates in one of the groups: in turn, a decomposition can be considered as a partitioned graph derived from the original graph g. we encode the partitioned graph as a binary matrix m, where m @xmath7, while m @xmath8. unfortunately, the annotated ground truth decompositions are unavailable in the majority of dtg training sets. therefore, the training objective becomes maximizing the marginal log likelihood of the output text y w.r.t. the latent m. as the number of input tuples increases, exploring all possible decompositions for each example is intractable. following @xcite ; @xcite , we approximate the marginal likelihood as: this results in the stochastic decomposition variable m being replaced with the deterministic value @xmath9. assuming that m follows a bernoulli distribution b(γ), with each element within the matrix being independent, we can represent the distribution of m as m ij ∼ b(γ ij ). thus, the expectation of the binary matrix m is the bernoulli parameter matrix γ. in section 3.1 we demonstrate two training methods to predict γ. ## experiments and results we compare our methods (cg-numerical, cg-nn, cg-rl) against fine-tuning t5 13 (kale and rastogi, 2020b) using the benchmarks introduced in section 2. additionally, we include another baseline called cg-random. it determines the number of groups, ranging from 1 to x and assigns each predicate to one of the groups randomly. comparing to cg-random allows us to gain insights into the impact of our methods on predicate decomposition and how it affects the text generation quality. to enhance readability given the large number of experiments conducted, we present a portion of the results that will be thoroughly discussed in this section. ## limitations and future work we have constructed our benchmark exclusively using webnlg 2017, as it exhibits several favorable characteristics (see appendix b). nonetheless, it would be advantageous to expand the benchmark to include data from a more diverse range of resources. additionally, we recognize the importance of including multiple languages in the benchmark. the multilingual divisions introduced in webnlg 2022 were not included in our benchmark due to the presence of training data generated automatically using translation models, which resulted in noisy data. in the future, we aim to expand our benchmark to include high-quality multilingual data resources. recent work @xcite explored the zero-shot ability of llms on dtg. across benchmarks including webnlg, both studies find that gpt-3 and chat-gpt achieve lower bleu scores compared to finetuned smaller scale models. in addition, llms still face challenges in comprehending the semantic relationships between entities, and the generated text often includes hallucinations. thus, we did not include llms in this study. due to the limited computational resources, we focused our performance testing on t5-base. however, it is important to evaluate the models in different sizes, including llms. we anticipate that tasks with longer inputs/outputs, such as multi-document summarization, may derive even greater benefits from the proposed "solving cg through learning to decompose" idea. in the future, we aim to extend this idea to other tasks.