Improving Compositional Generalization with Self-Training for Data-to-Text Generation (2022)

Abstract: Data-to-text generation focuses on generating fluent natural language responses from structured meaning representations (MRs). Such representations are compositional and it is costly to collect responses for all possible combinations of atomic meaning schemata, thereby necessitating few-shot generalization to novel MRs. In this work, we systematically study the compositional generalization of the state-of-the-art T5 models in few-shot data-to-text tasks. We show that T5 models fail to generalize to unseen MRs, and we propose a template-based input representation that considerably improves the model’s generalization capability. To further improve the model’s performance, we propose an approach based on self-training using fine-tuned BLEURT for pseudo-response selection. On the commonly-used SGD and Weather benchmarks, the proposed self-training approach improves tree accuracy by 46\%+ and reduces the slot error rates by 73\%+ over the strong T5 baselines in few-shot settings.

## introduction data-to-text generation @xcite is a critical component in today's taskoriented dialog systems for producing fluent natural language responses to users' requests. the task takes structured meaning representations (mrs) as input for natural language text response generation. such representations are compositional, which allows for the combination of atomic meaning units in various ways to express the rich semantics encoded in languages. recently, large pre-trained language models (lms) have shown impressive results on many language understanding and generation tasks @xcite @xcite , however it remains unclear how well these lms generalize compositionally to novel semantic representations. there have been many studies revealing that large lms often memorize the patterns from training data, while generalizing poorly to novel patterns. compositionality in languages @xcite further aggravates such issues as the number of novel structural combinations exponentially increases with the number of atomic semantic units. in recent years, we have seen progress on benchmarking and measuring compositional generalization for languages @xcite , from perspectives including specialized architectures @xcite and learning strategies @xcite . however, most of these works study the generalization for nlu tasks like question answering @xcite and semantic pars- query: is it jacket weather? [dg_no ] [dg_inform [condition light rain ] [humidity extremely humid ] [date_time [colloquial today ] ] [location [city palo alto ] ] ] [ds_justify [dg_recommend [attire_not jacket ] [location [city palo alto ] ] [date_time [colloquial today ] ] ] [dg_inform [condition_not cold ] [location [city palo alto ] ] [date_time [colloquial today ] ] ] ] (a) naive structured input query: is it jacket weather? [dg_no no ] [dg_inform there will be [condition light rain ] [humidity extremely humid ] [date_time at [colloquial today ] ] [location in [city palo alto ] ] ] [ds_justify [dg_recommend [attire_not jacket ] is not recommended [location in [city palo alto ] ] [date_time at [colloquial today ] ] ] , because [dg_inform there won't be [condition_not cold ] [location in [city palo alto ] ] [date_time at [colloquial today ] ] ] ] (b) template guided structured input [dg_no no ] , [ds_justify [dg_recommend leave the [attire_not jacket ] at home ] because [dg_inform it isn't [condition_not cold ] [date_time [colloquial today ] ] [location in [city palo alto ] ] ] . ] [dg_inform it'll be [humidity extremely humid ] with [condition light rain ] ] . response: no, leave the jacket at home because it isn't cold today in palo alto. it'll be extremely humid with light rain. ing @xcite . to the best of our knowledge, compositional generalization for natural language generation is still an under-explored problem, which is the focus of this work. to answer the question of whether pre-trained lms still suffer from lack of compositional generalization, we start with an empirical evaluation of t5 @xcite , the state-of-the-art model on data-to-text generation tasks @xcite . in our study, we use the weather dataset @xcite consisting of tree-structured compositional mrs along with treestructured output responses (see figure 2 for (a) naive mr and (c) target response). for evaluation, we compute the tree accuracy @xcite which measures exact match between input and generated tree-structures. in this study we observe 47%-80% (across different few-shot train splits) drop in the tree accuracy when evaluated on validation splits containing unseen treestructures in comparison to splits containing seen tree-structures (figure 1 ). furthermore, simply increasing the model size from t5-small to t5-large does not close the generalization gap ( inspired by @xcite , we examine whether template-guided mrs are effective over naive mrs for tackling compositional generalization in data-to-text tasks. we introduce a simple template engine that traverses the compositional mr in a top-down manner and converts it to a text representation (figure 2(b) ). we hypothesize that such a template-guided setup reduces the change in representation between lm pre-training and finetuning. with template-guided mrs, we report up to 2x increase in the tree accuracy over naive mrs on the validation split with unseen structures, demonstrating improved model generalization. we also propose to self-train the generation model to further boost performance by mitigating data sparsity in the low-data regime without requiring additional manual annotation. concretely, we augment the limited labeled mrs with unlabeled novel mrs to iteratively bootstrap the model. to filter out noisy pseudo responses during self-training, we repurpose bleurt @xcite , a learned metric, to be a quality estimator. we synthetically generate datasets for finetuning bleurt with the goal of identifying hallucinations, missing slot-values, and ungrammatical responses. in sum, our overall approach improves the tree accuracy on unseen structures of the fewshotweather dataset by 12.3%-46.4% over strong t5 baselines. on unseen schemata of the fewshotsgd dataset, we reduce the slot error rate by 54.4%-73.0%. ## case study: compositional generalization in data-to-text tasks in this section, we are interested in investigating the following with respect to data-to-text tasks: (q3) does scaling model size (and training data) trivially solve compositional generalization? problem setup data-to-text generation is the task of generating natural language text y from meaning representation (mr) x. in the context of task-oriented dialog systems, the choice of mr ranges from a flat list of slot-value pairs @xcite to a more expressive tree structure. balakrishnan et al. (2019) defines tree-structured mrs consisting of arguments, dialog acts, and discourse relations, which we use in this work. they report significant gains in the naturalness of the generated responses with tree-structured mrs on the weather domain dataset. figure 2 (a) visualizes an instantiation of such a tree-structured mr where the argument location is made up of a subargument (city), the dialog act recommend consists of three arguments (attire_not, location, date_time), and the discourse relation justify captures the relationship between two dialog acts (recommend, inform). we consider linearized versions of treestructured mr x and output response y. generating the tree structure in the output enables us to compute the tree accuracy which helps to assess the structural correctness of the predicted response. fewshotweather dataset due to the compositional nature of mrs, it is costly to collect responses for all combinations of discourse relations, dialog acts and arguments. in order to keep data labeling costs under control, we simulate a more realistic few-shot (or limited labeled data) setup. in the original weather dataset, we have 25, 390 training examples spanning 4, 690 unique tree-structured mrs. an unique tree-structured mr is defined as a novel composition of discourse relations, dialog acts and argument names. basically, they constitute non-terminals of a tree (figure 2 (a) without terminals or argument values like extremely humid, light rain, today, palo alto, jacket, and cold). for the weather dataset @xcite , we construct 4 few-shot splits: 1shot-250, 1shot-500, 1shot-750, and 1shot-1000, where 1shot-x denotes training split to include one example per unique tree-structured mr and in total x unique tree-structured mrs. further, all x examples in 1shot-x are included while constructing 1shot-y splits, where @xmath0. we also make sure each discourse relation, dialog act and argument name is represented at least once in our few-shot splits. however, all combinations of these may not exist, thus allowing us to simulate structural shifts and evaluate compositional generalization. based upon these splits, we construct two evaluation sets: seen tree-structures (overlapping with tree-structured mrs from 1shot-250) and unseen tree-structures (disjoint with tree-structured mrs from 1shot-1000) (see section 4.1 for more details). henceforth, all of the above splits constitute the fewshotweather dataset. we release these splits for future studies. ## self-training as discussed earlier, the compositional nature of mrs makes it difficult to collect responses for all combinations. however, with access to data simulators @xcite , it is feasible to automatically generate large amounts of unlabeled mrs. given limited labeled mrs, @xmath1, y i } n @xmath2, and assuming access to unlabeled mrs, @xmath3, we investigate self-training @xcite , a semisupervised learning approach to effectively use u to improve compositional generalization. self-training starts from a model trained on labeled data s, iteratively applies the current model to generate pseudo-labels on unlabeled data u , and then re-trains the current model on the augmented version of s and (subset of) u . for self-training to be effective, one needs to carefully select confident pseudo labels to alleviate the risk of reinforcing the model's mistakes @xcite . this issue gets further exacerbated in the context of generation tasks, where neural models are prone to hallucinate additional content not supported by the input @xcite . with recent developments in learned evaluation metrics that penalize the model for hallucination, fluency, etc., we pose the question: can we repurpose those metrics to assess the quality of pseudoresponses during self-training? formally, given a pair of template guided mr (source) and model predicted response (candidate), we want a model that estimates the response quality by looking for hallucinations, fluency, coverage of argument valuepairs. ideally, to learn such a model we require a large amount of positive and negative text pairs. to alleviate this requirement, we propose synthesizing the examples using the limited labeled task dataset. furthermore, we initialize our quality estimation model using a pre-trained bleurt @xcite , which is shown to be sample efficient and robust to data shifts as a learned evaluation metric. soruce (text-to-text input): there will be light freezing fog with a temperature high of 74 low of 61 at next friday positive candidate (target response): next friday will have a high of 74 , a low of 61 , and a light freezing fog negative candidates: ## conclusion and future work we systematically study the problem of compositional generalization for data-to-text generation and show that existing state-of-the-art generation models do not generalize to unseen structures. we propose a simple and generic way to produce template guided text representation for response generation, and demonstrate its effectiveness on both seen and unseen structures. further, we introduce a generic self-training approach that leverages finetuned bleurt for pseudo response selection and show significant improvements over vanilla selftraining on existing few-shot data-to-text genera-tion benchmarks. while our method requires only a small number of templates to start with, we still need to manually generate them for every unseen mr. automatically generating templates by priming gpt-style models is an interesting line of future work. furthermore, the effectiveness of our self-training method is highly dependent on the quality of the underlying bleurt model (see