Neural Data-to-Text Generation with LM -based Text Augmentation (2021)

Abstract: For many new application domains for data-to-text generation, the main obstacle in training neural models consists of a lack of training data. While usually large numbers of instances are available on the data side, often only very few text samples are available. To address this problem, we here propose a novel few-shot approach for this setting. Our approach automatically augments the data available for training by (i) generating new text samples based on replacing specific values by alternative ones from the same category, (ii) generating new text samples based on GPT-2, and (iii) proposing an automatic method for pairing the new text samples with data samples. As the text augmentation can introduce noise to the training data, we use cycle consistency as an objective, in order to make sure that a given data sample can be correctly reconstructed after having been formulated as text (and that text samples can be reconstructed from data). On both the E2E and WebNLG benchmarks, we show that this weakly supervised training paradigm is able to outperform fully supervised sequence-to-sequence models with less than 10% of the training set. By utilizing all annotated data, our model can boost the performance of a standard sequence-to-sequence model by over 5 BLEU points, establishing a new state-of-the-art on both datasets.

## introduction neural data-to-text generation has been the subject of much recent research. the task aims at transforming source-side structured data into target-side natural language text @xcite . while neural end-to-end systems afford the advantage of easy adaptability @xcite , huge amounts of data-text pairs are still necessary to perform on par with their rule-based counterparts the blue spice is a restaurant that serves english cuisine. table only u l figure 1: few-shot scenario: the model is expected to learn data-to-text generation with few labeled instances (i.e. table-text pairs). the example is taken from the e2e dataset. to address the limited-data challenge, we propose a simple yet effective way of augmenting the text side with the pretrained language model (lm) gpt-2 @xcite . unlike other text augmentation work employed in data-to-text generation systems @xcite , our proposal assumes little to no domain-dependent heuristics. it consists of two steps: (1) information augmentation by slot-value replacement and (2) lm augmentation by gpt-2 generation. once we have augmented the set of text samples, we are essentially in a similar setting as previously proposed semi-supervised approaches to data-totext generation @xcite ; @xcite ; @xcite , which assume the presence of vast amounts of unpaired data and text instances. these approaches exploit a cycle consistency objective in order to learn a pairing for the data samples. the cycle consistency objective tries to make sure that data samples can be reconstructed correctly from their textual formulations, and similarly that texts can be reconstructed after having been parsed into a data representation. as the automatically generated text samples from gpt-2 might be very noisy and not pair well with data samples, we align each augmented text sample with its most similar unlabeled data sample, as defined in their encoded vector space. this idea is inspired by recent work on representation matching in mt @xcite . to ensure good quality of the training data, only pairs above a certain similarity threshold are retained as pseudo pairs for training. the quality of the pseudo pairs will gradually improve as the encoder improves in the training process. in return, the learning of the encoder will also be facilitated with the improved quality of pseudo pairs as a virtuous cycle. on two data-to-text benchmarks e2e @xcite and webnlg @xcite , we show that our lm-augmented weakly supervised model succeeds on outperforming fully supervised seq2seq model, though utilizing less than 10% of the data annotations. it even outperforms previous work which additionally has access to all unpaired text samples. when trained with full data annotations, it is able to boost the model performance by up to 5 bleu points, establishing a new state-of-the-art on both datasets. in summary, this work makes the following contributions: ## related work building neural data-to-text systems with few paired samples (but a large set of unpaired samples) has been a hot research topic recently. most works adopt the idea of cycle consistency @xcite , which has been used in many text generation tasks like machine translation @xcite and style transfer @xcite . schmitt and schütze (2019); qader et al. (2019); su et al. (2020); chang et al. (2020, 2021a,b) applied this idea to the task of data-to-text generation and reported promising results. ma et al. (2019) ## problem formulation we represent the data samples as d and the text samples as t. in our work, we do not restrict the format of the data. each @xmath0, as in figure 1 , or in form of rdf triples as in @xcite . each text @xmath1. in few-shot settings, we are assumed to have (1) k labeled pairs (d l , t l ) and ( 2 ) large quantities of unlabeled data d u where |d u | @xmath2. this, we believe, is a more realistic setting as unlabeled data are usually abundant and also can be easily fabricated from predefined schemata. notably, we assume no access to outside resources containing in-domain text. the k annotations are all we know about the text side. ## approach in this section, we first explain our proposed new method for text sample augmentation, and then discuss methods to remove noise and automatically align the data by elaborating on the ideas of cycle consistency and representation matching. finally, we summarize the approach and present the detailed algorithm. ## results and analysis in this section, we present experiment results and analysis. we first compare our model with other baselines on both datasets, then perform a set of ablation studies on the e2e dataset to see the effects of each component. finally, we analyze how text augmentation helps improves the model, include example outputs and show the human evaluation results in the end. 2019 ) further utilized all the groundtruth unpaired text samples, while the other models run only on the few-shot annotations. we also include the results of slug @xcite and melbourne @xcite , the overall winner on automatic metrics in the e2e and webnlg challenge respectively(both seq2seqbased). slug uses a heuristic slot aligner based on a set of handcrafted rules and combines a complex pipeline of data augmentation, selection, model ensemble and reranker. the results show that our proposed model significantly improves over the baseline on both the few-shot and fully supervised setting. the improvement is more evident when only 10% annotations are available, with a leap of 11 and 7 bleu scores on e2e and webnlg respectively. it also outperforms systems relying on task-dependent heuristics. in comparison, @xcite , though with access to all text samples at all percentages, still underperforms our model with tangible margin. difference compared with the seq2seq baseline as no more extra data is incorporated in the training process. as such, we also observe that the text augmentation from finetuned gpt-2 model helps the proposed model on the few-shot setting, but its advantage also vanishes when all data annotations are available. in figure 4 , we draw the model performance with varying number of data annotations. all models are trained from scratch with 10 different random initializations and the standard deviation of the bleu-4 score is visualized. we can see our model (lstm-based), though with a relatively larger standard deviation due to the uncertainty of text augmentation sampling, still consistently outperforms other baselines significantly and even surpasses the fully supervised seq2seq model with less than 10% of data annotations. ## conclusion we study few-shot data-to-text generation with only limited annotated data. we propose text augmentation with slot-value replacement followed by gpt-2 generation. the augmented text, when combined with cycle consistency and representation matching, is shown to help the model to generalize to unseen new tokens and patterns of token combinations. with less than 10% annotations, it outperforms supervised seq2seq model trained on 100% annotations and is extensible enough to be combined with pretraining techniques.