Few-Shot NLG with Pre-Trained Language Model (2020)

Abstract: Neural-based end-to-end approaches to natural language generation (NLG) from structured data or knowledge are data-hungry, making their adoption for real-world applications difficult with limited data. In this work, we propose the new task of few-shot natural language generation. Motivated by how humans tend to summarize tabular data, we propose a simple yet effective approach and show that it not only demonstrates strong performance but also provides good generalization across domains. The design of the model architecture is based on two aspects: content selection from input data and language modeling to compose coherent sentences, which can be acquired from prior knowledge. With just 200 training examples, across multiple domains, we show that our approach achieves very reasonable performances and outperforms the strongest baseline by an average of over 8.0 BLEU points improvement. Our code and data can be found at https://github.com/czyssrs/Few-Shot-NLG

## introduction natural language generation (nlg) from structured data or knowledge @xcite is an important research problem for various nlp applications. some examples are taskoriented dialog, question answering @xcite @xcite @xcite and interdisciplinary applications such as medicine @xcite and healthcare @xcite . there is great potential to use automatic nlg systems in a wide range of real-life applications. recently, deep neural network based nlg systems have been developed, such as those seen in the e2e challenge @xcite , weather-gov @xcite , as well as more complex ones such as wikibio @xcite and ro-towire @xcite . compared to traditional slot-filling pipeline approaches, such neural-based systems greatly reduce feature engineering efforts and improve text diversity as well as fluency. although they achieve good performance on benchmarks such as e2e challenge @xcite and wikibio @xcite , their performance depends on large training datasets, e.g., 500k table-text training pairs for wikibio @xcite in a single domain. such data-hungry nature makes neural-based nlg systems difficult to be widely adopted in real-world applications as they have significant manual data curation overhead. this leads us to formulate an interesting research question: motivated by this, we propose the new task of fewshot natural language generation: given only a handful of labeled instances (e.g., 50 -200 training instances), the system is required to produce satisfactory text outputs (e.g., @xmath0. to the best of our knowledge, such a problem in nlg community still remains under-explored. herein, we propose a simple yet very effective approach that can generalize across different domains. in general, to describe information in a table, we need two skills to compose coherent and faithful sentences. one skill is to select and copy factual content from the table -this can be learned quickly by reading a handful of tables. the other is to compose grammatically correct sentences that bring those facts together -this skill is not re- input table attribute (r) value (v) name walter extra nationality german occupation aircraft designer and manufacturer ... ... table encoder attention weights walter extra is ... pre-trained language model walter extra german name name nationaltily table values attribute names position information walter extra is a … ... the swicth policy name name ----... ----... ## conclusion in this paper, we propose the new research problem of few-shot natural language generation. our approach is simple, easy to implement, while achieving strong performance on various domains. our basic idea of acquiring language modeling prior can be potentially extended to a broader scope of generation tasks, based on various input structured data, such as knowledge graphs, sql queries, etc. the deduction of manual data curation efforts for such tasks is of great potential and importance for many real-world applications. ods. each evaluation unit is assigned to 3 workers to eliminate human variance. the first study attempts to evaluate how well a generated text can correctly convey information in the table. each worker is present with both the input table and a generated text, and asked to count how many facts in the generated text are supported by the table, and how many are contradicting with or missing from the table, similar as in @xcite . the we calculate the average number of supporting and contradicting facts for the texts generated by each method. the second study aims to evaluate whether the generated text is grammatically correct and fluent in terms of language, regardless of factual correctness. each worker is present with a pair of texts generated from the same input table, by two different methods, then asked to select the better one only according to language naturalness, or "tied" if the two texts are of equal quality. the input table is not shown to the workers. each time a generated text is chosen as the better one, we assign score of 1.0. if two texts are tied, we assign 0.5 for each. we then calculate the average score for the texts generated by each method, indicating its superiority in pairwise comparisons with all other methods. the significance test is conducted respectively on all three measures: number of supporting facts and number of contradicting facts for the first study; the assigned score for the second study. we use the tukey hsd post-hoc analysis of an anova with the worker's response as the dependent variable, the method and worker id as independent variables.