KGPT : Knowledge-Grounded Pre-Training for Data-to-Text Generation (2020)

Abstract: Data-to-text generation has recently attracted substantial interests due to its wide applications. Existing methods have shown impressive performance on an array of tasks. However, they rely on a significant amount of labeled data for each task, which is costly to acquire and thus limits their application to new tasks and domains. In this paper, we propose to leverage pre-training and transfer learning to address this issue. We propose a knowledge-grounded pre-training (KGPT), which consists of two parts, 1) a general knowledge-grounded generation model to generate knowledge-enriched text. 2) a pre-training paradigm on a massive knowledge-grounded text corpus crawled from the web. The pre-trained model can be fine-tuned on various data-to-text generation tasks to generate task-specific text. We adopt three settings, namely fully-supervised, zero-shot, few-shot to evaluate its effectiveness. Under the fully-supervised setting, our model can achieve remarkable gains over the known baselines. Under zero-shot setting, our model without seeing any examples achieves over 30 ROUGE-L on WebNLG while all other baselines fail. Under the few-shot setting, our model only needs about one-fifteenth as many labeled examples to achieve the same level of performance as baseline models. These experiments consistently prove the strong generalization ability of our proposed framework.

## introduction data-to-text generation, i.e., generating textual description from structured data, is an important task with many real-world applications such as generating weather reports @xcite , sports news @xcite , dialog response @xcite , etc. neural gener-1 https://github.com/wenhuchen/kgpt moses malone, hakeem olajuwon, and james harden have been named the nba's most valuable player while playing for the rockets, for a total of four mvp awards. ation models based on different strategies like softtemplate @xcite , copy-mechanism @xcite , content planning @xcite , and structure awareness @xcite have achieved impressive results. however, existing studies are primarily focused on fully supervised setting requiring substantial labeled annotated data for each subtask, which restricts their adoption in real-world applications. ## dataset construction the construction process has two stages, namely the crawling stage and the selection stage: ## model we formally define the problem setting and kgpt's architectures in this section. ## experiments we experiment with three different down-stream tasks, which covers various table-to-text applications to verify the generalization capability of kgpt. besides the fully supervised learning, we also evaluate zero-shot and few-shot learning.