T a KG : A New Dataset for Paragraph-level Table-to-Text Generation Enhanced with Knowledge Graphs (2022)

Abstract: We introduce TaKG, a new table-to-text generation dataset with the following highlights: (1) TaKG defines a long-text (paragraph-level) generation task as opposed to well-established short-text (sentence-level) generation datasets. (2) TaKG is the first large-scale dataset for this task, containing three application domains and ~750,000 samples. (3) To address the divergence phenomenon, TaKG enhances table input using external knowledge graphs, extracted by a new Wikidata-based method. We then propose a new Transformer-based multimodal sequence-to-sequence architecture for TaKG that integrates two pretrained language models RoBERTa and GPT-2. Our model shows reliable performance on long-text generation across a variety of metrics, and outperforms existing models for short-text generation tasks.

## introduction data-to-text generation refers to semanticpreserving conversion from structured data to (unstructured) text. table -to-text generation is a class of data-to-text generation tasks where the input data takes the form of fact tables @xcite . table -to-text generation has widespread applications from biography generation @xcite to event summarisation @xcite . thus developing a fluent, truthful and informative table-to-text generation system has attracted considerable attention @xcite @xcite . a critical factor in building such a system is to prepare reliable and large-scale table-to-text datasets. however, existing table-to-text generation benchmarks have some clear limitations. first, most existing datasets, such as e2e @xcite and @xcite , focus on singlesentence generation tasks, which severely limits their use for tasks that involve the generation of long texts, e.g., entire paragraphs. then, the few datasets that involve long (paragraph-level) text generation, such as mlb @xcite and rotowire @xcite , consist of too few samples (less than 30k). last, real-world data-to-text generation tasks tend to exhibit the so-called divergence phenomenon, where the input data fail to provide all the key information in the target text description @xcite @xcite . this is illustrated by an example in figure 1 for the dutch painter jacoba surie (extracted from wikibio dataset @xcite ). existing table-to-text datasets in general lack of sufficient external knowledge required to generate the target text. to address these issues, we introduce a new table-to-text generation dataset: takg (table -and-knowledge graph) with the following highlights foot_0 : first, samples in takg contain long text (i.e., paragraphs) and their corresponding infoboxes (tables) extracted from wikipedia. thus takg amounts to a long-text generation task. takg contains three domains: biography, place, school, with a total of 745,574 samples, considerably larger than existing table-to-text datasets. to resolve the divergence issue, we employ external knowledge to "fill" the information in text description that is missing from the input infobox. in particular, we exploit another large-scale knowledge graph (kg) repository wikidata foot_1 . the kgs are added in takg as auxiliary input. figure 1 (upper right) shows an example kg. the goal of this paper is two-fold. (1) we first introduce the takg dataset. in a nutshell, takg defines a task that takes a fact table (i.e., infobox) about a target entity and a wikidata kg as input, and seeks a paragraph-level text description of the target entity. section 3 provides more details. (2) table and bottom text description are extracted from wikipedia. the blue colour words are the item with hyperlinks. the right upper knowledge graph is retrieved from wikidata. in lower biography, the red color words indicate the information missing from wikipedia table but can be found in the knowledge graph. we then demonstrate how takg may be used as a worthy benchmark to train a model for paragraphlevel table-to-text tasks. generating text with multiple data sources is challenging on two axes: table-kg information fusion and high-fidelity natural text generation. to address these challenges, we leverage pretrained language models (plms), such as roberta @xcite and gpt-2 (radford et al., 2019), for their abilities to acquire crossdomain knowledge. a seq2seq architecture is proposed for our task and utilizes two plms that fuses multiple input sources together. more details are provided in section 4. we validate our model's ability to generate long-text using the takg dataset. to further verify our method's wide applicability, we also demonstrate that, over standard short-text (sentence-level) generation tasks such as wikibio, our method also outperforms state-of-the-art benchmarks, with large margin on wikibio up to 7.3% (bleu) and 9% (rouge) increment. see section 5. our contributions are summarized below: 1. creating a large-scale paragraph-level dataset takg for table-to-text generation enhanced with knowledge graphs. 2. long-text generation: designing a new seq2seq model using two plms to accomplish takg tasks. 3. short-text generation: demonstrating that our new model outperforms benchmarks for sentencelevel table-to-text generation tasks. table-to-text datasets. existing table-to-text generation datasets are either single-sentence or multisentence generation tasks. the former, such as e2e (restaurant domain) @xcite , totto @xcite , and wikibio (biography domain) @xcite , are limited in terms of what the task seeks to generate. the latter, such as mlb (26.3k samples) @xcite , rotowire (4.9k samples) @xcite , uk-place (12k samples) and uk-school (5k samples) @xcite contain very few samples and are thus too small-scale. wikibio dataset above differs from the other datasets in the sense that each of its samples contains in fact a full paragraph of biography of a person. nevertheless, the task wikibio specifies only the first sentence of the paragraph as the ground truth output. indeed, all benchmarks tested on wik-ibio used the dataset as a single-sentence text generation task. due to the presence of paragraph-level texts in wikibio, we include wikibio samples in takg by incorporating the entire paragraph as ground truth output. divergence has been a common issue in multisentence generation @xcite @xcite . to address this issue, in the uk-place and uk-school datasets @xcite complements the input tables with some background knowledge. to obtain the background knowledge, the authors take hyperlinked keywords in the wikipedia infobox and extract one-hop facts of those keywords from wikidata, a large-scale open-domain knowledge graph repository containing close to 100 million data items @xcite . these one-hop facts are then used as background knowledge. however, we point out that this method may produce irrelevant background knowledge that distracts the generation of target text. this is because the hyperlinked keywords in the infoboxes are often not item-specific. for example, 'painting' and 'amsterdam' are keywords for the instance illustrated in figure 1 , which are clearly insufficient to deriving specific facts about the dutch painter jacoba surie. in our work, we will integrate samples from uk-place and uk-school into takg while adopting a different way to derive external knowledge graph. ## experiments in our work, there are two types of tasks: sentencelevel generation means to generate one sentence from input data, paragraph-level generation generates long text (more than one sentence). we borrow the idea of linearisation @xcite on table and kg data. since gpt-2 can generate text with common sense, to some extent it is not necessary to re-train a language model from scratch. observe that we have the same text-generation goal as the pretraining target of gpt-2 had. hence we select roberta as encoder and gpt-2 as the decoder. via fine-tuning roberta and gpt-2 for text generation, the proposed model treats roberta as a feature extractor and gpt-2 as a black-box with encoded text input and text output. for roberta and gpt-2, we use the pre-generated vocabulary and fine-tune the embedding layer. in particular, we have three types of layer settings: 1-layer, 2-layer and 12-layer. here, 1-layer means fine-tuning the first layer of roberta and gpt-2 in the studied seq2seq model. similarly, 2-layer and 12-layer mean the corresponding layers to be fine-tuned. there are added cross-attention @xcite layers in gpt-2, which are trained from scratch. decoding strategy also needs to be imposed during data-to-text generation. here, we use nucleus sampling (@xmath0.9) and top-k (@xmath1. ## conclusion in this study, we introduce takg (745,574 samples), the first large-scale kg-enhanced tableto-text dataset. different from existing wellestablished sentence-level generation datasets, takg defines a paragraph-level generation task. each sample of takg includes three parts: fact table (wikipedia), knowledge graph (wikidata) and paragraph-level description of an entity @xcite . we then propose a new transformer-based sequence-to-sequence architecture for takg that integrates two pretrained language models roberta and gpt-2. for paragraph-level generation, to generate text with multiple structured data sources, we use the simple yet effective concatenation-based fusion to combine the multiple structured data representation. our model shows the ability to generate reliably long texts using multiple data sources (table and kg) with the evaluation on bleu, parent and semantic similarity score (sts-roberta). to further verify the ability of the proposed method, we conduct the experiments on sentence-level text generation using wikibio. our method outperforms the best benchmark models with large margin on wikibio with 7.3% (bleu) and 9% (rouge) increment.