W iki G raphs: A W ikipedia Text - Knowledge Graph Paired Dataset (2021)

Abstract: We present a new dataset of Wikipedia articles each paired with a knowledge graph, to facilitate the research in conditional text generation, graph generation and graph representation learning. Existing graph-text paired datasets typically contain small graphs and short text (1 or few sentences), thus limiting the capabilities of the models that can be learned on the data. Our new dataset WikiGraphs is collected by pairing each Wikipedia article from the established WikiText-103 benchmark (Merity et al., 2016) with a subgraph from the Freebase knowledge graph (Bollacker et al., 2008). This makes it easy to benchmark against other state-of-the-art text generative models that are capable of generating long paragraphs of coherent text. Both the graphs and the text data are of significantly larger scale compared to prior graph-text paired datasets. We present baseline graph neural network and transformer model results on our dataset for 3 tasks: graph -> text generation, graph -> text retrieval and text -> graph retrieval. We show that better conditioning on the graph provides gains in generation and retrieval quality but there is still large room for improvement.

## introduction parallel datasets that pair data from different sources and modalities have enabled large amounts of research on cross modality learning. paired image-caption datasets enable models to describe visual scenes in natural language @xcite , paired streams of speech and transcription data makes it possible to train speech recognition systems @xcite or text-to-speech synthesis models @xcite , and parallel corpus of text in different languages enable learned machine translation models @xcite . 1 the data and the code to reproduce our baseline results are available at https://github.com/deepmind/ deepmind-research/tree/master/wikigraphs "where the streets have no name" is a song by irish rock band u2. it is the opening track from their 1987 album the joshua tree and was released as the album's third single in august 1987. the song 's hook is a repeating guitar arpeggio using a delay effect, played during the song's introduction and again at the end. lead vocalist bono wrote... type.object.name where the streets have no name key/wikipedia.en people.person.date of birth 1960-05-10 the edge key/wikipedia.en 1.77m people.person.date of birth 1961-08-08 people.person.height meters wikitext-103 ns/music.composition.language david howell evans, better known by his stage name the edge, is a british-born irish musician, songwriter... ns/common.topic.description ns/m.02h40lc we present a new dataset of wikipedia text articles each paired with a relevant knowledge graph (kg), which enables building models that can generate long text conditioned on a graph structured overview of relevant topics, and also models that extract or generate graphs from a text description. there has been many prior efforts trying to build datasets for learning @xmath0. however, existing graph-text paired datasets are mostly small scale, where the graphs tend to have 10-20 or even less nodes, and the text typically only contains one or a few sentences. this represents a significant contrast with the state-ofthe-art text generation models @xcite , which can already generate very fluent and long text that spans thousands of tokens over multiple paragraphs. we attempt to bridge this gap, with the goal of advancing the state-of-the-art @xmath1, graph representation learning models and also text-conditioned graph generative models. each text document in our dataset is a full-length wikipedia article, and we pair each of them with a kg that are significantly bigger than prior datasets of similar nature and includes much richer information. hand labelling text articles with kgs is expensive and not scalable @xcite , therefore we utilize an existing and established knowledge base, freebase @xcite , and designed an automated process to extract a relevant subgraph from it for each wikipedia article. to make the text generation results on our dataset directly comparable to the state-of-the-art, we chose the set of wikipedia articles from the established language modeling benchmark wikitext-103 @xcite , which contains a subset of high-quality wikipedia articles. this gives us a dataset of 23,522 graph-text pairs in total, covering 82.3% of wikitext-103 articles. on average each graph has 38.7 nodes and 48.3 edges, and each text article contains 3,533.8 tokens. in addition to structural information, our graphs also contain rich text information with an average of 895.1 tokens in each graph. furthermore, the automatic process we used to create this dataset can be extended to pair any wikipedia document with freebase, and can be scaled up to create over 3m graph-text pairs. out of many exciting new tasks that this dataset enables, we present 3 possibilities: @xmath2, @xmath3, and @xmath4. we benchmarked a few baseline models on these tasks. the models we considered were based on the recent transformer-xl @xcite model, and we adapted it to condition the text generation on the kg in different ways. our results show that better conditioning on the graph indeed improves the relevance of the generated text and the retrieval quality. however, there is still significant room for improvement on these tasks, which makes this an exciting dataset for research. our data and code for baseline models will be made publicly available. ## related work graph-text paired data there has been a lot of prior work on creating graph-text paired datasets. example applications include generating text summaries conditioned on abstract meaning representation graphs @xcite , generating the abstract of a scientific article given a kg and title @xcite and generating text from rdf triples @xcite . in the following we will mostly review related work on kg -text paired datasets. annotating kg or text to create paired datasets is expensive, as a good quality annotation requires annotators that understand the content and structure of the text and the corresponding kg @xcite 2020). therefore previous kg-text paired datasets that rely on human annotation have limited scale. among these, @xcite crowdsourced human annotators to verbalize rdf triplets taken from dbpedia @xcite to a few sentences (webnlg) and this caused errors in annotation that were fixed with a few updates through years. parikh et al. (2020) paired wikipedia table with one sentence text that is created by annotators that revise wikipedia text. another line of research focuses on eliminating the need of human annotations by automatically matching kg-text pairs or generating kgs from text using existing tools. @xcite automatically matched wikipedia infobox of biographies with their first sentence. @xcite utilized an earlier information extraction system that extracts entities, co-reference and relations from given text to build kg's. the gen-wiki dataset @xcite ) is automatically constructed by querying kgs in dbpedia with the title of articles in wikipedia followed by filtering and entity annotation. we construct our wikigraphs dataset by extracting a subgraph from freebase @xcite for each wikipedia article following a scalable automatic process. compared to previous work, our wikigraphs dataset contains significantly larger graphs and longer text ( models for graph-text paired data recent state of art language models are based on the transformer architecture @xcite that uses the self attention mechanism. the transformer-xl @xcite model further introduces a segment level recurrence with a novel positional encoding resulting in impressive performance in long sequences by capturing dependencies beyond a fixed length window. graph neural networks (gnns) @xcite learn representations for graph structured data through a message passing process. this class of models naturally exploit the graph structures, making them a good fit for graph data. gnns have been used in many applications on kg's @xcite @xcite . fundamentally, transformers can also be understood as a special type of gnns with a fully-connected graph structure. the most recent prior work on graph-to-text generation follows an encoder-decoder architecture @xcite , where the graph part is encoded with a gnn model, e.g. graph attention network (gat) @xcite . the text part is typically modeled using an attention based decoder with a copy mechanism (e.g. bilstms as in @xcite ) to process input from both the kg and text. the models we benchmarked for graph-to-text generation were based on the transformer-xl architecture and conditioned on the graph through a gnn, making full use of the graph structure and capable of generating very long text comparable to the state-of-the-art. ## dataset in this section we first present some properties of our dataset, and then describe the process that we used to create it. ## experiments we perform a set of experiments to showcase how the text and graph information can be combined in a language model. specifically, we consider three tasks: text generation conditioned on the graph, graph retrieval given the text, and text retrieval given the graph. ## conclusion in this paper, we present wikigraphs, a new graphtext paired dataset with significantly larger graphs and longer text compared to previous datasets of similar nature. we show that the text part of this data is a good benchmark for state-of-the-art text generation models, and the paired dataset can help us benchmark models that are capable of generating long and coherent text conditioned on a graph structure. in the first set of experiments on this dataset we showcase 3 different tasks using our dataset, and demonstrate the benefit of better models that make more use of the graph structure. there is still significant room for improvement for these tasks on our dataset, and we hope the release of the data and baseline code can help spur more interest in developing models that can generate long text conditioned on graphs, and generate graphs given text, which is another exciting direction our dataset enables but we did not explore, and eventually bridging the graph and text modalities. ns/m.02h40lc "inglés" key/wikipedia.en "/en/english" ns/type.object.id "english" ns/common.topic.alias "english is a west germanic language that was first spoken in early medieval england and is now a global lingua franca. it is an official language of almost 60 sovereign states, the most commonly spoken language in the united kingdom, the united states, canada, australia, ireland, and new zealand, and a widely spoken language in countries in the caribbean, africa, and south asia. it is the third most common native language in the world, after mandarin and spanish. it is widely learned as a secon...