Knowledge Graph Based Synthetic Corpus Generation for Knowledge-Enhanced Language Model Pre-training (2021)

Abstract: Prior work on Data-To-Text Generation, the task of converting knowledge graph (KG) triples into natural text, focused on domain-specific benchmark datasets. In this paper, however, we verbalize the entire English Wikidata KG, and discuss the unique challenges associated with a broad, open-domain, large-scale verbalization. We further show that verbalizing a comprehensive, encyclopedic KG like Wikidata can be used to integrate structured KGs and natural language corpora. In contrast to the many architectures that have been developed to integrate these two sources, our approach converts the KG into natural text, allowing it to be seamlessly integrated into existing language models. It carries the further advantages of improved factual accuracy and reduced toxicity in the resulting language model. We evaluate this approach by augmenting the retrieval corpus in a retrieval language model and showing significant improvements on the knowledge intensive tasks of open domain QA and the LAMA knowledge probe.

## introduction data-to-text generation @xcite involves converting knowledge graph (kg) triples of the form @xcite into a natural language sentence(s). there are many standard datasets for this task such as webnlg @xcite and many systems have been developed to improve performance on these datasets. however, to the best of our knowledge, no prior work has attempted to verbalize a full knowledge graph. verbalizing a full kg has additional challenges over small benchmark datasets, such as entity and relation coverage and the lack of grouped sets of triples that can produce coherent sentences together. in this paper, we convert the english wikidata kg @xcite into natural language text (figure 1 ). the first, the entity subgraphs on the left are created and then converted to the sentence on the right. we evaluate the quality of the generated corpus through human evaluation of a random sample. we further showcase the utility of this corpus in language model pre-training. text represents a limited coverage of the world knowledge. therefore, we expect the language models to be restricted to facts that are expressed in natural language. moreover, facts may not be expressed as explicitly in text as they are in kgs, and the variability in the quality of text can eventually cause biases in the resulting models @xcite @xcite . building models that handle structured data and free form text seamlessly has been a long sought-after goal. however, their integration is challenging due to different structural formats. kg verbalization provides a simple way to integrate kgs with natural text. we illustrate this by augmenting the realm @xcite retrieval corpus with the kelm corpus. we evaluate ## tekgen one of the challenges in converting an entire kg to text is the wide variety of entities and relations. wikidata consists of ∼6m entities and ∼1500 relations. in comparison, the webnlg dataset has ∼600 entities and ∼20 relations. in this section, we discuss the various components of tekgen, also illustrated in figure 2 -1. create a large yet noisy training dataset using distant supervision. ## kelm corpus in this section, we utilize the tekgen model and filtering mechanism to build a synthetic corpus that captures the kg in natural language format. ## knowledge enhanced lms in this section, we showcase an application of the generated kelm corpus as a way to integrate kgs into natural text corpora for pre-training language models (lms), as shown in figure 5 . we choose realm @xcite as a representative of the recently introduced family of retrieval language models and therefore we expect our work to be equally applicable to other such language models. we show gains on lama knowledge probe and open domain qa with augmentation. we also perform experiments where we integrate raw wikidata triples instead of kelm corpus to confirm the effectiveness of verbalization. ## conclusion in this paper, we converted an entire kg (wikidata) to natural text (kelm corpus), tackling various challenges over verbalizing domain-specific benchmark datasets. we further showcase that kg verbalization can be used to integrate kgs and natural text corpora by including the verbalized kg as additional pre-training data. we augment a retrieval-based language model with the generated synthetic kelm corpus as a retrieval corpus. we evaluated the augmented model on open domain qa and a knowledge probe, showing improvements on both. the kelm corpus is publicly available at https://github. com/google-research-datasets/ kelm-corpus .