Utilising Knowledge Graph Embeddings for Data-to-Text Generation (2020)

Abstract: Data-to-text generation has recently seen a move away from modular and pipeline architectures towards end-to-end architectures based on neural networks. In this work, we employ knowledge graph embeddings and explore their utility for end-to-end approaches in a data-to-text generation task. Our experiments show that using knowledge graph embeddings can yield an improvement of up to 2 – 3 BLEU points for seen categories on the WebNLG corpus without modifying the underlying neural network architecture.

## introduction data-to-text generation is concerned with building systems that can produce meaningful texts in a human language from some underlying non-linguistic representation of information @xcite . traditionally, most applications for data-totext generation have relied on rule-based systems which are designed using a modular pipeline architecture @xcite . however, there has been a shift recently towards end-to-end architectures using neural networks to convert data in the input to text in a natural language in the output. this trend is largely inspired by the success of the end-to-end approaches in the related task of machine translation as well as the availability of large corpora for data-to-text generation such as the wik-ibio @xcite or the rotowire @xcite datasets, which contain input data in the form of a table consisting of rows and columns. however, the structure and representation of the input data can vary significantly depending on the task at hand. for example, the input can also be a knowledge graph (kg) represented as a set of rdf-triples like the webnlg corpus @xcite or a dialogue-act-based meaning representation like the e2e dataset @xcite . in this work, we employ pre-trained knowledge graph embeddings (kges) for data-to-text generation with a model which is trained in an end-toend fashion using an encoder-decoder style neural network architecture. these embeddings have been shown to be useful in similar end-to-end architectures especially in domain-specific and underresourced scenarios for machine translation @xcite . we focus on the webnlg corpus which contains rdf-triples paired with verbalisations in english. we compare the use of kges to two baseline models -the standard sequenceto-sequence model with attention @xcite and the transformer model @xcite . we also do a comparison with pre-trained glove word-embeddings @xcite . ## dataset description the webnlg corpus consists of data units made up of rdf-triples extracted from dbpedia @xcite , paired with reference text lexicalisations. these texts contain sequences of one or more short sentences in english, verbalising the data units in the input. the corpus contains triplesets from 15 dbpedia categories and is divided into two subsets, seen and unseen for evaluation. the ten seen categories are airport, astronaut, building, city, comicscharacter, food, monument, sport-steam, university and writtenwork and the five unseen categories are artist, athlete, celestialbody, company, meanoftransportation and politician. the corpus 1 contains 16,095 rdf-triples in the ## methodology knowledge graph embeddings to leverage kges, we train the embeddings on around 4.2 million entities and 661 relations represented in the entire dbpedia repository, where each subject s, or object o, of an rdf-triple can be associated as a point in a continuous vector space. the relation or the predicate p between the subject and the object entities can be modelled as displacement vector (s + @xmath0. in this work, we use fast-text @xcite , which is based on a bagof-words representation and considers the subject s and the object o entities along with the predicate p as a unique discrete token. thus, fasttext models co-occurrences of these entities and relations and learns a word representation using a hierarchical softmax. this allows us to create semanticallyenriched kges which are 500-dimensional vectors that are used to initialise the encoder and decoder embedding layers in our end-to-end system. as a comparison to the usage of kges, we engage 300-dimensional glove embeddings as pre-trained textual embeddings. for each model, we compare the results with and without a delexicalisation step. delexicalisation before training a model, we perform a delexicalisation step where we modify the rdf-triple in the input similar to @xcite . the subject of the rdf-triple is replaced by the dbpedia category and the object is replaced by the predicate using a pre-defined delexicalisation dictionary 2 . we also split the predicate in each triple on the camelcase using a regular expression. we perform this delexicalisation step on the reference text too based on the corresponding entities in the input, as shown in ## experimental setup we follow the webnlg baseline system @xcite as one of the baseline architectures for our experiments, which is a vanilla sequenceto-sequence lstm model with attention where the rdf triples in the input are linearised as a sequence and the output text is tokenised before training. we use another baseline based on the transformer architecture similar to the end-to-end architecture setup by @xcite . the models are trained using the opennmt library @xcite . we use the default parameters for the baseline model: two hidden layers with 500 lstm units per hidden layer and word embeddings of 500 dimensions. dropout is applied with value 0.3 and the lstm models are trained with stochastic gradient descent, starting with a learning rate of 1.0 and learning rate decay enabled. for the transformer model, the encoder-decoder setup contains 6 layers with 512 hidden units. the word embeddings are 512-dimensional and the feed-forward sublayers are 2048-dimensional. each multi-head attention sublayer consists of 8 attention heads. dropout is applied with value 0.1 and the model 2 https://gitlab.com/webnlg/webnlg-baseline/ is trained using adam optimizer @xcite . the overlap between the model vocabulary and glove embeddings is 44.36% while for kges it is 45.66%. for the delexicalised corpus, the overlap increases to 46.70% for glove and 47.40% for kges respectively. since the size of the vocabulary of the training corpus is limited to a few thousand words (around 5,000 unique words), we enable the options for dynamic dictionary and shared vocabulary to allow the model to share tokens on between the source and the target side. all models are trained for 100,000 steps and for evaluation on the test set we take the checkpoint with the best bleu score on the validation set. ## results and discussion in this section, we report the results of our experiments in terms of two commonly used evaluation metrics, bleu @xcite and me-teor @xcite . we also report scores in terms of precision and recall of the input entities covered in the output generations. for statistical significance, we use multeval @xcite to perform bootstrap resampling and report the results on three test sets consisting of instances from seen, unseen and all categories in for the test set consisting of seen categories, using kges shows consistent improvement over the baseline models for both lstm and transformer architectures. this improvement is observed in both cases whether the delexicalisation step is performed or not. for the models trained on the delexicalised corpus, glove embeddings appear to produce better results than kges. this is due to the fact that the kges are not trained on delexicalised entities. nonetheless, we observe delexicalisation to be useful in generating text for unseen categories where almost none of the entities and properties are present in the training set. our results are consistent with @xcite , who have also shown that end-to-end neural approaches perform well on seen categories but struggle to generalise on the unseen ones. similar to @xcite , we propose entity-based evaluation metrics to measure the coverage of subject and object entities in the output generation. for each subject and object entity in the triples, we count unique entities extracted from the output text, and calculate precision and recall scores. a drawback of this approach is that it does not penalise multiple repetitions of the same entity in the output. however, it is useful as a measure of how well the entities are represented in the output. our results suggest both kges and glove embeddings yield an improvement in precision and recall, especially in the seen categories. delexicalisation also leads to an overall improvement in the scores, however, neither kges nor glove embeddings appear to do well on unseen categories for the delexicalised corpus in terms of these metrics. usually, data-to-text generation tasks involve only non-linguistic entities in the input, which makes it difficult to generate fluent and coherent text in the output without explicitly defining the rules for a mapping between the input entities and output text. however, in the webnlg corpus, the predicate inside each triple can be considered as a linguistic entity around which the output sentence is structured. ## conclusion and future work in this work we showed that using kges to initialise the encoder and decoder embeddings in neural data-to-text generation consistently achieves better performance by up to 2 or 3 bleu points over the baseline models for the seen categories on the webnlg corpus. for future work, we plan to leverage kges with delexicalised dbpedia entries, specifically to target the unseen data.