Generating Diverse Descriptions from Semantic Graphs (2021)

Abstract: Text generation from semantic graphs is traditionally performed with deterministic methods, which generate a unique description given an input graph. However, the generation problem admits a range of acceptable textual outputs, exhibiting lexical, syntactic and semantic variation. To address this disconnect, we present two main contributions. First, we propose a stochastic graph-to-text model, incorporating a latent variable in an encoder-decoder model, and its use in an ensemble. Second, to assess the diversity of the generated sentences, we propose a new automatic evaluation metric which jointly evaluates output diversity and quality in a multi-reference setting. We evaluate the models on WebNLG datasets in English and Russian, and show an ensemble of stochastic models produces diverse sets of generated sentences while, retaining similar quality to state-of-the-art models.

## introduction semantic graphs are an integral part of knowledge bases that integrate and store information in a structured and machine-accessible way @xcite . they are usually limited to specific domains, describing concepts, entities and their relationships in the real world. generating descriptions from semantic graphs is an important application of natural language generation (nlg) and can be framed in a graph-to-text transduction approach. in recent years, approaches to graph-to-text generation can be broadly categorised into two groups. the first uses a sequence-to-sequence model @xcite @xcite : the key step in this approach is to linearise the input graph to a sequence. sequence-to-sequence models have been proved to be effective for tasks like question answering @xcite ), text summarisation @xcite , and constituency parsing @xcite . however, when dealing with graph inputs, this method does not take full advantage of the graph structure. another approach is to handle the graph directly, using a graph-to-sequence model @xcite @xcite . this approach has been recently widely adopted as it shows better performance for generating text from graphs @xcite . the models used in previous work are deterministic: given the same input graph, they will always generate the same text (assuming a deterministic decoding algorithm is used). however, it is widely known that many graphs admit multiple valid descriptions. this is evidenced by the presence of multiple references in datasets such as webnlg @xcite and it is a common phenomenon in other generation tasks such as machine translation and image captioning. in this work, we propose to use models that generate sets of descriptions instead of a single one. in particular, we develop stochastic models with latent variables that capture diversity aspects of semantic graph descriptions, such as lexical and syntactic variability. we also propose a novel evaluation methodology that combines quality and diversity into a single score, in order to address caveats of previously proposed diversity metrics. our findings show that stochastic models perform favourably when generating sets of descriptions, without sacrificing the quality of state-of-the-art architectures. marcheggiani and perez-beltrachini (2018) proposed an encoder based on graph convolutional networks @xcite , which directly exploit the input structure. similar to @xcite , gcn layers can be stacked, resulting in representations that take into account non-adjacent, longdistance neighbours. @xcite used gated graph neural networks @xcite by extending networks on graph architectures with gating mechanisms, similar to gated recurrent units @xcite . @xcite proposed graph transformer encoder by extending transformers @xcite to graph-structured inputs, based on the graph attention network @xcite architecture. this graph encoder generates node embeddings by attending over its neighbours through a self-attention strategy. ribeiro et al. (2020) propose new models to encode an input graph with both global and local node contexts. to combine these two node representations together, they make a comparison between a cascaded architecture and a parallel architecture. latent variable models within neural networks, a standard approach for generative models with latent variables is the variational autoencoder (vae) @xcite . the generative process is represented as: p θ (x, z) = p θ (x | z)p θ (z), where p θ (z) is the prior from which the latent variable is drawn, p θ (x | z) is the likelihood of data point x conditioned on the latent variable z, typically calculated using a deep non-linear neural network, and θ denotes the model parameters. @xcite proposed a pioneering variational autoencoder for text generation to explicitly learn the global features using a continuous latent variable. they adapt the vae to text data using an lstm @xcite for both the encoder and the decoder, using a gaussian prior to build a sequence autoencoder. this architecture can be extended to conditional tasks (when there is an input guiding the generation). @xcite proposed an end-to-end variational model for neural machine translation (nmt), using a continuous latent variable to capture the semantics in source sentences and guide the translation process. @xcite proposed a more expressive word-level machine translation model incorporating a chain of latent variables, modelling lexical and syntactic variation in parallel corpora. ## stochastic graph-to-sequence model in this section we introduce the proposed approach to generate diverse descriptions from semantic graphs. we start from the state-of-the-art model of @xcite , which is a deterministic graph-to-sequence architecture. then we incorporate a latent variable and a variational training procedure to this model, in order to turn the model stochastic. this latent variable aims at capturing linguistic variations in the descriptions and is responsible for increasing the diversity at generation time. the architecture is shown in figure 1 . ## joint evaluation of diversity and quality addressing diversity in language generation is a recent topic that attracted attention in particular in image captioning. this led to the development of metrics that aim at measuring the diversity of a set of sentences, such as self-bleu @xcite . however, these metrics are based only on the generated output space, ignoring the references in the gold standard. this lead to spurious measurements, such as unconditional language models having excellent performance according to these metrics, even though they have no practical use as they ignore the input. to address these caveats, we propose a new evaluation procedure that assesses diversity and quality jointly. our key insight (and assumption) is based on using the reference set as a gold standard for both aspects. given a graph, the set of references acts as the "end goal", containing high-quality descriptions with sufficient levels of diversity. 1 we call this procedure multi-score (ms). the idea behind multi-score is shown pictorially in figure 2 . in this example, we have a single instance with three references and three predicted descriptions generated by a model. given a sentencelevel quality metric we can calculate it among all possible pairs between each prediction and reference, obtaining a weighted bipartite graph. we then solve the respective maximum matching problem for this bipartite graph and take the average weight of the edges corresponding to the optimal matching. we show the full procedure to calculate multi-score in algorithm 1. as a metric, multi-score has a number of desirable properties: ## results we assess each model on the test set of english and russian datasets respectively and report the quality and diversity results. the quality evaluation scores (bleu: @xcite , chrf++: popovic (2017)) are calculated based on the average score of the three outputs. we report the original bleu and chrf++ score to show the quality of the generated sentences from each model. the diversity evaluation scores (self-bleu, multi-score) are computed using the three outputs. as we describe in section 4, our proposed diversity evaluation metrics require a sentence-level quality evaluation metric to compute the score of two sentences. we adopt sentence-level bleu and chrf++ and refer to their corresponding multi-score versions as ms-bleu and ms-chrf. the diversity scores of these two datasets are shown in ## conclusion and future work in this work, we first propose stochastic graphto-text models to generate diverse sentences from semantic graphs. this was implemented through latent variable models that aim to capture linguistic variation and ensembling techniques. furthermore, to solve the limitation of the existing diversity eval-uation metrics, we also propose multi-score, a new automatic evaluation metric assessing diversity and quality jointly. it provides a general and effective way to assess the diversity of generated sentences for any text generation task. we perform experiments on english and russian datasets and results demonstrate the generated sentences from the stochastic ensemble have both high diversity and high quality. since multi-score is based on using the reference set as the gold standard, it has a limitation that the variety of the reference sentences can largely influence the metric. datasets containing reference sentences with higher quality and diversity will likely generate a more accurate multi-score for the predicted sentences. in other words, multi-score evaluates diversity implicitly through the references, as opposed to explicit judgements of diversity. however, explicit human evaluation requires a formal definition of diversity which is difficult to establish (as compared to quality judgements, for instance). nevertheless, addressing this challenge could provide a pathway to reduce the need for multiple references in evaluating diversity. to the best of our knowledge this is the first work that incorporates a latent variable within a graphto-sequence model. this in turn leads to many promising research avenues to explore in future work. our analysis showed that the latent variable mostly helps in syntactic variation but less in other aspects such as semantics. analysing the behaviour of the latent variable when modelling linguistic information is an important avenue that will enhance the understanding of stochastic models.