Enhancing Sequence-to-Sequence Modelling for RDF triples to Natural Text (2020)

Abstract: establishes key guidelines on how, which and when Machine Translation (MT) techniques are worth applying to RDF-to-Text task. Not only do we apply and compare the most prominent MT architecture, the Transformer, but we also analyze state-of-the-art techniques such as Byte Pair Encoding or Back Translation to demonstrate an improvement in generalization. In addition, we empirically show how to tailor these techniques to enhance models relying on learned embeddings rather than using pretrained ones. Automatic metrics suggest that Back Translation can significantly improve model performance up to 7 BLEU points, hence, opening a window for surpassing state-of-the-art results with appropriate architectures.

## introduction a knowledge base (kb) is a large source of information represented in a structured way. the information structure is based on resource description framework (rdf), which consist of three elements: subject, predicate, object . thus, it establishes relations (predicate) between entities @xcite . it is known that kbs are being widely considered in the industry for applications such as question answering (q&a) systems @xcite , search engines @xcite , recommender systems @xcite , etc. however, this data representation is not human-friendly, i.e. is not in a language form, hence, it is hard for human to comprehend the information embedded in these triples. the team address this problem under the context of natural language generation (nlg). this task can be divided into: text-to-text generation or datato-text generation, according to @xcite . the latter is the most common approach taken to solve the rdf-to-text task, since it is based on a mapping from structured data to text. however, we focus on applying mt architectures and techniques, which are considered for text-totext generation. to further improve the quality of the text generated by our models, we assess the advantages of back-translation (bt) @xcite for this task, as well as, compare the benefits of learning the embeddings from scratch to the alternative of providing pretrained embeddings. the specific contributions of our work to the field are: the rdf-to-text task aims to generate natural text from a set of rdf, which are entities in the form of words establishing relations between them. the input to our system is a kb that can be denoted as a set of rdf, i.e. k := {r 1 , ..., r n }. each element in rdf r i can be defined as s i , p i , o i , these elements stand for subject, predicate and object, respectively. type of text lexicalise delexicalise rdf rome, capital of, italy city, capitalof, country target sentence rome is the capital of italy. city is the capital of country. finally, we aim to generate a sequence of sentences s, which consists of a sequence of words [w 1 , ..., w m ]. the resulting sentences in s should be grammatically correct and should also contain all the information present in the kb k. in order to assess the correctness of s, the following metrics are studied: bleu @xcite , meteor @xcite , chrf++ @xcite ; for which greater values mean better performance, and ter @xcite ; for which lower values means better performance. ## related work the present task was also proposed in the webnlg challenge 2017 foot_1 . submissions to this challenge included different approaches, such as: neural machine translation (nmt), statistical machine translation (smt) and pipeline systems @xcite . the best model regarding automatic metrics was submitted by the university of melbourne. such model consisted of an encoderdecoder bidirectional long short-term memory architecture with attention foot_2 . although the model falls in nmt field, it is not an strict end-to-end approach as it matches input and output with delexicalised templates. recently, most of the taken approaches to solve this nlg task are based on encoder-decoder architectures as well as graph neural networks (gnn). the main idea behind these methods is to exploit the input structure, which can be seen as a graph. it can be empirically shown how encoderdecoder gnn architectures can achieve similar results to the best submission in webnlg challenge 2017 @xcite or even improve these benchmarks @xcite . ## system architecture this section describes the pipeline architecture from preprocessing k to postprocessing the out-put of intermediate models. however, the system will be adapted to allow different experiments. ## ablation experiment and results the first experiment analyses the influence of the number of bpe subwords. the second experiment study the improvements of pretrained embeddings against learned embeddings. last experiment is designed to enlarge training data by means of different bt approaches. ## models summary in the following, the best models obtained in each of the considered experiments as well as their performance are presented. moreover, we also provide a comparison between our models and some of the most relevant models in the rdf-to-text domain. webnlg alfred moore scales, successor, daniel gould fowle , daniel gould fowle, alma mater, princeton university