Semantic Triples Verbalization with Generative Pre-Training Model (2020)

Abstract: The paper devoted to the problem of automatic text generation from RDF triples. This problem was formalized and proposed as a part of the 2020 WebNLG challenge. We describe our approach to the RDF-to-text generation task based on a neural network model with the Generative Pre-Training (GPT-2) architecture. In particular, we outline a way of base GPT-2 model conversion to a model with language and classification heads and discuss the text generation methods. To research the parameters’ influence on the end-task performance a series of experiments was carried out. We report the result metrics and conclude with possible improvement directions.

## introduction the idea of semantic web has a long history of research and development. the resource description framework (rdf) is the most common and known standard for semantic data interchange developed by the world wide web consortium @xcite . the rdf data model allows to encode knowledge in a form of (subject, predicate, object) statements known as triples. thus providing a way of creating common knowledge databases understandable for machine and human. this idea together with recent development of transformer language models unlock a set of promising research directions. two of them were highlighted at this year's workshop on natural language generation from the semantic web (webnlg) @xcite . the webnlg organizers proposed: 1) rdf-to-text generation task and 2) text-to-rdf semantic parsing task (reverse of the first one). each task was suggested for english and russian language. basically the first task is the following: given an input rdf triples set, build a system to yield its verbal logical equivalent in natural language. for example, given the rdf set of three triples: adare manor|completiondate|1862 adare manor|architect|augustus pugin adare manor|buildingstartdate|1700 a good answer would be "the construction of adare manor began in 1700 and was completed in 1862. the manor was designed by augustus pugin." such a system, if proven to show human comparable performance, potentially can have several practical applications. based on the system a tool for specific domain rdf databases verbalization can be created. generally making interaction with knowledge graphs and databases more natural for lay users. another use case includes an application in the dialog systems development process. often conversational agents use retrieval-based @xcite or hybrid @xcite approaches for best utterance selection. those approaches require a predefined candidate response set, which can be automatically verbalized from an appropriate rdf database. in this work, we restrict ourselves only to the rdf-to-text generation task for the russian language. our ultimate goal was to benchmark performance of a neural network language model with gpt-2 architecture @xcite together with some decoding methods. ## data the workshop organizers provided the dataset (in the xml file format) for each language along with the split: train, dev and test. the russian data was collected from 9 dbpedia @xcite categories. absolute numbers for each category and split part can be found in basically each data entry may contain up to 7 rdf triplets and a few variants of this triplet set verbalization. based on the train data, the median value of an entry triplets is 3. except for a few outliers the number of verbalisations for an entry also no more than 3. each entry also includes some additional parameters, for example specs on triplet positional arrangement, but we didn't use it except for the translation links. to stay in the single language space we also had to explicitly translate predicate statements (227 unique elements) thus finalizing triplets conversion to the russian language. furthermore, as there were no restrictions on external data use, we tried to experiment with additional data (see section 3.2). for this purpose baidu ske dataset foot_0 was automatically translated from chinese to russian foot_1 and used as an augmentation data. the dataset (much larger is size) has a similar structure, each entry is a set of associated tuples @xcite and a text description. the number of unique predicate statements is 50 and there are 194,747 entries in total. ## proposed approach since their introduction the neural network models with transformer-based architecture @xcite has become the default choice when approaching a natural language processing problem. this success primarily is the performance boost and usability across a wide range of tasks. the same base model can be finetuned for a specific task (in an end-to-end fashion) with a low cost effort and relatively small amount of data. in this work we exploit the same strategy by taking russian gpt-2 model foot_2 (24 transformer block layers, 1024-dimensional word embeddings and 16 self-attention heads, with about 350m parameters in total) as base and finetune it for the rdf-to-text generation task. the gpt-2 essentially is a language model that is trained to predict the next token (a sub-word) given all previous tokens of a sequence. technically the cross-entropy loss function over n vocabulary tokens {t 1 , ..., t n } used to estimating the conditional probability distribution p(t i |t 1 , ..., t i-1 ). but we cannot directly apply such a model to our task as the genuine gpt-2 model has a lack of control over the generation process and succeeds only in spawning of text endings given a beginning. our goal is to force the model to wrap (join and rephrase) the given rdf set triplets in the form of coherent, syntactically and semantically correct natural language phrases. for this purpose we defined the model with double heads. first one is the same (as in gpt-2) language head and the second one is the classification head. the aim for the second head is exactly to distinguish between correct or wrong (a sample was randomly selected to represent a negative example) verbalisation of an input rdf set. thus our final loss function is the sum of two cross-entropy components. by minimizing such loss our model learns to automatically translate an rdf set to an adequate natural language phrase. the sample input data for the model is a pool of indices lists and masks. the main part of this pool is the list of token indices of an input sequence, which could be broken down into two parts: rdfpart and phrase-part. these parts are separated from each other by the special mediator token (by start/end tokens analogy). in the inference mode there is only an rdf-part and the mediator token is the trigger to start a phrase generation process (section 3.1). we trained the unified model for all categories and triplet cardinalities. ## conclusions in this paper we presented our rdf triples verbalization system based on the neural network model with gpt-2 architecture. specifically the model with language and classification heads were used to generate appropriate text given a set of russian triples. we described the model implementation details and briefly discussed common decoding methods and its recent sampling variants. the data provided in webnlg challenge allowed us to train the model and experiment with the generation pipeline. from this we can conclude that the verbalisation task from rdf data is more restricted (compared to open-ended language generation) and sustainable results can be obtained from a wide range of generation parameters. at the same time metric comparison with the other participant's systems and synthetic benchmark revealed the limitation of such approach. while this model can be used for verbalisation candidate generation the procedure of best candidate selection obviously should be further improved. a recap of processed test entries showed that sometimes not all parts of triples are mentioned in text or repetitions are encountered, while the candidate pool contained more appropriate variants. the mediocre results of human evaluation on the data coverage criteria are another confirmation of such system inefficiency. we'll consider to build a separate ranking model which hopefully would be able to better distinguish such cases.