Dynamic Topic Tracker for KB -to-Text Generation (2020)

Abstract: Recently, many KB-to-text generation tasks have been proposed to bridge the gap between knowledge bases and natural language by directly converting a group of knowledge base triples into human-readable sentences. However, most of the existing models suffer from the off-topic problem, namely, the models are prone to generate some unrelated clauses that are somehow involved with certain input terms regardless of the given input data. This problem seriously degrades the quality of the generation results. In this paper, we propose a novel dynamic topic tracker for solving this problem. Different from existing models, our proposed model learns a global hidden representation for topics and recognizes the corresponding topic during each generation step. The recognized topic is used as additional information to guide the generation process and thus alleviates the off-topic problem. The experimental results show that our proposed model can enhance the performance of sentence generation and the off-topic problem is significantly mitigated.

## introduction in recent years, many knowledge bases (kbs) have been built to incorporate different kinds of human knowledge into a structured triple representation such as freebase @xcite , dbpedia @xcite , yago @xcite and wikidata @xcite . many tasks including question answering and recommendation systems have benefited from kbs @xcite as external knowledge sources to improve the results. though kbs have achieved great success in supporting and improving various text mining tasks, they are still incomprehensible to humans due to the over-rigid structured format. reading a bunch of triples always annoys people since the form is not easily understandable especially to people who have never heard about kbs. in order to address this problem, recently some researchers have proposed the kb-to-text generation task @xcite @xcite to bridge the gap between kbs and natural language. this kb-to-text generation problem aims at directly converting a group of kbs triples into human-readable sentences. for example, given a triple group ( bill gates, birthplace, seattle , bill gates, founderof, microsoft bill gates, was born in seattle, 1955. he founded microsoft in 1975 <bill gates, birthplace, seattle>, <bill gates, birthyear, 1955> <bill gates, founderof, microsoft>, <microsoft, startyear, 1975> person topic off-topic while bill is a commonly used name in the usa. company topic figure 1 : the off-topic problem. during the generation process, the given data ranges from the topic of person topic to the topic of company. however, the models are prone to generate off-topic sentences just because the models associate this kind of information with the word "bill". to solve the off-topic problem, we propose to utilize the topic information as a piece of clue in the sentence generation process. unfortunately, the corresponding topic information is not available and there is no existing dataset containing the topic annotations. therefore, it is difficult to adopt supervised learning approaches for detecting topics. moreover, it is even more expensive to annotate the dynamic change of topic information in one sentence, as exemplified by the topic changes from "person" to "company" in figure 1 . therefore, we investigate the task of automatically detecting the hidden topic information and incorporating such information for the generation of sentences. many works have been proposed to utilize the static topic information to improve the generation performance. @xcite and @xcite propose to represent the topic for each sentence as a learnable vector. the topic is predicted by the input sentence and is used to enhance the generating phase. @xcite and @xcite detect the topic representation by applying a pre-trained lda model on the input sequence. moreover, @xcite and @xcite predict the topic representation directly from the input sequence using recurrent neural networks (rnn). all the above methods make an assumption that during generation the topic does not change so as to make the problem tractable, which scarifies the advantage of modeling the dynamic nature of topic information. we propose a novel dynamic topic tracker (dtt) neural model to tackle the problem. different from existing models, our proposed dtt model learns how the target sentence topic dynamically evolves and how to use the topic information to guide the generation process simultaneously. specifically, our dtt model is a neural model composed of four parts, namely, the state tracker, the topic attention, the global topic bank, and the topic memory. the state tracker captures the decoder state for each generation step. the topic attention uses the captured decoder state to focus on the input hidden representation to get a local topic state. the topic bank learns a global hidden topic representation and it calculates the most suitable local topic representation for each local topic state. the topic memory is used to memorize the previous local topic representation and computes the dynamic topic state for each generation step to guide the target sentence generation procedure. ## related work recently, various data-to-text tasks have been proposed handling different kinds of data. @xcite construct the webnlg dataset which aims at generating text descriptions based on dbpedia @xcite triples. @xcite and @xcite propose to generate a person's biography based on wikipedia's infobox. @xcite build the wikievent dataset aiming at generating text based on an event chain. @xcite generate restaurant reviews based on the information of restaurant attributes. @xcite generate basketball match descriptions based on the game records. moreover, @xcite propose to directly train the model on partially-aligned data called wita while @xcite propose to train a model based on purely unaligned data unsupervised with a dual learning framework. all of the above problems aim at converting some formatted data into natural language texts facilitating more understandability. some models have proposed to solve the kb-to-text problem by utilizing various information of the kbs. @xcite propose to directly rank the triples by relation frequency and flatten the triples to pure text. the flattened text is used as the input for a sequence-to-sequence model to generate the output text. @xcite propose to use a triple encoder to encode each triple into a hidden vector. the decoder input is constructed by simply concatenating all of the hidden vectors. @xcite propose a gtr-lstm model to encode not only the triple information, but also the structure information of the entity graph into hidden semantic space. @xcite exploit a mixed hierarchical attention based encoder-decoder model to leverage the structure and content information. @xcite propose to use delexicalization and copy mechanism to enhance the performance of the sequence-to-sequence framework. @xcite and @xcite propose to use template based methods to generate the text by using the extracted template information in the training set. @xcite propose to generate text description for entities by utilizing the knowledge distilled from the existing knowledge base. however, none of the above works consider the topic information in the kb-to-text generation process and thus not directly comparable to our work proposed in this paper. some works in text generation @xcite have been proposed to incorporate the topic information to help generate the text. these ideas can be adopted in kb-to-text generation. @xcite and @xcite add an extra topic tag into the source sentence for incorporating the topic information into the model. the whole model is built based on the sequence-to-sequence @xcite @xcite framework with standard attention @xcite . @xcite as well as @xcite propose to use the topic information as extra features to enhance the performance of the language model and word embedding. @xcite and @xcite use the same idea to utilize the topic feature to enhance the generation of the text. however, all these methods assume that the topic information is known in advance. some methods investigate the problem setting that the topic information is not given and needs to be detected. for example, the topic information can be detected from latent dirichlet allocation (lda). @xcite @xcite detect the topic distribution of words via topic model to enhance the translation procedure. @xcite propose a ta-seq2seq framework which uses the word topic information from lda to generate the responses in chatbot dialog systems. moreover, some researchers propose to directly detect the topic vector from the input sentences in the sequence-to-sequence framework. for example, @xcite propose to train a classifier to predict the topic of the source sentence and use it to help generate the dialog response. @xcite also propose to predict the topic vector directly from the input sequence. @xcite propose to use the topic information as a reward function. however, none of the existing works can capture the dynamic topic information suitable for the kb-to-text generation problem. ## conclusions and future work in this paper, we recognize the off-topic problem in the kb-to-text problem. we consider to utilize the dynamic topic information to alleviate this problem and improve the generation performance. to achieve this, we propose a dtt model which can learn the hidden representation of the topic information. during the sentence generating process, it can utilize the learned topic information in the sequence-to-sequence framework for enhancing the generation process. more importantly, the topic information is dynamic for each step in the generation process, and thus enables stronger capability than existing works. experimental results on a benchmark dataset show that our model can effectively capture the dynamic topic information at each step in the decoder. despite the promising result our model has achieved, there are some remaining challenges: (1) the model stacks too many lstm layers which leads the gradient hard to back-propagate if more layers are going to be added. some new technologies such as the transformer or gated cnn can be used to tackle this problem. (2) the topic representation only contains information from our training set. nevertheless, the novel architecture makes it possible to use any set of the topic vectors which can be pre-trained on a larger unannotated dataset.