A Sentiment-Controllable Topic-to-Essay Generator with Topic Knowledge Graph (2020)

Abstract: Generating a vivid, novel, and diverse essay with only several given topic words is a promising task of natural language generation. Previous work in this task exists two challenging problems: neglect of sentiment beneath the text and insufficient utilization of topic-related knowledge. Therefore, we propose a novel Sentiment Controllable topic-to- essay generator with a Topic Knowledge Graph enhanced decoder, named SCTKG, which is based on the conditional variational auto-encoder (CVAE) framework. We firstly inject the sentiment information into the generator for controlling sentiment for each sentence, which leads to various generated essays. Then we design a Topic Knowledge Graph enhanced decoder. Unlike existing models that use knowledge entities separately, our model treats knowledge graph as a whole and encodes more structured, connected semantic information in the graph to generate a more relevant essay. Experimental results show that our SCTKG can generate sentiment controllable essays and outperform the state-of-the-art approach in terms of topic relevance, fluency, and diversity on both automatic and human evaluation.

## introduction topic-to-essay generation (teg) task aims at generating human-like paragraph-level texts with only several given topics. it has plenty of practical applications, e.g., automatic advertisement generation, intelligent education, or assisting in keyword-based news writing @xcite . because of its great potential in practical use and scientific research, teg has attracted a lot of interest. @xcite . however, in teg, two problems are left to be solved: the neglect of sentiment beneath the text and the insufficient utilization of topic-related knowledge. love is a kind of emotion. love experience is important to every one.... a well-performed essay generator should be able to generate multiple vivid and diverse essays when given the topic words. however, previous work tends to generate dull and generic texts. one of the reason is that they neglect the sentiment factor of the text. by modeling and controlling the sentiment of generated sentences, we can generate much more diverse and fascinating essays. as shown in figure 1 , given the topic words "love", "experience" and "emotion", the "without sentiment" model generates monotonous article. in contrast, the sentiment-attach model generates positive statements such as "fall in love with my boyfriend" when given the "positive" label, and generates negative phrases such as "addicted to smoking", "broke up" when given the "negative" label. in addition, sentiment control is especially essential in topicto-essay generation task, which aims to generate multiple sentences. as the number of sentences increases, the search space for generation model is exponentially enlarged by controlling the sentiment polarity for each of the sentence. therefore, the ability to control sentiment is essential to improve discourse-level diversity for the teg task. as for the other problem, imagine that when we human beings are asked to write articles with some topics, we heavily rely on our commonsense knowledge related to the topics. therefore, the proper usage of knowledge plays a vital role in the topic-to-essay generation. previous state-of-the-art method @xcite extracts topic-related concepts from a commonsense knowledge base to enrich the input information. however, they ignore the graph structure of the knowledge base, which merely refer to the concepts in the knowledge graph and fail to consider their correlation. this limitation leads to concepts being isolated from each other. for instance, given two knowledge triples (law, antonym, disorder) and (law, part of, theory), about the topic word law, @xcite simply uses the neighboring concepts disorder and theory as a supplement to the input information. however, their method fails to learn that disorder has opposite meaning with law while theory is a hypernym to law, which can be learned from their edges (correlations) in the knowledge graph. intuitively, lacking the correlation information between concepts in the knowledge graph hinders a model from generating appropriate and informative essays. to address these issues, we propose a novel sentiment-controllable topic-to-essay generator with a topic knowledge graph enhanced decoder, named sctkg, which is based on the conditional variational auto-encoder (cvae) framework. to control the sentiment of the text, we inject the sentiment information in the encoder and decoder of our model to control the sentiment from both sentence level and word level. the sentiment labels are provided by a sentiment classifier during training. to fully utilize the knowledge, the model retrieves a topic knowledge graph from a largescale commonsense knowledge base conceptnet @xcite . different from @xcite , we preserve the graph structure of the knowledge base and propose a novel topic graph attention (tga) mechanism. tga attentively reads the knowledge graphs and makes the full use of the structured, connected semantic information from the graphs for a better generation. in the meantime, to make the generated essays more closely surround the semantics of all input topics, we adopt adversarial training based on a multi-label discriminator. the discriminator provides the reward to the generator based on the coverage of the output on the given topics. our contributions can be summarized as follow: ## task formulation traditional teg task takes as input a topic sequence @xmath0, • • • , x m ) with m words, and aims to generate an essay with m sentences (l 1 , • • • , l m ) corresponding with topic sequence x. in this paper, we provide a sentiment sequence @xmath1, • • • , s m ), each of which corresponds to a target sentence in essay. each sentiment can be positive, negative, or neutral. essays are generated in a sentence-by-sentence manner. the first sentence l 1 is generated only conditioned on the topic sequence x, then the model takes all the previous generated sentences as well as the topic sequence to generate the next sentence until the entire essay is completed. in this paper, we denote the previous sentences l 1:i-1 as context. ## model description in this section, we describe an overview of our proposed model. our sctkg generator based on a cvae architecture consists of an encoder and a topic knowledge graph enhanced decoder. the encoder encodes topic sequence, sentiment, and context and regards them as conditional variables c. then a latent variable z is computed from c through a recognition network (during training) or prior network (during inference). the decoder attaches with a topic knowledge graph and sentiment label to generate the texts. at each decoding step, the tga is used to enrich input topic information through effectively utilizing the topic knowledge graph. we adopt a two-stage training approach: (1) train the sctkg generator with the conventional cvae loss; (2) after the first step is done, we introduce a topic label discriminator to evaluate the performance of sctkg generator. we adopt adversarial training to alternately train the generator and the discriminator to further enhance the performance of the sctkg generator. ## results and analysis in this section, we introduce our experimental results and analysis from two part: the "text quality" and "sentiment control". then we show case study of our model. ## conclusions in this paper, we make a further step in a challenging topic-to-essay generation task by proposing a novel sentiment-controllable topic-to-essay generator with a topic knowledge graph enhanced decoder, named sctkg. to get better representation from external knowledge, we present tga, a novel topic knowledge graph representation mechanism. experiments show that our model can not only generate sentiment-controllable essays but also outperform competitive baselines in text quality.