Building the Directed Semantic Graph for Coherent Long Text Generation (2021)

Abstract: Generating long text conditionally depending on the short input text has recently attracted more and more research efforts. Most existing approaches focus more on introducing extra knowledge to supplement the short input text, but ignore the coherence issue of the generated texts. To address aforementioned research issue, this paper proposes a novel two-stage approach to generate coherent long text. Particularly, we first build a document-level path for each output text with each sentence embedding as its node, and a revised self-organising map (SOM) is proposed to cluster similar nodes of a family of document-level paths to construct the directed semantic graph. Then, three subgraph alignment methods are proposed to extract the maximum matching paths or subgraphs. These directed subgraphs are considered to well preserve extra but relevant content to the short input text, and then they are decoded by the employed pre-trained model to generate coherent long text. Extensive experiments have been performed on three real-world datasets, and the promising results demonstrate that the proposed approach is superior to the state-of-the-art approaches w.r.t. a number of evaluation criteria.

## introduction recently, in the domain of natural language generation (nlg), the conditional long text generation @xcite has been investigated with flourishing results @xcite @xcite . essentially, this task aims to generate text in line with the main semantic meanings of the input text; and meanwhile, it requires that the length of the generated text is significantly longer than that of the short input text. the key to this challenging task is how to generate extra but relevant content that is well positioned to express the semantic meanings of the * corresponding author. although the aforementioned issues were partially alleviated by existing approaches, they obviously ignored the logic coherency issue between the generated texts. to simplify the problem, hereinafter, the logic coherency refers to the generated paragraphs, with each generated to represent the same semantic meanings, and to have the correct sequence order. apparently, it is a non-trivial task to generate extra content having appropriate sequence order. first, it still remains an open challenge in the literature to generate extra content given the short input text, especially for some domain-specific applications, e.g., financial report generation @xcite , where there usually exist multiple underlying semantic meanings for each piece of short news. second, even if the underlying semantic content could be well generated, how to properly determine the logical order of the generated texts is another challenging issue. this paper is thus motivated to address aforementioned two research issues, i.e., logic coherency and extra content generation. to achieve this goal, we propose this two-stage approach which first constructs a directed semantic graph, and then generates coherent long text via a pre-trained model with the maximum matching subgraphs, extracted from the constructed semantic graph, as its input. particularly, we first build a document-level path for each long text with each sentence embedding as its node and the sentence order as its directed edge. a revised self-organising map (som) is proposed to merge similar nodes belonging to different document-level paths into one to represent a coarse level semantic meaning. the intuitive design behind this is that human writers may have their own writing styles, their sentences might be slightly different, and thus, by merging similar nodes the underlying semantic meanings could be acquired. after som merging step, the document level graphs are interwove with each other and thus a corpuslevel graph is acquired which is called the directed semantic graph. for the text generation stage, each input text is matched with the top-k similar input texts (training data) and their output texts are then constructed into paths or subgraphs. these paths or subgraphs are aligned with the directed semantic graph, and the embeddings of the maximum matching paths or subgraphs are fed into the employed pre-trained model for text generation. the major contributions of this paper are summarized as follows. ## related work recently, long text generation @xcite has received significant research attention which focuses more on generating previously unseen but relevant content @xcite . the existing approaches could be roughly categorized into three groups, e.g., sequence-to-sequence based approaches @xcite , generative model based approaches @xcite and gan based approaches (kusner and hernández-lobato, 2016; @xcite . recently, various pre-trained models @xcite @xcite have been proposed. for instance, @xcite employs a pre-trained module as the decoder to summarize text; while the gpt-based approaches @xcite further improve the readability of the generated text. to generate relevant content, @xcite proposed a vari-ational autoencoder based approach which generates sentences sampled from the underlying topics learnt from the input data. similarly, @xcite proposed a multi-level variational autoencoder based approach where both high-level document features and low-level word features are respectively extracted and aggregated for the text generation. to learn external domain-specific knowledge, the cvae-kd @xcite proposed a knowledge distillation method that employs the teacher-student architecture. the teacher component is to learn external knowledge from the extracted sub set of financial reports and then guides the student component to generate relevant text w.r.t. the input news. notably, most existing approaches ignore the logic coherency issue during the process of the text generation. alternative to the writing logic, @xcite utilized a storyline to generate stories given a title. similarly, @xcite proposed the news-to-report generation task where the authors first generate the outline for the input news and then generate the financial reports guided by the outline. however, both the storyline and outline are essentially defined as multiple semantic topics for the text generator to generate more elaborating content, whereas the writing logic issue, raised in this paper, constrains the relationship among the generated texts. similar to storyline, there also exist a number of document planningbased @xcite @xcite ) text generation approaches. these approaches either pre-define a document plan for well-structured text @xcite or dynamically learn such plan by extracting a set of keywords @xcite from the training data. although the extracted concepts or keywords @xcite could well preserve word level contextual information, but apparently they ignore the underlying semantic coherence between the generated sentences or paragraphs whereas the proposed approach aims to capture such semantic coherence. ## preliminaries the corpus set consists of (x, y ) pairs where @xmath0, x 2 , ..., x n } denotes the input text set, @xmath1, y 2 , ..., y n } denotes the corresponding output long text set, and y j is used to construct the directed semantic graph. let y @xmath2, s 2 k , ...s p k } with e(s p k ) = (a 1 , a 2 , ..., a n ) denoting the embed-ding of a sentence s p k , where s p k denotes the p -th sentence of the k -th document. a document-level path is to be constructed for each input or output text with a sentence s p k as its node, and (s i k , s j k ) is the directed edge between two nodes. thus, a document-level path is defined as p @xmath3, s m k ) p which starts from vertex s n k and ends at vertex s m k allowing repeated vertices. the directed semantic graph @xmath4, e) is to be constructed with the centroid of a group of similar path nodes {s j i } as its graph node, and the majority direction of the contained edges as its directed edge between two graph nodes. more details will be presented in section 4.1. ## the proposed approach the proposed approach consists of two stages: (1) the directed semantic graph construction; (2) maximum subgraph matching for text generation. details of each stage are illustrated in the following subsections. ## experiments in this section, we first briefly introduce the experimental datasets, evaluation criteria and baseline models. then, we present how the experiments are prepared as well as the parameter settings. at last, extensive experiments were performed on three real-world datasets to answer the following research questions. ## conclusion in this paper, we proposed a two-stage approach to generate coherent long text given the short input text. particularly, we first build a document-level path and then construct the directed semantic graph. three kinds of matching methods are proposed to extract the best matched paths or subgraphs for the later text generation. extensive experiments are evaluated on three real-world datasets and the promising experimental results demonstrated that the proposed approach outperforms both baseline and the sota approaches.