Correcting Texts Generated by Transformers using Discourse Features and Web Mining (2021)

Abstract: Recent transformer-based approaches to NLG like GPT-2 can generate syntactically coherent original texts. However, these generated texts have serious flaws: global discourse incoherence and meaninglessness of sentences in terms of entity values. We address both of these flaws: they are independent but can be combined to generate original texts that will be both consistent and truthful. This paper presents an approach to estimate the quality of discourse structure. Empirical results confirm that the discourse structure of currently generated texts is inaccurate. We propose the research directions to correct it using discourse features during the fine-tuning procedure. The suggested approach is universal and can be applied to different languages. Apart from that, we suggest a method to correct wrong entity values based on Web Mining and text alignment.

## introduction natural language generation (nlg) task is one of the most challenging and important tasks in nlp. there are various types of nlg tasks: text summarization, machine translation, knowledge aggregation and multimedia information construction such as music generation. we consider tasks where the main goal is to construct a text that cannot be distinguished from a human-written text, by a human or a recognition system. the most successful and universal models for solving nlp tasks are models based on the idea of transformers. hence gpt @xcite and its larger modifications gpt-2 @xcite and @xcite successfully perform text generation tasks. however, they still have drawbacks. first of all, fragments in some generated texts do not cohere well with each other, despite the correct syntactic structure. ko and li 6 [who survives a major accident] 7 [and is saved by a state of the art experimental operation] 8 [that turns him into a robotic machine-like agent] 9 [who has tools and contraptions of all sorts] 10 [built into his body at his use] 11 [when he says] 12 ...". we conducted experiments for gpt-2 and distinguished two types of its mistakes. firstly, it does not generate well an overall discourse structure (rst is described in sect. 3). accordingly, in some cases, contradictions can be found in it. we fine-tuned gpt-2 on lower-cased movie reviews. here are examples of typical mistakes in the generated texts. let us consider the example demonstrated in figure 1 . the sentence has too many "elaboration" and "joint" rhetorical relations, which are default ones. moreover, thought structure is not reflected in this discourse tree as it looks like a chain. generally, genuine discourse trees are more balanced. another illustrative example of generated text: with enough fine performances from all of them , this is one of the best cult films ever made. . . . it does have some nice gore and some pretty well chosen actors but it is definitely not one of the best cult films of all time . in this case, the core text idea is contained in the last sentence. the first part is used to elaborate it. however, there is a contradiction, and a contrast relation does not contribute to coherence. apart from that, the "final" summary is in the middle of the text in some cases. it is not followed by the "end of sequence" token and continues by elaboration. as a result, the text is duplicated. in addition, for specific areas like medical texts, gpt-2 uses incorrect values of entities since it does not utilize any knowledge base. our example is from the 19th century literature domain. the seed to gpt content generation is in bold. pushkin wrote the originial chapters of yevgenii onegin, then alexander pushkin sent letters to nikolay gogol, mikhil lermontov, and others. while in saint petersburgh, pushkin was approached by rodion romanovich krafft. krafft wanted pushkin to be an english translator of a german edition of his poem here, 'then' part of the first sentence seems plausible. then the reader proceeds to the invented person krafft, and a random, implausible text starts. to continue impress the reader with a smooth flow of text, we need to find a real person like krafft related to pushkin via translation. the bottom sentences need to be replaced with following: in the 1960's, vladimir nabokov, the only writer to simultaneously hold the position as a giant of both russian and english literature, published an english translation of pushkin's masterpiece. hence most entities from gpt text generation need to be replaced. this paper presents empirical proof that gpt-2 generates wrong discourse structure in some cases and proposes ideas for the development of text generation in two directions: besides, we propose a method to estimate the overall quality of the discourse structure. these directions complement each other and can be used sequentially. so, firstly, we correct the discourse structure of the text using additional fine-tuning. then, we utilize external knowledge to correct the exact specific entity values and get meaningful text. our paper is organized as follows. firstly, we summarize related work and introduce base concepts. then, we discuss our ideas, preliminary results, and propose directions for further research. ## rhetorical structure theory rhetorical structure theory (rst) proposed by @xcite allows to represent any text as a binary tree. its nodes correspond to text spans and edges conform to discourse relations between them. the tree is constructed step by step. at the first step, the leaves of the tree are elementary discourse units (shortly edus). they contain introductory words, single thoughts, or clauses. further, some of these nodes are connected via corresponding discourse relations like "elaboration", "joint" and "summary", and form new nodes associated with bigger text spans. then, the updated set of nodes is connected due to the corresponding discourse relations and so forth. in the end, the text span associated with the root node is the full entire text. rst distinguishes two types of nodes: nucleus and satellite. nodes of the first type contain key information necessary to understand meaning of the text. nodes of the second type comprise supplementary information. figure 2 demonstrates an example of a discourse tree constructed by the open-source alt documentlevel discourse parser @xcite fot the following text: here, edus are limited by brackets. ## conclusion in this paper, we investigated the current transformer-based approaches to natural language generation. we proposed two challenging research directions: an improvement of overall discourse structure and a correction of entity values to construct meaningful texts. these directions are independent but complement each other well. by learning how to solve both problems, we expect to be able to generate coherent long texts with meaningful, plausible thoughts. our idea is to first generate discursively and syntactically coherent text using the custom fine-tuned gpt-2 model. further, utilizing the nonlearning technique based on web mining and text alignment, we replace values of wrong factoid entities with truthful ones. we proposed a method to evaluate the quality of overall discourse structure and experimentally confirm that gpt-2 generates texts with a mistaken and inconsistent structure in some cases. we suggested some ideas to integrate additional knowledge about discourse into the gpt-2 fine-tuning and generation processes. apart from that, we suggested a way to correct wrong entity values in generated texts using web mining. we propose the partition of information types from two sources (generated texts and texts from external knowledge base) to apply a syntactic and semantic alignment. we suggested some ways to develop a universal approach that will not be just english languagespecific and can be applied to other languages. at the moment, there are open-source discourse parsers for russian, german, spanish, and we can utilize them without modifying the approach.