Towards Informative Open-ended Text Generation with Dynamic Knowledge Triples (2023)

Abstract: Pretrained language models (PLMs), especially large language models (LLMs) demonstrate impressive capabilities in open-ended text generation. While our statistical results show that LLMs often suffer from over-concentrated information, where the generated texts overly focus on the given prompt and fail to provide sufficient background and detailed information as humans do. To address this issue, we propose a dynamic knowledge-guided informative open-ended text generation approach, that utilizes a knowledge graph to help the model generate more contextually related entities and detailed facts. Specifically, we first employ a local knowledge filter to extract relevant knowledge from the comprehensive knowledge graph for a given topic sentence. Then we introduce a dynamic knowledge selector to predict the entity to be mentioned in the subsequent sentence. Finally, we utilize a knowledge-enhanced text generator to produce a more informative output. To evaluate the effectiveness of our approach, we evaluate the proposed approach in two scenarios: fine-tuning for small PLMs and prompt tuning for LLMs. Experimental results show that our approach could generate more informative texts than baselines.

## introduction open-ended text generation is a complex undertaking task, encompassing the need for fluent sentences, a natural and coherent flow, as well as informative and non-trivial content @xcite . in recent years, much of the research has concentrated on addressing grammatical correctness concerns such as inter-sentence coherence and ensuring relevance between the input prompt and the generated story content @xcite @xcite . however, thanks to the advancements in large language models (llms) like gpt-3 @xcite and chatgpt @xcite , many of these aforementioned challenges have been substantially mitigated @xcite . plms, particularly llms, have made significant advancements in open-ended text generation. however, our analysis reveals that current llms still face the challenge of over-concentrated information, wherein they prioritize events and entities mentioned in the prompt and fail to provide relevant entities and sufficient information as humans would. we define over-concentrated information as follows: over-concentrated information: texts that predominantly concentrate on the events and entities mentioned in the prompt often face challenges in encompassing relevant entities' comprehensive information. this discrepancy is evident in figure 1 , where we compare texts generated by human writers with those generated by powerful llms such as gpt-3.5-turbo and text-davinci-003 using the headline "elon musk sets low expectations before first spacex launch of starship, most powerful rocket ever built". humanwritten text not only includes information directly related to the headline but also provides additional contextual details, such as the collaboration between nasa, faa, and spacex, as well as specific information about the rocket and the launch location. on the other hand, texts generated by gpt-3.5-turbo and text-davinci-003 tend to be focused solely on the provided information and lack the depth of additional relevant details beyond the headline. in addition to analyzing specific cases, we also conducted human evaluation to compare the informativeness of texts written by humans with those generated by llms. 1 we select a diverse range of plms and ask annotators to rank the texts based on the informativeness. the results are listed in to address the issue of over-concentrated information, we propose a dynamic knowledge-guided informative open-ended text generation approach, infogen, in which a knowledge graph is utilized to help the model generate more related entities and detailed facts. infogen consists of several key components. first, we employ a local knowledge filter to extract relevant information from a comprehensive knowledge graph. second, we introduce a dynamic knowledge selector, which predicts the entity to be mentioned in the subsequent sentence. finally, we utilize a knowledge-enhanced text generator that leverages the filtered local knowledge and selected entities to produce a more informative 1 specific details are listed in appendix b and coherent output. furthermore, we construct a chinese news generation dataset and annotate it, along with two existing english datasets, using named entity recognition and entity-linking techniques. to evaluate the effectiveness of infogen, we conduct experiments in two scenarios: fine-tuning for small plms and prompt learning for llms. the experimental results demonstrate that infogen consistently outperforms baseline models in terms of text informativeness. our contributions are summarized as the following points: ## conclusion in this paper, we address an important and unresolved issue in knowledge-intensive text generation, namely over-concentrated information. through manual evaluations of news texts generated by both human writers and models, we discovered substantial disparities between the two, emphasizing the difficulty of generating informative content. drawing from these insights, we propose a dynamic knowledge-guided approach to openended text generation called infogen. our method involves filtering relevant knowledge triples and dynamically selecting entities as content plans. we conduct experiments in two scenarios: fine-tuning small plms and prompt learning for llms. the results demonstrate that our approach effectively enhances the informativeness of the generated texts, thereby addressing the limitations observed in previous methods.