I nfo D iffusion: Information Entropy Aware Diffusion Process for Non-Autoregressive Text Generation (2023)

Abstract: Diffusion models have garnered considerable interest in the field of text generation. Several studies have explored text diffusion models with different structures and applied them to various tasks, including named entity recognition and summarization. However, there exists a notable disparity between the “easy-first” text generation process of current diffusion models and the “keyword-first” natural text generation process of humans, which has received limited attention. To bridge this gap, we propose InfoDiffusion, a non-autoregressive text diffusion model. Our approach introduces a “keyinfo-first” generation strategy and incorporates a noise schedule based on the amount of text information. In addition, InfoDiffusion combines self-conditioning with a newly proposed partially noising model structure. Experimental results show that InfoDiffusion outperforms the baseline model in terms of generation quality and diversity, as well as exhibiting higher sampling efficiency.

## introduction non-autoregressive (nar) generation refers to a method of generating sequences where each element is generated independently, without relying on previously generated elements, allowing for faster parallel generation but potentially sacrificing the generation accuracy @xcite . recently, diffusion models have demonstrated powerful generative capabilities in image generation tasks, gradually becoming a new paradigm in generative models. the successful application of diffusion models to continuous data such as images and audio has motivated researchers to introduce them to discrete data like text. previous researches have attempted to incorporate diffusion models into nonautoregressive text generation, designing different text diffusion model structures and applying them to various text generation tasks, such as named entity recognition @xcite and summarization @xcite . however, these works have failed to recognize a fundamental difference between the process of generating text with diffusion models and the actual process of human text generation, which may be a reason why text diffusion models have consistently fallen short in terms of generation efficiency and quality. previous research has found that the text diffusion models seem to follow an "easy-first" principle @xcite in the decoding process. the "easy-first" principle means the model tends to generate tokens that are most frequently observed (and least surprising) in the training corpus, in order to achieve a higher likelihood. as the context becomes richer, more details are incorporated into the sequence. figure 1 illustrates the decoding process of some existing text diffusion models, where it can be observed that the model tends to prioritize generating simple, highfrequency, and semantically poor words like "the" and "of" before generating less frequent but more informative and semantically rich words like "structure" and "remember". this differs from the actual order in which humans process or generate text. people tend to prioritize the core parts of a sentence or a paragraph, which contain crucial information @xcite . for example, when asked:"what are your upcoming plans?", you would answer:"i am going to finish a research paper." in this process, the words that come to your mind first are most likely "paper" or "finish", as they carry key information or have higher information entropy, rather than meaningless words like "a" or "the". it is difficult to imagine how someone whose first reaction is "the" would answer the question mentioned above. similarly, this is also disastrous for a language model to which we expect to impart language abilities and even thoughts. ## infodiffusion in this section, we introduce the detailed design of infodiffusion. the overall model architecture of infodiffusion is depicted in figure 2 . infodiffusion incorporates an information entropy aware noise schedule, enabling the model to follow a "keyinfo-first" generation strategy, thereby achieving text generation that aligns with human-like processes. additionally, infodiffusion combines selfconditioning and partially noising to achieve faster and superior text generation. ## conclusion this paper, we propose infodiffusion, a novel nonautoregressive text diffusion model. by designing an information entropy aware noise schedule, we enable the diffusion model to follow a "keyinfofirst" text generation process that is more aligned with human text generation, thereby achieving improved effectiveness and efficiency in text generation. experimental results on four benchmark datasets confirm the effectiveness of infodiffusion. this study is the first research on the decoding order of diffusion models and the first attempt to alter the decoding order of diffusion text models. future work could explore the use of the proposed noise schedule to replace the existing noise in related tasks based on diffusion models, in order to further enhance the model's performance.