Visualize Before You Write: Imagination-Guided Open-Ended Text Generation (2023)

Abstract: Recent advances in text-to-image synthesis make it possible to visualize machine imaginations for a given context. On the other hand, when generating text, human writers are gifted at creative visualization, which enhances their writings by forming imaginations as blueprints before putting down the stories in words. Inspired by such a cognitive process, we ask the natural question of whether we can endow machines with the same ability to utilize visual information and construct a general picture of the context to guide text generation. In this work, we propose iNLG that uses machine-generated images to guide language models (LM) in open-ended text generation. The experiments and analyses demonstrate the effectiveness of iNLG on open-ended text generation tasks, including text completion, story generation, and concept-to-text generation in both few-shot and full-data scenarios. Both automatic metrics and human evaluations verify that the text snippets generated by our iNLG are coherent and informative while displaying minor degeneration.

## introduction one great resource human writers cherish is the ability of imagination, with which they render mental images about an actual or vicarious experience and link knowledge that would later make the writing more concrete, sensible, and intriguing. cognitive studies show that visual imagery improves comprehension during language processing @xcite @xcite , and that mental imagery facilitates humans' written language expression at young ages @xcite . when it comes to the study of artificial intelligence (ai), one classic challenge for ai systems is to generate informative and coherent text snippets. open-ended text generation is such a task that provides an input context, and asks the model to 1 our code & data: https://github.com/vegb/inlg . : and the individual adds them to the pan. ## conclusion in this work, we propose inlg, a framework that introduces machine-generated images to guide open-ended text generation. this endows the machines with the ability of creative visualization that human writers often demonstrate. we distill the vision information from the pre-trained multimodal models and further construct visual prefixes to guide language models to perform text generation with the teacher forcing and the contrastive objective. extensive experiments show the effectiveness of inlg in open-ended text generation tasks, including text completion, story generation, and concept-to-text generation in few-shot settings.