Learning to Imagine: Visually-Augmented Natural Language Generation (2023)

Abstract: People often imagine relevant scenes to aid in the writing process. In this work, we aim to utilize visual information for composition in the same manner as humans. We propose a method, LIVE, that makes pre-trained language models (PLMs) Learn to Imagine for Visually-augmented natural language gEneration. First, we imagine the scene based on the text: we use a diffusion model to synthesize high-quality images conditioned on the input texts. Second, we use CLIP to determine whether the text can evoke the imagination in a posterior way. Finally, our imagination is dynamic, and we conduct synthesis for each sentence rather than generate only one image for an entire paragraph. Technically, we propose a novel plug-and-play fusion layer to obtain visually-augmented representations for each text. Our vision-text fusion layer is compatible with Transformer-based architecture. We have conducted extensive experiments on four generation tasks using BART and T5, and the automatic results and human evaluation demonstrate the effectiveness of our proposed method. We will release the code, model, and data at the link: https://github.com/RUCAIBox/LIVE .

## introduction natural language generation (nlg) is a fundamental technique for supporting a variety of downstream applications @xcite , e.g., text summarization, story generation, and data-to-text generation. as the mainstream nlg approach, pre-trained language models (plms) can produce human-like text under the guidance of input conditions. despite their success, these models are pre-trained on the text-only corpora, and they cannot well capture visuallygrounded semantics, e.g., visual commonsense @xcite , making it difficult to achieve desired results when visual knowledge is required. ## related work pre-trained models. in recent years, large-scale pre-training has achieved remarkable success and has become the dominant technique in the nlp community @xcite @xcite . pre-trained on massive text corpora, models can learn contextualized representations that include both linguistic and world knowledge @xcite . since plms are trained with pure text corpora without connection to the visual world, vision-language pretraining (vlp) leverages image-text pairs to learn cross-modal representations @xcite @xcite . it has been discovered that vlp models have more visual knowledge than plms @xcite , however, they cannot perform well on text-only tasks such as language understanding @xcite . in this work, we mainly focus on incorporating visual knowledge to enhance the performance of natural language generation tasks based on existing text-only models. visually-augmented language learning. considering the leakage of visual knowledge in language models, many researchers attempt to en-hance text-only tasks with visual information, which is known as visually-augmented (aided or grounded) language learning. vokenization @xcite and iace @xcite propose to treat contextualized-related images as vokens and pre-train a text model to predict them for fusing visual information. similarly, vid-lankd @xcite extends finite image vokens to diverse video frames and employs a knowledge distillation method to acquire visual knowledge. the subsequent works leverage clip @xcite as the vision source to integrate visual information into plms via clip output embeddings @xcite or knowledge transfer methods @xcite . the majority of these works can outperform plms on language understanding tasks. as for natural language generation tasks, researchers mainly focus on finding suitable images and fusing the visual representations into text-only models using a shallow module. some works apply generation models, such as gan-based models @xcite and vaebased models @xcite , to synthesize (latent) images, while @xcite , @xcite , and @xcite propose to employ contextualized text embeddings to retrieve relevant images. in our work, we utilize the superior diffusion model @xcite to synthesize high-quality images and propose a plugand-play vision-text fusion layer to deeply integrate visual knowledge into plms and obtain visuallyaugmented text representations. ## further analysis in this section, we conduct various experiments to test the efficacy of our methods. the tuning details are identical to those introduced in section 4.1.4. ## conclusion in this paper, we present the live method for natural language generation. first, we propose an imagination-based method, imitating the process of human writing. it is a relevant, selective, and dynamic approach that leverages stable diffusion to synthesize images for each input sentence and discard the images with lower text visuality computed by clip. furthermore, we introduce a plug-andplay vision-text fusion layer to deeply incorporate visual knowledge into plms and obtain visuallyaugmented text representations for text generation. extensive experiments have demonstrated that our live methods are compatible with two plms (i.e., bart and t5) and can achieve superior performance over all the baseline models. in future work, we will investigate how to synthesize more relevant images based on the input prompt and design a finer fusion method for better aligning different words and images. we will also attempt to extend our methods to more tasks (e.g., language understanding) and plms (e.g., bert). besides, it is meaningful to explore the probability of combining our live method with existing large language models @xcite to enhance their representation and generation capabilities.