V i PE : Visualise Pretty-much Everything (2023)

Abstract: Figurative and non-literal expressions are profoundly integrated in human communication. Visualising such expressions allow us to convey our creative thoughts, and evoke nuanced emotions. Recent text-to-image models like Stable Diffusion, on the other hand, struggle to depict non-literal expressions. Recent works primarily deal with this issue by compiling humanly annotated datasets on a small scale, which not only demands specialized expertise but also proves highly inefficient. To address this issue, we introduce ViPE: Visualise Pretty-much Everything. ViPE offers a series of lightweight and robust language models that have been trained on a large-scale set of lyrics with noisy visual descriptions that represent their implicit meaning. The synthetic visual descriptions are generated by GPT3.5 relying on neither human annotations nor images. ViPE effectively expresses any arbitrary piece of text into a visualisable description, enabling meaningful and high-quality image generation. We provide compelling evidence that ViPE is more robust than GPT3.5 in synthesising visual elaborations. ViPE also exhibits an understanding of figurative expressions comparable to human experts, providing a powerful and open-source backbone to many downstream applications such as music video and caption generation.

## introduction recent efforts in addressing this challenge have mostly focused on constructing datasets for figurative language, such as metaphors, similes, and idioms @xcite @xcite . however, these datasets are often small in size and require expert knowledge for expansion. moreover, despite the benefits of these datasets, the fundamental issue of text-to-image models remains unresolved. to address these limitations, we present vipe: visualise pretty-much everything. vipe eliminates the need for human annotations or images with metaphorical contents, yet effectively assists text-to-image models in visualising figurative and abstract phrases, and even arbitrary textual input. the core idea behind our approach is to unfold the implicit meaning through a new textual description (elaboration) containing visual symbols. following @xcite , we use the term visual elaboration to refer to the visualisable textual description of a piece of text with figurative content. as illustrated in figure 1 , vipe transforms the input into a detailed image caption while preserving the intended meaning. therefore, it facilitates the visualisation of figurative language. building vipe involves three main stages. (1) a large scale lyric dataset: we compile a largescale collection of lyrics (≈ 10m lines) as a rich source of figurative language. (2) synthetic visual elaborations: we construct a supervised dataset we call lyriccanvas, by employing a large language model(llm) to generate noisy visual elaborations for all the lyrics. (3) knowledge distillation: we conduct knowledge distillation to build a robust model by fine-tuning a set of lightweight language models on lyriccanvas. vipe, our approach, addresses the limitations found in previous works by leveraging two key findings. the first finding is that lyrics serve as a rich repository of knowledge, embodying a wide spectrum of figurative language, including metaphors, similes, idioms, and beyond @xcite @xcite . the second finding stems from the observation that the task of vipe is akin to style transfer using machine translation (mt) @xcite @xcite , which often benefits from large amounts of data @xcite @xcite , including noisy data @xcite @xcite . therefore, we propose to create a large-scale dataset, the lyriccanvas dataset, from publicly available lyrics with automated but potentially noisy visual elaborations generated by an llm, gpt3.5 foot_0 , instructed via prompting. subsequently, we build vipe by fine-tuning two lightweight language models, gpt2-small, and gpt2-medium @xcite on the lyriccanvas dataset. we will show that vipe, despite its size (s: 117m and m: 345m parameters), is more robust than gpt3.5 with 175b parameters in synthesising zero-shot visual elaborations. figure 2 demonstrates two challenging examples for dall.e 2, highlighting the improvement depictions based on vipe. overall, our contributions are the following. 1. we release a robust and powerful model tailored to assist all text-to-image models in visualising nonliteral expressions. 2. we introduce the largest dataset available for generating visual elaborations, which we refer to as lyriccanvas. with approximately 10 million samples, lyriccanvas proves to be adequate, unlike existing datasets, for finetuning powerful language models like gpt2. moreover, we provide our scraper framework, allowing researchers to acquire the exact training inputs at no additional cost. 3. we eliminate the expensive and time-consuming involvement of human expert annotations for abstract and figurative visualisations. 4. we show that vipe's generation is highly robust and is competitive with human experts. vipe's powerful zero-shot capability paves the way for its usage in downstream applications such as synthetic caption generation from keywords, abstract art visualisations, and music video generations. the source code, pre-trained vipe, and the lyriccanvas dataset are available at foot_1 . ## vipe we present vipe, a set of robust and lightweight language models designed to generate visual elaborations from arbitrary text input. the development of vipe comprises three stages, illustrated in figure 3 . firstly, we perform data collection by scraping and preprocessing an extensive collection of lyrics (≈ 10m lines) sourced from genius 6 . secondly, we utilise a large language model (llm) to generate noisy visual elaborations for the lyrics by appropriate prompt design. finally, the paired data of lyrics and generated visual elaborations are used to train lightweight language models. they are fine-tuned using a causal language modeling objective tailored specifically for visual elaborations. the primary goal is to generate detailed textual descriptions of visual scenes (visual elaborations) to convey the intended meaning of the rich figurative phrases in lyrics. the generated elaboration can then be passed as an input prompt to any text-toimage synthesiser to visualise the original input. ## evaluation assessing figurative language visualisation is a complex task due to its highly subjective nature (figure 2 ). moreover, existing evaluation procedures differ, ranging from visual entailment ## implementation details training on lyriccanvas: two versions of vipe are developed: vipe-m (based on gpt2-small) and vipe-s (based on gpt2-medium). the models are fine-tuned on lyriccanvas for 5 epochs using 8 a100 nvidia gpus, each with 40 gb ram. we use the adamw optimizer @xcite with a learning rate of 5e-05 and a linear scheduler with 1000 warmup steps. for vipe-s, the batch size is generally 50, except with a context size of 7, where a batch size of 32 is utilised. in the case of vipe-m, the batch sizes vary for different context sizes: {32, 32, 20, 16, 8} for context sizes {0, 1, 3, 5, 7}, respectively. 10 % of lyriccanvas (≈ 1m samples) is used for validation. image-text retrieval: we load a blip @xcite ) checkpoint trained on coco, initialised on vit-b @xcite and bert-base @xcite . to finetune, we use a batch size of 16 for 10 epochs using adamw, a learning rate of 1e -4, and a batch size of 128 with reranking for fast inference, commonly used in retrieval @xcite . emotion classification: bert-base-uncased is fine-tuned for 5 epochs using adamw optimizer, a learning rate of 5e -05 and a batch size of 256. figurative qa: we made use of the provided evaluation framework foot_10 by @xcite and trained with the batch size of 32 for 5 epochs using adamw optimizer, with a learning rate of 5e -05. the test results are publicly available under the name imaginate emnlp2023 on their leaderboard. 6 applications music video generation: besides its power to produce well-depictable prompts for text-to-image synthesis, we utilise vipe as a versatile assistant to generate stylish visuals for music videos. we employ a zero-shot text-to-video synthesis approach, incorporating motion information for temporal consistency @xcite and smooth visual transitions between lines by interpolating image and text embeddings. more specifically, our approach comprises (1) extracting lyrics and timestamps from a given audio file, (2) generating visual elaborations from lyrics using vipe, (3) creating a cohesive video narrative that encompasses the composition of the song, including all the lyrics and music. in appendix b, we detail our pipeline. below, we summarise our key points. ## conclusion in this paper, we introduced vipe, the first automated model for visualising figurative expressions in text-to-image models. vipe efficiently gener-ates diverse image captions, or visual elaborations, from arbitrary textual input. our approach involves training lightweight language models on a novel dataset, lyricscanvas, comprising 10 million lines of lyrics paired with visual elaborations generated by gpt3.5. our key achievements are as follows: overall, vipe enables accessible solutions for complex research questions and paves the way for automated pipelines. in the future, we plan to apply vipe in investigating the interplay between language and perception in related disciplines such as psycho-linguistics and cognitive science.