Creative Painting with Latent Diffusion Models (2022)

Abstract: Artistic painting has achieved significant progress during recent years. Using a variational autoencoder to connect the original images with compressed latent spaces and a cross attention enhanced U-Net as the backbone of diffusion, latent diffusion models (LDMs) have achieved stable and high fertility image generation. In this paper, we focus on enhancing the creative painting ability of current LDMs in two directions, textual condition extension and model retraining with Wikiart dataset. Through textual condition extension, users’ input prompts are expanded with rich contextual knowledge for deeper understanding and explaining the prompts. Wikiart dataset contains 80K famous artworks drawn during recent 400 years by more than 1,000 famous artists in rich styles and genres. Through the retraining, we are able to ask these artists to draw artistic and creative paintings on modern topics. Direct comparisons with the original model show that the creativity and artistry are enriched.

## introduction artistic painting has achieved significant progress during recent years thanks to the appearing of hundreds of gan variants @xcite . however, adversarial training has been reported to be notoriously unstable and can lead to mode collapse. to escape from adversarial training and inspired by non-equilibrium thermodynamics, diffusion probabilistic models @xcite , such as noise-conditional score network (ncsn) @xcite , denoising diffusion probabilistic models (ddpm) @xcite , stable diffusion models in latent spaces @xcite have achieved ganlevel sample quality without adversarial training. these diffusion models are appealing with rather flexible model architectures, exact log-likelihood computation, and inverse problem solving without re-training models. there are two markov chain style processes in a typical diffusion model. the first process is a forward diffusion process which appends multiplescale random noise to a given data sample "step by step" or "in jump" until the disturbed sample slip into a predefined isotropic gaussian distribution. this process does not include trainable parameters. the second process is a reverse diffusion process which generates a target distribution data sample from pure noise guided by some (user-input) pregiven conditions. a parameterized deep learning model is required in this reverse process. intuitively speaking, the forward diffusion process can be recognized as "directional blasting of a building" x 0 to "ruins with dusts" x t . the learning algorithm is a reverse engineering which learns how to (re-)construct a building (expressed by p θ (x t-1 |x t ) with a parameter set θ and @xmath0, ..., t }) from each step of inverse directional blasting (expressed by q(x t-1 |x t , x 0 )) of each given building sample x 0 . in one step of this reverse engineering, x t-1 represents "one complete wall" in a building and x t represents "concrete and sands" that can be used to construct the complete wall x t-1 in a reconstruction process or can be obtained from the complete wall x t-1 in a forward "blasting" process. the reconstruction process is learned from the blasting process with targets such as noise prediction in ddpm @xcite or score prediction using score matching strategy in ncsn @xcite . we follow a recent impressive work of highresolution image synthesis with ldms by given textual or visual conditions foot_0 @xcite . there are several proposals in this ldm. the first proposal is applying the encoder part of a pretrained variational autoencoder to project images into low-dimension latent spaces and then perform diffusion/construction processes. training diffusion models on such a low-dimension representation space allows us to reach a near-optimal point between computation complexity reduction and detail preservation to boost virtual fidelity of constructed images. the second is a cross-attentionenhanced @xcite u-net framework @xcite in the diffusion model where general conditioning inputs such as text or bounding boxes are taken as memory (i.e., keys and values in the cross-attention layers) for the query (latent representations of images to be generated) to retrieve information on. finally, the decoder module in the variational autoencoder is applied to recover the target image into high-resolution. we aim at improving the creativity of image synthesis, or painting, using conditional ldms. it is relatively difficult to precisely define the concept of creativity since it is subjective and influenced by culture, history, and region. the color, style, objects included in painting reflect rich emotions of numerous topics. for example, when we are given a textual condition, "a painting of a virus monster playing guitar", we can recognize noun entities such as "virus monster" and "guitar" and a verbal action "playing". what are the emotions involved in this textual hint? happy, surprise and funny should be the major emotions. the painting requires less imagination since we should better include the entries with a determined action. however, there are challenges for the models to draw painting for rather high-level topics such as "urbanization of china" or "asian morning". these textual hints should be enriched and extended with concrete objects and actions to tell a story in a painting or in a series of paintings. extensions to "urbanization of china" include "originally a collection of fishing villages, shenzhen rapidly grew to be one of the largest cities in china", "a train runs on the snow-capped mountains of the qinghai-tibet plateau", and "left-behind children running in wheat-field". given an initial textual hint, we leverage wikipedia and large-scale pretrained language models to execute this extension. in addition, we retrain existing checkpoints by the wikiart paintings dataset foot_1 which has a collection of 81,444 fine-art paintings from 1,119 artists, ranging from fifteenth century to modern times. this dataset contains 27 different styles (e.g., mini-malism, symbolism, realism) and 45 different genres. as far as our knowledge, it is currently the largest digital art datasets publicly available for research usage. this dataset was used to train an @xcite where conditions such as categorical label information was used for artwork synthesis. in this paper, we embed the textual information of artists, year, styles, and genres as additional conditions to the ldm. through this way, we can explicitly invite vincent van gogh or rembrant to help us "draw" artworks of modern topics such as "urbanization of china". this paper is organized as follows. in section 2, we briefly review the background knowledge required for understanding the stable diffusion models @xcite . in particular, we describe the two processes defined in ddpm @xcite , the variational autoencoder framework and loss functions used in it @xcite , cross attention enhanced u-net which acts as the backbone of the diffusion model, and pseudo numerical methods integrated with ddims for fast sampling. in section 3, we describe our proposal of extending users' prompts by pretrained language models and existing knowledge resources. in section 4, we show detailed information of the wikiart dataset and our pipeline of retraining. we describe the experiments in section 5 and finally conclude in section 6. ## background diffusion models have been successfully used in image generation @xcite ), text-tospeech synthesis @xcite , sing synthesis and conversion @xcite , music generation @xcite and healthcare medical anomaly detection @xcite . surveys can be find in @xcite @xcite . we limit our discussion to text-to-image generation by leveraging the ldms @xcite and existing checkpoints foot_2 . we briefly review the core processes and target objectives of ddpms @xcite that are used in ldms. in addition, variational autoencoders enhanced with kl-divergence, cross-attention embedded u-net @xcite , clip pretrained language models @xcite , as "reference" to pθ(xt-1|xt) q(xt|xt-1) diffusion process reconstruction process 2021) and sampling algorithms such as that used in denoising diffusion implicit models (ddims) @xcite and pseudo numerical methods @xcite will be briefly reviewed. ## textual condition extension we perform textual condition extension by leveraging wikipedia as the knowledge base and largescale pretrained language models as implicit knowledge graphs. the pipeline is depicted in figure 2 . given a textual prompt, we first match it with the title list in wikipedia. at the same time, the input prompt is sent to ( 1 ) a pretrained language model, t5 @xcite , to continue writing by taking the given prompt as a prefix hint and to ( 2 ) a pretrained dialog model, dialogpt foot_7 @xcite that takes the input prompt as "query" and consequently generate "responses". wikipedia's titles and contents are used for matching the input prompt and t5/dialogpt's outputs. we use bm25 @xcite here to algorithm 1: pseudo linear multi-step (plms) algorithm enhanced by ddim 1 x t ∼ n (0, i); 2 for @xmath1, t -1, ..., 1 do 3 e @xmath2, t); 4 if @xmath3; 6 x t-1 , f θ (x t , t) = ddim(x t , e t , t); 7 e t-1 = ϵ θ (x t-1 , t -1); 8 e ′ @xmath4; 9 else if @xmath5; 11 e ′ @xmath6; 12 else if @xmath7; 14 e ′ @xmath8; 15 else 16 # plms-4th (adams-bashforth) ; 17 e ′ @xmath9; 18 x t-1 , f θ (x t , t) = ddim(x t , e ′ t , t); 19 return x 0 ; in addition, we encourage the result prompts to include spatial and temporal information. we leverage a named entity recognizer 10 and regular expressions to recognize place/region names, addresses, time, and date. the number of spatial and temporal entities discounted by a hyper parameter 9 https://laion.ai/blog/laion-5b/ 10 https://github.com/kamalkraj/bert-ner input prompt wikipedia t5/dialogpt pretrained model 1 retrieve by title matching 2 continue generating/response generating 3 enriched retrieval by content matching entry document result prompt list 6 sentence importance weighting and selecting 4 sentence selecting 5 document selecting λ 2 is added with w(v) for the final scoring of a prompt. ## retraining with wikiart different artists have quite different numbers of paints in wikiart dataset. the top-3 artists are vincent van gogh, nicholas roerich, and pierre auguste renoir with 1,889, 1,860, and 1,400 paintings, respectively. the top-10, top-20, and top-30 artists share 14.18%, 21.80%, and 27.62% of the samples, respectively. figure 3 shows the distribution of the number of paintings and their authors. we first retrain the clip text encoder with the same tokenizer with the ldm fixed. this stage is expected to map the captions used in wikiart to stable diffusion's latent space. then, we fine-tune the text encoder and the ldm jointly. this stage is expected to help the ldm to enrich its knowledge of artworks from different artists, in different styles and genres. ## experiments we use a dgx-a100-80gb server with 8 nvidia a100-80gb gpu cards. the original code and settings of the stable diffusion model's checkpoint v1-4 is reused. during inferencing, single gpus are used with @xmath10.0, @xmath11, height and width are both 512, and scale is set to be 5.0. ## conclusion in order to improve the creativity of ldms, we have proposed two directions of extending the input prompts and of retraining the original model by the wikiart dataset. we take the 1,000 artists in recent 400 years as the major source of both creativity and artistry. with these proposals, the resulting diffusion models can ask these famous artists to draw novel and expressive paints of modern topics. we believe this is an interesting topic and has industrial design requirements for real-world ap-plications, such as cloth designing, advertisement posters, and game character designing. through drawing the real-world's topics with the help of hundreds to thousands famous artists, it is reasonable to learn the creativity and fertility from these artists' eyes.