CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders
Abstract
CLIPDraw is an algorithm that synthesizes novel drawings from natural languageinput.
It does not require any additional training; rather, a pre-trained CLIPlanguage-image encoder is used as a metric for maximizing similarity between thegiven description and a generated drawing.
Crucially, CLIPDraw operates overvector strokes rather than pixel images, which biases drawings towards simplerhuman-recognizable shapes.
Results compare CLIPDraw with other synthesisthrough-optimization methods, as well as highlight various interesting behaviorsof CLIPDraw, such as satisfying ambiguous text in multiple ways, reliably producing drawings in diverse styles, and scaling from simple to complex visualrepresentations as stroke count increases.
Introduction
When humans hear a description of a scene, it’s easy to imagine what it may look like.
Conversely,when we construct a mental image, it’s easy to then describe that scene.
At some level, humans havea deeply coupled representation for textual and visual structures key to understanding our world.
The recent introduction of CLIP (Radford et al., 2021), a dual language-image encoder, is a largestep towards unifying textual and visual information.
In a CLIP model, both text and images aremapped onto the same representational space, thus enabling the similarity between images and textualdescriptions to be measured.
When trained on large amounts of data, CLIP representations have beenshown to solve a robust range of image-based recognition tasks.
This work presents CLIPDraw , an algorithm that synthesizes novel drawings based on naturallanguage input.
CLIPDraw does not require any training; rather a pre-trained CLIP model is used asa metric for maximizing similarity between the given description and a generated drawing.
Ratherthan photorealistic images, CLIPDraw aims to synthesize simple drawings that nevertheless matchthe prompt.
Thus, CLIPDraw optimizes a set of vector strokes rather than pixel images, a constraintthat biases drawings towards simple human-recognizable shapes.
The aim of this work is to present CLIPDraw a testbed for exploring language-image relationshipsand synthesizing AI-assisted artwork, as well as to showcase various nuances of the method.
Results compare between CLIPDraw and other optimization-based text-to-image methods, along withhighlighting several interesting behaviors:
• By adjusting descriptive adjectives, such as “watercolor” or “3D rendering”, CLIPDrawproduces drawings of vastly different styles.
• CLIPDraw often matches the description prompt in creative ways, such as writing wordsfrom the prompt inside the image itself, or interpreting ambiguous nouns in multiple ways.
• Running CLIPDraw with a low stroke count results in cartoonish drawings, while highstroke counts tend to result in realistic renderings.
• By giving CLIPDraw abstract prompts, such as “happiness” or “self”, we can examine whatvisual concepts the CLIP model associates with them.
• CLIPDraw behavior can be further controlled through the use of negative prompts, such as“a messy drawing”, to encourage the opposite behavior.
Related Work
Text-to-Image Synthesis.
This work greatly draws from the field of text-to-image synthesis, whoseprimary aim is to generate images that correctly match a given textual description.
In recent years,focus has been on methods that aim to learn a direct text-to-image mapping function, often in theform of a conditional GAN (Goodfellow et al., 2014a; Mirza and Osindero, 2014;
Reed et al., 2016;
Frolov et al., 2021).
Commonly used datasets include Oxford-120 Flowers (Nilsback and Zisserman,2008), CUB-200 Birds (Wah et al., 2011), and COCO (Lin et al., 2014), all of which contain naturalimages and captions describing them.
While GAN-based methods have enabled considerable progresstowards photorealistic image synthesis, strong autoregressive models have achieved similar qualityresults (Oord et al., 2017; Chen et al., 2020), with the recent DALL-E model (Ramesh et al., 2021)showcasing the benefit of scaling text-to-image synthesis networks to a large capacity.
In comparisonto text-to-image generative models, which require large amounts of training, this work follows theframework of synthesis through optimization , in which images are generated through evaluation-timeoptimization against a given metric.
Synthesis Through Optimization.
Instead of directly learning an image generation network, analternative method of image synthesis is to optimize towards a matching image during evaluationtime.
This framework is often referred to as activation maximization (Erhan et al., 2009; Nguyenet al., 2016; Mordvintsev et al., 2015), where a random image is optimized through backpropogationto increase certain neuron activations of a pretrained network.
Activation-maximization methods haveproduced highly realistic images, however it is a challenge to understand the meaning of a neuronactivation.
CLIPDraw builds off a set of methods where rather than maximizing an activation, theobjective is to minimize the distance between the produced image and a given description phrase,as defined by a powerful CLIP language-image encoder (Fernando et al., 2021; Murdock, 2021;
Crowson et al., 2022; Galatolo et al., 2021).
A key issue in synthesis through optimization is that theproduced images often leave the space of natural images (Nguyen et al., 2015), or fool the systemthrough adversarial means (Goodfellow et al., 2014b), thus a body of work aims to discover ‘naturalimage priors’ to constrain produced images (Nguyen et al., 2016, 2017).
While a typical solution isto constrain optimization to the generative space of a GAN, this setup can be expensive to evaluate,and only allows synthesis of images producible by the GAN generator.
Because CLIPDraw focuseson synthesizing drawings rather than realistic pictures, CLIPDraw instead limits optimization to a setof vector curves.
This constraint results in stroke-based images, which must capture larger featuressuch as shapes and outlines, rather than fine-grained textures.
Starting from a random set of Bézier curves, the position and colors of the curves are optimized so that the generated drawings best match the given description prompt.
Before being passed into the
CLIP encoder, drawings are augmented into multiple perspective-shifted copies.
Vector Graphics.
This work builds largely from work by Li et al.
(2020), which introduces adifferentiable renderer for vector graphics.
Image generation methods that operate over vector imageshave traditionally required a vector-based dataset, however recent work has shown how differentiablerenderers can be used to bypass this limitation (Reddy et al., 2021;
Shen and Chen, 2021; Kotovenkoet al., 2021).
CLIPDraw uses a differentiable renderer as a representation for generating drawings;namely a set of RGBA Bézier curves are optimized rather than pixels.
Follow Up Work to CLIPDraw.
A draft of this work was previously released to the publicresearch community, and a number of follow-up papers have since been published.
StyleCLIPDraw(Schaldenbrand et al., 2021) introduces an explicit image-based style loss to the CLIPDraw framework.
CLIP-CLOP (Mirowski et al., 2022) extends CLIPDraw to use image patches rather than strokes,and CLIPasso (Vinker et al., 2022) uses CLIP to extract sketches from photos.
Tian and Ha (2022)pursure optimization via evolutionary strategies rather than gradient descent.
CLIPDraw has alreadyinfluenced the image synthesis community in its pre-release form, and we hope a formal publicationwill continue to encourage collaboration.
Method
The objective of CLIPDraw is to synthesize a drawing that matches a given description prompt(Figure 1).
Specifically, a pre-trained CLIP model is used as a judge.
A CLIP model contains twonetworks – an image encoder and a textual encoder – which both map their respective inputs into ashared encoding space of a 512-length vector.
Similarity is measured via the cosine distance betweentwo encodings.
Thus, the goal of CLIPDraw is to produce an image which, when encoded via CLIP,matches the CLIP encoding of the prompt.
Drawings in CLIPDraw are represented by a set of differentiable RGBA Bézier curves, following themethod by Li et al.
Each curve is parametrized by 3-5 control points, along with thicknessand an RGBA color vector.
Drawings initially begin with curves randomly distributed throughout theimage, with a white background.
During optimization, the number of curves and control points isfixed, however the positions of the points along with the thickness and color vectors can be optimizedvia gradient descent.
The CLIPDraw algorithm (Algorithm 1) works by running evaluation-time gradient descent, as shownin Figure 2.
First, the description phrase is encoded via the CLIP model, and a random set of N
Bézier curves are initialized.
During each iteration, the curves are rendered to a pixel image viathe differentiable renderer, and the resulting image is then duplicated D times and augmented by arandom perspective shift and random crop-and-resize.
The resulting batch of augmented images is
Algorithm
Input: Description Phrase desc ; Iteration Count I ; Curve Count N ; Augment Size D ; Pre-trained
CLIP model.
Begin:
Encode Description Phrase.
EncPhr = CLIP(desc)
Initialize Curves.
Curves ..N = RandomCurve() for i =
to I do
Render Curves to Pixels.
Pixels = DiffRender(Curves)
Augment the Image.
AugBatch ..
D = Augment(Pixels)
Encode Image.
EncImg = CLIP(AugBatch)
Compute Loss.
Loss = − CosineSim(EncPhr, EncImg)
Backprop.
Curves ← Minimize(Loss) end for passed into the CLIP image encoder, and the cosine distances to the description phrase are summedto form the loss value.
Because all operations are differentiable, gradient descent can be run throughthe entire loop, optimizing the parameters of the curves to decrease loss.
This procedure is repeated Itimes, until convergence.
The goal of the image augmentation is to force drawings to remain recognizable whenviewed through various distortions.
Without image augmentation, synthesis-through-optimizationmethods often result in adversarial images that fulfill the numerical objective but are unrecognizable to humans.
This work uses the torch.transforms.
RandomPerspective andtorch.transforms.
RandomResizedCrop functions in sequence.
Note that the specific details ofthe augmentation were not the focus of this work, furthermore, trials with different augmentationsshow that augmentation choice does not influence synthesis to a noticeable degree (see Appendix).
Figure 3 showcases the gradual synthesis of a typical CLIPDraw drawing.
Note that while theoptimization process is largely deterministic, there is randomness in the initial curves and imageaugmentations, thus multiple runs of CLIPDraw can result in different drawings.
Results
In the following sections, various interesting behaviors of CLIPDraw are highlighted through a varietyof examples.
With the exception of Figure 15, example images are picked to best convey the behaviorin consideration.
Focus is placed on qualitative observations, unusual behavior, or recurring trends in
CLIPDraw image synthesis.
Compared to methods that learn a direct generative model, optimization-based synthesis methodsdo not require prior training.
Instead, images are generated through an evaluation-time optimizationloop, aiming to maximize a given objective.
This work specifically focuses on synthesizing imagesthat match the CLIP encoding of a description prompt.
The following methods are compared:
• CLIPDraw , in which drawings are produced by a set of RGBA Bézier curves.
The controlpoints, thickness, and colors of the curves can all be adjusted.
• Pixel Optimization , which instead optimizes a 224x224x3 matrix of RGB pixels.
Otherwise,
all algorithmic aspects are the same as CLIPDraw, including image augmentation.
• BigGAN Optimization , in which images are produced using a pre-trained BigGAN gen-
erator.
The weights of the generator are frozen; only the latent Z vectors are optimized.
Samples are generated as by Murdock (2021).
• VQGAN Optimization , in which images are created through sampling a VQGAN codebook
(Esser et al., 2021).
Samples are generated as by Crowson et al.
• CLIPDraw
(No Augment) , which is identical to CLIPDraw, except no image augmentation
is applied to the synthesized drawings.
These methods are run on the same CLIP matching objective for 250 steps of gradient descent(Figure 15).
In CLIPDraw, stroke count is 256, and 8 duplicates are used during image augmentation.
Prompt
CLIPDraw
Pixel Opt.
BigGan Opt
VQGAN Opt (No Aug)
A drawing of a cat.
.376 ± .
.385 ± .
.325 ± .
.377 ± .
.240 ± .
A paint.
of a sunset.
.390 ± .
.215 ± .
.379 ± . .379 ± .
.325 ± .
Underwater.
.413 ± .
.385 ± .
.327 ± .
.358 ± .
.247 ± .
Sheep wearing a top hat.
.434 ± . .434 ± .
.321 ± .
.411 ± .
.215 ± .
A 3D rend.
of a temple.
.467 ± .
.385 ± .
.311 ± .
.430 ± .
.240 ± .
Watercol.
of a firetruck.
.507 ± .
.385 ± .
.318 ± .
.457 ± .
.233 ± .
[Table]
CLIPDraw tends to result in a diverse set of human-recognizable doodles.
Pixel Optimization createsinteresting textures but fails to compose colors and shapes.
BigGAN and VQGAN Optimizationsynthesize high-resolution images, but are constrained to the set of images the generator can produce.
Images from CLIPDraw without image augmentation score high numerically, but are nonsense to thehuman eye.
4.1 What kinds of visual techniques does CLIPDraw use to satisfy the textual description?
CLIPDraw often results in drawings that match their description prompts in multiple, unexpectedways, as shown in Figure 5.
A prime example is the prompt for “a painting of a starry night sky”.
Thedrawing’s background features a sky with a prominent moon and a few scattered stars.
The drawing isrendered in a painterly style, however it also features an actual canvas and painter.
Inside the canvas,black and blue swirls resemble Van Gogh’s 1889 “The Starry Night”.
Another interesting behavior of CLIPDraw is its tendency to write words in the drawing itself.
In“Yeti taking a selfie”, letters resembling “Yeti” can be seen in the top-right corner.
In “Third Eye”,again words resembling “third” and “eye” are scattered throughout the image.
At times, the drawingscontain symbols that are not literally the description, but are still associated, such as the prompt “ 自転 車 ” ( bicycle in Japanese) resembling a Google Maps screenshot with a Japanese-like character inthe corner.
Prompt ambiguity also presents intriguing results.
In “Fast Food”, a McDonald’s logo along with aset of hamburgers is shown.
However, also present are two joggers in a footrace, providing anotherinterpretation of “fast”.
Included in Figure 5 are the top words predicted by CLIP as closest to theimage, showing that CLIP recognizes both “jogging” and “hamburger” as related to the synthesizeddrawing of “fast food”.
4.2 Can CLIPDraw reliably produce drawings in different styles?
A useful feature of CLIPDraw is its ability to adjust not just the content of its drawings, but alsothe styles, based on the description prompts given.
Part of this flexibility is due to the robustness ofcurve-based images: in comparison to methods that use a pre-trained GAN generator, CLIPDrawdrawings are not limited to the space of natural images.
Thus, a variety of styles can be produced,and these styles are easily explorable through text.
As shown in Figure 6, a synthesized image of a cat can look vastly different depending on thedescriptor words included.
When asked for a “drawing of a cat”, CLIPDraw synthesized a cartoonishdepiction of a cat, comprised mostly of an outline and simple face.
A “realistic photograph” featuresmore detailed shading, while a “cat as 3D rendered in Unreal Engine” showcases complex lightingalong with a depth-based blurring.
Further styles feature a bias towards certain colors, such as thereds and greens of Japanese woodblock prints, or the multi-color blends of watercolors.
An interesting result is that adjusting descriptive adjectives not only changes the textures of the drawings, akin to Style Transfer methods (Gatys et al., 2015), but also changes its structural representationof the underlying content.
For example, prompting for “a drawing” produces a flat cartoonish cat,while prompts like “a 3D wireframe” produce a cat in perspective, with depth and shadows.
4.3 How does the stroke count affect what drawings CLIPDraw produces?
When the stroke count is low, CLIPDraw tends to produce cartoonish or abstract representationsof the given description prompt.
As stroke counts increase, drawings become more detailed andincorporate additional features.
Figure 7 shows “The Eiffel Tower” with various stroke counts.
In the16-stroke example, the tower is drawn as only a few straight lines.
Higher stroke count images beginmore details on the Eiffel Tower itself, along with additional features such as background colors andcomplex lighting.
A common thread in synthesis-through-optimization methods is that pure optimization leads toundesirable results; it is also necessary to constrain optimization to a suitable space of images, suchas the natural images generated by a GAN, or any image made of strokes in the case of CLIPDraw.
Limiting stroke count furthers this constraint.
When optimizing within the space of 16-stroke images,it is hard to achieve details or textures, thus synthesized drawings will reveal the most basic formsthat make up a visual concept.
As a tool for AI-assisted art, the stroke-count parameter presents aneasy way of adjusting between “simple” and “complex”.
4.4 What happens if abstract words are given as a description prompt?
When given an abstract prompt without a literal interpretation, CLIPDraw must utilize cultural connections to come up with visual concepts that relate to the description.
Often, this results in drawingsthat contain symbols relating to the given prompt, such as in Figure 8 with “Happiness” containing
At times, synthesized drawings demonstrate concepts through more complex relationships.
In theprompt “Self”, the resulting drawing features a body with multiple heads, evoking e.g. the ideathat a person’s self may contain multiple outward personalities.
When asked “What do you looklike, CLIPDraw?”, the synthesized drawing contains a smiling face followed by text resembling“CLIPDRAW”.
Finally, “The space between infinity” presents a dream-like landscape with an infinity symbol under a galaxy-filled sky.
4.5 Can synthesized drawings be fine-tuned via additional negative prompts?
A common pain point in AI-assisted image synthesis is that is hard to control what the AI will produce.
One potential tool for increased control in CLIP-based methods is to introduce negative prompts; theoptimization objective is to minimize cosine distance between the CLIP-encoded drawing and thedescription prompt, while maximizing distance between the drawing and a set of negative prompts.
Figure 9 shows pairs of drawings synthesized from the same random initialization, with the bottomrow utilizing additional negative fine-tuning prompts, weighted .3:1.
“Hashtag” contains manyinstances of the word “hashtag” written out.
By penalizing “Words and text”, the bottom examplecontains fewer words.
By penalizing “Purple” and “Red”, the main color of “A torii gate” switches togreen.
Lastly, the original drawing for “a realistic painting of a sailboat” features many sailboats onan ocean, and penalizing the phrase “many sailboats” results in only one sailboat.
While negative prompts present a richly semantic way to fine-tune image synthesis in CLIP-basedmethods, it remains tricky to locate prompts that consistently encourage the intended behavior.
Manytimes, negative prompts show negligible effects.
During experiments, a cure-all negative prompt suchas “a low-quality drawing”, with the goal of consistently improving drawing quality, was unable tobe found.
Further work remains on how to best influence CLIP-based synthesis-through-optimizationmethods through additional objectives, whether negative or positive.
Discussion
This work presents CLIPDraw, a text-to-drawing synthesis method based on the CLIP language-imageencoder.
CLIPDraw does not require any model training; rather, drawings are synthesized throughiterative optimization during evaluation time.
CLIPDraw is not the first method to use evaluation-timeoptimization for image synthesis; many recent works have also used CLIP as an objective.
However,by constraining synthesis to images made of RGBA Bézier curves, CLIPDraw biases towards sketchesof human-recognizable concepts.
At a higher level, CLIPDraw is work that takes advantage of an existing large pre-trained foundationmodel (Bommasani et al., 2021), and derives benefit from the representations provided.
As foundationmodels are often viewed as black-box models, it is important to study their use and affect ondownstream tasks.
Thus, the focus of this paper is to examine the nuances of CLIPDraw behavior,and experiments focus on specific questions and observations about synthesized drawings.
Theseobservations, such as prompt ambiguity or the compositionality of styles and content, are qualitatively distinct and interesting and likely apply to further CLIP-based work in the future, thus this paperaims to start an investigation that continues forward.
5.1 Limitations
Synthesizing high-resolution images is a challenge, and CLIPDraw will often fall short of methodsthat incorporate a high-functioning generative model.
This problem is related to a classic pitfall insynthesis-through-optimization methods, which is that an image may very closely match the CLIPobjective, while looking messy and ugly to a human.
Thus it is important to introduce auxiliaryobjectives or constraints.
In the case of CLIPDraw, drawings are constrained by the Bézier curverepresentation, however stricter constraints such as fooling a GAN discriminator may improve quality.
However, this work shows that simple image augmentation is enough to unlock the robustness of
CLIP representation to generate recognizable images.
A second limitation has to do with using CLIP encodings as a synthesis objective.
While CLIPprovides a rich textual representation for describing an image, in comparison to coarser neuronactivation objectives, it still remains a challenge to specify details.
For example, it is hard totell CLIPDraw to move a sailboat to the other side of the image.
Preliminary experiments alsoexplored negative prompts as a possible direction towards more fine-grained adjustments, however aconsistently satisfying method was hard to locate.
A promising path in future research can lie in howto correctly steer synthesized images, or introduce finer detail into description prompts via additionalobjectives.
5.2 Ethics and Social Biases
An important concept to keep in mind when dealing with human data is the existence of inherentsocial biases contained within.
The pre-trained CLIP model is trained on a large corpus of onlinedata, so its representations may include connections or biases that are undesirable.
As CLIPDrawdoes not learn a new model, but instead optimizes based on CLIP itself, the bias studies presented inthe CLIP paper (Radford et al., 2021) are highly relevant for CLIPDraw as well.
In Section 4.4, a usecase for CLIPDraw is mentioned as a tool for exploring visual connections in human culture.
It iscrucial to recognize that symbols or connections that are formed by CLIPDraw are not necessarilyreflective of human culture, but rather are artifacts of the data used to train the original CLIP model.
Thus, while CLIPDraw can be used to synthesize drawings that utilize cultural connections to evokeemotions or abstract concepts, it remains the duty of the user to ensure that the final product is up todesired standards.
5.3 Conclusion
Overall, the aim of this work is to introduce CLIPDraw as an easily accessible starting pointto experiment with natural language image synthesis.
Due to its focus on drawings rather thanphotorealistic rendering, CLIPDraw presents a straightforward method to examine language-imagerelationships without the overhead of realism.
The presented CLIPDraw implementation can generallysynthesize images within a minute on a typical Colab GPU.
This paper describes various interesting behaviors of CLIPDraw, however it is not exhaustive.
CLIPbased text-to-image synthesis remains a field with many promising directions, and we hope others willcontinue to use this work for additional research into the nuances of synthesis-through-optimization,or as a practical tool for AI-assisted art and other interactive visual applications.
To this end, sourcecode is available at: https://bit.ly/3qWTii2 .
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or[N/A] .
You are strongly encouraged to include a justification to your answer either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are
proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A Appendix
CLIPDraw is a deterministic algorithm, with the exception of the random initialization used to seed the Bezier curve positions.
Importantly, the random initialization of these curves play a large part in the positional composition of the final image.
For example, in "a painting of a sunset", the position of the sun can appear in many different locations of the image.
At the same time, the high-level content of the image depends greatly on the prompt, and the random initialization does not play a large factor.
In "a painting of a sunset", while the sun may move around, it is always rendered as a yellow circle with an orange border, surrounded by waves.
In "A 3D rendering of a temple", the architecture of the temple adjusts from seed to seed, yet the temple always features a prominent 45-degree angled tower and surrounding trees.