UnCLIP
Hierarchical Text-Conditional Image Generation with CLIP Latents is by Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, Mark Chen. The UnCLIP model in π€ Diffusers comes from kakaobrainβs karlo.
The abstract from the paper is following:
Contrastive models like CLIP have been shown to learn robust representations of images that capture both semantics and style. To leverage these representations for image generation, we propose a two-stage model: a prior that generates a CLIP image embedding given a text caption, and a decoder that generates an image conditioned on the image embedding. We show that explicitly generating image representations improves image diversity with minimal loss in photorealism and caption similarity. Our decoders conditioned on image representations can also produce variations of an image that preserve both its semantics and style, while varying the non-essential details absent from the image representation. Moreover, the joint embedding space of CLIP enables language-guided image manipulations in a zero-shot fashion. We use diffusion models for the decoder and experiment with both autoregressive and diffusion models for the prior, finding that the latter are computationally more efficient and produce higher-quality samples.
You can find lucidrains DALL-E 2 recreation at lucidrains/DALLE2-pytorch.
Make sure to check out the Schedulers guide to learn how to explore the tradeoff between scheduler speed and quality, and see the reuse components across pipelines section to learn how to efficiently load the same components into multiple pipelines.
UnCLIPPipeline
class diffusers.UnCLIPPipeline
< source >( prior: PriorTransformer decoder: UNet2DConditionModel text_encoder: CLIPTextModelWithProjection tokenizer: CLIPTokenizer text_proj: UnCLIPTextProjModel super_res_first: UNet2DModel super_res_last: UNet2DModel prior_scheduler: UnCLIPScheduler decoder_scheduler: UnCLIPScheduler super_res_scheduler: UnCLIPScheduler )
Parameters
-
text_encoder (
CLIPTextModelWithProjection
) — Frozen text-encoder. -
tokenizer (
CLIPTokenizer
) — ACLIPTokenizer
to tokenize text. - prior (PriorTransformer) — The canonical unCLIP prior to approximate the image embedding from the text embedding.
-
text_proj (
UnCLIPTextProjModel
) — Utility class to prepare and combine the embeddings before they are passed to the decoder. - decoder (UNet2DConditionModel) — The decoder to invert the image embedding into an image.
- super_res_first (UNet2DModel) — Super resolution UNet. Used in all but the last step of the super resolution diffusion process.
- super_res_last (UNet2DModel) — Super resolution UNet. Used in the last step of the super resolution diffusion process.
-
prior_scheduler (
UnCLIPScheduler
) — Scheduler used in the prior denoising process (a modified DDPMScheduler). -
decoder_scheduler (
UnCLIPScheduler
) — Scheduler used in the decoder denoising process (a modified DDPMScheduler). -
super_res_scheduler (
UnCLIPScheduler
) — Scheduler used in the super resolution denoising process (a modified DDPMScheduler).
Pipeline for text-to-image generation using unCLIP.
This model inherits from DiffusionPipeline. Check the superclass documentation for the generic methods implemented for all pipelines (downloading, saving, running on a particular device, etc.).
__call__
< source >(
prompt: typing.Union[str, typing.List[str], NoneType] = None
num_images_per_prompt: int = 1
prior_num_inference_steps: int = 25
decoder_num_inference_steps: int = 25
super_res_num_inference_steps: int = 7
generator: typing.Union[torch._C.Generator, typing.List[torch._C.Generator], NoneType] = None
prior_latents: typing.Optional[torch.FloatTensor] = None
decoder_latents: typing.Optional[torch.FloatTensor] = None
super_res_latents: typing.Optional[torch.FloatTensor] = None
text_model_output: typing.Union[transformers.models.clip.modeling_clip.CLIPTextModelOutput, typing.Tuple, NoneType] = None
text_attention_mask: typing.Optional[torch.Tensor] = None
prior_guidance_scale: float = 4.0
decoder_guidance_scale: float = 8.0
output_type: typing.Optional[str] = 'pil'
return_dict: bool = True
)
β
ImagePipelineOutput or tuple
Parameters
-
prompt (
str
orList[str]
) — The prompt or prompts to guide image generation. This can only be left undefined iftext_model_output
andtext_attention_mask
is passed. -
num_images_per_prompt (
int
, optional, defaults to 1) — The number of images to generate per prompt. -
prior_num_inference_steps (
int
, optional, defaults to 25) — The number of denoising steps for the prior. More denoising steps usually lead to a higher quality image at the expense of slower inference. -
decoder_num_inference_steps (
int
, optional, defaults to 25) — The number of denoising steps for the decoder. More denoising steps usually lead to a higher quality image at the expense of slower inference. -
super_res_num_inference_steps (
int
, optional, defaults to 7) — The number of denoising steps for super resolution. More denoising steps usually lead to a higher quality image at the expense of slower inference. -
generator (
torch.Generator
orList[torch.Generator]
, optional) — Atorch.Generator
to make generation deterministic. -
prior_latents (
torch.FloatTensor
of shape (batch size, embeddings dimension), optional) — Pre-generated noisy latents to be used as inputs for the prior. -
decoder_latents (
torch.FloatTensor
of shape (batch size, channels, height, width), optional) — Pre-generated noisy latents to be used as inputs for the decoder. -
super_res_latents (
torch.FloatTensor
of shape (batch size, channels, super res height, super res width), optional) — Pre-generated noisy latents to be used as inputs for the decoder. -
prior_guidance_scale (
float
, optional, defaults to 4.0) — A higher guidance scale value encourages the model to generate images closely linked to the textprompt
at the expense of lower image quality. Guidance scale is enabled whenguidance_scale > 1
. -
decoder_guidance_scale (
float
, optional, defaults to 4.0) — A higher guidance scale value encourages the model to generate images closely linked to the textprompt
at the expense of lower image quality. Guidance scale is enabled whenguidance_scale > 1
. -
text_model_output (
CLIPTextModelOutput
, optional) — Pre-definedCLIPTextModel
outputs that can be derived from the text encoder. Pre-defined text outputs can be passed for tasks like text embedding interpolations. Make sure to also passtext_attention_mask
in this case.prompt
can the be leftNone
. -
text_attention_mask (
torch.Tensor
, optional) — Pre-defined CLIP text attention mask that can be derived from the tokenizer. Pre-defined text attention masks are necessary when passingtext_model_output
. -
output_type (
str
, optional, defaults to"pil"
) — The output format of the generated image. Choose betweenPIL.Image
ornp.array
. -
return_dict (
bool
, optional, defaults toTrue
) — Whether or not to return a ImagePipelineOutput instead of a plain tuple.
Returns
ImagePipelineOutput or tuple
If return_dict
is True
, ImagePipelineOutput is returned, otherwise a tuple
is
returned where the first element is a list with the generated images.
The call function to the pipeline for generation.
UnCLIPImageVariationPipeline
class diffusers.UnCLIPImageVariationPipeline
< source >( decoder: UNet2DConditionModel text_encoder: CLIPTextModelWithProjection tokenizer: CLIPTokenizer text_proj: UnCLIPTextProjModel feature_extractor: CLIPImageProcessor image_encoder: CLIPVisionModelWithProjection super_res_first: UNet2DModel super_res_last: UNet2DModel decoder_scheduler: UnCLIPScheduler super_res_scheduler: UnCLIPScheduler )
Parameters
-
text_encoder (
CLIPTextModelWithProjection
) — Frozen text-encoder. -
tokenizer (
CLIPTokenizer
) — ACLIPTokenizer
to tokenize text. -
feature_extractor (
CLIPImageProcessor
) — Model that extracts features from generated images to be used as inputs for theimage_encoder
. -
image_encoder (
CLIPVisionModelWithProjection
) — Frozen CLIP image-encoder (clip-vit-large-patch14). -
text_proj (
UnCLIPTextProjModel
) — Utility class to prepare and combine the embeddings before they are passed to the decoder. - decoder (UNet2DConditionModel) — The decoder to invert the image embedding into an image.
- super_res_first (UNet2DModel) — Super resolution UNet. Used in all but the last step of the super resolution diffusion process.
- super_res_last (UNet2DModel) — Super resolution UNet. Used in the last step of the super resolution diffusion process.
-
decoder_scheduler (
UnCLIPScheduler
) — Scheduler used in the decoder denoising process (a modified DDPMScheduler). -
super_res_scheduler (
UnCLIPScheduler
) — Scheduler used in the super resolution denoising process (a modified DDPMScheduler).
Pipeline to generate image variations from an input image using UnCLIP.
This model inherits from DiffusionPipeline. Check the superclass documentation for the generic methods implemented for all pipelines (downloading, saving, running on a particular device, etc.).
__call__
< source >(
image: typing.Union[PIL.Image.Image, typing.List[PIL.Image.Image], torch.FloatTensor, NoneType] = None
num_images_per_prompt: int = 1
decoder_num_inference_steps: int = 25
super_res_num_inference_steps: int = 7
generator: typing.Optional[torch._C.Generator] = None
decoder_latents: typing.Optional[torch.FloatTensor] = None
super_res_latents: typing.Optional[torch.FloatTensor] = None
image_embeddings: typing.Optional[torch.Tensor] = None
decoder_guidance_scale: float = 8.0
output_type: typing.Optional[str] = 'pil'
return_dict: bool = True
)
β
ImagePipelineOutput or tuple
Parameters
-
image (
PIL.Image.Image
orList[PIL.Image.Image]
ortorch.FloatTensor
) —Image
or tensor representing an image batch to be used as the starting point. If you provide a tensor, it needs to be compatible with theCLIPImageProcessor
configuration. Can be left asNone
only whenimage_embeddings
are passed. -
num_images_per_prompt (
int
, optional, defaults to 1) — The number of images to generate per prompt. -
decoder_num_inference_steps (
int
, optional, defaults to 25) — The number of denoising steps for the decoder. More denoising steps usually lead to a higher quality image at the expense of slower inference. -
super_res_num_inference_steps (
int
, optional, defaults to 7) — The number of denoising steps for super resolution. More denoising steps usually lead to a higher quality image at the expense of slower inference. -
generator (
torch.Generator
, optional) — Atorch.Generator
to make generation deterministic. -
decoder_latents (
torch.FloatTensor
of shape (batch size, channels, height, width), optional) — Pre-generated noisy latents to be used as inputs for the decoder. -
super_res_latents (
torch.FloatTensor
of shape (batch size, channels, super res height, super res width), optional) — Pre-generated noisy latents to be used as inputs for the decoder. -
decoder_guidance_scale (
float
, optional, defaults to 4.0) — A higher guidance scale value encourages the model to generate images closely linked to the textprompt
at the expense of lower image quality. Guidance scale is enabled whenguidance_scale > 1
. -
image_embeddings (
torch.Tensor
, optional) — Pre-defined image embeddings that can be derived from the image encoder. Pre-defined image embeddings can be passed for tasks like image interpolations.image
can be left asNone
. -
output_type (
str
, optional, defaults to"pil"
) — The output format of the generated image. Choose betweenPIL.Image
ornp.array
. -
return_dict (
bool
, optional, defaults toTrue
) — Whether or not to return a ImagePipelineOutput instead of a plain tuple.
Returns
ImagePipelineOutput or tuple
If return_dict
is True
, ImagePipelineOutput is returned, otherwise a tuple
is
returned where the first element is a list with the generated images.
The call function to the pipeline for generation.
ImagePipelineOutput
class diffusers.ImagePipelineOutput
< source >( images: typing.Union[typing.List[PIL.Image.Image], numpy.ndarray] )
Output class for image pipelines.