Hugging Face's logo

Diffusers documentation

VAE Image Processor

You are viewing v0.24.0 version. A newer version v0.31.0 is available.
Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

VAE Image Processor

The VaeImageProcessor provides a unified API for StableDiffusionPipelines to prepare image inputs for VAE encoding and post-processing outputs once they’re decoded. This includes transformations such as resizing, normalization, and conversion between PIL Image, PyTorch, and NumPy arrays.

All pipelines with VaeImageProcessor accept PIL Image, PyTorch tensor, or NumPy arrays as image inputs and return outputs based on the output_type argument by the user. You can pass encoded image latents directly to the pipeline and return latents from the pipeline as a specific output with the output_type argument (for example output_type="latent"). This allows you to take the generated latents from one pipeline and pass it to another pipeline as input without leaving the latent space. It also makes it much easier to use multiple pipelines together by passing PyTorch tensors directly between different pipelines.

VaeImageProcessor

class diffusers.image_processor.VaeImageProcessor

< >

( do_resize: bool = True vae_scale_factor: int = 8 resample: str = 'lanczos' do_normalize: bool = True do_binarize: bool = False do_convert_rgb: bool = False do_convert_grayscale: bool = False )

Parameters

  • do_resize (bool, optional, defaults to True) — Whether to downscale the image’s (height, width) dimensions to multiples of vae_scale_factor. Can accept height and width arguments from image_processor.VaeImageProcessor.preprocess() method.
  • vae_scale_factor (int, optional, defaults to 8) — VAE scale factor. If do_resize is True, the image is automatically resized to multiples of this factor.
  • resample (str, optional, defaults to lanczos) — Resampling filter to use when resizing the image.
  • do_normalize (bool, optional, defaults to True) — Whether to normalize the image to [-1,1].
  • do_binarize (bool, optional, defaults to False) — Whether to binarize the image to 0/1.
  • do_convert_rgb (bool, optional, defaults to be False) — Whether to convert the images to RGB format.
  • do_convert_grayscale (bool, optional, defaults to be False) — Whether to convert the images to grayscale format.

Image processor for VAE.

binarize

< >

( image: Image ) → PIL.Image.Image

Parameters

  • image (PIL.Image.Image) — The image input, should be a PIL image.

Returns

PIL.Image.Image

The binarized image. Values less than 0.5 are set to 0, values greater than 0.5 are set to 1.

Create a mask.

convert_to_grayscale

< >

( image: Image )

Converts a PIL image to grayscale format.

convert_to_rgb

< >

( image: Image )

Converts a PIL image to RGB format.

denormalize

< >

( images: typing.Union[numpy.ndarray, torch.Tensor] )

Denormalize an image array to [0,1].

get_default_height_width

< >

( image: typing.Union[PIL.Image.Image, numpy.ndarray, torch.Tensor] height: typing.Optional[int] = None width: typing.Optional[int] = None )

Parameters

  • image(PIL.Image.Image, np.ndarray or torch.Tensor) — The image input, can be a PIL image, numpy array or pytorch tensor. if it is a numpy array, should have shape [batch, height, width] or [batch, height, width, channel] if it is a pytorch tensor, should have shape [batch, channel, height, width].
  • height (int, optional, defaults to None) — The height in preprocessed image. If None, will use the height of image input.
  • width (int, optional, defaults to None) -- The width in preprocessed. If None, will use the width of the image` input.

This function return the height and width that are downscaled to the next integer multiple of vae_scale_factor.

normalize

< >

( images: typing.Union[numpy.ndarray, torch.Tensor] )

Normalize an image array to [-1,1].

numpy_to_pil

< >

( images: ndarray )

Convert a numpy image or a batch of images to a PIL image.

numpy_to_pt

< >

( images: ndarray )

Convert a NumPy image to a PyTorch tensor.

pil_to_numpy

< >

( images: typing.Union[typing.List[PIL.Image.Image], PIL.Image.Image] )

Convert a PIL image or a list of PIL images to NumPy arrays.

postprocess

< >

( image: FloatTensor output_type: str = 'pil' do_denormalize: typing.Optional[typing.List[bool]] = None ) → PIL.Image.Image, np.ndarray or torch.FloatTensor

Parameters

  • image (torch.FloatTensor) — The image input, should be a pytorch tensor with shape B x C x H x W.
  • output_type (str, optional, defaults to pil) — The output type of the image, can be one of pil, np, pt, latent.
  • do_denormalize (List[bool], optional, defaults to None) — Whether to denormalize the image to [0,1]. If None, will use the value of do_normalize in the VaeImageProcessor config.

Returns

PIL.Image.Image, np.ndarray or torch.FloatTensor

The postprocessed image.

Postprocess the image output from tensor to output_type.

preprocess

< >

( image: typing.Union[torch.FloatTensor, PIL.Image.Image, numpy.ndarray] height: typing.Optional[int] = None width: typing.Optional[int] = None )

Preprocess the image input. Accepted formats are PIL images, NumPy arrays or PyTorch tensors.

pt_to_numpy

< >

( images: FloatTensor )

Convert a PyTorch tensor to a NumPy image.

resize

< >

( image: typing.Union[PIL.Image.Image, numpy.ndarray, torch.Tensor] height: typing.Optional[int] = None width: typing.Optional[int] = None ) → PIL.Image.Image, np.ndarray or torch.Tensor

Parameters

  • image (PIL.Image.Image, np.ndarray or torch.Tensor) — The image input, can be a PIL image, numpy array or pytorch tensor.
  • height (int, optional, defaults to None) — The height to resize to.
  • width (int, optional, defaults to None`) — The width to resize to.

Returns

PIL.Image.Image, np.ndarray or torch.Tensor

The resized image.

Resize image.

VaeImageProcessorLDM3D

The VaeImageProcessorLDM3D accepts RGB and depth inputs and returns RGB and depth outputs.

class diffusers.image_processor.VaeImageProcessorLDM3D

< >

( do_resize: bool = True vae_scale_factor: int = 8 resample: str = 'lanczos' do_normalize: bool = True )

Parameters

  • do_resize (bool, optional, defaults to True) — Whether to downscale the image’s (height, width) dimensions to multiples of vae_scale_factor.
  • vae_scale_factor (int, optional, defaults to 8) — VAE scale factor. If do_resize is True, the image is automatically resized to multiples of this factor.
  • resample (str, optional, defaults to lanczos) — Resampling filter to use when resizing the image.
  • do_normalize (bool, optional, defaults to True) — Whether to normalize the image to [-1,1].

Image processor for VAE LDM3D.

depth_pil_to_numpy

< >

( images: typing.Union[typing.List[PIL.Image.Image], PIL.Image.Image] )

Convert a PIL image or a list of PIL images to NumPy arrays.

numpy_to_depth

< >

( images: ndarray )

Convert a NumPy depth image or a batch of images to a PIL image.

numpy_to_pil

< >

( images: ndarray )

Convert a NumPy image or a batch of images to a PIL image.

rgblike_to_depthmap

< >

( image: typing.Union[numpy.ndarray, torch.Tensor] )

Returns: depth map

←Utilities