Diffusers documentation

IP-Adapter

Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

IP-Adapter

IP-Adapter is a lightweight adapter that enables prompting a diffusion model with an image. This method decouples the cross-attention layers of the image and text features. The image features are generated from an image encoder. Files generated from IP-Adapter are only ~100MBs.

Learn how to load an IP-Adapter checkpoint and image in the IP-Adapter loading guide.

IPAdapterMixin

class diffusers.loaders.IPAdapterMixin

< >

( )

Mixin for handling IP Adapters.

load_ip_adapter

< >

( pretrained_model_name_or_path_or_dict: Union subfolder: Union weight_name: Union **kwargs )

Parameters

  • pretrained_model_name_or_path_or_dict (str or os.PathLike or dict) — Can be either:

    • A string, the model id (for example google/ddpm-celebahq-256) of a pretrained model hosted on the Hub.
    • A path to a directory (for example ./my_model_directory) containing the model weights saved with ModelMixin.save_pretrained().
    • A torch state dict.
  • cache_dir (Union[str, os.PathLike], optional) — Path to a directory where a downloaded pretrained model configuration is cached if the standard cache is not used.
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to resume downloading the model weights and configuration files. If set to False, any incompletely downloaded files are deleted.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, for example, {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • local_files_only (bool, optional, defaults to False) — Whether to only load local model weights and configuration files or not. If set to True, the model won’t be downloaded from the Hub.
  • token (str or bool, optional) — The token to use as HTTP bearer authorization for remote files. If True, the token generated from diffusers-cli login (stored in ~/.huggingface) is used.
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, a commit id, or any identifier allowed by Git.
  • subfolder (str, optional, defaults to "") — The subfolder location of a model file within a larger model repository on the Hub or locally.

unload_ip_adapter

< >

( )

Unloads the IP Adapter weights

Examples:

>>> # Assuming `pipeline` is already loaded with the IP Adapter weights.
>>> pipeline.unload_ip_adapter()
>>> ...