Hugging Face's logo

PEFT documentation

Hotswapping adapters

Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

Hotswapping adapters

The idea of hotswapping an adapter is the following: We can already load multiple adapters, e.g. two LoRAs, at the same time. But sometimes, we want to load one LoRA and then replace its weights in-place with the LoRA weights of another adapter. This is now possible the hotswap_adapter function.

In general, this should be faster than deleting one adapter and loading the adapter in its place, which would be the how to achieve the same final outcome without hotswapping. Another advantage of hotswapping is that it prevents re-compilation in case the PEFT model is already compiled using torch.compile. This can save quite a lot of time.

import torch
from transformers import AutoModelForCausalLM
from peft import PeftModel
from peft.utils.hotswap import hotswap_adapter

model_id = ...
inputs = ...
device = ...
model = AutoModelForCausalLM.from_pretrained(model_id).to(device)

# load lora 0
model = PeftModel.from_pretrained(model, <path-adapter-0>)
model = torch.compile(model)  # optionally compile the model
with torch.inference_mode():
    output_adapter_0 = model(inputs)

# replace the "default" lora adapter with the new one
hotswap_adapter(model, <path-adapter-1>, adapter_name="default", torch_device=device)
with torch.inference_mode():
    output_adapter_1 = model(inputs).logits

Hotswapping works with transformers models and diffusers models. However, there are some caveats:

  • It only works for the same PEFT method, so no swapping LoRA and LoHa, for example.
  • Right now, only LoRA is properly supported.
  • The adapters must be compatible (e.g. same LoRA alpha, same target modules).
  • If you use torch.compile and want to avoid recompilation, the LoRA rank must be the same.

peft.utils.hotswap.hotswap_adapter

< >

( model model_name_or_path adapter_name torch_device = None **kwargs )

Parameters

  • model (~PeftModel) — The PEFT model with the loaded adapter.
  • model_name_or_path (str) — The name or path of the model to load the new adapter from.
  • adapter_name (str) — The name of the adapter to swap, e.g. "default". The name will stay the same after swapping.
  • torch_device — (str, optional, defaults to None): The device to load the new adapter onto.
  • **kwargs (optional) — Additional keyword arguments used for loading the config and weights.

Substitute old adapter data with new adapter data, keeping the rest the same.

As of now, only LoRA is supported.

This function is useful when you want to replace the loaded adapter with a new adapter. The adapter name will remain the same, but the weights and other parameters will be swapped out.

If the adapters are incomptabile, e.g. targeting different layers or having different alpha values, an error will be raised.

Example:

>>> import torch
>>> from transformers import AutoModelForCausalLM
>>> from peft import PeftModel
>>> from peft.utils.hotswap import hotswap_adapter

>>> model_id = ...
>>> inputs = ...
>>> device = ...
>>> model = AutoModelForCausalLM.from_pretrained(model_id).to(device)

>>> # load lora 0
>>> model = PeftModel.from_pretrained(model, "path-adapter-0")
>>> model = torch.compile(model)  # optionally compile the model
>>> with torch.inference_mode():
...     output_adapter_0 = model(inputs)

>>> # replace the "default" lora adapter with the new one
>>> hotswap_adapter(model, "path-adapter-1", adapter_name="default", torch_device=device)
>>> with torch.inference_mode():
...     output_adapter_1 = model(inputs).logits

peft.utils.hotswap.hotswap_adapter_from_state_dict

< >

( model state_dict adapter_name parameter_prefix = 'lora_' )

Parameters

  • model (nn.Module) — The model with the loaded adapter.
  • state_dict (dict[str, torch.Tensor]) — The state dict of the new adapter, which needs to be compatible (targeting same modules etc.).
  • adapter_name (str) — The name of the adapter that should be hot-swapped, e.g. "default". The name will remain the same after swapping.
  • parameter_prefix (str, optional, defaults to "lora_") — The prefix used to identify the adapter’s keys in the state dict. For LoRA, this would be "lora_" (the default).

Raises

RuntimeError

  • RuntimeError — If the old and the new adapter are not compatible, a RuntimeError is raised.

Swap out the adapter weights from the model with the weights from state_dict.

As of now, only LoRA is supported.

This is a low-level function that assumes that the adapters have been checked for compatibility and that the state_dict has been correctly mapped to work with PEFT. For a high level function that performs this work for you, use hotswap_adapter instead.

< > Update on GitHub

←Helpers