Hugging Face's logo

PEFT documentation

LoKr

You are viewing v0.9.0 version. A newer version v0.13.0 is available.
Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

LoKr

Low-Rank Kronecker Product (LoKr), is a LoRA-variant method that approximates the large weight matrix with two low-rank matrices and combines them with the Kronecker product. LoKr also provides an optional third low-rank matrix to provide better control during fine-tuning.

LoKrConfig

class peft.LoKrConfig

< >

( peft_type: Union = None auto_mapping: Optional = None base_model_name_or_path: Optional = None revision: Optional = None task_type: Union = None inference_mode: bool = False rank_pattern: Optional[dict] = <factory> alpha_pattern: Optional[dict] = <factory> r: int = 8 alpha: int = 8 rank_dropout: float = 0.0 module_dropout: float = 0.0 use_effective_conv2d: bool = False decompose_both: bool = False decompose_factor: int = -1 target_modules: Union = None init_weights: bool = True layers_to_transform: Union = None layers_pattern: Optional = None modules_to_save: Optional = None )

Parameters

  • r (int) — LoKr rank.
  • alpha (int) — The alpha parameter for LoKr scaling.
  • rank_dropout (float) — The dropout probability for rank dimension during training.
  • module_dropout (float) — The dropout probability for disabling LoKr modules during training.
  • use_effective_conv2d (bool) — Use parameter effective decomposition for Conv2d with ksize > 1 (“Proposition 3” from FedPara paper).
  • decompose_both (bool) — Perform rank decomposition of left kronecker product matrix.
  • decompose_factor (int) — Kronecker product decomposition factor.
  • target_modules (Optional[Union[List[str], str]]) — The names of the modules to apply the adapter to. If this is specified, only the modules with the specified names will be replaced. When passing a string, a regex match will be performed. When passing a list of strings, either an exact match will be performed or it is checked if the name of the module ends with any of the passed strings. If this is specified as ‘all-linear’, then all linear/Conv1D modules are chosen, excluding the output layer. If this is not specified, modules will be chosen according to the model architecture. If the architecture is not known, an error will be raised — in this case, you should specify the target modules manually.
  • init_weights (bool) — Whether to perform initialization of adapter weights. This defaults to True, passing False is discouraged.
  • layers_to_transform (Union[List[int], int]) — The layer indices to transform. If a list of ints is passed, it will apply the adapter to the layer indices that are specified in this list. If a single integer is passed, it will apply the transformations on the layer at this index.
  • layers_pattern (str) — The layer pattern name, used only if layers_to_transform is different from None.
  • rank_pattern (dict) — The mapping from layer names or regexp expression to ranks which are different from the default rank specified by r.
  • alpha_pattern (dict) — The mapping from layer names or regexp expression to alphas which are different from the default alpha specified by alpha.
  • modules_to_save (Optional[List[str]]) — List of modules apart from adapter layers to be set as trainable and saved in the final checkpoint.

Configuration class of LoKrModel.

LoKrModel

class peft.LoKrModel

< >

( model config adapter_name ) → torch.nn.Module

Parameters

  • model (torch.nn.Module) — The model to which the adapter tuner layers will be attached.
  • config (LoKrConfig) — The configuration of the LoKr model.
  • adapter_name (str) — The name of the adapter, defaults to "default".

Returns

torch.nn.Module

The LoKr model.

Creates Low-Rank Kronecker Product model from a pretrained model. The original method is partially described in https://arxiv.org/abs/2108.06098 and in https://arxiv.org/abs/2309.14859 Current implementation heavily borrows from https://github.com/KohakuBlueleaf/LyCORIS/blob/eb460098187f752a5d66406d3affade6f0a07ece/lycoris/modules/lokr.py

Example:

>>> from diffusers import StableDiffusionPipeline
>>> from peft import LoKrModel, LoKrConfig

>>> config_te = LoKrConfig(
...     r=8,
...     lora_alpha=32,
...     target_modules=["k_proj", "q_proj", "v_proj", "out_proj", "fc1", "fc2"],
...     rank_dropout=0.0,
...     module_dropout=0.0,
...     init_weights=True,
... )
>>> config_unet = LoKrConfig(
...     r=8,
...     lora_alpha=32,
...     target_modules=[
...         "proj_in",
...         "proj_out",
...         "to_k",
...         "to_q",
...         "to_v",
...         "to_out.0",
...         "ff.net.0.proj",
...         "ff.net.2",
...     ],
...     rank_dropout=0.0,
...     module_dropout=0.0,
...     init_weights=True,
...     use_effective_conv2d=True,
... )

>>> model = StableDiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5")
>>> model.text_encoder = LoKrModel(model.text_encoder, config_te, "default")
>>> model.unet = LoKrModel(model.unet, config_unet, "default")

Attributes:

  • model (~torch.nn.Module) — The model to be adapted.
  • peft_config (LoKrConfig): The configuration of the LoKr model.

←LoHa LoRA→