Hugging Face's logo

PEFT documentation

LoKr

You are viewing v0.7.1 version. A newer version v0.11.0 is available.
Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

LoKr

Low-Rank Kronecker Product (LoKr), is a LoRA-variant method that approximates the large weight matrix with two low-rank matrices and combines them with the Kronecker product. LoKr also provides an optional third low-rank matrix to provide better control during fine-tuning.

LoKrConfig

class peft.LoKrConfig

< >

( peft_type: typing.Union[str, peft.utils.peft_types.PeftType, NoneType] = None auto_mapping: typing.Optional[dict] = None base_model_name_or_path: typing.Optional[str] = None revision: typing.Optional[str] = None task_type: typing.Union[str, peft.utils.peft_types.TaskType, NoneType] = None inference_mode: bool = False rank_pattern: Optional[dict] = <factory> alpha_pattern: Optional[dict] = <factory> r: int = 8 alpha: int = 8 rank_dropout: float = 0.0 module_dropout: float = 0.0 use_effective_conv2d: bool = False decompose_both: bool = False decompose_factor: int = -1 target_modules: typing.Union[typing.List[str], str, NoneType] = None init_weights: bool = True layers_to_transform: typing.Union[typing.List[int], int, NoneType] = None layers_pattern: typing.Optional[str] = None modules_to_save: typing.Optional[typing.List[str]] = None )

Parameters

  • r (int) — LoKr rank.
  • alpha (int) — The alpha parameter for LoKr scaling.
  • rank_dropout (int) — The dropout probability for rank dimension during training.
  • module_dropout (int) — The dropout probability for disabling LoKr modules during training.
  • use_effective_conv2d (bool) — Use parameter effective decomposition for Conv2d with ksize > 1 (“Proposition 3” from FedPara paper).
  • decompose_both (bool) — Perform rank decomposition of left kronecker product matrix.
  • decompose_factor (int) — Kronecker product decomposition factor.
  • target_modules (Union[List[str],str]) — The names of the modules to apply LoKr to.
  • init_weights (bool) — Whether to perform initialization of LoKr weights.
  • layers_to_transform (Union[List[int],int]) — The layer indexes to transform, if this argument is specified, it will apply the LoKr transformations on the layer indexes that are specified in this list. If a single integer is passed, it will apply the LoKr transformations on the layer at this index.
  • layers_pattern (str) — The layer pattern name, used only if layers_to_transform is different from None and if the layer pattern is not in the common layers pattern.
  • rank_pattern (dict) — The mapping from layer names or regexp expression to ranks which are different from the default rank specified by r.
  • alpha_pattern (dict) — The mapping from layer names or regexp expression to alphas which are different from the default alpha specified by alpha.
  • modules_to_save (List[str]) — The names of modules to be set as trainable except LoKr parameters.

Configuration class of LoKrModel.

LoKrModel

class peft.LoKrModel

< >

( model config adapter_name ) → torch.nn.Module

Parameters

  • model (torch.nn.Module) — The model to which the adapter tuner layers will be attached.
  • config (LoKrConfig) — The configuration of the LoKr model.
  • adapter_name (str) — The name of the adapter, defaults to "default".

Returns

torch.nn.Module

The LoKr model.

Creates Low-Rank Kronecker Product model from a pretrained model. The original method is partially described in https://arxiv.org/abs/2108.06098 and in https://arxiv.org/abs/2309.14859 Current implementation heavily borrows from https://github.com/KohakuBlueleaf/LyCORIS/blob/eb460098187f752a5d66406d3affade6f0a07ece/lycoris/modules/lokr.py

Example:

>>> from diffusers import StableDiffusionPipeline
>>> from peft import LoKrModel, LoKrConfig

>>> config_te = LoKrConfig(
...     r=8,
...     lora_alpha=32,
...     target_modules=["k_proj", "q_proj", "v_proj", "out_proj", "fc1", "fc2"],
...     rank_dropout=0.0,
...     module_dropout=0.0,
...     init_weights=True,
... )
>>> config_unet = LoKrConfig(
...     r=8,
...     lora_alpha=32,
...     target_modules=[
...         "proj_in",
...         "proj_out",
...         "to_k",
...         "to_q",
...         "to_v",
...         "to_out.0",
...         "ff.net.0.proj",
...         "ff.net.2",
...     ],
...     rank_dropout=0.0,
...     module_dropout=0.0,
...     init_weights=True,
...     use_effective_conv2d=True,
... )

>>> model = StableDiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5")
>>> model.text_encoder = LoKrModel(model.text_encoder, config_te, "default")
>>> model.unet = LoKrModel(model.unet, config_unet, "default")

Attributes:

  • model (~torch.nn.Module) — The model to be adapted.
  • peft_config (LoKrConfig): The configuration of the LoKr model.

←LoHa LoRA→