Parameters

• n_frequency (int) — Num of learnable frequencies for the Discrete Fourier Transform. ‘n_frequency’ is an integer that is greater than 0 and less than or equal to d^2 (assuming the weight W has dimensions of d by d). Additionally, it is the number of trainable parameters required to update each delta W weight. ‘n_frequency’ will affect the performance and efficiency for PEFT. Specifically, it has little impact on training speed, but higher values of it (typically) result in larger GPU memory costs and better accuracy. With the same target_modules, the number of parameters of LoRA is (2dr/n_frequency) times that of FourierFT. The following examples of settings regarding ‘n_frequency’ can be used as reference for users. For NLU tasks with the RoBERTa-large model, adopting ‘n_frequency’: 1000 can almost achieve similar results as ‘r’: 8 in LoRA. At this time, the number of parameters of LoRA is about 16 times that of FourierFT. For image classification tasks with Vit-large models, adopting ‘n_frequency’: 3000 can almost achieve similar results as ‘r’: 16 in LoRA, where the number of parameters of LoRA is about 11 times that of FourierFT.
• scaling (float) — The scaling value for the delta W matrix. This is an important hyperparameter used for scaling, similar to the ‘lora_alpha’ parameter in the LoRA method. ‘scaling’ can be determined during the hyperparameter search process. However, if users want to skip this process, one can refer to the settings in the following scenarios. This parameter can be set to 100.0 or 150.0 for both RoBERTa-base and RoBERTa-large models across all NLU (GLUE) tasks. This parameter can be set to 300.0 for both LLaMA family models for all instruction tuning. This parameter can be set to 300.0 for both ViT-base and ViT-large models across all image classification tasks.
• random_loc_seed (int) — Seed for the random location of the frequencies, i.e., the spectral entry matrix.
• target_modules (Union[list[str],str]) — List of module names or regex expression of the module names to replace with FourierFT. For example, [‘q’, ‘v’] or ‘.decoder.(SelfAttention|EncDecAttention).*(q|v)$‘. Only linear layers are supported.
• fan_in_fan_out (bool) — Set this to True if the layer to replace stores weight like (fan_in, fan_out).
• bias (str) — Bias type for FourierFT. Can be ‘none’, ‘all’ or ‘fourier_only’.
• modules_to_save (list[str]) — List of modules apart from FourierFT layers to be set as trainable and saved in the final checkpoint. For example, in Sequence Classification or Token Classification tasks, the final layer classifier/score are randomly initialized and as such need to be trainable and saved.
• layers_to_transform (Union[list[int],int]) — The layer indexes to transform, is this argument is specified, PEFT will transform only the layers indexes that are specified inside this list. If a single integer is passed, PEFT will transform only the layer at this index.
• layers_pattern (str) — The layer pattern name, used only if layers_to_transform is different to None and if the layer pattern is not in the common layers pattern.
• n_frequency_pattern (dict) — The mapping from layer names or regexp expression to n_frequency which are different from the default specified. For example, {model.decoder.layers.0.encoder_attn.k_proj: 1000}.
• init_weights (bool) — The initialization of the Fourier weights. Set this to False if the spectrum are initialized to a standard normal distribution. Set this to True if the spectrum are initialized to zeros.

Parameters

• model (torch.nn.Module) — The model to be adapted.
• config (FourierFTConfig) — The configuration of the FourierFT model.
• adapter_name (str) — The name of the adapter, defaults to "default".

The FourierFT model.