Towards More Efficient Post-training via F ourier Domain Adapter Framework (2025)

Abstract: We introduce Fourier Domain Adapter (FDA), a novel and parameter-efficient framework for fine-tuning large-scale pre-trained language models. FDA reparameterizes the core projection operation of the adapter module directly in the Fourier domain. This involves transforming the input features via discrete Fourier transform (DFT), applying sparse learnable complex modulations in frequency space, and then back-transforming via inverse DFT, supplemented by highly compact auxiliary linear layers. This approach significantly reduces the number of trainable parameters while enhancing the model’s ability to capture salient frequency-based semantic information. Comprehensive experiments on GLUE, E2E NLG, and instruction tuning benchmarks show that our FDA consistently outperforms existing parameter-efficient fine-tuning (PEFT) methods. It can achieve better performance with nearly 100x fewer training parameters than traditional fine-tuning methods such as LoRA and AdapterH. Our results demonstrate that FDA is a robust and efficient solution for developing efficient and powerful language models.

## introduction large language models (llms) @xcite @xcite have revolutionized numerous areas of natural language processing by demonstrating an exceptional ability to store vast amounts of knowledge during pre-training and effectively recall this knowledge during inference. however, despite these capabilities, llms frequently "generating inaccurate or outdated information @xcite @xcite ," generating inaccurate or outdated information. to address this, the academic community has introduced fine-tuning methods. yet, full fine-tuning to adapt llms to specific downstream tasks or update their internal knowledge is often prohibitively expensive due to its substantial computational and time costs. consequently, parameter-efficient fine-tuning (peft) methods @xcite have emerged, aiming to update target knowledge or behaviors with minimal additional overhead while preserving the model's original capabilities. broadly, current peft methods primarily focus on achieving efficient adaptation by modifying a small number of parameters or introducing small additional modules while keeping the majority of llm parameters frozen. in this context, adapter @xcite modules and low-rank adaptation (lora) @xcite have become two highly influential mainstream paradigms. adapters typically insert small feed-forward network (ffn) modules between the layers of a pre-trained model. these modules comprise a down-projection layer with an activation function and an up-projection layer without one. lora, on the other hand, indirectly modifies model behavior with a small number of trainable parameters by decomposing the updates to pre-trained weights into the product of two lowrank matrices. both strategies aim to significantly reduce the number of trainable parameters during the fine-tuning process. despite the remarkable success of these methods in parameter efficiency, existing mainstream peft paradigms still face a critical bottleneck in balancing extreme parameter compression with the maintenance of high performance. specifically, although traditional adapters are termed "lightweight," their projection layers rely on dense weight matrices. this fundamental design means that even with small adapter dimensions, their total parameter count (often reaching millions) poses a severe deployment challenge for edge devices, which are highly sensitive to memory and computational resources. conversely, lora achieves substantial parameter reduction through low-rank approximation, but this inherent lowrank constraint itself may limit its capacity to fully express complex state transitions. when tackling tasks that require capturing high-frequency signals or subtle semantic differences, the low-rank assumption can become a limiting factor for model performance, leading to suboptimal outcomes in specific scenarios or even sacrificing some of the inherent powerful representation potential of llms. to address the aforementioned drawbacks, i.e., the parameter redundancy of traditional adapters and the potential representational limitations of lora, we explore a novel parameterization approach. instead of being confined to optimizing the scale of dense projections or relying on fixed low-rank decompositions, we attempt to parameterize the core projection operations of adapters in the fourier domain to map input features to the frequency domain via fourier transforms and learn adaptive transformations within this frequency space, thereby replacing traditional dense weight matrices while enhancing the adapter's sensitivity to frequency components. we posit that frequency-domain representations have the potential to efficiently capture multi-scale, periodic, or high-frequency features with fewer parameters, offering a way to overcome the limitations of traditional spatial-domain parameterization. notably, in our proposed architecture, rigorous experimental validation has shown that the non-linear activation function in the down-projection path is crucial for maintaining model performance and is retained. our main contributions are: i) the introduction of the fourier domain adapter (fda), a novel parameter-efficient fine-tuning (peft) paradigm that reparameterizes adapter projection layers directly in the fourier domain; ii) endowing the adapter with significantly enhanced frequency perception and adaptive capabilities. this is achieved by its unique fourier domain parameterization, which allows for targeted, sparse modulation of frequency components to effectively address the representational limitations of prior peft methods; iii) the comprehensive experiments demonstrate that fda consistently outperforms existing parameter-efficient fine-tuning methods on multiple benchmarks, including glue, e2e nlg, and instruction fine-tuning while requiring only a few trainable parameters. for example, when finetuning large models such as llama3-70b (such as the e2e nlg task), fda requires more than 151 times fewer trainable parameters than the traditional adapterh and 112 times less than lora. ## related work parameter-efficient fine-tuning (peft) methods have gained widespread application in the adaptive adjustment of large-scale pre-trained language models in recent years. traditional fullparameter fine-tuning methods @xcite @xcite ) require updating a large number of model parameters when dealing with specific tasks, leading to high computational and storage costs. to address this issue, researchers have proposed various peft methods, such as adapters @xcite and lora @xcite . adapters insert lightweight adapter modules between the layers of the model, fine-tuning only these new parameters can significantly reduce the number of parameters required for fine-tuning. lora reduces the scale of parameter updates through low-rank matrix decomposition. ## experiments we evaluate fda fine-tuned nlp models across three perspectives: (1) natural language understanding (nlu) tasks on the glue benchmark @xcite with roberta (base & large) @xcite , (2) natural language generation (nlg) tasks on the e2e nlg dataset @xcite using gpt2-small @xcite , deepseek-r1-distill-qwen-1.5b (deepseek-ai, 2025), llama2-7b (meta-ai, 2023), llama3-8b (meta-ai, 2024)and llama3-70b(meta-ai, 2024), and (3) instruction tuning tasks on mt-bench @xcite , vicuna eval @xcite , bbh @xcite , math @xcite , and alpaca @xcite with deepseek-r1-distill-qwen-1.5b, llama2-7b, qwen2-7b, and llama3-8b.for a detailed introduction to the dataset, see the supplementary materials section in the appendix e.1.in addition, we also designed frequency perception experiments and ablation experiments to test the specific frequency performance of the fda fine-tuned model and the impact of each component on the fda model. all experiments were performed on eight a100 gpus. throughout all experiments, the intermediate dimension d h of the fda modules was configured to d model /4, where d model denotes the input hidden dimension of the pre-trained model. for instance, in the case of roberta, where d @xmath0, the intermediate dimension d h was set to 768/4 = 192. ## conclusion this paper introduced the fourier domain adapter (fda), a novel approach for highly parameterefficient fine-tuning of large language models (llms). by reparameterizing adapter projection layers directly in the fourier domain using sparse learnable frequency modulations and complemented by highly compact auxiliary linear layers, our fda achieves a dramatic reduction in trainable parameters while simultaneously delivering superior performance. we present a robust and promising direction for making the adaptation of llms significantly more efficient and accessible.