SSH : Sparse Spectrum Adaptation via Discrete Hartley Transformation (2025)

Abstract: Low-rank adaptation (LoRA) has been demonstrated effective in reducing the trainable parameter number when fine-tuning a large foundation model (LLM). However, it still encounters computational and memory challenges when scaling to larger models or addressing more complex task adaptation.In this work, we introduce **Sparse Spectrum Adaptation via Discrete Hartley Transformation (SSH)**, a novel approach that significantly reduces the number of trainable parameters while enhancing model performance. It selects the most informative spectral components across all layers, under the guidance of the initial weights after a discrete Hartley transformation (DHT). The lightweight inverse DHT then projects the spectrum back into the spatial domain for updates.Extensive experiments across both single-modality tasks—such as language understanding and generation—and multi-modality tasks—such as video-text understanding—demonstrate that SSH outperforms existing parameter-efficient fine-tuning (PEFT) methods while achieving substantial reductions in computational cost and memory requirements. For instance, during instruction tuning on the LLaMA3.1 8B model, SSH achieves higher accuracy with only 0.048M trainable parameters compared to LoRA’s 33.5M, while reducing computational intensity up to 55% compared to FourierFT.

## introduction pretrained foundation models, such as gpt-4 (islam and moushi, 2024), llama3.1 @xcite , and vision transformers (vit) @xcite , have demonstrated remarkable performance across diverse natural language processing (nlp) @xcite and vision tasks @xcite . this success can largely be attributed to the unprecedented growth in model size @xcite . however, as these models scale up to billions of parameters, adapting them for downstream tasks in various domains presents significant computational and memory challenges. fully fine-tuning these large models becomes prohibitively expensive, both in terms of memory consumption and computational resources. for instance, fine-tuning the llama3.1 model with 8 billion parameters requires 60gb of gpu memory. while parameter-efficient finetuning (peft) methods like lora @xcite and qlora @xcite can reduce memory requirements to 16gb and 6gb, respectively, they still fall short when applied to larger models such as llama3.1 70b, which demands up to 48gb of memory, even with q-lora. lora and its successors @xcite @xcite have made substantial progress by introducing low-rank adaptations and quantization techniques to mitigate the memory overhead when fine-tuning large models. despite their ability to reduce the number of trainable parameters, as these methods primarily operate in the weight space of the original models, they still face limitations in terms of overall computational efficiency and gpu memory requirements, especially when scaling to massive models. a promising alternative to address this bottleneck is to leverage the frequency transformations, which offer a more compact representation of model weights with less trainable parameters. recent work on frequency-based peft, such as the discrete fourier transform (dft) approach @xcite , has shown that transforming weight matrices into the spectral domain and updating spectral components can significantly reduce the number of trainable parameters. however, dft operates in the complex domain, introducing potential computational overhead and numerical instability @xcite , particularly in large-scale models @xcite . such numerical inaccuracies inevitably degrade the performance. moreover, the asymmetry between dft and its inverse (idft) complicates forward and backward transformations, increasing computational intensity. to address these challenges, we propose a novel fine-tuning framework, sparse spectrum adaptation via discrete hartley transform (ssh). the advantage of leveraging dht over dft @xcite is twofold. firstly, dht integrates both cosine and sine components in a single operation. it avoids the imaginary number computation, simplifies the computations and improves the numerical stability. secondly, its symmetry-where the inverse transformation is identical to the forward-streamlines the fine-tuning process, allowing efficient transitions between time and frequency domains. ssh selectively fine-tunes the most critical frequency components, identifies through the energy compaction properties, and efficiently recovers weight updates via inverse dht. by capitalizing on the symmetrical nature of dht and reducing parameter usage, ssh provides a highly parameter-efficient and computationally optimized fine-tuning strategy. as shown in fig. 1(a ) & (b), ssh achieves comparable performance in both instruction tuning with the llama3.1-8b model and image classification with the vit 85.8m model, with significantly fewer parameters than lora and fourierft. fig. 1(c ) & (d) further demonstrate its superior computational efficiency, requiring 55% fewer gflops than fourierft. this is due not only to reduce the parameter count but also to avoid the complex num-ber handling required by fourierft, where real and imaginary parts must be processed separately. concretely, our contribution is threefold. frequency-based spectrum learning has been used to reduce trainable parameters while preserving model capacity. prior works @xcite @xcite showed the effectiveness of compact and sparse spectral representation learning. gao et al. @xcite applied the fourier transform to fine-tune a subset of spectral coefficients, highlighting the potential of sparse spectrum adaptation in large foundation models. however, the dft introduces complex operations, and the asymmetry between the dft and its inverse increases computational overhead. ssh addresses these issues with the real-valued dht, which eliminates complex arithmetic, reduces computational complexity, and enhances numerical stability through symmetric transforms. additionally, ssh's energy-based sparse selection further de- ## methodology we propose ssh, a novel frequency selection strategy based on the discrete hartley transform for peft, as illustrated in fig. 2 . it operates in the hartley spectral domain and learns a set of spectral coefficients. specifically, we employ the discrete hartley transform to the pretrained weights, using energy-based frequency selection to identify the most informative frequencies. in addition, we incorporate random sampling to ensure diversity in the selected frequencies while maintaining the representation ability. the ssh algorithm 1 selectively fine-tunes a pretrained layer's weights using the discrete hartley transform (dht) to capture the most significant frequency components. first, the layer's weights are transformed into the frequency domain using dht, and the top frequencies, based on energy, are selected for fine-tuning, while the remaining frequencies are randomly chosen and kept frozen. the trainable spectral coefficients are initialized using kaiming initialization, and during backpropagation, only the selected frequencies have their gradients updated. the inverse dht is then applied to these updated spectral coefficients, scaled by a factor α, to obtain the transformed weights in the spatial domain. these updates are merged with the original pretrained weights, resulting in the final transformed tensor. this approach ensures that only the most informative frequency components are fine-tuned, significantly reducing the number of trainable parameters while maintaining model performance. ## limitations while ssh shows great promise in terms of parameter and computation efficiency, there are several limitations that need to be addressed in the future: ## conclusion we introduced sparse spectrum adaptation via discrete hartley transformation (ssh), a novel parameter-efficient fine-tuning (peft) method that reduces the number of trainable parameters while maintaining competitive performance and computation efficiency. ssh leverages the real-valued dht and its symmetric forward and backward transform to selectively update the most informative spectral components, addressing the computational and memory challenges of fine-tuning large models. through extensive experiments across diverse tasks, ssh demonstrates robust versatility, excelling in single-modality nlp tasks such as natural language understanding (nlu), natural language generation (nlg), text summarization, and mathematical reasoning. furthermore, ssh extends its effectiveness to multi-modality applications, including vision-language image classification. ssh not only achieves state-of-the-art performance but also surpasses existing peft methods in both parameter and computational efficiency, positioning it as a scalable and lightweight solution for fine-tuning large models across various domains.